An open data synchronization service method, system, device and medium

Abstract

This application provides an open data synchronization service method, system, device, and medium. The method includes the following steps: acquiring the network status of distributed open data sources and their mirror sites, and generating dynamic health indicators based on availability and transmission performance; receiving and parsing synchronization requests, ranking candidate transmission paths based on health indicators, and selecting an execution path; identifying the target data source and verifying its metadata and compliance rules, and generating access control instructions; if access is granted, issuing collection tasks to collection nodes on the execution path and dynamically managing the task status; splitting the collected raw data stream into a content stream and a metadata stream in parallel, calculating content fingerprints for deduplication, and performing reference or distributed storage operations, while simultaneously extracting index features and storing them in an index database. Implementing the technical solution provided in this application can dynamically select the optimal transmission path based on network status and data source characteristics, improving the stability and efficiency of data synchronization.

Description

Technical Field

[0001] This application relates to the field of Internet data processing technology, and in particular to an open data synchronization service method, system, device and medium. Background Technology

[0002] With the deepening development of data-intensive scientific research paradigms and global digital collaboration, the scale of global open data resources, such as biological gene sequences, astronomical observation data, and academic literature metadata, is growing exponentially, and the distribution of data sources is increasingly fragmented and heterogeneous. How to efficiently, stably, and compliantly synchronize, collect, and distribute massive amounts of open data distributed around the world has become a core requirement for improving the efficiency of scientific research infrastructure and promoting the flow of data elements.

[0003] In existing technologies, the acquisition and synchronization of such data is typically achieved by writing customized single acquisition scripts or utilizing general peer-to-peer cloud transmission services. For example, crawler scripts based on HTTP (Hypertext Transfer Protocol) requests are written for specific academic websites, or data is transferred using physical migration equipment and dedicated line services provided by cloud service providers. However, in practical applications, existing technologies often employ script acquisition or peer-to-peer transmission methods with preset fixed strategies. When faced with dynamic fluctuations in the global network environment and changes in access rules or data structures of heterogeneous data sources, the lack of adaptive adjustment based on real-time status monitoring and a closed-loop feedback mechanism across the entire link leads to risks of poor data synchronization task execution stability and unexpected transmission link interruptions during large-scale cross-regional data synchronization. Summary of the Invention

[0004] In view of this, this application provides an open data synchronization service method, system, device and medium to solve the above problems.

[0005] Firstly, an open data synchronization service method is provided, which includes:

[0006] Obtain network status information of distributed open data sources and their mirror sites; determine availability and transmission performance indicators for each open data source based on the network status information; and generate dynamic health evaluation indicators based on the availability and transmission performance indicators.

[0007] Receive data synchronization requests, parse the source and destination attributes in the data synchronization requests, combine dynamic health evaluation indicators, prioritize candidate transmission paths with open data sources and their mirror sites as starting nodes, and determine the transmission path with the highest priority as the execution path.

[0008] The target data source pointed by the execution path is identified, the metadata of the target data source is obtained, the metadata is matched and verified with a preset digital compliance rule library, an access control instruction for the data synchronization request is generated according to a risk assessment result generated by the verification, and the access control instruction is sent to the source end.

[0009] In response to the access control instruction being in a pass state, a data collection task is established and distributed to a collection node deployed on the execution path, and a state management mechanism is used to dynamically control the state flow of the data collection task in a life cycle;

[0010] The original data stream returned by the collection node executing the data collection task is received, the original data stream is shunted, and content data stream and metadata stream are generated in parallel;

[0011] A unique fingerprint feature is calculated for the content data stream, a de-duplication verification is performed in a storage system based on the unique fingerprint feature, and a reference link establishment operation or a distributed storage operation is performed according to a verification result, the metadata stream is parsed to extract an index feature, and the index feature is stored in an index database.

[0012] The above technical solutions can prioritize a path with high availability and good transmission performance, reduce transmission interruptions and retry times, by collecting network states of distributed data sources and mirror sites in real time and sorting candidate transmission paths based on health degrees before synchronization; the access instruction can be issued based on the matching and verification of the metadata of the target data source and the compliance rules, and the data stream that is not compliant or high-risk can be blocked at the source end, reducing compliance penalties and subsequent manual intervention; the task is controlled by the state management mechanism according to the life cycle and the node computing resources are released, avoiding long-term occupation of resources; finally, parallel fingerprint de-duplication and index storage for content can avoid repeated writing, save storage resources, and improve subsequent retrieval efficiency and traceability.

[0013] Optionally, the availability indicators and transmission performance indicators of each open data source are determined based on network state information, and a dynamic health degree evaluation indicator is generated according to the availability indicators and transmission performance indicators, specifically including:

[0014] The connection success rate data contained in the network state information is extracted as the availability indicator, and the round-trip delay data and throughput data contained in the network state information are extracted as the transmission performance indicators;

[0015] The availability indicators and transmission performance indicators in the historical time window are weighted and averaged using a preset time decay algorithm to obtain a smoothed state value for the current time period;

[0016] The smoothed state value is input into a preset scoring model, the normalized function in the scoring model is used to map the smoothed state values of different dimensions to standard scores, and the standard scores are summed to generate the dynamic health degree evaluation indicator.

[0017] The above technical solution uses connection success rate as an availability indicator and round-trip latency and throughput as transmission performance indicators. By applying a time-decrease weighted average to the indicators within the historical window, it can suppress the impact of occasional outliers on the evaluation, making the current period's status value more reflective of the recent real situation. Furthermore, by normalizing the scoring, it maps indicators of different dimensions to comparable standard scores and sums them to obtain a unified dynamic health score. This makes it easy to use directly in subsequent path selection and priority comparison and avoids misjudgments caused by inconsistent dimensions.

[0018] Optionally, by combining dynamic health evaluation metrics, candidate transmission paths originating from open data sources and their mirror sites can be prioritized, specifically including:

[0019] A network topology graph is constructed based on source and destination attributes, and candidate transmission paths are mapped to directed connected links in the network topology graph.

[0020] Read the dynamic health evaluation index and calculate the link cost between nodes in the directed connected link by calculating the reciprocal of the dynamic health evaluation index.

[0021] The network topology is traversed using a preset shortest path search algorithm. The link cost of each link segment is accumulated to calculate the total path cost, and priority is generated in order of total path cost from low to high.

[0022] The above technical solution constructs the source / destination attributes into a network topology and maps candidate paths into directed links. It then uses the reciprocal of the health evaluation index as the link cost and calculates the total path cost by accumulating the results using the shortest path algorithm. This allows the network health and transmission performance to be quantified as path costs, thereby automatically selecting the transmission path with the minimum cumulative cost, reducing overall transmission delay and failure risk. Furthermore, the selection process is repeatable and auditable.

[0023] Optionally, the metadata of the target data source is obtained, and the metadata is matched and verified against a pre-defined digital compliance rule base. Based on the risk assessment results generated by the verification, an access control instruction for the data synchronization request is generated, specifically including:

[0024] Parse the header description file of the target data source and extract attribute fields containing data generation region, data sensitivity level and authorization agreement type as metadata;

[0025] Read the pre-stored conditional logic tree in the digital compliance rule base. The action types associated with the leaf nodes of the conditional logic tree include blocking, warning, and allowing.

[0026] The user's location information in the metadata and data synchronization request is input into the conditional logic tree for hierarchical judgment. When the judgment path falls into a leaf node associated with a blocking action type, an access control instruction indicating that synchronization is prohibited is generated.

[0027] The above technical solution extracts metadata such as region, sensitivity level, and authorization agreement by parsing the header description file of the target data source. It then uses a pre-stored conditional logic tree to make layer-by-layer judgments. The system can perform three types of actions, namely "blocking / warning / allowing", on data of different regions / sensitivity levels at the source. Data that violates local regulations or exceeds the scope of authorization is directly prohibited or marked before synchronization, reducing the outflow of illegal data and the cost of compliance review.

[0028] Optionally, a state management mechanism can be used to dynamically control the state transitions of data acquisition tasks throughout their lifecycle, specifically including:

[0029] When creating a data acquisition task, the task status identifier is initialized to a suspended state in the preset status storage unit;

[0030] When a data acquisition node sends a ready signal, the data acquisition task is distributed to the data acquisition node and the task status flag is updated to running status.

[0031] Receive the completion confirmation signal returned by the acquisition node, parse the status confirmation identifier in the completion confirmation signal, and if the status confirmation identifier is verified, update the task status identifier to archived status and release the computing resources of the acquisition node.

[0032] The above technical solutions set clear status identifiers for tasks (suspended → running → archived) when the data collection task is created, and synchronously update the status and release computing resources when the data collection node is ready or completed. This can prevent task concurrency conflicts, avoid node resource leakage, improve resource utilization, and provide clear status records for task monitoring and troubleshooting.

[0033] Optionally, the method also includes:

[0034] The system collects real-time statistics on the frequency of request failures during task execution. When the frequency of request failures exceeds a preset tolerance threshold, the task is deemed to be abnormal.

[0035] Call the proxy resource interface to filter out backup Internet Protocol addresses that are in different network subnets from the preset global proxy pool, and generate a configuration update command containing the backup Internet Protocol addresses.

[0036] Send configuration update instructions and reset signals to the status management mechanism, roll back the task status from running to suspended, and reinitialize the network connection session of the data acquisition task using the backup Internet Protocol address.

[0037] The above technical solution can count the frequency of request failures in real time during the operation of the task and automatically select a backup address from the global proxy pool that is in a different subnet than the current address when the threshold is exceeded. This can quickly switch the connection path when network segmentation, carrier failure or link quality drops sharply, reduce data loss or task interruption caused by long-term task failure, and improve the robustness and recoverability of the overall data collection task.

[0038] Optionally, deduplication verification is performed in the storage system based on the unique fingerprint feature, and reference link establishment or distributed storage operation is performed according to the verification result, specifically including:

[0039] The data block digest value of the content data stream is calculated using a preset hash algorithm as a unique fingerprint feature, and the index record corresponding to the data block digest value is queried in the global index table of the index database.

[0040] If an index record is found, the physical address pointer stored in the index record is extracted, a virtual file object pointing to the physical address pointer is constructed, and a write success signal is returned.

[0041] If no index record is found, the consistent hash ring algorithm is used to map the data block digest value to the virtual node space. The target write location is determined based on the mapping relationship between the virtual node space and the physical storage node. The content data stream is written to the target write location, and a new index entry containing the data block digest value and the target write location is created in the global index table.

[0042] The above technical solution uses a data block digest as a unique fingerprint and queries the global index table: if it already exists, it directly constructs a virtual file object pointing to the existing physical address and returns a successful write, avoiding duplicate writes; if it does not exist, it maps to the target physical storage node through a consistent hash ring and writes it while simultaneously adding an entry to the global index table, ensuring load balancing and scalability of writes, and achieving rapid existence judgment and subsequent location through index maintenance.

[0043] Secondly, an open data synchronization service system is provided, the system comprising:

[0044] The health assessment module is configured to obtain network status information of distributed open data sources and their mirror sites. Based on the network status information, it determines the availability and transmission performance indicators of each open data source and generates dynamic health evaluation indicators based on these indicators.

[0045] The path sorting module is configured to receive data synchronization requests, parse the source and destination attributes in the data synchronization requests, combine dynamic health evaluation indicators, prioritize candidate transmission paths starting from open data sources and their mirror sites, and determine the transmission path with the highest priority as the execution path.

[0046] The access verification module is configured to identify the target data source pointed to by the execution path, obtain the metadata of the target data source, match and verify the metadata with the preset digital compliance rule base, and generate access control instructions for the data synchronization request based on the risk assessment results generated by the verification.

[0047] The task scheduling module is configured to respond to access control commands in an approved state, establish data acquisition tasks, and distribute data acquisition tasks to acquisition nodes deployed on the execution path. It also uses a state management mechanism to dynamically control the state transition of data acquisition tasks throughout their lifecycle.

[0048] The data splitting module is configured to receive the raw data stream returned by the acquisition node when it performs data acquisition tasks, split the raw data stream, and generate content data stream and metadata stream in parallel.

[0049] The deduplication module is configured to calculate a unique fingerprint feature for the content data stream, perform deduplication verification in the storage system based on the unique fingerprint feature, and perform reference link establishment operation or distributed storage operation according to the verification result. It also parses the metadata stream to extract index features and stores the index features in the index database.

[0050] Thirdly, an electronic device is provided, including a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform any of the methods described above.

[0051] Fourthly, a computer-readable storage medium is provided, which stores instructions that, when executed, perform the method described in any of the above.

[0052] In summary, implementing one or more technical solutions provided in this application has at least the following technical effects or advantages:

[0053] By linking network health-driven path selection, source compliance access, task lifecycle management, and content deduplication into an automated, configurable closed-loop process, the time window from data triggering to availability is significantly shortened, synchronization success rate and response speed are improved, while reducing manual intervention and operation and maintenance costs. In addition, it supports hot rule updates and horizontal scaling, which can smoothly cope with data volume growth and network fluctuations while ensuring compliance and integrity, and enhance business continuity and audit traceability. Attached Figure Description

[0054] Figure 1 This is an exemplary system architecture diagram of an open data synchronization service method or an open data synchronization service system that applies this application;

[0055] Figure 2 This is a flowchart illustrating one of the open data synchronization service methods disclosed in this application;

[0056] Figure 3 This is a schematic diagram of a module of an open data synchronization service system disclosed in this application;

[0057] Figure 4 This is a schematic diagram of the structure of an electronic device disclosed in this application.

[0058] Explanation of reference numerals in the attached diagram: 100, System architecture; 101, First terminal device; 102, Second terminal device; 103, Third terminal device; 104, Network; 105, Server; 301, Health assessment module; 302, Path sorting module; 303, Access verification module; 304, Task scheduling module; 305, Data distribution module; 306, Deduplication and data entry module; 401, Processor; 402, Communication bus; 403, User interface; 404, Network interface; 405, Memory. Detailed Implementation

[0059] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0060] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0061] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0062] Figure 1 An exemplary system architecture diagram is shown, illustrating an embodiment of an open data synchronization service method or an open data synchronization service system to which this application can be applied.

[0063] like Figure 1 As shown, the system architecture 100 may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0064] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as model training applications, video recognition applications, web browser applications, social platform software, etc.

[0065] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays, including but not limited to smartphones, tablets, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptops, and desktop computers, etc. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0066] When terminals 101, 102, and 103 are hardware devices, video capture devices can also be installed on them. These video capture devices can be various devices capable of capturing video, such as cameras, sensors, etc. Users can use the video capture devices on terminals 101, 102, and 103 to capture video.

[0067] Server 105 can be a server that provides various services, such as a backend server for processing data displayed on terminal devices 101, 102, and 103. The backend server can analyze and process the received data and can feed back the processing results (such as recognition results) to the terminal devices.

[0068] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0069] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included. In particular, if the target data does not need to be obtained remotely, the above system architecture may exclude the network and include only terminal devices or servers.

[0070] Figure 2 This is a flowchart illustrating an open data synchronization service method according to an embodiment of this application. This method can be implemented using a computer program, a microcontroller, or run on an open data synchronization service system. The computer program can be integrated into an application or run as a standalone utility application. The specific steps of an open data synchronization service method are described in detail below.

[0071] S201: Obtain network status information of distributed open data sources and their mirror sites, determine availability and transmission performance indicators for each open data source based on the network status information, and generate dynamic health evaluation indicators based on the availability and transmission performance indicators.

[0072] For example, the system initiates continuous active probing of all registered open data sources and their redundant mirror sites by deploying probe nodes around the world. It collects multi-dimensional network metrics in real time, including connection status, response latency, and data transmission rate. The system then uses a weighted fusion algorithm to clean and normalize these raw metrics, and finally outputs a weighted and normalized scalar metric to characterize the service quality of the nodes, providing data support for subsequent routing decisions.

[0073] In one possible implementation, availability and transmission performance metrics of each open data source are determined based on network status information, and dynamic health evaluation metrics are generated based on these metrics. Specifically, this includes: extracting connection success rate data from the network status information as availability metrics, and extracting round-trip latency and throughput data from the network status information as transmission performance metrics; using a preset time decay algorithm to perform a weighted average calculation on the availability and transmission performance metrics within a historical time window to obtain a smoothed state value for the current time period; inputting the smoothed state value into a preset scoring model, mapping the smoothed state values ​​of different dimensions to standard scores through a normalization function in the scoring model, and summing the standard scores to generate dynamic health evaluation metrics.

[0074] In this application embodiment, the dynamic health evaluation index refers to a numerical metric parameter used to quantify the overall service quality of an open data source in the current network environment. It is used to represent the stability and transmission efficiency of the data source and its mirror sites at the network layer. For example, the index can be expressed as a score between 0 and 100. The higher the score, the more stable the network connection of the data source and the faster the transmission speed, making it more suitable as a target source for data synchronization.

[0075] Specifically, the system parses the acquired raw network probe logs, identifying and extracting successful statistics for TCP three-way handshakes or HTTP requests. The calculated connection success rate data is marked as an availability metric. Simultaneously, the system extracts round-trip latency data (representing response speed) and throughput data (representing the amount of data successfully transmitted per unit time) from the data packet transmission records, marking both as transmission performance metrics. Considering potential data noise caused by network jitter, instead of directly using instantaneous sample values ​​at a single moment, the system retrieves multiple historical data sequences within a preset historical time window (e.g., the past 5 minutes). A preset time decay algorithm (e.g., exponentially weighted moving average) is used to calculate a weighted average of the availability and transmission performance metrics. During this process, data closer to the current moment is assigned a larger weight coefficient, thus smoothing out occasional network fluctuations and obtaining a smoothed state value for the current time period. These smoothed state values ​​with different dimensions and physical meanings (e.g., lower latency is better, higher throughput is better) are input into a preset scoring model. The normalization function integrated into the scoring model (such as max-min standardization or Z-Score standardization) is used to map each data into a dimensionless standard score with a consistent numerical range (e.g., between 0 and 1). Finally, the standard scores of each dimension are linearly summed or weighted and accumulated according to the preset business weight strategy to generate the final dynamic health evaluation index.

[0076] Furthermore, when calculating the smoothed state value using the exponentially weighted moving average algorithm, the system employs the following dynamic calculation logic to adapt to the rapidly changing characteristics of the network: The calculation formula is set as follows... ,in This represents the smoothed state value for the current period. These are instantaneous indicators collected at the current moment. This is the smoothed value from the previous period. To quickly respond to network faults, the system is designed with a non-linear smoothing coefficient. Dynamic adjustment mechanism: When instantaneous indicators are detected When there is a significant deterioration compared to the historical average (such as a sudden increase in latency exceeding 50%), automatically increase the speed. A value (e.g., adjusted to 0.8) gives greater weight to currently deteriorating data, causing the score to drop rapidly and triggering a route switch; conversely, when network metrics are fluctuating smoothly, a smaller value is used. A value (e.g., 0.2) is used to maintain the stability of the score and avoid frequent path switching due to minor network fluctuations. In addition, for the connection success rate metric, a consecutive failure penalty factor is introduced. That is, if the detection fails for N consecutive time windows (e.g., N=3 or N=5, where N is a preset positive integer), the availability metric for that time window is directly forced to be set to 0, regardless of its historical performance, to ensure that the source of failure is eliminated immediately.

[0077] S202: Receive data synchronization request, parse the source and destination attributes in the data synchronization request, combine dynamic health evaluation indicators, prioritize candidate transmission paths starting from open data sources and their mirror sites, and determine the transmission path with the highest priority as the execution path.

[0078] For example, when responding to a user-initiated data synchronization request, the system parses the network locations of both requesting parties and constructs a logical network topology covering the source station and all its mirror sites based on the health indicators generated in previous steps. On this basis, the system transforms the abstract health score into algorithmically calculable path weights (i.e., lower cost represents higher quality), uses a weighted pathfinding algorithm to calculate the comprehensive transmission cost from each candidate source to the destination, and finally selects the link with the lowest cost as the execution path for this task, thereby achieving adaptive routing selection based on multi-dimensional network indicators.

[0079] In one possible implementation, a dynamic health evaluation index is used to prioritize candidate transmission paths originating from open data sources and their mirror sites. Specifically, this includes: constructing a network topology graph based on source and destination attributes, and mapping candidate transmission paths to directed connected links in the network topology graph; reading the dynamic health evaluation index, calculating the link cost between nodes in the directed connected link by reciprocal of the dynamic health evaluation index; traversing the network topology graph using a preset shortest path search algorithm, accumulating the link cost of each link segment to calculate the total path cost, and generating a priority ranking based on the total path cost from low to high.

[0080] In the embodiments of this application, the link cost value refers to a numerical weight parameter used to quantify the logical overhead or resistance required to transmit data between two nodes in a network topology graph. It is used to represent the "distance" or "cost" of a network path at the algorithm level, and this value is negatively correlated with the actual health status of the path. For example, in a pathfinding algorithm, the link cost value can be regarded as the "length" of a road segment. The higher the health status of the link, the shorter its "length" (the smaller the cost value), and thus it is easier to be determined as the optimal path during calculation.

[0081] Specifically, based on the parsed source attributes (such as data source IP and geographical location) and destination attributes (such as receiver IP and network access point), the system abstracts various node entities in the network space, constructs a network topology graph containing data source nodes, mirror site nodes, intermediate routing nodes, and destination nodes, and maps all theoretically feasible candidate transmission paths from the source to the destination as directed connected links in this graph. To adapt to the "minimum sum" calculation logic of conventional pathfinding algorithms, the previously generated dynamic health evaluation index is read and mathematically transformed. Specifically, the reciprocal of the dynamic health evaluation index is calculated (to avoid division by zero, a very small positive number can be added to the denominator, for example...). ,in For a preset very small positive floating-point number, for example, a value of Alternatively, when the health level is 0, a preset maximum path cost (e.g., MAX_INT) is assigned to the system to obtain the link cost between nodes in a directed connected link. A preset shortest path search algorithm (such as Dijkstra's algorithm or Floyd's algorithm) is used to traverse and search the network topology, accumulating the link costs of all network segments traversed along a complete path to calculate the total path cost.

[0082] Furthermore, a numerical comparison-based sorting logic is executed, using the calculated total path cost as the sorting key to perform an ascending sort operation on the candidate transmission path set (i.e., the smaller the value, the higher the ranking). The paths that are at the beginning or in the first X (X is a positive integer) positions of the sorted list are marked as high-priority paths, generating the final priority ranking to ensure that the selected execution paths meet the preset path cost minimization condition.

[0083] It should be noted that the network topology map described above is not static. In addition to a static IP geolocation database, this system integrates BGP (Border Gateway Protocol) routing table information and Traceroute probe data. It periodically (e.g., hourly) retrieves AS (Autonomous System) path information from core routers or public routing servers to dynamically correct the connectivity between nodes in the logical topology, ensuring the topology map reflects the actual physical link paths. Furthermore, when generating priority rankings, it is not limited to locking onto a single optimal path but supports variations of the equal-cost multi-path routing strategy: when multiple candidate paths with similar total path costs (e.g., cost differences within 5%) are calculated, the suboptimal path is not discarded directly. Instead, all these paths are selected as execution paths, and the data collection task is divided into multiple subtasks. Traffic weights are allocated based on the inverse of the path cost, and these subtasks are distributed in parallel to different collection nodes on these paths, thereby achieving cross-link load balancing and maximizing the utilization of egress bandwidth.

[0084] S203: Identify the target data source pointed to by the execution path, obtain the metadata of the target data source, match and verify the metadata with the preset digital compliance rule base, and generate an access control instruction for the data synchronization request based on the risk assessment result generated by the verification.

[0085] For example, before formally initiating the data transmission link, the system performs a pre-emptive automated compliance audit. By extracting the legal and security attributes of the target data source and projecting them onto the digital compliance strategy for multi-dimensional matching, the system logically determines whether the current request meets cross-border transmission regulations or copyright authorization requirements. Based on the determination result, deterministic conditional branch control is executed: once the logic tree traversal result hits any blocking condition, an interruption command is immediately triggered to terminate the current session; an access token is generated only when all verification items pass, thereby physically blocking the inflow of illegal data into subsequent processing stages.

[0086] In one possible implementation, the metadata of the target data source is obtained and matched against a pre-defined digital compliance rule base. Based on the risk assessment results generated by the verification, an access control instruction for the data synchronization request is generated. Specifically, this includes: parsing the header description file of the target data source and extracting attribute fields containing the data generation region, data sensitivity level, and authorization agreement type as metadata; reading a pre-stored conditional logic tree in the digital compliance rule base, where the leaf nodes of the conditional logic tree are associated with action types including blocking, warning, and allowing; inputting the metadata and the user's location information in the data synchronization request into the conditional logic tree for hierarchical judgment; and generating an access control instruction indicating that synchronization is prohibited when the judgment path falls into a leaf node associated with a blocking action type.

[0087] In this application embodiment, a conditional logic tree refers to a data structure that transforms complex legal provisions or compliance requirements into hierarchical decision logic that can be executed by a computer. It is used to represent a series of nested Boolean judgment conditions and their corresponding decision flow paths. For example, the tree structure can take "data jurisdiction" as the root node, "data sensitivity level" and "cross-border transmission protocol" as intermediate branch nodes, and through the step-by-step traversal of these nodes, it eventually points to the leaf node that defines the specific compliance operation (such as "allow" or "block").

[0088] Specifically, the system establishes a connection session with the target data source (such as an HTTP / HTTPS Keep-Alive connection) and obtains the metadata carrier of that data source. This carrier specifically includes: HTTP standard header fields in the response message (such as Content-Disposition, X-Rate-Limit), and data embedded in the HTML page. <meta> Tags (such as tags conforming to the Dublin Core metadata standard) or machine-readable index files in the site's root directory (such as sitemap.xml, robots.txt, or dcat.json data directory files). The parser extracts fields from the above standardized format, identifying and extracting key attribute fields related to compliance judgment. These fields specifically include the data generation region that identifies the legal jurisdiction of the data, the data sensitivity level that identifies the confidentiality and privacy attributes of the data, and the authorization agreement type that identifies the scope of data usage permissions. These extracted attribute fields are combined to establish the metadata to be verified. A pre-stored conditional logic tree is retrieved from the system's pre-built digital compliance rule base. The terminal leaf nodes of this conditional logic tree are pre-associated with action types for different compliance states. These action types cover blocking to intercept high-risk operations, warning to indicate potential risks, and allowing normal transmission. The extracted metadata and the user's location information parsed from the original data synchronization request are used as input parameters and substituted into the conditional logic tree for top-down hierarchical judgment and path matching. If, after conditional judgments at each level of nodes, the final judgment path falls into a leaf node associated with a blocking action type (for example, when the judgment data originates from a "GDPR jurisdiction" and the user's location is "overseas" and the data sensitivity level is "high"), then the risk control mechanism is triggered according to the definition of the leaf node, generating an access control instruction that explicitly prohibits synchronization.

[0089] Furthermore, considering the dynamic nature of global compliance policies, this digital compliance rule base supports hot reloading and conflict resolution mechanisms. The rule base is not hard-coded into the program but stored in the configuration center in DSL (Domain-Specific Language) or JSON format. The system employs a double-buffering mechanism or a copy-on-write (COW) strategy. After rule updates are issued from the configuration center, the admission verification module constructs a new conditional logic tree object instance in the background memory area. Once constructed, the CPU's atomic swap instruction (such as Compare-and-Swap) instantly switches the global reference pointer pointing to the old logic tree to the new instance, thus completing the hot update of rules with zero downtime. Meanwhile, in response to potential rule conflicts (for example, the law of the country where the data source is located allows the transmission, but the law of the receiving end prohibits entry), the system has a preset "strictest policy priority" algorithm: when constructing or traversing the conditional logic tree, if multiple contradictory action types are derived from different regulations on the same leaf node path (such as both "allow" and "block"), the system will automatically determine the higher priority "block" action, or prioritize the adoption of specific rules formulated for a specific data source ID to cover general regional rules, so as to ensure that compliance risks are minimized.

[0090] S204: In response to the access control command being in the pass state, a data acquisition task is established and distributed to the acquisition nodes deployed on the execution path. The state management mechanism is used to dynamically control the state transition of the data acquisition task during its lifecycle.

[0091] For example, once the access command is confirmed and allowed, the system encapsulates the request into a standardized data collection job and sends it to the designated node. At the same time, it initializes a monitoring process with a status feedback link, maintains a real-time view of the entire link status of the task from creation, scheduling, execution to termination, and supports automatic recovery of the task through status rollback and resource switching mechanisms when execution is detected to be blocked, thereby ensuring the consistency of the data synchronization process and the ability to recover from faults.

[0092] In one possible implementation, a state management mechanism is used to dynamically control the state transition of data acquisition tasks throughout their lifecycle. Specifically, this includes: when a data acquisition task is created, the task status identifier is initialized to a suspended state in a preset state storage unit; when the acquisition node sends a ready signal, the data acquisition task is distributed to the acquisition node, and the task status identifier is updated to a running state; a completion confirmation signal is received from the acquisition node, the status confirmation identifier in the completion confirmation signal is parsed, and if the status confirmation identifier is verified, the task status identifier is updated to an archived state and the computing resources of the acquisition node are released.

[0093] In the embodiments of this application, the task status identifier refers to a variable symbol or status code used to uniquely mark and indicate the logical stage of the data acquisition task at the current life cycle node within the system. It is used to indicate whether the task is in a real-time state of waiting to be scheduled, being executed, or having been processed. For example, the identifier can be represented as an enumerated field in a database status table, and its value can be "PENDING" representing pending, "RUNNING" representing running, and "ARCHIVED" representing archived.

[0094] Specifically, during the initialization phase of a data acquisition task—that is, when the task is generated and ready to enter the scheduling process—the system creates a control record for the task in a pre-defined state storage unit (such as a high-concurrency key-value database or a distributed caching system). The task status identifier in this record is initialized to a suspended state, indicating that the task has been effectively created in the system but has not yet been allocated execution resources. Subsequently, the system enters the scheduling monitoring phase. When a ready signal is detected from a data acquisition node deployed on the execution path, the serialized task parameter packet (Protobuf or JSON format) is pushed to the consumption queue of that data acquisition node using low-latency remote procedure calls (gRPC) or message queue middleware (MessageQueue). Simultaneously, to prevent multiple nodes from concurrently claiming the same task, an atomic update operation is performed using the SETNX command of a distributed caching system (such as Redis) or the optimistic locking mechanism (CAS, Compare-And-Swap) of the database. This updates the record in the state storage unit from a suspended state to a running state, locking the task to prevent it from being claimed repeatedly by other nodes.

[0095] Furthermore, after the task execution is completed, a completion confirmation signal is received from the acquisition node. The status confirmation identifier (such as task completion check code or digital signature) used to verify the validity and integrity of the execution result is parsed from the signal message. If the status confirmation identifier is verified and the task is confirmed to be completed correctly and the data is intact, the status storage unit is updated again, and the task status identifier is finally updated to the archived status and formed into a historical log. At the same time, a resource reclamation instruction is triggered to release the computing resources of the acquisition node so that it can process subsequent tasks.

[0096] Optionally, the method further includes: real-time statistics on the request failure frequency of the data acquisition task while it is running; when the request failure frequency exceeds a preset tolerance threshold, the task execution is determined to be abnormal; calling the proxy resource interface to select a backup Internet Protocol address from a preset global proxy pool that is in a different network subnet than the currently used Internet Protocol address, and generating a configuration update instruction containing the backup Internet Protocol address; sending the configuration update instruction and reset signal to the state management mechanism to roll back the task status from the running state to the suspended state, and re-initializing the network connection session of the data acquisition task using the backup Internet Protocol address.

[0097] In this embodiment, a global proxy pool refers to a collection of proxy server resources distributed across different geographical locations around the world and belonging to different operators. It represents a network relay facility that the system can dynamically invoke when performing data collection tasks, designed to hide the true source of requests and circumvent access frequency restrictions on the target data source. For example, the proxy pool can consist of tens of thousands of Internet Protocol address nodes that support HTTP, HTTPS, or SOCKS5 protocols. These nodes cover different Class C network segments (C-Segments) and can simulate the access behavior of discrete users from different regions around the world, thereby reducing the risk of a single IP address being blocked due to high-frequency access.

[0098] Specifically, during task execution, the background monitoring service continuously tracks network interaction logs and collects real-time statistics on the frequency of request failures during task execution. This statistics cover abnormal situations such as connection timeouts and server refusal to respond (e.g., HTTP 403 / 529 status codes). When the calculated request failure frequency exceeds a preset tolerance threshold (e.g., the number of consecutive failures reaches the upper limit or the failure rate per unit time is too high), the logical judgment mechanism will determine that the currently used network exit has been identified or blocked by the target source, thus determining that the task execution is abnormal. At this time, the fault tolerance recovery process is triggered, calling the proxy resource interface to search the resource library, following the anti-association principle, and executing subnet matching logic based on binary bitmasks: using a configurable subnet mask (e.g., 255.255.255.0 or / 24 CIDR) to perform a bitwise AND operation on the binary sequences of the currently blocked IP and the candidate IPs in the proxy pool to calculate the network number. Based on the network number, nodes belonging to the same network segment are located and removed from the pre-set inverted index or hash bucket, thus selecting a backup Internet Protocol (IP) address. This ensures that the new access identity is unrelated to the old blocked identity at the network layer, and a configuration update instruction containing the new address information is generated. This configuration update instruction and reset signal are sent to the core state management mechanism to perform a task context reset operation. Specifically, the system sends an interrupt signal (such as SIGINT) or a termination instruction to the worker thread currently executing the task, forcibly triggering the closing and recycling process of the underlying TCP socket file descriptors, clearing the temporary data of the task in the memory stack. Atomic operations are used to roll back the task status identifier in the state storage unit from the running state to the suspended state, allowing the scheduler to recapture the task and build a new HTTP proxy tunnel using the backup IP address carried in the configuration update instruction, reinitializing the network connection session, and achieving automatic task migration and retry.

[0099] S205: Receives the raw data stream returned by the data acquisition node when it performs the data acquisition task, performs split processing on the raw data stream, and generates content data stream and metadata stream in parallel.

[0100] In the embodiments of this application, the raw data stream refers to the mixed binary data packet captured by the acquisition node during the interaction with the target source, which has not yet undergone decoupling processing. It is used to represent a complete communication message containing the payload content and control information describing the attributes of the payload. For example, the data stream can be a complete HTTP response, which encapsulates both the HTML page source code or image file as the actual business data and header description information including status code, content type and server timestamp.

[0101] Specifically, the system establishes a TCP / IP-based streaming data transmission channel with the acquisition node in execution mode, receiving the raw data stream, which may have been compressed or chunked, from the node in real time. A pre-built streaming parsing engine based on a Finite State Machine (FSM) performs real-time scanning of this input stream. This engine maintains a byte-level sliding window in memory, comparing byte-by-byte with specific boundary symbols defined by the network protocol (e.g., the double newline character represented as 0D 0A 0D 0A in hexadecimal in the HTTP protocol) through state transition logic. When the state machine transitions to the "boundary matching" state, a byte sequence matching algorithm identifies the physical separation point between control information and the entity carrier, thereby splitting the raw data stream in memory. During this process, the payload portion after the segmentation point (such as webpage source code, document binary stream, and JSON data packets) is extracted and constructed into an independent content data stream, which is directly directed to the storage deduplication module. Simultaneously, the header description portion and collection context (such as URL address, response header, and collection time) before the segmentation point are extracted and constructed into an independent metadata stream, which is directed to the index database module. The extraction and encapsulation of these two streams are performed simultaneously through a dual-threaded or asynchronous I / O mechanism, without blocking each other, thus generating the content data stream and metadata stream in parallel.

[0102] Furthermore, to prevent OutOfMemoryError (OOM) caused by the downstream (storage or indexing module) processing speed being slower than the upstream (collection network I / O) receiving speed, a backpressure feedback mechanism is introduced into the distribution pipeline. The distribution module maintains bounded blocking queues for content data streams and metadata streams. When the queue fill rate reaches a high watermark threshold (e.g., 80%), the distribution module pauses reading new data packets from the network socket, using the sliding window mechanism of the TCP protocol to force the sender to reduce its sending rate until the downstream consumer frees up sufficient buffer space. In addition, when generating data streams in parallel, to reduce the number of data copies between user mode and kernel mode, the system employs zero-copy technology (such as using the splice or sendfile interface of the Linux system). For large content data streams, the parser only passes memory address references and length information instead of copying actual data blocks, directly transferring data from the network buffer to the storage write buffer via DMA (Direct Memory Access), significantly reducing CPU utilization in high-bandwidth scenarios and improving the overall throughput performance of the system.

[0103] S206: Calculate a unique fingerprint feature for the content data stream, perform deduplication verification in the storage system based on the unique fingerprint feature, and perform reference link establishment operation or distributed storage operation according to the verification result. Parse the metadata stream to extract index features and store the index features in the index database.

[0104] For example, after obtaining the split content data stream and metadata stream, the system processes the two types of data in parallel to achieve content-addressed deduplication storage and retrieval. On the one hand, for the content data stream, the system anchors the data content by calculating a fixed-length encrypted digest value with anti-collision properties to determine whether the data already exists in the system. For duplicate data, only logical references are established to save storage space, while for new data, it is routed to the best physical node for persistent storage according to a distributed strategy. On the other hand, the system synchronously parses the metadata stream, extracts feature information with retrieval value such as webpage title and publication time, and establishes an index to ensure that the corresponding original data can be quickly located using keywords.

[0105] In one possible implementation, deduplication verification is performed in the storage system based on a unique fingerprint feature, and a reference link establishment operation or distributed storage operation is performed according to the verification result. Specifically, this includes: calculating the data block digest value of the content data stream using a preset hash algorithm as a unique fingerprint feature, and querying the index record corresponding to the data block digest value in the global index table of the index database; if an index record is found, extracting the physical address pointer stored in the index record, constructing a virtual file object pointing to the physical address pointer, and returning a write success signal; if no index record is found, mapping the data block digest value to the virtual node space using a consistent hash ring algorithm, determining the target write location according to the mapping relationship between the virtual node space and the physical storage node, writing the content data stream to the target write location, and creating a new index entry in the global index table containing the data block digest value and the target write location.

[0106] In this application embodiment, the consistent hash ring algorithm refers to a special distributed hash algorithm designed to solve the large-scale data migration problem caused by the dynamic addition or removal of nodes in a distributed cache or storage system. It represents mapping data objects and storage nodes to the same closed ring space (typically 0 to 2). 32 -1) topological logic; for example, the algorithm places the data digest value and the server IP hash value on a ring, and the data is always stored on the first server node encountered in a clockwise direction. When a server is added, only a small part of the data near the new node on the ring needs to be repositioned, without rearranging the entire data.

[0107] Specifically, the system extracts features from the incoming content data stream and uses a preset hash algorithm (such as SHA-256 or MD5) to perform a full calculation on the binary data block, generating a fixed-length, irreversible hexadecimal string, i.e., the data block digest value of the content data stream, as a unique fingerprint feature (this feature has anti-collision properties). This digest value is used as a unique key value and quickly searched in the global index table of the index database that maintains the distribution status of files across the entire network to check whether an index record corresponding to the data block digest value already exists. If an index record is found, it indicates that a copy of the data block already exists in the system. At this time, the "instant transfer" logic is triggered. Without performing physical data transmission, the physical address pointer stored in the index record is directly extracted (i.e., the metadata index node Inode Number or object storage object key of the existing file in the storage system). Based on this pointer, a system call (such as the `link` or `symlink` function in Linux) is executed at the file system level to create a hard link or symbolic link (i.e., the aforementioned virtual file object) that does not allocate a new physical disk block (data block) but only adds a mapping relationship to the same inode in the directory entry (dentry), and directly returns a write success signal to the requesting end. Conversely, if no index record is found, indicating that the data is newly added, the storage routing logic is initiated, using a consistent hash ring algorithm to map the calculated data block digest value to a ring-shaped integer space (e.g., 0 to 2) determined by the bit width of the hash function output. 32 Within the hash value range of -1, a clockwise search is performed along the ring space to find the nearest virtual node. Based on the pre-configured mapping relationship between virtual node space and physical storage nodes (such as a mapping table from virtual node to physical server IP), the target write location responsible for the data segment is determined. The content data stream is then physically written to the target write location via the network. After the writing is completed, a new index entry containing the data block digest value and the target write location is created synchronously in the global index table, thereby completing the registration and storage of the new data.

[0108] Figure 3 This is a schematic diagram of a module of an open data synchronization service system according to an embodiment of this application. This system can be implemented through software, hardware, or a combination of both, becoming all or part of the system. For example... Figure 3 As shown, the system includes:

[0109] The health assessment module 301 is configured to obtain network status information of distributed open data sources and mirror sites of open data sources, determine the availability indicators and transmission performance indicators of each open data source based on the network status information, and generate dynamic health evaluation indicators based on the availability indicators and transmission performance indicators.

[0110] The path sorting module 302 is configured to receive data synchronization requests, parse the source and destination attributes in the data synchronization requests, combine dynamic health evaluation indicators, prioritize candidate transmission paths with open data sources and mirror sites of open data sources as starting nodes, and determine the transmission path with the highest priority as the execution path.

[0111] The access verification module 303 is configured to identify the target data source pointed to by the execution path, obtain the metadata of the target data source, match and verify the metadata with the preset digital compliance rule base, and generate access control instructions for the data synchronization request based on the risk assessment results generated by the verification.

[0112] The task scheduling module 304 is configured to respond to the access control command to the pass state, establish a data acquisition task and distribute the data acquisition task to the acquisition nodes deployed on the execution path, and use the state management mechanism to dynamically control the state flow of the data acquisition task during its life cycle.

[0113] The data splitting module 305 is configured to receive the raw data stream returned by the data acquisition node when it performs the data acquisition task, split the raw data stream, and generate content data stream and metadata stream in parallel.

[0114] The deduplication module 306 is configured to calculate a unique fingerprint feature for the content data stream, perform deduplication verification in the storage system based on the unique fingerprint feature, and perform reference link establishment operation or distributed storage operation according to the verification result. It also parses the metadata stream to extract index features and stores the index features in the index database.

[0115] Based on the above embodiments, as an optional embodiment, the health assessment module 301 is specifically used to: extract connection success rate data contained in network status information as availability indicators, and extract round-trip delay data and throughput data contained in network status information as transmission performance indicators; use a preset time decay algorithm to perform a weighted average calculation on availability indicators and transmission performance indicators within a historical time window to obtain a smoothed state value for the current time period; input the smoothed state value into a preset scoring model, map smoothed state values ​​of different dimensions to standard scores through a normalization function in the scoring model, and sum the standard scores to generate a dynamic health evaluation index.

[0116] Based on the above embodiments, as an optional embodiment, the path sorting module 302 is specifically used to: construct a network topology graph based on source and destination attributes, and map candidate transmission paths to directed connected links in the network topology graph; read dynamic health evaluation indicators, and obtain the link cost value between each node in the directed connected link by calculating the reciprocal of the dynamic health evaluation indicators; traverse the network topology graph using a preset shortest path search algorithm, accumulate the link cost value of each link segment to calculate the total path cost, and generate a priority sort according to the total path cost from low to high.

[0117] Based on the above embodiments, as an optional embodiment, the access verification module 303 is specifically used to: parse the header description file of the target data source, extract attribute fields containing the data generation region, data sensitivity level and authorization protocol type as metadata; read the pre-stored conditional logic tree in the digital compliance rule base, wherein the action types associated with the leaf nodes of the conditional logic tree include blocking, warning and allowing; input the metadata and user location information in the data synchronization request into the conditional logic tree for hierarchical judgment, and when the judgment path falls into the leaf node associated with the blocking action type, generate an access control instruction indicating that synchronization is prohibited.

[0118] Based on the above embodiments, as an optional embodiment, the task scheduling module 304 is specifically used to: initialize the task status identifier to the suspended state in the preset state storage unit when creating a data acquisition task; when the acquisition node sends a ready signal, distribute the data acquisition task to the acquisition node and update the task status identifier to the running state; receive the completion confirmation signal returned by the acquisition node, parse the status confirmation identifier in the completion confirmation signal, and if the status confirmation identifier is verified, update the task status identifier to the archived state and release the computing resources of the acquisition node.

[0119] Based on the above embodiments, as an optional embodiment, the task scheduling module 304 is specifically used for: real-time statistical data collection task request failure frequency in the running state; when the request failure frequency exceeds a preset tolerance threshold, determining that the task execution is abnormal; calling the proxy resource interface to filter out backup Internet Protocol addresses in different network subnets from the preset global proxy pool, and generating a configuration update instruction containing the backup Internet Protocol address; sending the configuration update instruction and reset signal to the state management mechanism, reverting the task status identifier from the running state to the suspended state, and reinitializing the network connection session of the data collection task using the backup Internet Protocol address.

[0120] Based on the above embodiments, as an optional embodiment, the deduplication and data entry module 306 is specifically used to: calculate the data block digest value of the content data stream as a unique fingerprint feature using a preset hash algorithm, and query the index record corresponding to the data block digest value in the global index table of the index database; if an index record is found, extract the physical address pointer stored in the index record, construct a virtual file object pointing to the physical address pointer, and return a write success signal; if no index record is found, map the data block digest value to the virtual node space using a consistent hash ring algorithm, determine the target write location according to the mapping relationship between the virtual node space and the physical storage node, write the content data stream to the target write location, and create a new index entry in the global index table containing the data block digest value and the target write location.

[0121] It should be noted that the system provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0122] This embodiment also discloses an electronic device, as shown in the reference. Figure 4 The electronic device may include: at least one processor 401, at least one communication bus 402, user interface 403, network interface 404, and at least one memory 405.

[0123] The communication bus 402 is used to enable communication between these components.

[0124] The user interface 403 may include a display screen and a camera. Optionally, the user interface 403 may also include a standard wired interface and a wireless interface.

[0125] The network interface 404 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0126] The processor 401 may include one or more processing cores. The processor 401 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 405, and by calling data stored in memory 405. Optionally, the processor 401 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 401 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor 401.

[0127] The memory 405 may include random access memory (RAM) or read-only memory. Optionally, the memory 405 may include a non-transitory computer-readable storage medium. The memory 405 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 405 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 405 may also be at least one storage device located remotely from the aforementioned processor 401. Figure 4 As shown, the memory 405, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for an open data synchronization service method.

[0128] exist Figure 4In the electronic device shown, the user interface 403 is mainly used to provide an input interface for the user and to obtain the user input data; while the processor 401 can be used to call an application program stored in the memory 405 that provides an open data synchronization service method. When executed by one or more processors 401, the electronic device executes one or more methods as described in the above embodiments.

[0129] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0130] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0131] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be through some service interfaces; indirect couplings or communication connections between apparatuses or units may be electrical or other forms.

[0132] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0133] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0134] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory 405 and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory 405 includes various media capable of storing program code, such as a USB flash drive, external hard drive, magnetic disk, or optical disk.

[0135] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of other embodiments of this disclosure upon considering the disclosure in this specification. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are considered exemplary only, and the scope of this application is defined by the claims.

Claims

1. An open data synchronization service method, characterized in that, The method specifically includes: Obtain network status information of distributed open data sources and their mirror sites; determine availability and transmission performance indicators for each open data source based on the network status information; and generate dynamic health evaluation indicators based on the availability and transmission performance indicators. Receive a data synchronization request, parse the source and destination attributes in the data synchronization request, combine the dynamic health evaluation index, prioritize the candidate transmission paths starting from the open data source and the mirror site of the open data source, and determine the transmission path with the highest priority as the execution path. Identify the target data source pointed to by the execution path, obtain the metadata of the target data source, match and verify the metadata with the preset digital compliance rule base, and generate an access control instruction for the data synchronization request based on the risk assessment result generated by the verification. In response to the access control command being in the pass state, a data acquisition task is established and distributed to the acquisition nodes deployed on the execution path. The state management mechanism is used to dynamically control the state transition of the data acquisition task during its lifecycle. Receive the raw data stream returned by the acquisition node when it executes the data acquisition task, split the raw data stream, and generate content data stream and metadata stream in parallel; A unique fingerprint feature is calculated for the content data stream. Based on the unique fingerprint feature, deduplication verification is performed in the storage system. According to the verification result, a reference link establishment operation or a distributed storage operation is performed. The metadata stream is parsed to extract index features, and the index features are stored in the index database.

2. The method according to claim 1, characterized in that, The step of determining the availability and transmission performance metrics of each of the open data sources based on the network status information, and generating dynamic health evaluation metrics based on the availability and transmission performance metrics, specifically includes: The connection success rate data contained in the network status information is extracted as the availability index, and the round-trip latency data and throughput data contained in the network status information are extracted as the transmission performance index. The availability index and the transmission performance index within the historical time window are weighted and averaged using a preset time decay algorithm to obtain the smoothed state value of the current time period. The smoothed state value is input into a preset scoring model. The normalization function in the scoring model maps the smoothed state value of different dimensions to standard scores. The standard scores are then summed to generate the dynamic health evaluation index.

3. The method according to claim 2, characterized in that, The step of prioritizing candidate transmission paths originating from the open data source and its mirror sites, based on the dynamic health evaluation indicators, specifically includes: A network topology graph is constructed based on the source end attributes and the destination end attributes, and the candidate transmission paths are mapped to directed connected links in the network topology graph. Read the dynamic health evaluation index, and calculate the link cost between each node in the directed connected link by calculating the reciprocal of the dynamic health evaluation index; The network topology is traversed using a preset shortest path search algorithm. The link cost of each link segment is accumulated to calculate the total path cost, and the priority sort is generated in order of the total path cost from low to high.

4. The method according to claim 1, characterized in that, The process of acquiring metadata from the target data source, matching and verifying the metadata against a preset digital compliance rule base, and generating access control instructions for the data synchronization request based on the risk assessment results generated by the verification, specifically includes: Parse the header description file of the target data source and extract attribute fields containing data generation region, data sensitivity level and authorization agreement type as the metadata; Read the pre-stored conditional logic tree in the digital compliance rule base. The action types associated with the leaf nodes of the conditional logic tree include blocking, warning, and allowing. The metadata and the user's location information in the data synchronization request are input into the conditional logic tree for hierarchical determination. When the determination path falls into a leaf node associated with a blocking action type, the access control instruction indicating that synchronization is prohibited is generated.

5. The method according to claim 1, characterized in that, The use of a state management mechanism to dynamically control the state transition of the data acquisition task throughout its lifecycle specifically includes: When creating the data acquisition task, the task status identifier is initialized to a suspended state in the preset status storage unit; When the acquisition node sends a ready signal, the data acquisition task is distributed to the acquisition node, and the task status identifier is updated to running status; The system receives a completion confirmation signal returned by the acquisition node, parses the status confirmation identifier in the completion confirmation signal, and if the status confirmation identifier is verified, updates the task status identifier to the archived status and releases the computing resources of the acquisition node.

6. The method according to claim 5, characterized in that, The method further includes: The frequency of request failures during the running state of the data acquisition task is statistically analyzed in real time. When the frequency of request failures exceeds a preset tolerance threshold, the task execution is determined to be abnormal. Call the proxy resource interface to filter out backup Internet Protocol addresses that are in different network subnets from the preset global proxy pool and generate a configuration update instruction containing the backup Internet Protocol address. Send the configuration update command and reset signal to the state management mechanism, roll back the task status identifier from the running state to the suspended state, and reinitialize the network connection session of the data acquisition task using the backup Internet Protocol address.

7. The method according to claim 1, characterized in that, The process of performing deduplication verification in the storage system based on the unique fingerprint feature, and then performing reference link establishment or distributed storage operations based on the verification result, specifically includes: The data block digest value of the content data stream is calculated using a preset hash algorithm as the unique fingerprint feature, and the index record corresponding to the data block digest value is queried in the global index table of the index database. If the index record is found, the physical address pointer stored in the index record is extracted, a virtual file object pointing to the physical address pointer is constructed, and a write success signal is returned; If the index record is not found, the data block digest value is mapped to the virtual node space using the consistent hash ring algorithm. The target write location is determined according to the mapping relationship between the virtual node space and the physical storage node. The content data stream is written to the target write location, and a new index entry containing the data block digest value and the target write location is created in the global index table.

8. An open data synchronization service system, characterized in that, The system includes: The health assessment module is configured to obtain network status information of distributed open data sources and their mirror sites, determine availability and transmission performance indicators for each open data source based on the network status information, and generate dynamic health evaluation indicators based on the availability and transmission performance indicators. The path sorting module is configured to receive data synchronization requests, parse the source and destination attributes in the data synchronization requests, combine the dynamic health evaluation index, prioritize the candidate transmission paths starting from the open data source and the mirror site of the open data source, and determine the transmission path with the highest priority as the execution path. The access verification module is configured to identify the target data source pointed to by the execution path, obtain the metadata of the target data source, match and verify the metadata with the preset digital compliance rule base, and generate access control instructions for the data synchronization request based on the risk assessment results generated by the verification. The task scheduling module is configured to respond to the access control command as an approved state, establish a data acquisition task, and distribute the data acquisition task to the acquisition nodes deployed on the execution path. The module also uses a state management mechanism to dynamically manage the state transition of the data acquisition task during its lifecycle. The data splitting module is configured to receive the raw data stream returned by the acquisition node when it performs the data acquisition task, split the raw data stream, and generate content data stream and metadata stream in parallel. The deduplication and data entry module is configured to calculate a unique fingerprint feature for the content data stream, perform deduplication verification in the storage system based on the unique fingerprint feature, and perform reference link establishment operation or distributed storage operation according to the verification result, parse the metadata stream to extract index features, and store the index features into the index database.

9. An electronic device, characterized in that, The device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions. The user interface and the network interface are both used to communicate with other devices. The processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed, perform the method as described in any one of claims 1-7.

Images