Multi-source data full life cycle registration, subscription, synchronization, development and disposal system and method

Abstract

The application discloses a multi-source data full life cycle registration, subscription, synchronization, development and treatment system and method, and belongs to the technical field of big data management and data resource management, and comprises the following: a data registration module, which provides a standardized registration template for multi-source data, and assigns a globally unique resource identifier after verification; a unified directory management module, which constructs a global integrated data directory, binds the identifier and dynamically updates; a data subscription module, which performs shunt approval and configures a subscription strategy based on a data sensitivity level; an intelligent synchronization module, which performs intelligent synchronization in double modes according to the strategy and monitors a synchronization state in real time; a safe development linkage module, which realizes data safe development and achievement shelving through environment isolation and data desensitization processing; and a compliance treatment module, which performs hierarchical differentiated treatment based on a data sensitivity level and generates an official treatment report. The application forms a traceability link by connecting each link in series with the identifier as the main line. The application realizes integrated closed-loop management and control of multi-source data from registration to treatment.

Description

Technical Field

[0001] This invention belongs to the field of big data governance and data resource management technology, specifically involving the Internet and cloud computing, big data services in the new generation of information technology industry, and particularly involving a system and method for the synchronous development and processing of multi-source data throughout its entire lifecycle of registration, subscription and processing. Background Technology

[0002] With the deepening of urbanization, government, enterprise, and industry data resources are characterized by cross-entity, cross-domain, and heterogeneous formats. Data, as a generative element, needs to achieve orderly flow and efficient utilization among multiple entities. However, these three types of data belong to different management entities and differ in data formats, interface standards, network environments, and security levels, placing higher demands on unified management and cross-domain collaboration of data resources. Urban data service platforms need to simultaneously serve three roles: data providers, data developers, and data users, undertaking a series of processes including data resource registration, catalog display, subscription delivery, development testing, and compliance handling.

[0003] Most existing data management systems only cover a single step in the aforementioned process. Traditional government data management platforms are only for internal government data, with functions limited to data entry and static catalog display. They do not support external enterprise or industry data access, lack subscription and automated synchronization capabilities, and are not equipped with a secure data development environment or standardized data processing procedures. General-purpose data development tools focus on data processing and analysis, lacking a unified data registration and catalog management system. Data sources are chaotic, ownership is unclear, and they cannot meet government compliance requirements. Data flow and processing lack oversight processes.

[0004] In summary, to address the technical problems of existing solutions, such as fragmented processes, inconsistent standards, weak subscription and synchronization capabilities, lack of security in the development phase, non-standardized handling procedures, and lack of ownership and traceability, a multi-source data full lifecycle registration, subscription, synchronization, development, and handling system and method is proposed. Summary of the Invention

[0005] The main objective of this invention is to provide a multi-source data full lifecycle registration, subscription, synchronization, development, and disposal system and method to solve the technical problems existing in the prior art, such as fragmented processes, inconsistent standards, weak subscription and synchronization capabilities, lack of security in the development process, non-standard disposal processes, and lack of ownership and traceability.

[0006] Based on the first major aspect of the present invention, a multi-source data full lifecycle registration, subscription, synchronization, development, and processing system is provided, comprising:

[0007] The data registration module is used to provide standardized registration templates for data providers of multi-source data to register data, collect basic information, ownership information, classification and grading information, data sensitivity level and metadata information of the data, and assign a globally unique resource identifier to each multi-source data after passing the qualification and content verification, generate data archives and archive them.

[0008] The unified directory management module is used to automatically build a unified data directory for the entire domain by classifying the registered multi-source data according to multiple dimensions, providing hierarchical retrieval and display to the outside world, and automatically refreshing the directory when the status of multi-source data changes;

[0009] The data subscription module is used to perform traffic distribution and approval based on the data sensitivity level and the configured synchronization rules when the data user submits a subscription application for target data in the whole domain integrated data catalog, and to manage the status of the subscription task throughout its entire lifecycle.

[0010] The intelligent synchronization module is used to perform dual-mode intelligent synchronization based on subscription strategy-driven automated data flow. It synchronizes multi-source data from the data source to the target environment, monitors the synchronization status in real time, and triggers warnings and automatic retries in case of anomalies.

[0011] The secure development linkage module is used to receive multi-source data that has been intelligently synchronized for developers to use for testing, through network isolation and data anonymization between the production environment and the development and testing environment, and to deploy the developed code to the production environment for execution. The development results can be uploaded to the global integrated data catalog.

[0012] The compliance handling module is used to implement graded and differentiated handling strategies based on the data sensitivity level. After the multi-source data that has completed intelligent synchronization is reviewed, graded and differentiated handling will be carried out on multi-source data that has exceeded the usage period, is invalid or violates regulations, and a formal handling report will be generated.

[0013] Through the integration of the above-mentioned data registration module, unified directory management module, data subscription module, intelligent synchronization module, secure development linkage module, and compliant disposal module, each module is sequentially linked and interconnected according to the natural flow order of multi-source data access to destruction, forming an end-to-end closed-loop management architecture covering six major stages: registration, directory display, subscription, synchronization, development, and disposal. This architecture fundamentally solves the defects of existing solutions, such as severely fragmented system processes and independent operation of each stage. By integrating functional modules that were originally scattered across multiple independent platforms into one, and realizing the automatic flow and transmission of multi-source data through standardized module indirect interfaces, multi-source data can automatically advance along a preset full lifecycle path from registration to final disposal, eliminating the need for manual export and import between different systems and improving the management efficiency of the entire lifecycle of multi-source data.

[0014] As a further preferred embodiment, in the aforementioned system, the steps for performing data registration are as follows:

[0015] The data providers include, but are not limited to, government departments, enterprises, and industry organizations. Dedicated standardized registration templates are configured for different data providers. The required fields in the standardized registration templates include at least the data name, the subject of the data, the data ownership, the data classification and grading, the data sensitivity level, the data format, the update cycle, the metadata information, and the scope of legal use.

[0016] After the data provider fills in the standardized registration template and submits it, the system automatically verifies the compliance of the submitted entity's qualifications and data format by combining the submitted multi-source data with the third-party entity qualification database and data format fields.

[0017] Based on the data classification and sensitivity level entered, the final data classification and sensitivity level labels are manually confirmed and reviewed. After verification, a globally unique resource identifier is generated for each piece of multi-source data.

[0018] Using the globally unique resource identifier as the main thread, the operation logs of each link in the system can be linked together to form a complete traceability link and output the source traceability record;

[0019] Finally, metadata is automatically collected by connecting to the data source interface and automatically capturing metadata that includes at least the data structure, field definitions, and flow records. This metadata is then merged and archived with the manually registered information filled in by the data provider to form a complete data archive.

[0020] By setting unified mandatory fields in the registration template, it is ensured that multi-source data from different entities follow a unified description standard when entering the system, eliminating confusion in data naming, coding, and classification. Meanwhile, the global unique resource identifier is assigned in real time when the data is accessed and the metadata archive is established synchronously, providing an indispensable primary key foundation for the traceability of each subsequent operation, avoiding the problems of inconsistent standard systems and lack of ownership and traceability mechanisms.

[0021] As a further preferred option, in the aforementioned system, the integrated data directory is a tree-shaped directory structure, and its hierarchical relationship consists of three layers of nodes: the first layer of root nodes are divided according to data type, the second layer of nodes are divided according to industry, and the third layer of leaf nodes are divided according to field.

[0022] The integrated data catalog also provides keyword search and multi-condition combination filtering, which can quickly locate target data. When displaying data, basic information is shown, and core fields are hidden for sensitive data, only a de-identified summary is shown.

[0023] In this integrated data catalog, each catalog entry is bound to a globally unique resource identifier. Clicking on the corresponding catalog entry allows you to view the data archive and its usage status.

[0024] In the above technical solutions, multi-dimensional hierarchical organization breaks through the limitation of data being difficult to discover under a single classification dimension. Data users can quickly locate target data from any dimension based on their familiar knowledge system, easily achieving the retrieval and identification of multi-source data. The hierarchical display control ensures that the core fields of sensitive data are hidden at the unauthorized discovery stage, allowing data security protection to be implemented from the directory level, strengthening the protection measures for sensitive data.

[0025] As a further preferred embodiment, in the aforementioned system, the execution steps for performing the triage approval are as follows:

[0026] Data users select target data in the data catalog, submit a subscription application, and fill in the usage scenario, usage period, and access frequency. The data subscription module automatically sorts and approves data based on its sensitivity level.

[0027] The automatic approval process follows these rules: when the data sensitivity level is "general," it is automatically approved without human intervention.

[0028] When the data sensitivity level is important or core data, it will be pushed to the data provider for manual approval.

[0029] Once approved, the data subscription module allows data users to customize synchronization rules; these rules include synchronization method, synchronization period, data push target address, and data output format; after configuring the synchronization rules, the data user obtains the subscription strategy.

[0030] The data subscription module also supports renewal and cancellation of subscriptions. It automatically reminds data users to send renewal reminders before the subscription expires, and the corresponding subscription policy automatically expires after the expiration date.

[0031] In data subscription, the visual strategy configuration enables data users to complete all the preliminary steps of multi-source data acquisition independently without offline communication. The configuration of the automated synchronization strategy provides clear parameter instructions for the subsequent intelligent synchronization module, enhancing subscription and synchronization capabilities. The data sensitivity level-driven classification approval replaces the inefficient manual judgment of each order. The automatic approval of general data improves efficiency, while the manual approval of sensitive data ensures security, avoiding a one-size-fits-all approval process and achieving a balance between efficiency and security.

[0032] As a further preferred option, in the aforementioned system, the intelligent synchronization module is equipped with a multi-source adaptive synchronization interface, which is compatible with, but not limited to, database interfaces, API interfaces, file interfaces, and data stream interfaces, and adapts to different network environments.

[0033] When performing intelligent synchronization, a dual-mode synchronization of incremental and full data is adopted. In normal scenarios, incremental synchronization is performed to transmit only newly added and modified data, while full synchronization is triggered periodically to verify data integrity and correct data deviations.

[0034] Real-time monitoring of synchronization status includes tracking synchronization progress, data transmission volume, and success rate. In case of disconnection, data loss, or abnormal format, automatic retry is performed and an alarm signal is sent to the administrator.

[0035] After synchronization is complete, the synchronization time, data volume, and data recipient information will be automatically recorded.

[0036] The above technical solutions connect data sources in different network environments through multi-type interface adaptation, ensure the accuracy and consistency of multi-source data when it flows between multiple entities through automatic switching between incremental and full modes and periodic integrity checks, and realize the automatic handling of synchronization anomalies through monitoring and alarm mechanisms. This solves the technical problems in the existing technology that multi-source data cannot be automatically synchronized according to different update frequencies, manual handling is inefficient, and anomalies cannot be detected in a timely manner.

[0037] As a further preferred embodiment, in the aforementioned system, the secure development linkage module receives data that has been intelligently synchronized, sets up mutually isolated formal operating environments and development testing environments, and stores formal encrypted data in the external network area; the development testing environment provides anonymized sample data;

[0038] The secure development collaboration module has built-in development tools, including at least data cleaning tools, data transformation tools, data modeling tools, and data analysis tools. It supports multi-person collaborative development and version management. After secure development is completed, the code will be deployed to the production environment for execution.

[0039] During secure development, development operations are restricted to the development and testing environment only, recording editing, calculations, and code export operations.

[0040] During development, physical network isolation between the formal environment and the development and testing environment blocks the direct exposure of raw multi-source data to the development side. By providing only anonymized sample data, it is ensured that data operations during development and testing do not touch the real raw multi-source data. At the same time, the code release mechanism and operation log recording ensure that development behavior is completed within the controlled scope of development and is traceable. This solves the problems in existing technologies where data development and flow processes are disconnected, raw multi-source data is easily leaked due to the lack of an isolated development environment, and development behavior is not traceable.

[0041] As a further preferred option, in the aforementioned system, the compliance handling module supports configuring data strategies by database and table dimension, setting data retention time, and automatically destroying multi-source data that exceeds the retention period;

[0042] A tiered and differentiated handling strategy is implemented based on the data sensitivity level, specifically as follows:

[0043] General data is destroyed directly after being taken down. Important or core data is forcibly desensitized after being taken down, and then destroyed. Core data is destroyed in a closed loop and is prohibited from being transferred to a secondary source.

[0044] The compliance handling module supports two handling modes: manual initiation and automatic system triggering. All handling operations must be reviewed and approved by the data provider before execution.

[0045] After the disposal process is completed, including removal from shelves, mandatory desensitization, and destruction, the disposal results are automatically verified, and a formal disposal report is automatically generated upon completion.

[0046] In the above technical solution, the delisting, desensitization, and destruction operations at the end of the entire life cycle are deeply bound to the data sensitivity level. Different levels of data correspond to different sequences of handling actions. A closed-loop compliance handling process is formed through pre-approval, result verification, and report generation. This standardizes the handling process, avoids the lack of handling records, and provides legally admissible evidence support for compliance audits.

[0047] Based on the second main aspect of the present invention, a method for registering, subscribing, synchronizing, developing, and processing multi-source data throughout its entire lifecycle is provided, applied to the aforementioned system, comprising the following steps:

[0048] Acquire multi-source data and provide a standardized registration template for data registration. After passing the qualification and content verification, assign a globally unique resource identifier to each multi-source data and establish a data archive. The globally unique resource identifier is linked with the operation logs of each step in the method to form a complete traceability link and output the traceability record.

[0049] The registered multi-source data will be automatically classified into a unified data catalog across the entire domain according to multiple dimensions, providing hierarchical retrieval and display to the outside world, and automatically refreshing the catalog when the data status changes;

[0050] Data users select target data in the unified data catalog and submit a subscription application. The data sensitivity level is combined with the traffic approval process, and a subscription strategy is generated based on the synchronization rules.

[0051] Based on the subscription strategy, multi-source data is synchronized from the data source to the target environment through multiple data interfaces, and the synchronization status is monitored in real time. In case of anomalies, alarms are triggered and automatic retry is performed.

[0052] The system receives multi-source data that has been intelligently synchronized for developers to test, deploys the developed code to the production environment for execution, and uploads the development results to the unified global data catalog.

[0053] Based on the data sensitivity level, a graded and differentiated handling strategy is implemented. After the multi-source data that has completed intelligent synchronization is reviewed, the multi-source data that has exceeded its usage period, is invalid, or violates regulations will be subject to graded and differentiated handling, and a formal handling report will be generated.

[0054] Based on a third key aspect of the present invention, an electronic device is provided, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0055] The memory stores a computer program, which, when executed by the processor, causes the processor to perform the aforementioned method for multi-source data full lifecycle registration, subscription, synchronous development, and processing.

[0056] Based on the fourth key aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed, implements the aforementioned method for multi-source data full lifecycle registration, subscription, synchronous development, and processing.

[0057] Compared with existing technologies, this invention provides a multi-source data full lifecycle registration, subscription, synchronization, development, and disposal system and method. Firstly, this invention connects six key stages—data registration, unified cataloging, subscription, synchronization, secure development, and compliant disposal—to form an integrated closed-loop management process covering the entire data lifecycle from access to destruction. Each module operates sequentially according to the data flow order. Any status change of data at any stage is automatically synchronized to the catalog and associated with a globally unique resource identifier. This achieves standardized access, automated flow, and unified presentation of multi-source heterogeneous data without manual coordination, eliminating the problems of repetitive operations and information inconsistencies caused by fragmented processes in traditional management models.

[0058] Secondly, this invention establishes a unified registration standard and catalog system for multi-source data from different sources, compatible with heterogeneous data formats, and achieves centralized and standardized management of cross-domain data. Based on this, it implements differentiated subscription tiered approval and automated incremental or full synchronization according to data sensitivity levels. It deeply integrates the data flow link after synchronization with a secure, isolated development and testing environment, blocking channels for original data leakage and balancing data availability and security. At the end of the data lifecycle, it matches sensitivity levels with tiered disposal rules for removal, de-identification, or destruction, replacing manual judgment and operation.

[0059] Finally, using a globally unique resource identifier as the main thread to connect the entire process of operation logs, we can achieve full traceability of each piece of data from registration and access to compliant disposal, providing complete technical support for the large-scale governance of multi-entity and multi-type data at the city level. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, obtaining other drawings based on these drawings without creative effort still falls within the scope of the present invention.

[0061] Figure 1 This diagram illustrates a system architecture of a multi-source data full lifecycle registration, subscription, synchronous development, and processing system according to an embodiment of the present invention.

[0062] Figure 2 The diagram illustrates the execution flow of a multi-source data full lifecycle registration, subscription, synchronous development, and processing method according to an embodiment of the present invention. Detailed Implementation

[0063] The preferred embodiments of the present invention will be described in detail below to provide a clearer understanding of the purpose, features, and advantages of the invention. It should be understood that the following embodiments are not intended to limit the scope of the invention, but are merely illustrative of the essential spirit of the technical solution of the invention.

[0064] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments in order to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the art will recognize that embodiments may be practiced without one or more of these specific details. In other instances, well-known techniques associated with the invention may not have been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0065] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, the appearance of "in an embodiment" or "an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any manner in one or more embodiments.

[0066] In one feasible embodiment, the present invention provides a multi-source data full lifecycle registration, subscription, synchronization, development, and processing system, comprising:

[0067] The data registration module is used to provide standardized registration templates for data providers to register data, collect basic information, ownership information, classification and grading information, data sensitivity level and metadata information of the data, and assign a globally unique resource identifier to each data after qualification and content verification, generate data archives and archive them.

[0068] The unified directory management module is used to automatically build a unified data directory for the entire domain by classifying the registered multi-source data according to multiple dimensions, providing hierarchical retrieval and display to the outside world, and automatically refreshing the directory when the data status changes;

[0069] The data subscription module is used to perform traffic distribution and approval based on the data sensitivity level and the configured synchronization rules when the data user submits a subscription application for target data in the whole domain integrated data catalog, and to manage the status of the subscription task throughout its entire lifecycle.

[0070] The intelligent synchronization module is used to perform dual-mode intelligent synchronization based on subscription strategy-driven automated data flow. It synchronizes multi-source data from the data source to the target environment, monitors the synchronization status in real time, and triggers warnings and automatic retries in case of anomalies.

[0071] The secure development linkage module is used to receive multi-source data that has been intelligently synchronized for developers to use for testing, through network isolation and data anonymization between the production environment and the development and testing environment, and to deploy the developed code to the production environment for execution. The development results can be uploaded to the global integrated data catalog.

[0072] The compliance handling module is used to implement graded and differentiated handling strategies based on the data sensitivity level. After the multi-source data that has completed intelligent synchronization is reviewed, graded and differentiated handling will be carried out on multi-source data that has exceeded the usage period, is invalid or violates regulations, and a formal handling report will be generated.

[0073] Combination Figure 1 As shown, this invention is a multi-source data full lifecycle management and control system, which can be divided into seven core modules: data registration module, unified directory management module, data subscription module, intelligent synchronization module, secure development linkage module, compliance handling module, and full-process traceability and auditing module. Each module is linked in sequence according to the data flow order to form a closed-loop management system, and uniformly connects to three types of data sources: government data, enterprise data, and industry data.

[0074] The data registration module is described in one of the following feasible embodiments.

[0075] The data registration module is the data entry point and source of the entire system. Its core function is to transform the multi-source heterogeneous raw data provided by government departments, enterprises, and industry organizations (hereinafter referred to as data or multi-source data) into a unified data resource that the system can identify, manage, and trace through a standardized and regulated registration process. This provides basic data support for subsequent catalog display, subscription, synchronization, development, processing, and traceability.

[0076] This module has built-in standardized registration templates for government data, enterprise data, and industry data. Each template uniformly includes required fields such as data name, owner, data ownership, data classification and grading, data sensitivity level, data format, update cycle, metadata information, and legal scope of use.

[0077] Data providers select the corresponding standardized registration template based on their various types of heterogeneous multi-source data, and fill in the above fields one by one on the interface to complete the entry of basic registration information.

[0078] After the data provider submits the registration information form, the data registration module initiates a dual verification process for qualifications and content. The submitted entity's qualifications are automatically verified by connecting to third-party entity qualification databases, such as business registration information and government agency directories, automatically comparing the submitting entity's name, unified social credit code, or organization code to verify its legitimacy.

[0079] Perform format compliance checks on multi-source data samples to verify whether structured data conforms to field type definitions and whether semi-structured or unstructured data meets basic format requirements.

[0080] For data that passes automatic verification, the system will push the registration information to the management terminal, where managers will manually confirm and label the data classification, grading, and sensitivity level.

[0081] After verification, the data registration module assigns a globally unique resource identifier to each data entry. This identifier uniquely identifies the data entry across the entire system, officially incorporating the data into system management.

[0082] During the registration and verification process, the data registration module simultaneously initiates an automatic metadata collection process. By connecting to the data interface and after authorization from the data provider, it automatically retrieves metadata information such as the data source's structure definition, field types and lengths, data update frequency, historical flow records, and access permissions.

[0083] Automatically collected metadata and manually entered registration information are combined and stored in the same data archive, forming a complete data archive containing both technical and business metadata.

[0084] After completing the aforementioned verification, identifier allocation, and metadata collection, the data registration module archives and stores the registration records and data files in real time. The archived content includes the subject information of the registration operation, operation data, verification records, the assigned globally unique resource identifier, and complete metadata and business information.

[0085] Archived records serve as the foundational data for subsequent data catalog display, subscription approval, synchronization, development and use, and compliance handling, with a globally unique resource identifier indexed throughout the entire process.

[0086] The unified directory management module is described in one of the following feasible embodiments.

[0087] The unified catalog management module takes over the data that has been registered by the data registration module and is responsible for building a multi-source integrated catalog system covering the entire domain. It is the core discovery hub connecting the data supply side and the data consumption side.

[0088] This module organizes, classifies, and displays scattered multi-source data according to unified standards, providing data users with centralized data resource retrieval and discovery capabilities. This enables data users to quickly locate the data they need and understand its basic information, thereby triggering the subsequent subscription process.

[0089] Based on the multi-source data that has been registered by the data registration module, the catalog building process is triggered after the data registration and archiving are completed.

[0090] The unified directory management module reads the registered data entries and their classification attributes, such as global unique resource identifier, data type, industry, field, and data sensitivity level, from the archive storage of the data registration module. It then classifies the data entries into layers according to three dimensions: data type (e.g., government data, enterprise data, industry data), industry (e.g., finance, transportation, healthcare), and field (e.g., public services, urban governance, industrial economy), and constructs a tree-shaped directory structure.

[0091] The root node of the tree-structured directory is divided according to data type. Each data entry and the classification attribute in the registration information are attached to the leaf node of the corresponding level, forming a unified data directory across the entire domain.

[0092] The unified directory management module provides centralized display and retrieval services. When a data user enters the directory, this module determines the scope of data that the user can access based on their permissions and displays the corresponding level of data entries in the tree-structured directory.

[0093] Multi-source data displays basic information in the data catalog, including data description information, registration information, and listing information, such as data name, owner, data format, and update cycle.

[0094] Data users can locate target data by searching with keywords or by selecting multiple conditions such as data type, industry, and field. The search and filtering operations are applied to all nodes in the tree-shaped directory, returning a list of data entries that meet the conditions and their respective directory paths.

[0095] This module establishes a mechanism for associating directory entries with data archives. Each directory entry stores a corresponding globally unique resource identifier. When a data user clicks on any data entry in the directory interface, the unified directory management module retrieves the complete registration archive of that data from the data registration module based on the stored globally unique resource identifier. This archive includes metadata information, ownership information, classification and grading tags, and usage status, and is displayed to the data user via a page jump or pop-up window.

[0096] Meanwhile, the unified directory management module marks the current usage status of each piece of multi-source data in the interface. This status is updated in real time by each downstream module to the archived record of the data registration module during the multi-source data flow process. The unified directory management module displays the status by reading the status field in the archived record.

[0097] A dynamic update channel for the directory is established between the unified directory management module, the data registration module, and the compliance handling module.

[0098] Once the data registration module completes the registration of new data and assigns a globally unique resource identifier, the data registration module automatically triggers a directory update signal. After receiving the signal, the unified directory management module adds the new multi-source data to the corresponding node in the tree directory according to the classification attributes of the data.

[0099] When changes occur in multi-source data, the data registration module will synchronize the change information to the unified directory management module and update the display content of the corresponding directory entries.

[0100] Once the compliance handling module completes the data removal operation, it sends an invalidation signal to the directory entry of the removed data. The unified directory management module immediately removes the entry from the tree directory or marks it as inaccessible, ensuring that the directory display content is consistent with the actual data status in real time.

[0101] The data subscription module is described in one of the following feasible embodiments.

[0102] The data subscription module is the core hub connecting the unified directory management module and the intelligent synchronization module, acting as a bridge from multi-source data discovery to multi-source data acquisition. After discovering the required multi-source data in the directory, the data user submits a multi-source data acquisition request through the data subscription module. After approval, the data synchronization strategy is configured, ultimately driving the intelligent synchronization module to complete the actual data transmission and delivery.

[0103] The data subscription module initiates the subscription process after the data user locates the target data through the unified directory management module.

[0104] Specifically, the data user selects the target data entry in the catalog interface. The data subscription module reads the globally unique resource identifier stored in the data user and the target data entry, calls the archived record of the data registration module, and obtains the data sensitivity level information of the data entry.

[0105] The module presents a subscription application interface to the data user. The data name, subject, and data format fields are automatically filled in by the system and cannot be edited. The data user needs to fill in three pieces of information in the interface: usage scenario, usage period, and access frequency, and submit the subscription application. The subscription application binds the data user's identifier, globally unique resource identifier, and the usage information filled in.

[0106] The data subscription module performs triage and approval based on the data sensitivity level of the target data. For general data, a subscription certificate is generated after the data provider's approval is completed.

[0107] If the data is important or core data, it will be manually approved by the data provider and then transferred to the platform operators for approval. During this process, the data provider and operators can view the identity information of the data user and the data usage intent on the terminal, and choose to agree or reject the subscription application.

[0108] The approval result is recorded by the module and fed back to the data user who applied. If the approval is approved, a subscription certificate is generated. If the approval is rejected, the subscription is terminated and the data user is notified.

[0109] Once approved, the data subscription module opens the subscription strategy configuration interface to data users. Data users can customize synchronization rules in the interface, including synchronization method, synchronization period, data push target address, and data output format.

[0110] The module collects the parameters configured by the data user, binds them to a globally unique resource identifier and a data user identifier, and stores the complete subscription policy in the subscription task record. This subscription policy is then read by the intelligent synchronization module as the basis for driving automated data flow.

[0111] The data subscription module performs full lifecycle status management for all subscription tasks, maintaining a list of subscription task records. Each record includes the subscription policy, approval history, effective date, and current status. Data users can view all submitted subscription tasks and their statuses in the subscription management interface, and can renew or cancel subscriptions.

[0112] The module sends a renewal reminder to the data user's terminal before the subscription expires. If the data user does not renew after the reminder, the module will automatically mark the subscription task as invalid after the subscription expires, and the corresponding subscription policy will no longer be executed by the smart synchronization module.

[0113] All subscription requests, approval processes, policy configurations, and status changes generate operation logs within the module. These logs record the operation subject, operation time, operation content, and execution result, are associated with a globally unique resource identifier, and are transmitted to the full-process traceability and auditing module.

[0114] The intelligent synchronization module is described in one of the following feasible embodiments.

[0115] The intelligent synchronization module is the core hub of the entire system's data flow. It receives the subscription strategy output by the data subscription module and is responsible for automatically transmitting multi-source data from the data provider to the target environment specified by the data user according to a predetermined strategy. Located between the subscription and development phases, this module completes the material flow of data from the supplier to the user, and is a key executor for achieving automated data delivery.

[0116] The intelligent synchronization module initiates the synchronization process after the data subscription module generates the subscription policy. This module establishes a policy reading channel with the data subscription module to obtain newly added and updated subscription policies in real time.

[0117] The intelligent synchronization module identifies the source data interface type based on the data source address type field in the subscription strategy. This module contains multiple data source interfaces, such as database connection interface, API call interface, file transfer interface, and data stream access interface, which are compatible with different network environments such as government intranets, enterprise private networks, and industry platforms.

[0118] The intelligent synchronization module starts synchronization tasks according to the synchronization period configured in the subscription strategy. For tasks configured for incremental synchronization, the module completes the full baseline extraction and writing of the source data on the first execution. On subsequent periodic triggers, it only reads the new and modified data from the source since the last synchronization time.

[0119] For tasks configured for full synchronization, the module reads the complete dataset of the data entry from the source each time it is triggered and overwrites it at the target address.

[0120] Synchronization tasks are triggered by timed triggers and event triggers. Timed triggers are executed according to the synchronization cycle in the subscription strategy, while event triggers are driven by the data update signal from the source end. The module supports both trigger modes and is marked in the subscription strategy.

[0121] Throughout the entire synchronization task execution process, the module collects and monitors multi-source data in real time, including the current synchronization progress, data transmission volume, and success rate. During synchronization, the module checks the data transmission status line by line. If data transmission is interrupted, data is missing, or the format is abnormal, automatic retry is initiated, and an alarm signal is sent to the administrator terminal.

[0122] After the synchronization is completed, the intelligent synchronization module generates a flow log, which includes the globally unique resource identifier of this synchronization, the synchronization time, the amount of data, and the recipient. The flow log is archived in real time after it is generated.

[0123] The secure development linkage module is described in one of the following feasible embodiments.

[0124] It is the core hub connecting data supply and data value creation, located before the intelligent synchronization module and the compliance handling module. The core mission of this module is to provide data developers with a development and testing environment based on real business scenarios, while ensuring the security of raw data. This allows them to conduct multi-source data exploration, model development, and algorithm verification based on anonymized sample data. After development, the results are deployed to the production environment for execution, ultimately producing data products that can be sold and circulated.

[0125] After completing data synchronization, the intelligent synchronization module sends a multi-source data arrival signal to the secure development linkage module. Upon receiving the signal, the secure development linkage module initiates the multi-source data receiving process, distinguishing between development / test data and formal data based on the globally unique resource identifier and target environment identifier carried in the multi-source data arrival signal.

[0126] For official data, the module will directly write the multi-source data completed synchronously into the encrypted storage node of the government extranet area, and use the encryption algorithm recognized by the State Cryptography Administration to encrypt and store the data content. The encryption key is generated independently by the module and stored in a secure area.

[0127] For development and testing data, the module automatically generates de-identification tasks based on the data structure definition in the synchronized data. The de-identification tasks include field-level de-identification rule configurations, which perform de-identification operations such as replacement, masking, or offset on sensitive personal fields such as name, ID number, phone number, and address, generating de-identified sample datasets.

[0128] The secure development collaboration module establishes a network isolation mechanism between the production environment and the development / testing environment. The production environment is deployed in the government extranet area or the enterprise production network area, while the development / testing environment is deployed in a separate test network segment. Access control isolation between the two network segments is achieved through a firewall, and there is no direct data channel between the data encryption storage node of the production environment and the data storage node of the development / testing environment.

[0129] The anonymized sample dataset is transferred unidirectionally from the production environment to the development and testing environment by the module through a data transfer mechanism at the network isolation boundary. After the transfer is completed, the temporary anonymized data copy is immediately deleted on the production environment side to ensure that the original multi-source data does not flow into the deployment area due to development and testing operations. At the same time, the computing resources, storage resources and network configuration in the development and testing environment are consistent with those in the production environment.

[0130] After the development and testing environment is deployed, the data development team uses the visual development tools integrated into the secure development collaboration module to perform data exploration, data cleaning, data transformation, data modeling, and data analysis. The visual development tools include data cleaning, data transformation, data modeling, and data analysis components, supporting multi-user collaborative development, version management, and online retention of results.

[0131] Within the module, the scope of operations for data developers is restricted through the development behavior submodule, prohibiting them from accessing any data resources in the storage nodes of the development and testing environment other than the currently authorized de-identified sample dataset. All operations performed by data developers in the development and testing environment are recorded: data exploration, data script development, model training, data testing, workflow configuration and execution, etc., and each operation record is transmitted in real time to the end-to-end traceability and auditing module.

[0132] After the data developer completes development and testing, they submit a review application. The security development collaboration module packages the generated code and scripts and submits them to the review process. Once the review is approved, the module imports the code and scripts from the development and testing environment into the production environment via the code deployment channel. In the production environment, the code and scripts are executed by connecting to the encrypted storage of the original data.

[0133] After execution, the module performs desensitization, encryption, and watermarking on the output structure. For example, it desensitizes sensitive fields in the results, encrypts and stores the complete result file, and embeds digital watermark information such as usage identifier and delivery time in the header or footer area of ​​the result file.

[0134] The processed result files are output to the data users through a secure delivery channel or after being anonymized. At the same time, the module supports uploading the development results to the unified directory management module. When submitting the application for listing, the data developer fills in the name of the result, the type of result, and the instructions for use. After the listing is approved, the result is included as a new data product in the whole-domain integrated data catalog to realize the circulation of data resources.

[0135] The compliance handling module is described in one of the following feasible embodiments.

[0136] The compliance handling module is the final link in the data lifecycle management, undertaking the function of handling data at the end of its lifecycle. The core mission of this module is to ensure that multi-source data that has expired, become invalid, violated regulations, or is outdated can be disposed of through standardized processes and differentiated strategies, including removal, de-identification, and destruction. At the same time, it reduces human intervention and mitigates the risk of data residue due to human negligence by using automated retention strategies.

[0137] The compliance handling module's handling process is initiated by two trigger modes: manual initiation and automatic system triggering.

[0138] In the manual initiation mode, administrators process disposal applications in the compliance disposal module, locate the data entries to be disposed of through the globally unique resource identifier or the integrated data directory of the entire domain, select disposal operations such as de-listing, de-identification or destruction, fill in the disposal reason and submit the disposal application.

[0139] Under automatic system triggering, a data lifecycle monitoring channel is established between the data registration module and the compliance handling module. Each data entry in the data registration module records its subscription validity period and retention period. When multi-source data in the system exceeds the subscription validity period or retention period, the data registration module generates an overdue signal and sends it to the compliance handling module. The module automatically constructs a corresponding data handling application and marks the handling operation as destruction, without involving manual action initiated by the data provider.

[0140] Upon receiving a disposal application, the compliance handling module reads the globally unique resource identifier of the target data, obtains the data sensitivity level of the data through the archived records of the data registration module, and matches the corresponding graded and differentiated handling strategy based on the data sensitivity level.

[0141] The tiered differentiated handling strategy is as follows: For resources and products that do not contain any data after the destruction operation is performed according to the retention strategy, a direct removal and destruction strategy is adopted. The module first performs a catalog removal operation, sends a removal instruction to the unified catalog management module to remove the catalog display of the data entry, and then performs a destruction operation to delete the original data file existing in the production environment. If any data exists in the resources and products after the destruction operation is performed according to the retention strategy, the removal operation is not performed.

[0142] For resources and products for which no data exists after the destruction operation is performed according to the retention policy configuration, a strategy of removal, desensitization, and then destruction is adopted. The module first performs the catalog removal operation, then calls the desensitization process to perform field-level desensitization on the data content, generates a desensitized data copy for archiving, and finally performs the destruction operation. If any data exists in the resources and products after the destruction operation is performed according to the retention policy configuration, the removal operation will not be performed.

[0143] The core data adopts a closed-loop destruction strategy. For resources and products that do not contain any data after the destruction operation is performed according to the retention policy configuration, the module first performs a catalog removal operation, then marks the data as a destruction pending approval status, and performs data overwriting and deletion within the production environment. It is prohibited to output the data content to any location outside the production environment. After the destruction is completed, the module independently generates and stores the destruction certificate. Data copies or exported files are generated throughout the process. If any data exists in the resources and products after the destruction operation is performed according to the retention policy configuration, the removal operation will not be performed.

[0144] After the disposal application is submitted, the compliance disposal module performs triage and approval based on the data sensitivity level of the target data, similar to the approval process in the data subscription module.

[0145] Applications for processing general data are automatically approved by the system without human intervention, and the process begins immediately after approval.

[0146] Applications for the handling of important or core data are routed to the data provider's terminal for manual approval. The data provider views the reason for handling, data identifier, and data content summary in the application on the terminal, and chooses to approve or reject the handling operation. The approval result is recorded by the module and fed back to the application initiator. Only after approval can the execution phase begin.

[0147] During the execution of the disposal operation, the compliance disposal operation automatically verifies the disposal results. For resources and products that no longer have data after the disposal operation due to exceeding the timeliness requirement, a delisting operation is performed, and the delisted resources and products will not be displayed in the data catalog.

[0148] For the desensitization operation, the module reads the desensitized data copy output after desensitization, verifies whether all sensitive fields have been replaced or overwritten, confirms that all desensitized fields have been desensitized, and records the desensitization completion status after the verification is passed.

[0149] For the destruction operation, the module physically deletes the data files on the data storage nodes in the production environment. After deletion, it overwrites the data on the storage nodes. After overwriting, it reads the space release confirmation information returned by the storage nodes to confirm that the multi-source data cannot be recovered. If the verification passes, it records the destruction completion status. If the verification fails, the module pauses the current processing operation and sends an alarm signal to the administrator terminal.

[0150] The compliance handling module automatically generates a handling report after the handling operation is completed and verified. The handling report includes information such as handling type, operation time, operator, and data identifier. Auditors can query and access the report through the compliance audit interface by data identifier, handling type, and time range.

[0151] The disposal report is also linked to the process traceability audit module, and is merged into the complete traceability chain of the data using a globally unique resource identifier as an index. After the disposal is completed, the compliance disposal module will write the final status back to the data registration module, and the data registration module will update the lifecycle status of the corresponding data entry.

[0152] In the approval process of the data subscription module and the compliance handling module, the approval process for important or core data will be changed from single-level approval to multi-level joint approval, which will strengthen the control of core multi-source data without changing the main process of the entire life cycle.

[0153] The system can configure different approval levels for data with different sensitivity levels. For example, important data can be configured for Level 1 approval, which only requires approval from the person in charge of the data provider to take effect, while core data can be configured for Level 3 approval. During the approval process, the business department head of the data provider first conducts a business compliance review and signs off, then it is transferred to the data security management department for security compliance checks and signs off, and finally it is transferred to the supervisor in charge for final approval.

[0154] Each level of approver independently reviews the application data. If any level of approval fails, the entire approval process is terminated, and the system provides feedback to the applicant regarding the specific reasons for rejection and the approval process.

[0155] In the compliance handling module, when performing de-listing, de-identification, or destruction operations on data that meets the disposal conditions, a pre-assessment of data resource revitalization can be added before formal disposal. For data that has been idle for a long time but has not met the prescribed destruction conditions, an asset value assessment should be conducted first, and the subsequent disposal path should be determined based on the assessment results.

[0156] Specifically, the system periodically scans the data storage area to identify idle multi-source data that has not been accessed for a long time or has zero or very low subscription counts. This data is then included in the asset revitalization assessment list. First, an assessment notification is sent to the data provider to determine whether the long-term idle data under their name needs to be assessed. If the data provider confirms that an assessment is needed, the system collects the historical subscription counts and access frequency of the multi-source data, the data size and quality score, the supply and demand of similar data in the market, and the potential application scenarios and value estimates of the multi-source data.

[0157] After the evaluation is completed, the system will provide feedback to the data provider based on the evaluation results. If the evaluation determines that the multi-source data still has potential value, it will recommend that the data provider re-describe the multi-source data or adjust the usage conditions before re-uploading it.

[0158] For data deemed to have low value after assessment but whose retention period has not yet expired, it is recommended that the data provider migrate multi-source data from hot storage to cold archive storage to reduce storage costs, and then destroy it after the retention period expires.

[0159] For data that is no longer valuable and can be destroyed, it is recommended to proceed directly to the destruction process.

[0160] The following feasible embodiment illustrates the full-process traceability and auditing module.

[0161] The end-to-end traceability and auditing module runs through all the above links, enabling full-chain traceability of multi-source data from registration to disposal.

[0162] The end-to-end traceability and auditing module serves as the infrastructure throughout all aspects of the system. It does not participate in the processing of business data, but rather acts as a horizontal support layer of the system architecture through end-to-end monitoring. It establishes connections with the data registration module, unified directory management module, data subscription module, intelligent synchronization module, security development linkage module, and compliance handling module. After each operation of each business module is completed, the module automatically pushes the operation log to that module.

[0163] The end-to-end traceability and auditing module organizes its storage structure using a globally unique resource identifier as the primary index key. All operation logs corresponding to the globally unique resource identifier are assigned the same index key value when written and stored in the log storage partition.

[0164] The module extracts globally unique resource identifiers, synchronization time, synchronization method, data volume, and success rate from the flow logs received from the intelligent synchronization module; and extracts the handling type, operation time, approval records, and final handling status from the handling reports received from the security development linkage module. Each module's logs are appended with a device or node information field when written, recording the network address and terminal identifier where the operation occurred.

[0165] Within the module, each log record is assigned an incrementing sequence number. Log records with the same globally unique resource identifier are arranged in sequence number order, forming a time series organized chronologically.

[0166] The end-to-end traceability and auditing module provides a multi-dimensional query interface. After auditors enter a globally unique resource identifier through this interface, the module retrieves all log records corresponding to that identifier from the log storage area and displays the complete event chain of the multi-source data from registration and access to disposal and destruction in a timeline format according to the sequence. The event chain is marked by each event node with the operation steps of registration, directory access, subscription, synchronization, development, and disposal, as well as the corresponding operation time.

[0167] It also supports combined searches by time period, by operation subject, and by data type. The module summarizes and displays log records that meet the filtering criteria in chronological order. The module also returns a registration summary of the data entry in the query response, including the data name, owner, ownership, sensitivity level, and format, to assist auditors in verifying the data's identity.

[0168] The end-to-end traceability and auditing module implements anti-tampering protection for the stored log data. When writing log records, the module calculates the hash value of each log record, generates a fixed-length hash digest using the SHA-256 algorithm, and stores the hash digest together with the original log record.

[0169] The module periodically performs integrity checks on stored log records, recalculates the hash value of the record and compares it with the stored hash digest. If they do not match, the record is marked as tampered, the tamper detection time and record identifier are recorded, and an integrity alarm is sent to the administrator terminal.

[0170] The module configures the retention period for log storage. The retention period shall not be less than the audit data archiving period required by laws and regulations. Log records that exceed the retention period shall be archived by the module, migrated from online storage to offline archiving storage media, and after the migration is completed, the corresponding log records shall be deleted from the online storage area and an archive list shall be generated.

[0171] In one feasible embodiment, combined with Figure 2 As shown, the present invention also provides a method for multi-source data full lifecycle registration, subscription, synchronization, development, and processing, applied to the system, comprising the following steps:

[0172] S100: Acquire multi-source data and provide a standardized registration template for data registration. After passing the qualification and content verification, assign a globally unique resource identifier to each multi-source data and establish a data archive. The globally unique resource identifier is linked with the operation logs of each step in the method to form a complete traceability link and output the traceability record.

[0173] S200 automatically constructs a unified data catalog across the entire domain by classifying the registered multi-source data according to multiple dimensions, providing hierarchical retrieval and display to the outside world, and automatically refreshing the catalog when the data status changes;

[0174] S300: Data users select target data in the unified data catalog and submit a subscription application. The data sensitivity level is combined with the traffic approval process, and a subscription strategy is generated based on the synchronization rules.

[0175] S400: Based on the subscription strategy, multi-source data is synchronized from the data source to the target environment through multiple data interfaces, and the synchronization status is monitored in real time. In case of an anomaly, an alarm is triggered and an automatic retry is performed.

[0176] S500 receives multi-source data that has been intelligently synchronized for developers to test, deploys the developed code to the production environment for execution, and the development results can be uploaded to the global integrated data catalog.

[0177] S600 implements a graded and differentiated handling strategy based on data sensitivity level. After reviewing multi-source data that has completed intelligent synchronization, it performs graded and differentiated handling on multi-source data that has exceeded its usage period, is invalid, or violates regulations, and generates a formal handling report.

[0178] In one feasible embodiment, the system and method of the present invention are illustrated using a specific use case.

[0179] Scenario: Data providers register data resources and synchronize them to the data catalog for display. Data developers find the required data in the data catalog based on the data requirements put forward by the data demanders. After subscribing, the data provider synchronizes the data to the data developer for data product development. The data provider manages the historical data slices provided to the data developer by configuring retention policies.

[0180] Pain points of traditional methods (comparison): Fragmented processes: Data access, display, development, and supply processes are independent, resulting in cumbersome operations and low efficiency; Lack of unified data registration standards and specifications: Data dictionaries and metadata information are messy and redundant, leading to high costs and low efficiency in cross-departmental, cross-business, and cross-company data research; Lack of a unified data subscription and synchronization entry point: Management is chaotic, with no unified management of data synchronization methods, sharing permissions, scope, and operational controls; Lack of data security guarantees during data development and delivery: Lack of environmental isolation, making it easy to leak original data, and development activities lack traceability; Lack of data processing linkage: Data redundancy, high storage costs, and no retention strategy management for data provided to data developers and users, making data security difficult to guarantee.

[0181] Technical implementation steps of the present invention:

[0182] Step 1: Data registration and access (accessing data and registering metadata).

[0183] Data providers register data on the interface according to a unified data resource and data product registration template, and complete the content including industry classification, update frequency, data type, field content, etc., as metadata for management, ensuring a comprehensive description of the data from all dimensions. By configuring retention policies, data that exceeds the time limit is destroyed or archived.

[0184] Step 2: Data catalog display (data display and discovery).

[0185] Based on the industry classification and data structure information collected during the registration of data resources and data products, the data is displayed hierarchically in the data resource catalog and data product catalog, allowing data developers and data users to find the data resources or data products they need based on their own data and application requirements.

[0186] Step 3: Data subscription (apply for the required data resources or data products).

[0187] Once data developers and data users discover the data resources or data products they need, they can apply for a data subscription to specify the delivery method, timeliness requirements, and other information. The data provider will review the subscription requests from the data developers and data users, and the subscription will take effect once both parties agree.

[0188] Step 4: Data synchronization (delivering development test data and real data).

[0189] After the subscription takes effect, data is delivered in two categories: development / test data and real data. The data provider configures data synchronization tasks through the delivery interface, and performs data anonymization tasks to de-identify different scenarios and data types. The anonymized development / test data is then synchronized to the development / test environment for data developers and users to explore and develop. Real data is synchronized to the production environment through configuration tasks, and retention policies are configured to manage the destruction and disposal of timed-out data. The production environment and the development / test environment are isolated using different network segments.

[0190] Step 5: Data Development (Data Product Development and Testing).

[0191] In addition to the data, a development and testing environment is created that is completely identical to the production environment in terms of configuration items such as environment, dependencies, and table structure. All operations performed by the data development team can only be performed in the development and testing environment. By configuring data masking tasks and selecting consistent data masking, the same data content from different sources is masked while ensuring data consistency. This supports scenarios such as data association and intersection, and ensures that the masked data is sufficient to reflect real business scenarios while ensuring security. Data exploration, development, and testing processes are documented through management logs. After development is completed and approved, the code and scripts are imported into the production environment for execution.

[0192] Step 6: Data processing (data retention and archiving).

[0193] Data products produced by data developers and data resources provided by data providers can be configured with retention policies to manage the timeliness requirements of data storage and to destroy or archive expired data.

[0194] This invention integrates six core elements—registration, cataloging, subscription, synchronization, development, and disposal—to form a complete lifecycle management and control chain, enabling a single system to complete all operations.

[0195] Meanwhile, this invention designs differentiated rules for data from different sources and with different data sensitivity levels, unifying basic management standards while taking into account differentiated security and compliance requirements. Functions such as automated synchronization, isolated development, and hierarchical disposal are deeply integrated into business processes, improving data flow efficiency while strengthening security and compliance at all stages of flow, use, and destruction. The end-to-end traceability capability also makes up for the shortcomings of existing systems in defining responsibilities, making it more suitable for the large-scale management needs of city-level multi-entity and multi-type data.

[0196] The technical terms, principles, or means related to the technical solutions of the present invention mentioned in the above embodiments, which are not described in detail above, are all well-known technologies or common practices that are known to those skilled in the art.

[0197] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A multi-source data full lifecycle registration, subscription, synchronous development, and processing system, characterized in that, include: The data registration module is used to provide standardized registration templates for data providers of multi-source data to register data, collect basic information, ownership information, classification and grading information, data sensitivity level and metadata information of the data, and assign a globally unique resource identifier to each multi-source data after passing the qualification and content verification, generate data archives and archive them. The unified directory management module is used to automatically build a unified data directory for the entire domain by classifying the registered multi-source data according to multiple dimensions, providing hierarchical retrieval and display to the outside world, and automatically refreshing the directory when the status of multi-source data changes; The data subscription module is used to perform traffic distribution and approval based on the data sensitivity level and the configured synchronization rules when the data user submits a subscription application for target data in the whole domain integrated data catalog, and to manage the status of the subscription task throughout its entire lifecycle. The intelligent synchronization module is used to perform dual-mode intelligent synchronization based on subscription strategy-driven automated data flow. It synchronizes multi-source data from the data source to the target environment, monitors the synchronization status in real time, and triggers warnings and automatic retries in case of anomalies. The secure development linkage module is used to receive multi-source data that has been intelligently synchronized for developers to use for testing, through network isolation and data anonymization between the production environment and the development and testing environment, and to deploy the developed code to the production environment for execution. The development results can be uploaded to the global integrated data catalog. The compliance handling module is used to implement graded and differentiated handling strategies based on the data sensitivity level. After the multi-source data that has completed intelligent synchronization is reviewed, graded and differentiated handling will be carried out on multi-source data that has exceeded the usage period, is invalid or violates regulations, and a formal handling report will be generated.

2. The multi-source data full lifecycle registration, subscription, synchronous development, and processing system according to claim 1, characterized in that, The steps for data registration are as follows: The data providers include, but are not limited to, government departments, enterprises, and industry organizations. Dedicated standardized registration templates are configured for different data providers. The required fields in the standardized registration templates include at least the data name, the subject of the data, the data ownership, the data classification and grading, the data sensitivity level, the data format, the update cycle, the metadata information, and the scope of legal use. After the data provider fills in the standardized registration template and submits it, the system automatically verifies the compliance of the submitted entity's qualifications and data format by combining the submitted multi-source data with the third-party entity qualification database and data format fields. Based on the data classification and sensitivity level entered, the final data classification and sensitivity level labels are manually confirmed and reviewed. After verification, a globally unique resource identifier is generated for each piece of multi-source data. Using the globally unique resource identifier as the main thread, the operation logs of each link in the system can be linked together to form a complete traceability link and output the source traceability record; Finally, metadata is automatically collected by connecting to the data source interface and automatically capturing metadata that includes at least the data structure, field definitions, and flow records. This metadata is then merged and archived with the manually registered information filled in by the data provider to form a complete data archive.

3. The multi-source data full lifecycle registration, subscription, synchronous development, and processing system according to claim 1, characterized in that, The integrated data directory is a tree-shaped directory structure, with a hierarchical relationship consisting of three levels of nodes: the first level root node is divided according to data type, the second level nodes are divided according to industry, and the third level leaf nodes are divided according to field. The integrated data catalog also provides keyword search and multi-condition combination filtering, allowing data users to quickly locate target data. Basic information is displayed during the data presentation, while core fields are hidden for sensitive data, and only a de-identified summary is shown. In this integrated data catalog, each catalog entry is bound to a globally unique resource identifier. Clicking on the corresponding catalog entry allows you to view the data archive and its usage status.

4. The multi-source data full lifecycle registration, subscription, synchronous development, and processing system according to claim 1, characterized in that, The execution steps for the triage approval process are as follows: Data users select target data in the data catalog, submit a subscription application, and fill in the usage scenario, usage period, and access frequency. The data subscription module automatically sorts and approves data based on its sensitivity level. The automatic approval process follows these rules: when the data sensitivity level is "general," it is automatically approved without human intervention. When the data sensitivity level is important or core data, it will be pushed to the data provider for manual approval. Once approved, the data subscription module allows data users to customize synchronization rules; these rules include synchronization method, synchronization period, data push target address, and data output format; after configuring the synchronization rules, the data user obtains the subscription strategy. The data subscription module also supports renewal and cancellation of subscriptions. It automatically reminds data users to send renewal reminders before the subscription expires, and the corresponding subscription policy automatically expires after the expiration date.

5. The multi-source data full lifecycle registration, subscription, synchronous development, and processing system according to claim 1, characterized in that, The intelligent synchronization module is equipped with a multi-source adaptive synchronization interface, which is compatible with, but not limited to, database interfaces, API interfaces, file interfaces, and data stream interfaces, and adapts to different network environments. When performing intelligent synchronization, a dual-mode synchronization of incremental and full data is adopted. In normal scenarios, incremental synchronization is performed to transmit only newly added and modified data, while full synchronization is triggered periodically to verify data integrity and correct data deviations. Real-time monitoring of synchronization status includes tracking synchronization progress, data transmission volume, and success rate. In case of disconnection, data loss, or abnormal format, automatic retry is performed and an alarm signal is sent to the administrator. After synchronization is complete, the synchronization time, data volume, and data recipient information will be automatically recorded.

6. The multi-source data full lifecycle registration, subscription, synchronous development, and processing system according to claim 1, characterized in that, The secure development linkage module receives data that has been intelligently synchronized and sets up mutually isolated formal operating environments and development testing environments, and stores formal encrypted data in the external network area; the development testing environment provides anonymized sample data; The secure development collaboration module has built-in development tools, including at least data cleaning tools, data transformation tools, data modeling tools, and data analysis tools. It supports multi-person collaborative development and version management. After secure development is completed, the code will be deployed to the production environment for execution. During secure development, development operations are restricted to the development and testing environment only, recording editing, calculations, and code export operations.

7. The multi-source data full lifecycle registration, subscription, synchronous development, and processing system according to claim 1, characterized in that, The compliance handling module supports configuring data strategies by database and table dimension, setting data retention time, and automatically destroying multi-source data that exceeds the retention period. A tiered and differentiated handling strategy is implemented based on the data sensitivity level, specifically as follows: General data is destroyed directly after being taken down. Important or core data is forcibly desensitized after being taken down, and then destroyed. Core data is destroyed in a closed loop and is prohibited from secondary circulation. The compliance handling module supports two handling modes: manual initiation and automatic system triggering. All handling operations must be reviewed and approved by the data provider before execution. After the disposal process is completed, including removal from shelves, mandatory desensitization, and destruction, the disposal results are automatically verified, and a formal disposal report is automatically generated upon completion.

8. A method for multi-source data full lifecycle registration, subscription, synchronization, development, and processing, applied to the system described in any one of claims 1-7, characterized in that, Includes the following steps: Acquire multi-source data and provide standardized registration templates for data registration. After passing the qualification and content verification, assign a globally unique resource identifier to each multi-source data and establish a data archive. The globally unique resource identifier is used to link the operation logs of each step in the method to form a complete traceability link and output the source traceability record; The registered multi-source data will be automatically classified into a unified data catalog across the entire domain according to multiple dimensions, providing hierarchical retrieval and display to the outside world, and automatically refreshing the catalog when the data status changes; Data users select target data in the unified data catalog and submit a subscription application. The data sensitivity level is combined with the traffic approval process, and a subscription strategy is generated based on the synchronization rules. Based on the subscription strategy, multi-source data is synchronized from the data source to the target environment through multiple data interfaces, and the synchronization status is monitored in real time. In case of anomalies, alarms are triggered and automatic retry is performed. The system receives multi-source data that has been intelligently synchronized for developers to test, deploys the developed code to the production environment for execution, and uploads the development results to the unified global data catalog. Based on the data sensitivity level, a graded and differentiated handling strategy is implemented. After the multi-source data that has completed intelligent synchronization is reviewed, the multi-source data that has exceeded its usage period, is invalid, or violates regulations will be subject to graded and differentiated handling, and a formal handling report will be generated.

9. An electronic device, comprising: The processor, communication interface, memory, and communication bus are connected, with the processor, communication interface, and memory communicating with each other via the communication bus. The feature is that the memory stores a computer program, which, when executed by the processor, causes the processor to execute the multi-source data full lifecycle registration, subscription, synchronous development and processing method as described in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed, it implements the multi-source data full lifecycle registration, subscription, synchronous development and processing method as described in claim 8.

Images