Public data fusion development platform

Abstract

The invention discloses a public data fusion development platform which comprises a platform portal website, a business center, a capability support plate and an infrastructure layer. The platform web portal serves as a user layer to provide an external service window; the business center serves as a business layer and comprises an operation management system, a second-level development system and a competition management system which serve a first-level development subject, a second-level development subject and the public respectively. The capability supporting plate serves as a middle table layer and comprises a data management system, a resource management system, a privacy computing system and an operation and maintenance management system which are used for providing data management, resource scheduling, security computing and full-life-cycle operation and maintenance support for the service center; the infrastructure layer provides one or more infrastructures including a cloud resource pool, a privacy computing environment, a computing engine, containerization, a scheduling engine, intrusion protection, threat detection, a firewall, and a bastion host for the capability support section.

Description

Technical Field

[0001] This invention relates to the field of public data processing technology, and more specifically to a public data fusion development platform. Background Technology

[0002] With the development of the digital economy, public data, as an important production factor, has extremely high development value. However, the current architecture of public data development platforms is fragmented, making it difficult to simultaneously meet the layered needs of multiple roles, including government-authorized operating agencies, enterprise research institutions, and the general public, on a single platform. At the same time, there is room for further optimization of the architecture layers. For example, some platforms integrate user access points with business operation functions, causing interference between the operation of government-authorized operating agencies and the development needs of enterprise research institutions. Some platforms lack a unified middleware layer, with core capabilities such as data governance, resource scheduling, secure computing, and operation and maintenance management scattered across different modules, forming information silos and resulting in cumbersome operations.

[0003] Therefore, it is necessary to build a public data fusion development platform with a complete architecture and clear hierarchy to solve the above problems and promote the compliant and efficient development and utilization of public data. Summary of the Invention

[0004] The purpose of this invention is to provide a public data fusion development platform with a complete architecture and clear hierarchy.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A public data fusion development platform, comprising: The platform portal website, as the user layer, provides a window for external services; The business center, as the business layer, includes the operation management system, the secondary development system, and the competition management system, which respectively serve the primary development entities, the secondary development entities, and the general public. The capability support module, as the middle platform layer, includes a data management system, a resource management system, a privacy computing system, and an operation and maintenance management system that provide data governance, resource scheduling, secure computing, and full lifecycle operation and maintenance support to the business centers, respectively. The infrastructure layer provides one or more infrastructure components for the capability support module, including cloud resource pools, privacy computing environments, computing engines, containerization, scheduling engines, intrusion prevention, threat detection, firewalls, and bastion hosts.

[0006] Furthermore, the data management system includes a data integration unit configured as follows: supporting data synchronization of relational databases, non-relational databases, and distributed databases, providing offline table synchronization, offline database synchronization, interface parsing synchronization, and file parsing synchronization methods; supporting the synchronization of structured data, unstructured data, semi-structured data, and API data; providing unified management of data synchronization tasks, including creating new synchronization tasks, starting / stopping tasks, deleting tasks, and task scheduling monitoring; and supporting data synchronization via message queues, whereby the data source writes incremental data to the message queue through database triggers or logs.

[0007] Furthermore, the resource management system includes a data resource allocation unit configured as follows: after a data query is completed, a temporary table is output from the dataset based on the user's query results. The temporary table only contains the fields required by the user, and the dataset to which the field belongs must be authorized by the corresponding data owner. During the scenario development configuration process, it supports selecting datasets available on the development platform, adding exploration libraries and exploration tasks. If the usage restrictions of the dataset do not match the application usage method, or if the usage restrictions of the dataset involved in the exploration library or exploration task do not match the application usage method, then the creation of a development scenario is prohibited. The system configures the computing and storage resources required for the development scenario, and allocates the corresponding resources to the scenario after the scenario is approved.

[0008] Furthermore, the resource management system includes a tenant management unit configured as follows: building a container image repository, supporting users to upload model image files and generate corresponding call URLs, and loading model images from the image repository based on these URLs when starting training tasks; configuring the container runtime environment, supporting the setting of image addresses and hardware acceleration parameters, including the number of GPUs, the number of CPU cores, and the memory size; achieving container isolation through namespaces, limiting the upper limits of CPU, memory, and disk resources for container process groups, and avoiding mutual interference between different container applications; and supporting elastic resource scaling, dynamically allocating platform computing resources according to the real-time resource needs of third-party applications.

[0009] Furthermore, the data management system also includes a data reconciliation unit configured as follows: importing national and industry data standards and specifications, establishing platform data standards and configuring their business, technical, and security attributes; creating new verification rules, and creating data exploration tasks for imported or treated data and outputting results through one or more of the following: standard dictionary exploration, numerical value range checking, uniqueness checking, null value checking, and length checking; cleaning data using platform master data, automatically correcting data that does not conform to master data specifications, filtering out data that cannot be automatically corrected and then manually intervening, recording and discarding data that cannot be corrected; and using one or more of the following encryption algorithms to achieve data desensitization: full masking, fixed position masking, first and last character number masking, fixed character masking, fixed character replacement, MD5, SHA1, SHA256, HMAC, DES, and AES.

[0010] Furthermore, the privacy-preserving computation system includes a federated learning framework and multi-party secure computation functions configured as follows: the federated learning framework supports horizontal federated learning, vertical federated learning, and federated transfer learning; the multi-party secure computation supports privacy intersection, hidden query, basic operations, joint statistics, logistic regression, and linear regression.

[0011] Furthermore, the resource management system includes a data development space and development tool unit configured as follows: it supports calling data analysis, data mining, and model training tools available on the platform; it supports containerized deployment of Rstudio and JupyterLab tools; it supports the deployment of user-defined development tools, which, after being certified and approved by the platform administrator, process authorized data through the platform's storage service, and are deployed in a containerized manner; it has a built-in AI model training unit, providing a base image that supports deep learning. The base image includes a Python runtime environment, TensorFlow, Keras deep learning frameworks, and SciPy and NumPy data processing libraries, allowing users to add applications to the base image and deploy them to the platform.

[0012] Furthermore, the secondary development system also includes a development process management unit configured as follows: one application scenario corresponds to one development space, and each development space provides development tools and mutually isolated system environments; the running results of development instances in the development space can only be exported to the platform to generate new datasets, and scenarios requiring output review can only have their results exported to the platform's designated area to generate new datasets after being reviewed and approved by the platform operator.

[0013] Furthermore, the operation management system includes a development data catalog management unit and an application scenario management unit: the development data catalog management unit compiles a development data catalog based on the public data resource catalog, which includes data business applications, technical parameter information, synchronously generates sample data, and supports CSV, Excel, JSON data formats and API interfaces; The application scenario management unit collects application scenario construction requirements and imports them into the platform. The platform updates and organizes the scenario names, descriptions, and technical parameter information to generate or update the application scenario catalog. Both the development data directory and the application scenario directory support directory query, directory details viewing, directory export, directory version management, directory reporting and publishing. When the application scenario directory is published, it is automatically associated with the original application scenario requirement summary content and published through the platform portal website.

[0014] Furthermore, the operation management system includes a visual cockpit configured as follows: It displays data authorization information and data lineage information. Data authorization information includes data processing application information from primary and secondary development entities, as well as resource provider authorization information. Data lineage information includes data associations and dependencies, data sources, and the data flow and processing process. It also displays data resource management, data requirement management, and data application management. Data resource management visually displays the daily inventory of public data resource catalogs and sample data details. Data requirement management visually displays the results of secondary development entity information completion, material uploading, and approval processes. Data application management visually displays the review results and approval progress of data interfaces, data packets, data models, and data report-type data applications. Finally, it displays data service management, task scheduling management, data application management, and data operation and maintenance management. Data service management visually displays the encapsulation, registration, and release of data service interfaces. Task scheduling management visually displays task monitoring and alarm status. Data application management visually displays the data application development status and progress. Data operation and maintenance management visually displays the operation and maintenance status of each sub-platform, including the monitoring and alarm center, resource overview, fault emergency handling, and log management information.

[0015] By adopting the above technical solution, the present invention has the following advantages compared with the prior art: 1. This application constructs a four-tiered collaborative architecture consisting of a user layer, a business layer, a middle platform layer, and an infrastructure layer. It clarifies the functional positioning and support relationships of each layer, resulting in a clear architectural hierarchy. This architecture enables layered services to be provided simultaneously to primary development entities (operators), secondary development entities (developers), and the general public (participants), covering the core needs of different roles and resolving the problem of isolated or interfering services among multiple entities. The middle platform layer provides data, resource, security, and operational support for the development and utilization of public data. 2. This application covers the synchronization of multi-source heterogeneous data, solving the integration problem caused by the complexity of public data sources, and realizing the full lifecycle management of data synchronization tasks to avoid synchronization anomalies caused by task chaos; this application also realizes real-time incremental data synchronization based on message queues, reducing data synchronization delay and ensuring the timeliness of public data; 3. This application adheres to the principle of minimum availability, outputting only a temporary table containing the fields required by the user after the data query is completed, and requires authorization from the data owner to avoid exposing all data and reduce the risk of sensitive data leakage; at the same time, by verifying the compatibility between the usage restrictions of the dataset / exploration library / exploration task and the application method, the creation of non-compliant scenarios is prohibited, ensuring the compliance of scenario development and reducing the risk of data misuse; the computing and storage resources required by the scenario are configured and allocated as needed to improve resource utilization. 4. This application constructs a container image repository and generates a call URL to enable fast loading of model images, solving the problems of disordered image management and low call efficiency; it isolates containers through namespaces and limits resource limits to avoid interference between different container applications; it supports elastic scaling of resources to dynamically allocate computing resources according to the real-time needs of third-party applications, avoiding insufficient or idle resources; it provides standardized communication interfaces between containers and storage providers, and supports mounting, unmounting, and expanding storage volumes to ensure stable container storage communication. 5. This application imports national / industry data standards and configures business, technical, and security attributes to unify multi-source data standards at the data source and solve the problem of data standard confusion; through verification rules, data exploration tasks, data cleaning, and other data governance methods, it reduces the impact of invalid data on development and outputs high-quality data that can be directly used for development; it uses multiple encryption algorithms to de-identify sensitive data in the output data, taking into account both data development and privacy protection. 6. This application supports three federated learning modes: horizontal, vertical, and transfer learning. It can cover scenarios with the same features but different samples, the same samples but different features, and little overlap between features and samples. It enables multiple parties to jointly model without sharing the original data. It supports a variety of multi-party secure computation functions to meet the needs of collaborative computation in multiple scenarios, making the data usable but not visible, and ensuring that the original data is not leaked during the computation process. 7. This application supports the use of tools listed on the platform and containerized deployment of Rstudio and JupyterLab, thereby reducing tool configuration costs and lowering the development threshold for secondary development entities; it supports the containerized deployment of user-defined development tools after review, realizing tool diversity. At the same time, containerized deployment is isolated from the main platform, ensuring platform security; it includes a built-in base image containing a Python runtime environment, deep learning framework, and data processing libraries, eliminating the need for manual configuration of AI model training environments and improving AI development efficiency. 8. This application adopts a model of one application scenario corresponding to one development space, which realizes the isolation of development spaces and avoids cross-contamination of data in different scenarios; it restricts the development results to be exported only to the platform, and the export is subject to approval by the operator after the scenario is reviewed, in order to avoid the leakage of original data or sensitive results and further control the security of development results; 9. This application employs a dual-directory design, consisting of a development data directory and an application scenario directory. The development data directory focuses on the data supply side, clearly defining the business applications, technical parameters, and sample data of public data. This helps secondary development entities quickly determine whether the data is suitable for their own development needs. The application scenario directory focuses on the demand side. By aggregating approved development scenarios and associating them with the data resources required by those scenarios, primary development entities can accurately supplement data based on scenario requirements, while secondary development entities can directly locate the required data through the scenario directory, avoiding blindly searching for data or having idle data. By establishing a connection between data information and scenario requirements through the dual-directory design, secondary development entities do not need to repeatedly communicate with primary development entities about the purpose of data and scenario suitability, reducing cross-entity collaboration costs and shortening the development cycle. 10. This application addresses the issues of opaque authorization processes and uncontrollable data flow by visually displaying data authorization information; it also enables real-time monitoring of data processing status and timely detection of processing anomalies by visually displaying data resource management, data demand management, and data application management, ensuring controllable processing flows; and it facilitates rapid location of fault nodes, optimization of service quality, and improvement of operation and service efficiency by visually displaying data service management, task scheduling management, and data operation and maintenance management, thus resolving the problems of difficult fault location and low operation and maintenance efficiency. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall architecture of the present invention; Figure 2 This is a visual illustration of the data resource list of the present invention; Figure 3 This is a visualization diagram of the data resource details of the present invention; Figure 4 This is a schematic diagram illustrating a scenario of the present invention; Figure 5 This is a schematic diagram of the bloodline information data of the present invention; Figure 6 This is a visual illustration of data resource management according to the present invention; Figure 7 This is a visual diagram illustrating the data service management of the present invention. Figure 8 This is a visual diagram illustrating the data application management of the present invention; Figure 9 This is a schematic diagram of the data integration unit of the present invention; Figure 10 This is a schematic diagram of the data governance topology of the present invention; Figure 11 This is a visual schematic diagram of the operation and maintenance management system of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Example

[0018] Please refer to Figure 1 As shown, this invention discloses a public data fusion development platform, which includes a platform portal website, a business center, and an infrastructure layer.

[0019] The platform portal website, as the user layer, provides a window for external services; The business center, as the business layer, includes the operation management system, the secondary development system, and the competition management system, which respectively serve the primary development entities, the secondary development entities, and the general public. The capability support module, as the middle platform layer, includes a data management system, a resource management system, a privacy computing system, and an operation and maintenance management system that provide data governance, resource scheduling, secure computing, and full lifecycle operation and maintenance support to the business centers, respectively. The infrastructure layer provides one or more infrastructure components for the capability support module, including cloud resource pools, privacy computing environments, computing engines, containerization, scheduling engines, intrusion prevention, threat detection, firewalls, and bastion hosts.

[0020] The platform portal website serves as the user layer, providing external service windows. It includes resource catalog units, data application units, registration units, policy and regulation units, frequently asked questions units, and user center.

[0021] The resource catalog unit includes a data resource list unit, a data resource details unit, and a search and filter unit. For example... Figure 2 As shown, the data resource list unit displays the published development data catalog, supporting viewing the catalog list by category and sorting by rules such as update time and popularity, allowing users to quickly filter target data resources. Figure 3 As shown, the data resource details section displays essential information about the data resource catalog, including but not limited to the data resource name, resource type, source organization, and resource description. It provides sample data viewing functionality, and the details page includes an application entry point that redirects to the backend to initiate the data usage application process. The search and filtering section supports keyword search, displaying all data resources that meet the criteria; it also supports filtering by industry, type, source organization, and other custom tags, helping data users easily obtain accurate data product information.

[0022] The data application unit includes an application scenario list unit, an application scenario details unit, and a search and filtering unit. The application scenario list unit displays approved and published application scenarios and application scenario requirements. It supports viewing the list by category and sorting by update time, popularity, and other rules, allowing users to quickly locate scenarios of interest. The application scenario details unit displays necessary information about the application scenario, including but not limited to the application scenario name, domain, description, and data resources used. The details page includes an application function that redirects users to the backend to initiate the scenario participation or development application process. The search and filtering unit also supports keyword search, presenting all application scenarios that meet the criteria; it also supports filtering by industry, type, source organization, and other custom tags, accurately displaying matching application scenarios to the searcher.

[0023] The registration and onboarding process includes a user registration unit, a user review unit, and a user login unit. In the user registration unit, users provide the required information to complete account registration, including but not limited to basic business registration information and legal representative information. In the user review unit, after the registration application is submitted, the platform operator reviews the submitted qualification materials and categorizes them according to dimensions such as company size, industry, and needs, providing a basis for accurate allocation of functional permissions and data resources. In the user login unit, after the user registration is approved, they can log in to the platform using their username and password. After logging in, they can view their own needs, operation records, and other content, and view information within the corresponding scope according to their applied permissions.

[0024] The FAQ section helps the platform establish a knowledge Q&A area, compiling common questions and their answers to form a FAQ database. This provides timely and accurate help and answers to platform users, improving the platform's usability and user satisfaction.

[0025] The user center includes a username and password management section, a personal basic information management section, and a message notification and feedback section. The username and password management section provides users with the ability to modify their username and password, allowing them to adjust their login information at any time and ensuring account security. Users can view and modify their personal basic information through the personal basic information management section, and can also upload avatars, set signatures, complete their profiles, and showcase their personality. The message notification and feedback section centrally displays important notifications and messages sent to users by the platform, making it convenient for users to view relevant reminders in a timely manner.

[0026] The Policy and Regulations section serves as the core information display and external service foundation of the platform portal, comprising the Homepage, Information Disclosure, Ecosystem Partners, Notices and Announcements, News and Information, Help Center, and About Us sections. The Homepage displays the platform's background, business scope, processes, and latest news. The Information Disclosure section is used for publicizing market entity registration, third-party institution announcements, and data application submissions. The Ecosystem Partners section showcases the platform's ecosystem partners, including event organization and partner introductions. The Notices and Announcements section publishes platform rules and other notices and announcements. The News and Information section provides the latest industry news, market trends, and platform news. The About Us section introduces the platform's background, functions, advantages, and operator information.

[0027] Within the business center, the operations management system includes units for organization and user management, development data catalog management, application scenario catalog management, business process approval management, data application management, visualization dashboard, and data subject authorization management.

[0028] The organization and user management unit supports configuring different roles such as operators, developers, and regulators, and flexibly allocating data access permissions; it manages the onboard secondary development entities and individual users, maintains their business licenses, authorization agreements and other qualification materials, and ensures that user management is compliant and orderly.

[0029] The development data catalog management unit compiles a development data catalog based on public data resource catalogs to ensure data source compliance. Its core catalog information includes data business applications (applicable development scenarios), technical parameter information (such as data format and update frequency), and synchronously generates sample data to help developers better understand and use data resources. It supports CSV, Excel, and JSON data formats and provides API interfaces to adapt to different development needs. Additionally, it provides API interfaces to facilitate data access and processing by developers.

[0030] The application scenario directory management unit integrates and reviews development scenario requirements to form a standardized application scenario directory. It collects scenario applications from secondary development entities (such as...). Figure 4As shown, the platform facilitates scenario application, scenario query, and classification evaluation, generating and publishing an application scenario directory. Scenario applications can be initiated by secondary development entities, with application content including scenario description, applicant organization information, and developers. Scenario queries are presented in list format, displaying information such as the industry, secondary development entity, review status, authorization status, and details. Clicking on details allows viewing the development scenario details. The scenario classification evaluation module supports the evaluation and review of application scenario requirements submitted by secondary development entities. Secondary development entities can submit application scenario requirements through the platform, including scenario descriptions, business requirements, and technical requirements. This requirement information will be submitted to the primary development entity for evaluation. After application scenarios approved by the competent authority are imported into the platform, the platform automatically generates or updates the application scenario directory, updating and organizing information such as scenario names, descriptions, and technical parameters to ensure the accuracy and completeness of the directory. Finally, after associating the summary content of the original application scenario requirements, the scenario directory is reasonably classified and organized, and published through the portal.

[0031] Both the development data directory and the application scenario directory support directory creation, directory query, directory details viewing, directory export, directory version management, and directory submission and publishing. Directory query allows users to search for directory information based on directory name, type, cataloging status, and associated resources. Directory details allow viewing detailed information about resource directories, such as basic directory attributes, information items, and associated resources. Clicking on a resource's directory name will take you to its details page. Directory export allows exporting directory data to Excel or other files based on directory categories. Directory version management automatically assigns a unique version number to each directory after creation or modification, facilitating subsequent management and tracking. Historical version records, including the content, version number, and approval status of each version, allow users to view and compare different versions and understand the directory's evolution. Version control and rollback operations are supported; if a new version has issues or needs correction, a rollback can be performed to revert to a previous version, ensuring data accuracy and consistency. Consistency; Catalog submission and release refers to the process where, after internal approval, a new version of the development data catalog can be exported and submitted to the municipal big data authority. Once the catalog is approved, the system marks it as publishable, preparing for subsequent release operations. Catalog release is supported through simple operations. Users can select the catalog version to be released and trigger the release process. The platform will automatically release the selected catalog version to the designated location or channel for other users or systems to access and use. Once the catalog is successfully released, the new version will automatically take effect and be enabled. Users can then access and use the new version of the catalog to obtain the required data resource information. At the same time, the platform will update the catalog version information to ensure that users are using the latest version of the catalog.

[0032] In this way, the development data catalog clearly defines the attributes and usage methods of supply-side data, while the application scenario catalog clearly defines the development direction and data requirements of the demand side. The linkage between the two catalogs allows secondary development entities to quickly locate the required data through scenarios, and also allows primary development entities to supplement data resources based on scenario requirements, solving the problem of low efficiency in matching data with scenarios. At the same time, the development data catalog is compiled based on the public data resource catalog, while the application scenario catalog is generated after platform review. Both catalogs follow standardized management processes, ensuring the compliance of data use and scenario development from the source. Meanwhile, catalog version management and reporting and release mechanisms ensure the traceability of the development process.

[0033] The business process approval management unit provides approval functions such as institutional access, scenario application, application deployment, and data application release. It supports customized approval processes and can be integrated with other systems to ensure compliant and controllable business processes. Each approval process supports approval and rejection operations, and allows users to input remarks such as reasons for rejection.

[0034] The data application management unit manages different data applications, including data interfaces, data packets, data models, and data reports, and provides corresponding review methods and approval processes. It supports adding third-party organizations during the data application deployment approval process and provides a workbench for these organizations to complete compliance review operations. After security encapsulation (including providing external business system IP access control, authorization authentication, and return format verification functions), the data is released to the production environment. Furthermore, it manages application versions and data lineage to ensure that the source, processing, and usage of data are traceable and manageable.

[0035] like Figure 5 As shown, the visualization cockpit unit displays data authorization information and data lineage information through a graphical interface. The data authorization information includes data processing application information of the primary development entity and the secondary development entity, as well as authorization information of the resource provider. The data lineage information includes data association and dependency relationships, data sources and flow processing processes.

[0036] The visualization cockpit unit also displays the status of data resource management, data demand management, and data application management through a graphical interface, such as... Figure 6 As shown, the data resource management system visually displays the daily inventory of public data resource catalogs and sample data details; the data requirement management system visually displays the information filling, material uploading, and approval process results of secondary development entities; and the data application management system visually displays the review results and approval progress of data interfaces, data packages, data models, and data reports.

[0037] The visualization cockpit unit also displays the status of data service management, task scheduling management, data application management, and data operation and maintenance management through a graphical interface, such as... Figure 7 As shown, the data service management visualization displays the encapsulation, registration, and publishing status of data service interfaces; the task scheduling management visualization displays task monitoring and alarm status; and the data application management visualization displays the development status and progress of data applications. Figure 8 As shown, the data operation and maintenance management visualization displays the operation and maintenance status of each sub-platform, including the monitoring and alarm center, resource overview, fault emergency handling, and log management information.

[0038] The data subject authorization management unit uniformly manages the data subject authorization information of public data such as personal information and trade secrets involved in the data application of secondary development entities, synchronizes the authorization records on the application side, and conducts compliance checks.

[0039] Specifically, the secondary development system includes resource center unit, development process management unit, data development management unit, and results management unit.

[0040] The resource center unit manages the data resources, computing resources, and development scenarios authorized by the secondary development entities, including application progress, usage status, and usage statistics. It also provides search engine services and supports the uploading and management of non-public data.

[0041] The development process management unit follows the principle of minimum availability and provides data encryption, data exploration, data square, and data authorization functions. It records the entire development and utilization process and ensures data output security through de-identification processing.

[0042] The data authorization specifically includes: for unauthorized datasets, platform users can only see the header field names and a summary of the dataset, but not the actual data values. This allows platform users to accurately discover the number of data records within the platform that meet their needs through a database-like query. On the other hand, before viewing and using the query results, platform users must obtain authorization from the data owner associated with the query results, and the authorized data dimension is limited to the data fields required by the requester, not the entire dataset or table, ensuring that the data obtained by the requester does not exceed the scope of use.

[0043] Furthermore, data authorization further ensures security through the following methods: Platform data is organized as datasets, each with a clearly defined owner (administrator). The platform's file encryption system can encrypt different datasets using different keys. Only the dataset owner and authorized users possess the data access key and use the data through decryption. Platform users can only use data within the platform after being authorized by the dataset owner. The platform data authorization process involves the dataset owner sharing the data access key with the authorized user in a secret manner. This avoids the problem of other authorized users or platform administrators transferring data usage rights without authorization, ensuring minimal scope of authorization. Authorized data cannot leave the platform through downloads or other means; it can only be processed within the platform using built-in applications or user-developed programs. Data usage authorization methods include visible and invisible authorization. When platform users process data through applications within the platform, if the authorized data is authorized in a visible manner, then the platform user can see the authorized data. In some cases, users must see the raw data to process it, such as when platform users manually annotate or clean the raw data. In other scenarios, platform users don't necessarily need to access the raw data when processing data using applications. For example, when using labeled data for model training, users don't need to see the labeled data; they only need the training results and the trained model parameters. By using two different authorization methods, the flexibility of data owners in authorizing data can be increased, achieving a balance between different levels of data security and ease of use.

[0044] Meanwhile, the development process management unit configures a corresponding development space for each application scenario. Each development space provides development tools and an isolated system environment. Secondary development entities can independently configure the development work interface within the development space, manage data and documents within the space, and perform version management and export operations on the application. The running results of development instances within the development space can only be exported to the platform to generate new datasets. Scenarios requiring output review can only have their results exported to the platform's designated area to generate new datasets after being approved by the platform operator.

[0045] The data development management unit supports secondary development entities in developing applications such as scripts, ETL, and models within the allocated development space. It manages the start and stop of development instances, log viewing, and connects to the privacy computing environment. It records the application development process, including the scenario itself and the datasets input to the scenario, and establishes the lineage relationship between the output dataset and the input dataset. In this way, for any dataset on the platform, it can be traced back to the original dataset by its origin and lineage.

[0046] The results management unit manages the data applications developed, including application review and submission, progress tracking, release and delisting applications, displays application service data (number of applications, number of calls, etc.), provides an application tool marketplace, and aggregates general and third-party development tools.

[0047] Specifically, the competition management system, as an innovative participation system serving the public, includes a contestant unit, a judging unit, a competition management unit, a skills competition management unit, and an online programming unit. The contestant unit supports contestant registration and application, manages individual participation information, submission records, and competition results, and provides participation guidance and notifications. The judging unit provides a judging platform for experts, allowing them to view entries, score them, submit judging comments, and record the judging process and results. The competition management unit manages the overall competition process, including event announcement, registration management, competition scheduling, and results publication, and supports setting competition rules and awards. The skills competition management unit provides specialized management for skills competitions, clarifies the content and assessment standards of skills competitions, organizes contestants to conduct skills competitions, and records competition process data. The online programming unit provides an online programming environment, supporting contestants to complete code writing, debugging, and running within the platform, ensuring the fairness and security of the competition process.

[0048] The data management system includes a data integration unit and a data reconciliation unit.

[0049] like Figure 9 As shown, the data integration unit realizes the synchronization of multi-source heterogeneous data, supports the synchronization of relational, non-relational, and distributed database data, as well as structured and unstructured data, and provides multiple synchronization methods such as table offline, whole database offline, interface parsing synchronization, and file parsing; it supports the synchronization of structured data, unstructured data, semi-structured data, and API data; it can create, start, stop, delete, and schedule and monitor synchronization tasks, and synchronize incremental data through message queues to ensure timely data updates.

[0050] The data reconciliation unit imports national and industry standards to establish platform data standards and configures attributes across multiple dimensions, including business, technology, and security. It verifies data using rules such as standard dictionary exploration, numerical range checks, uniqueness checks, null value checks, and length checks, and creates new exploration tasks to output results. It cleans data using master data, automatically correcting data that does not conform to master data specifications. Data that cannot be automatically corrected is filtered out and manually intervened; data that cannot be corrected is recorded and discarded. Data desensitization is achieved using one or more encryption algorithms, including full masking, fixed-position masking, first and last character count masking, fixed character masking, fixed character replacement, MD5, SHA1, SHA256, HMAC, DES, and AES.

[0051] Thus, as Figure 10As shown, the data management system integrates with public data resource platforms through its integration tools, performs data governance on the public data to create standardized data, and provides this standardized data for secondary development entities to apply for use. Authorized public data can be used for data application development within the development space, integrating with the non-public data of secondary development entities. Once developed, the data applications can be output through corresponding data services. Non-public data is accessed by secondary development entities within the development space after review and approval, and a non-public data catalog must be compiled upon access. Non-public data is pushed to the integrated development data resource pool as needed.

[0052] The resource management system includes a data resource allocation unit, a data development space and development tool unit, a tenant management unit, and a data storage and computing management unit.

[0053] The data resource allocation unit follows the principle of minimum availability. After a data query is completed, a temporary table is output from the dataset based on the user's query results. This temporary table only contains the fields required by the user, and the dataset to which these fields belong must be authorized by the corresponding data owner. During scenario development configuration, users can select datasets available on the development platform, add exploration libraries and exploration tasks, and verify the compatibility between data usage restrictions and application methods in scenario development. If the dataset usage restrictions do not match the application usage methods, or if the usage restrictions of the datasets involved in the exploration libraries or exploration tasks do not match the application usage methods, the creation of the development scenario is prohibited. The required computing and storage resources for the scenario are configured and allocated as needed to improve resource utilization.

[0054] The data development space and development tool unit allocate independent development space for secondary development entities, managing data and documents within the space; it integrates data analysis and mining tools, supports containerized deployment of tools such as Rstudio, and supports the deployment of user-defined development tools. After being certified and approved by the platform administrator, user-defined development tools are processed through the platform's storage service, and deployed in a containerized manner; it also includes a built-in AI model training unit, providing a base image that supports deep learning. The base image includes a Python runtime environment, TensorFlow, Keras deep learning framework, and SciPy and NumPy data processing libraries, allowing users to add their applications to the base image and deploy them to the platform, lowering the development threshold.

[0055] The tenant management unit builds a container image repository, generates image call URLs for training tasks to load, configures the container runtime environment, supports setting image addresses and hardware acceleration parameters, including the number of GPUs, CPU cores, and memory size, and isolates containers and limits resource limits through namespaces. It supports elastic resource scaling, provides standardized storage communication interfaces, and ensures a stable and efficient development environment. The data storage and computing management unit manages the platform's data storage resources, supports mounting, unmounting, and expanding storage volumes, schedules computing resources according to development needs, connects to the computing engine, provides computing support for tasks such as data synchronization and model training, and achieves dynamic resource adaptation.

[0056] The privacy-preserving computation system includes federated learning units, multi-party secure computation units, and node management units. The node management unit manages the nodes participating in privacy-preserving computation, configures node communication parameters and security protocols, verifies the legitimacy of node identities, and ensures the security of data transmission and interaction between nodes to support multi-party collaborative computation.

[0057] Specifically, the federated learning framework in this platform works by building a privacy-preserving computation network. This allows clients to train models on their own terminals using local data, upload and aggregate model updates, and then merge these updates from different terminals to optimize the prediction model. The client terminal then downloads the updated model locally, and this process is repeated continuously. Throughout the entire process, the platform ensures that terminal data is always stored locally, thus avoiding the risk of data leakage.

[0058] This platform's federated learning framework integrates horizontal federated learning, vertical federated learning, and federated transfer learning. Horizontal federated learning is characterized by similar business (features) but different users (samples). Vertical federated learning is characterized by a large number of duplicate user samples in a distributed dataset; the server uses this data to train models for the corresponding users but does not acquire related data. Federated transfer learning performs transfer learning without exchanging data, and is suitable for scenarios where there is little overlap in features and samples among participants, such as joint learning between hospitals and insurance companies in different regions.

[0059] Once the federated learning model is trained, model developers can create a new dataset on the platform to export the trained model to the platform's secure storage database. The platform implements strict approval controls on new result datasets, ensuring that users can only export data locally after obtaining approval from the data owner.

[0060] This platform supports privacy-preserving intersection, hidden queries, basic operations, joint statistics, logistic regression, and linear regression. It allows multiple data owners to perform collaborative computations without mutual trust, outputting computation results and ensuring that no party can obtain any information other than the computation results they are entitled to.

[0061] The operations and maintenance management system, as the core system providing full lifecycle operations and maintenance support, includes a log management unit, a resource operation status unit, and a service monitoring unit. The log management unit conducts unified operations and maintenance management, covering system deployment, troubleshooting, version updates, etc.; it implements data security management, configures access control policies, monitors data access behavior, and prevents data leakage risks. The resource operation status unit monitors resource operation status, statistics on CPU, memory, and other resource consumption; it builds security threat models, identifies and warns of security risks during development, and ensures the security of the development environment. The service monitoring unit displays data authorization information and lineage information, clearly presenting the data source and flow; such as... Figure 11 As shown, the visualization presents the data resources, requirements, application management status, and service, scheduling, and operation and maintenance status, covering the monitoring needs of the entire lifecycle of data development and utilization.

[0062] In the infrastructure layer, the cloud resource pool integrates basic hardware resources such as CPU, memory, storage, and network to form a dynamically allocated resource cluster. It elastically supplies computing and storage resources according to the platform's business load (such as peak data synchronization tasks and model training needs), avoiding the waste of idle physical resources and providing a stable resource foundation for the entire process of data management, development, and secure computing.

[0063] The privacy computing environment is built on an isolated computing space based on Trusted Execution Environment (TEE) and homomorphic encryption technologies. It provides a dedicated operating environment for tasks such as federated learning and multi-party secure computation, ensuring that the original data is stored locally when multiple parties are collaborating, and only encrypted model parameters or intermediate computation results are transmitted. This ensures that the data is available but not visible from both the hardware and environment levels.

[0064] The computing engine integrates distributed computing frameworks such as Spark and Flink, supporting both batch and stream processing modes. It provides efficient computing power support for computationally intensive tasks such as synchronous cleaning, statistical analysis, and model training of massive amounts of public data, improving data processing speed and computational accuracy to meet the needs of large-scale data development and utilization. Containerization, based on technologies such as Docker, packages development tools, applications, and dependent environments into standardized container images, ensuring consistency across development, testing, and production environments and avoiding development problems caused by inconsistencies. Simultaneously, container encapsulation achieves application isolation, ensuring that development instances of different secondary development entities are independent, improving platform resource reuse and deployment efficiency.

[0065] The scheduling engine employs scheduling tools such as Kubernetes and Airflow to uniformly schedule data synchronization tasks, container resource allocation, and model training tasks within the platform. It dynamically adjusts execution plans based on task priority and resource requirements to achieve optimal resource and task matching. Intrusion prevention is deployed at the platform's network boundary and core nodes. Through technologies such as feature matching and abnormal behavior analysis, it monitors and blocks malicious attacks against the platform in real time, intercepting illegal data packets and access requests to protect core assets such as platform servers and databases from intrusion and damage.

[0066] Threat detection uses machine learning algorithms to build security threat models, continuously collects platform logs, analyzes user behavior and system operation status, identifies abnormal behaviors (such as unauthorized account logins and batch downloads of sensitive data) and triggers alarms, enabling early detection and early warning of security threats.

[0067] As the first line of defense for platform network security, the firewall filters and controls network traffic entering and leaving the platform according to preset security policies, clearly allowing or prohibiting access requests for specific IP addresses, ports, and protocols, and isolating networks of different security levels, such as the Internet zone and the government intranet zone, to prevent external network risks from penetrating into the platform.

[0068] The bastion host centralized management platform restricts the operation and maintenance access permissions of the core servers and databases. It requires operation and maintenance personnel to log in through the bastion host, record and log all operation and maintenance operations in the entire process, so as to achieve full traceability of operation and maintenance operations and avoid security risks caused by unauthorized or malicious operations by operation and maintenance personnel.

[0069] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A public data fusion development platform, characterized in that, include: The platform portal website, as the user layer, provides a window for external services; The business center, as the business layer, includes the operation management system, the secondary development system, and the competition management system, which respectively serve the primary development entities, the secondary development entities, and the general public. The capability support module, as the middle platform layer, includes a data management system, a resource management system, a privacy computing system, and an operation and maintenance management system that provide data governance, resource scheduling, secure computing, and full lifecycle operation and maintenance support to the business centers, respectively. The infrastructure layer provides one or more infrastructure components for the capability support module, including cloud resource pools, privacy computing environments, computing engines, containerization, scheduling engines, intrusion prevention, threat detection, firewalls, and bastion hosts.

2. The public data fusion development platform according to claim 1, characterized in that, The data management system includes a data integration unit configured as follows: Supports data synchronization for relational databases, non-relational databases, and distributed databases, providing offline table synchronization, offline database synchronization, interface parsing synchronization, and file parsing synchronization methods; Supports the synchronization of structured data, unstructured data, semi-structured data, and API data; Unified management of data synchronization tasks, including creating new synchronization tasks, starting / stopping tasks, deleting tasks, and task scheduling monitoring. It also supports data synchronization via message queues, where the data source writes incremental data to the message queue through database triggers or logs.

3. The public data fusion development platform according to claim 1, characterized in that, The resource management system includes a data resource allocation unit configured as follows: After the data query is completed, a temporary table is output from the dataset based on the user's query results. The temporary table only contains the fields required by the user, and the dataset to which the field belongs must be authorized by the corresponding data owner. During the scenario development and configuration process, it is possible to select datasets available on the development platform, add exploration libraries and exploration tasks. If the usage restrictions of the dataset do not match the application usage method, or if the usage restrictions of the dataset involved in the exploration library or exploration task do not match the application usage method, the creation of the development scenario is prohibited. Configure the computing and storage resources required for the development scenario, and allocate the corresponding resources to the scenario after the scenario is approved.

4. The public data fusion development platform according to claim 1, characterized in that, The resource management system includes a tenant management unit configured as follows: Build a container image repository that allows users to upload model image files and generate corresponding call URLs. When starting a training task, the model image in the image repository is loaded based on the URL. Configure the container runtime environment, supporting the setting of image address and hardware acceleration parameters, including the number of GPUs, the number of CPU cores, and the memory size; Namespaces are used to isolate containers, limit the CPU, memory, and disk resources of container process groups, and prevent different container applications from interfering with each other; elastic resource scaling is supported, and platform computing resources are dynamically allocated according to the real-time resource needs of third-party applications.

5. The public data fusion development platform according to claim 1, characterized in that, The data management system also includes a data reconciliation unit configured as follows: Import national and industry data standards and specifications, establish platform data standards, and configure their business, technical, and security attributes; Create new verification rules, and use one or more of the following methods to create a new data exploration task for imported or treated data and output the results: standard dictionary exploration, numerical range check, uniqueness check, null value check, and length check. The platform uses master data to clean data, automatically corrects data that does not conform to master data specifications, filters out data that cannot be automatically corrected and then manually intervenes, and records and discards data that cannot be corrected. Data desensitization can be achieved by using one or more of the following encryption algorithms: full masking, fixed position masking, first and last character number masking, fixed character masking, fixed character replacement, MD5, SHA1, SHA256, HMAC, DES, and AES.

6. The public data fusion development platform according to claim 1, characterized in that, The privacy-preserving computation system includes a federated learning framework and multi-party secure computation capabilities configured as follows: The federated learning framework supports horizontal federated learning, vertical federated learning, and federated transfer learning. The multi-party secure computation supports privacy intersection, hidden query, basic operations, joint statistics, logistic regression, and linear regression.

7. The public data fusion development platform according to claim 1, characterized in that, The resource management system includes a data development space and development tool unit configured as follows: It supports accessing data analysis, data mining, and model training tools available on the platform; Supports containerized deployment of Rstudio and JupyterLab tools; It supports the deployment of user-defined development tools. After being certified and approved by the platform administrator, user-defined development tools are processed through the platform storage service to handle authorized data, and the user-defined development tools are deployed in a containerized manner. It has a built-in AI model training unit and provides a base image that supports deep learning. The base image includes a Python runtime environment, TensorFlow, Keras deep learning framework, and SciPy and NumPy data processing libraries. Users can add their applications to the base image and deploy them to the platform.

8. The public data fusion development platform according to claim 1, characterized in that, The secondary development system also includes a development process management unit configured as follows: One application scenario corresponds to one development space, and each development space provides development tools and mutually isolated system environments; The results of development instances within the development space can only be exported to the platform to generate new datasets. For scenarios requiring output review, the results can only be exported to the designated area of ​​the platform to generate new datasets after the platform operator has approved them.

9. The public data fusion development platform according to claim 1, characterized in that, The operation management system includes a development data catalog management unit and an application scenario management unit: The development data catalog management unit compiles a development data catalog based on the public data resource catalog, which includes data business applications, technical parameter information, and synchronously generates sample data. It supports CSV, Excel, and JSON data formats and API interfaces. The application scenario management unit collects application scenario construction requirements and imports them into the platform. The platform updates and organizes the scenario name, description, and technical parameter information, generates or updates the application scenario directory, and automatically associates the original application scenario requirement summary content when the application scenario directory is published, and publishes it through the platform's portal website. Both the development data directory and the application scenario directory support directory query, directory details viewing, directory export, directory version management, directory submission and publishing.

10. The public data fusion development platform according to claim 1, characterized in that, The operation management system includes a visual cockpit configured as follows: Display data authorization information and data lineage information. Data authorization information includes data processing application information of primary and secondary development entities and authorization information of resource providers. Data lineage information includes data association and dependency relationships, data sources and circulation and processing processes. The system displays the status of data resource management, data demand management, and data application management. The data resource management system provides a visual display of the daily inventory of public data resource catalogs and sample data details. The data demand management system provides a visual display of the information filling, material uploading, and approval process results for secondary development entities. The data application management system provides a visual display of the review results and approval progress of data interfaces, data packages, data models, and data reports. It displays the status of data service management, task scheduling management, data application management, and data operation and maintenance management. Data service management visualizes the encapsulation, registration, and release of data service interfaces. Task scheduling management visualizes task monitoring and alarm status. Data application management visualizes the development status and progress of data applications. Data operation and maintenance management visualizes the operation and maintenance status of each sub-platform, including the monitoring and alarm center, resource overview, fault emergency handling, and log management information.

Images