Construction method of industrial park big data center and big data center system

Abstract

The invention discloses an industrial park big data center construction method and a big data center system, and belongs to the technical field of edge computing and artificial intelligence, and the method comprises the steps: data collection, obtaining of data required by a big data platform from multiple data sources, data access, and visual configuration and task scheduling management. Managing and controlling the process of aggregation and access of various types of data; data calculation: extracting data from a data source, processing the data, and storing the data, so as to complete data reconstruction; the data service comprises data calling, data persistence, security management, basic data service, data collision service, holographic query service, relationship-based service and location-based service; and data application: unified convergence access is carried out on structured and unstructured data of all departments and the Internet by providing a standard data unified access mechanism. The problem that data cannot be efficiently circulated and shared in the operation process of the professional park of the oral health industry is solved.

Description

Technical Field

[0001] The present invention relates to the field of edge computing and artificial intelligence technology, and specifically to a method for constructing a big data center in an industrial park and a big data center system. Background Art

[0002] During the operation of the industrial professional park, oral industry data is stored in different enterprises in the upstream and downstream industrial chains, and the internal data of enterprises are stored in different systems, applications or geographical locations, and some are even stored in paper documents; enterprises in the upstream and downstream industrial chains understand and define data from the perspective of their own enterprises or departments, and some of the same data are given different meanings. There is no unified data standard between the institutions in the upstream and downstream industrial chains, the data interface is not open, and the data sharing mechanism is not sound. There is a lack of effective connection and interaction between data islands, and the data utilization efficiency is low. It is difficult for decision makers to fully understand the production and operation status of the oral health industry / enterprises, which affects the efficiency and quality of decision-making. Summary of the invention

[0003] The technical task of the present invention is to address the above shortcomings and provide a method for constructing an industrial park big data center and a big data center system to solve the problem that data cannot be efficiently circulated and shared during the operation of oral health industry professional parks, provide accurate, proactive and timely data services for enterprises, park operating organizations and governments, and provide data support for decision-making on business operations and industrial development.

[0004] The technical solution adopted by the present invention to solve its technical problem is:

[0005] A method for constructing an industrial park big data center, comprising:

[0006] Data collection: obtain the data required by the big data platform from multiple data sources, access the data, and manage the process of data aggregation and access through visual configuration and task scheduling management;

[0007] Data computing: extracting data from data sources, processing data, and storing data, thereby completing data reconstruction to meet the data format requirements of big data search applications and other data mining applications;

[0008] Data services, including data call, data persistence, security management, basic data services, data collision services, holographic query services, relationship-based services, and location-based services;

[0009] Data application, by providing a standard unified data access mechanism, can uniformly aggregate and access structured and unstructured data from various departments and the Internet; and provide unified cleaning, conversion, deduplication and cataloging functions for the accessed data to form a data model that meets unified storage requirements; use distributed databases and distributed file systems to store structured and unstructured data, establish a high availability, high performance and scalability mechanism for information resources, and use the elastic expansion and dynamic scaling mechanism of cloud computing to achieve flexible storage of big data.

[0010] This method gathers multi-source heterogeneous data, mines implicit, unknown, and potentially valuable relationships, patterns, and trends from a large amount of data (including text), uses machine learning to establish a model for decision support through the mined knowledge and rules, and provides predictive decision support relationships, tools, and processes. It includes: data acquisition, data calculation, data service, and data application. Among them, data acquisition is to automatically collect non-electrical or electrical signals from analog and digital units such as sensors and other devices under test through the Internet of Things platform, and send them to the host computer for analysis and processing, and finally form various physical quantities, analog quantities, or digital quantities. Data calculation is connected with data acquisition, and the collected data is stored after transmission, filtering, cleaning, disambiguation, etc.; data service is supported as a service interface for data application, and all data display sources will be supported by data service; data application is the display window of the system, and the user has the most intuitive interface display and interface operation.

[0011] Furthermore, data collection and data access methods include:

[0012] Offline data synchronization: For departments that have not built business systems and only have electronic files or the original business systems cannot be connected due to maintenance issues, we dynamically generate Excel template files based on the information items provided by the departments. After the departments fill in the data, we provide the ability to parse and import the data.

[0013] Real-time synchronization: When the business database cannot be collected through the front-end database and real-time collection is required, incremental data capture of the database is achieved by parsing the database log, without changing the original network structure or modifying the business system, minimizing the impact. By parsing the database log file through RDS, DML, DDL and DCL data, as well as data source information, can be captured. The data collected by RDS is more comprehensive, which can better ensure data consistency and become a useful supplement for data operation behavior auditing.

[0014] Data connection: Data connection management supports editing and modifying data connections. After a data connection is created, users can modify the parameters of the data connection according to their needs. When users do not need to use a data connection, they can delete the data connection in data management.

[0015] Task scheduling: By running, stopping, and resuming jobs, the jobs can be effectively scheduled and executed in a pre-set logical order;

[0016] Breakpoint resume: The encrypted compressed data packet is processed into pieces, and each part is uploaded / downloaded using a thread. If a network failure occurs, it supports continuing to upload / download the unfinished part from the part that has been uploaded / downloaded. Through the breakpoint resume capability, in an unstable network, when sending a large file, it is automatically split into database packets. If a high-priority task or network interruption occurs, the transmission can be temporarily stopped; when the transmission can be continued, it will continue from the last interruption without resending;

[0017] Cross-network routing: In different network segments and layers, based on the directional communication between exchange node services, data packets are directed forwarded through the https encrypted transmission protocol to achieve cross-segment and cross-layer data transmission.

[0018] Furthermore, the task scheduling supports single scheduling and periodic scheduling, supports running, pausing, resuming, and terminating jobs, supports viewing the running details of jobs and task nodes in jobs, supports dependency scheduling between jobs, provides visual workflow design functions, and has built-in process design node components.

[0019] Furthermore, the data calculation includes:

[0020] 1) Data attribute design:

[0021] Formulate coding rules and standardize data for the park information system, and formulate and maintain information coding and maintenance procedures;

[0022] Create data elements and related entities, describe data elements and their attributes using data modeling, provide a common vocabulary for information exchange and sharing, and support integrated business between government departments;

[0023] Create a standard file for the value range table of a data element;

[0024] Ensure the quality of data elements;

[0025] Compile the required data dictionary set to form the data level standard of application software, provide standards and compliance for data sharing, intercommunication, data collection and integration, and support SQL statements to access database data dictionary;

[0026] 2) Data model management:

[0027] Create data modeling, abstractly organize various types of data, determine the scope of the database, the organization form of the data, and transform it into a real database;

[0028] According to the conceptual data model, data entities are mapped into tables, entity attributes are mapped into fields, and entity relationships are mapped into constraints, gradually converting the gain year data model into a physical data model;

[0029] The logical data model includes the overall logical model of the design system, the cross-business model, the business information association model, the service object information association model, the service object information association model and the business system logical data model design;

[0030] 3) Metadata design:

[0031] Access the metadata of the data management system, develop an adapter package according to the adapter interface specification requirements, upload the adapter package through the adaptation function, and complete the extraction of this type of metadata;

[0032] Metadata information maintenance supports query, addition, modification, and deletion operations, and supports retaining historical version information;

[0033] Divide metadata into three categories: business metadata, technical metadata, and operational metadata;

[0034] Supports querying metadata change records, including change content and creation time;

[0035] Metadata identifies the version, including the version name, version description, and version finalization time;

[0036] Support metadata version comparison;

[0037] 4) Data collection:

[0038] Data collection supports SDK collection of stock data and subsequent incremental data collection;

[0039] Supports summarizing and organizing the collected data and storing the database in a structured form;

[0040] Supports unstructured data storage database, retrieval and knowledge mining;

[0041] Supports converting fields in the target data into corresponding fields in the business system and supports deleting fields;

[0042] Data acquisition monitoring monitors and controls on-site operating equipment.

[0043] 5) Data backup:

[0044] The original business data of the park is stored in the buffer, extracted to the data center through data extraction tools, and standardized cleaning and conversion are performed in the area;

[0045] Load business processes and original business data into the sedimentation area for conversion, store the data in the park's original business data area, and provide data summaries for the data warehouse of basic data;

[0046] The data support area includes the main database, index library, code library, and provides databases for data management and basic support;

[0047] Supports backup and restore of system data, business models, data models, and data tables;

[0048] Collect system status at time intervals, record the time series of system status, divide the system status time series according to rules, and calculate statistics for each shard based on data characteristics;

[0049] 6) Data association:

[0050] The collected, converted and loaded internal and external data are associated according to the associated primary key to form an information view related to the park, enterprise and event;

[0051] 7) Data extraction:

[0052] Data association forms a data pool, and objects and metadata storage mapping rules are configured in the data pool. Metadata is extracted from the extraction source to the specified target metadata database, supporting incremental extraction and full extraction;

[0053] 8) Data Fusion:

[0054] Use data matching algorithms to compare the similarity of duplicate data collected from various campus systems, identify possible duplicate record pairs from the data, and integrate the information of duplicate record pairs through automatic fusion strategies and manual intervention to form unique and correct records;

[0055] 9) Data standard design:

[0056] Establish data entity standards and data relationship standards, continuously implement data standards and handle non-compliant data, and continuously evaluate data standards to promptly correct issues where data standards have expired;

[0057] 10) Data evaluation:

[0058] Evaluate the data, discover hidden problems in the data, assist in the formulation of data standards or rules, and provide reference for the formulation of data cleaning solutions and quality plans;

[0059] 11) Data cleaning

[0060] According to the requirements of the basic database system model, the extracted metadata is converted, cleaned, split, and summarized. The identified erroneous data is repaired automatically or manually, incomplete data is completed or enhanced, missing data is automatically filled, and non-standard data is normalized.

[0061] Furthermore, the data calls:

[0062] Deploy an intermediate database to realize the rapid exchange of different data on park services, park security, and park affairs;

[0063] Establish rules for process sharing, rule sharing, and service sharing to ensure that park services, park security, park affairs, etc. are interconnected at the business level.

[0064] The data is persisted:

[0065] Provide a unified data access mechanism to shield the restrictions of specific database system platforms on the system implementation layer;

[0066] Provide unified programming for DB2, Oracle, and SQL Server databases;

[0067] Build a bridge between business logic data objects and physical databases to reduce the difficulty of changing physical databases;

[0068] Configure the same calling method for different remote access protocols;

[0069] The security management:

[0070] Support user memory permission allocation and control mechanism;

[0071] Supports data encryption, encapsulation and transmission using a unique encryption method;

[0072] The basic data services:

[0073] Provide data services including address services, encoding services, encoding conversion and address translation;

[0074] The data collision service:

[0075] Provide specified data association, comparison and analysis services;

[0076] The holographic query service:

[0077] Support external personal and corporate comprehensive credit information query services. External applications pass in specific information and query scope, and the corresponding information of the entity will be returned;

[0078] The relationship-based services:

[0079] Based on data mining technology, the existing information of the three basic databases is mined and stored in the relationship graph; at the same time, the data service platform provides social relationship data of each entity based on population and legal person relationship analysis, and provides it to external applications through the data service interface;

[0080] The location-based services:

[0081] The geographic information GIS system provides location-based API services, including returning GPS coordinates based on addresses, as well as location services such as proximity queries and geographic analysis.

[0082] Furthermore, the data application provides unified cleaning, conversion, deduplication and cataloging functions for the accessed data, including management of data definition, data structure, data identification, data encoding, data cataloging, responsible persons, sources, conversion relationships, targets, quality levels, dependencies, and security permissions.

[0083] Furthermore, the data application specifically includes:

[0084] 1) Data warehouse management:

[0085] A data warehouse is a structured data environment for decision support systems and online analytical application data sources, providing all types of data support including business process improvement, monitoring time, cost, quality, and control:

[0086] 2) Real-time computing and analysis:

[0087] Provide a unified real-time computing platform to meet the requirements of business development, refined operations and monitoring;

[0088] 3) Offline calculation analysis:

[0089] Supports inputting all known and unchanged data before the calculation starts, and the calculation results can be obtained immediately after running the calculation;

[0090] 4) Data discovery and analysis:

[0091] The classification engine scans the database and identifies columns containing potentially sensitive data, allowing you to review and apply corresponding classification suggestions or manually classify data columns;

[0092] Mark, which permanently marks the column with a sensitivity classification label;

[0093] View the database classification status in the detailed report, and directly print / export the report if it is required in compliance and audit matters;

[0094] 5) Exploratory Data Analysis:

[0095] Starting from objective data, explore the inherent data regularity from actual data;

[0096] 6) Accurate data services:

[0097] On the basis of the existing information system, build data channels between systems to drill and integrate current, historical and scattered business data;

[0098] Based on the user's behavior data, error processing is performed on the user's service data to analyze the user's service habits. When the user's service gradually stabilizes on a certain type and related items, a scheduled push service is provided to the user.

[0099] The label accurate identification service supports the construction of service label portraits of enterprises and individuals in the park, providing support for the rapid matching of government service resources and needs;

[0100] Based on the information in the service item database, the service items are classified according to various themes, and precise government services are provided in the form of themes for users who meet the requirements, including the association of service themes with tags and the maintenance of service themes;

[0101] Obtain user needs through technical algorithms, provide push methods including broadcast push, tag push, scheduled push, and associated push, and push the right service content to the right people at the right time;

[0102] By analyzing and presenting the active service objects, service content and service evaluation data in multiple dimensions; providing analysis including service objects, access channels, visitor trajectories, specific objects, service themes, popular services, service evaluations and service needs;

[0103] Based on the historical service database, the matters are classified by park level and by the matters of each functional department. Through the peak service volume analysis of one day, one week and one year, the peak service volume of each time period is analyzed. During the peak service period, the public is actively reminded to avoid the peak service period and guide the public to handle the matters during the off-peak service period.

[0104] The user's mathematical model is calculated based on the user's behavioral characteristics, and the needs that the user may be interested in are found and recommended to the user.

[0105] The present invention also claims protection for an industrial park big data center system, comprising:

[0106] The data collection module is used to obtain the data required by the big data platform from multiple data sources, access the data, and control the process of data aggregation and access through visual configuration and task scheduling management;

[0107] The data computing module is used to extract data from the data source, process data, and store data, thereby completing data reconstruction to meet the data format requirements of big data search applications and other data mining applications;

[0108] Data service module, including data call, data persistence, security management, basic data service, data collision service, holographic query service, relationship-based service and location-based service;

[0109] The data application module provides a standard unified data access mechanism to aggregate and access structured and unstructured data from various departments and the Internet in a unified manner; it also provides unified cleaning, conversion, deduplication and cataloging functions for the accessed data to form a data model that meets the unified storage requirements; it uses distributed databases and distributed file systems to store structured and unstructured data, establishes a high availability, high performance and scalability mechanism for information resources, and uses the elastic expansion and dynamic scaling mechanism of cloud computing to achieve flexible storage of big data;

[0110] The system is specifically constructed through the above-mentioned method of constructing the industrial park big data center.

[0111] The present invention also claims protection for a construction device for an industrial park big data center, comprising: at least one memory and at least one processor;

[0112] The at least one memory is used to store a machine-readable program;

[0113] The at least one processor is used to call the machine-readable program to implement the above method.

[0114] The present invention also claims protection for a computer-readable medium having computer instructions stored thereon, which implement the above method when executed by a processor.

[0115] Compared with the prior art, the method for constructing an industrial park big data center and the big data center system of the present invention have the following beneficial effects:

[0116] With the help of the construction of the Yagu Industrial Smart Park, the present invention acquires oral health industry big data and oral industry chain data through multiple channels and all factors, including data on industrial park investment promotion, services, operation management and development, dynamic data on oral industry policies and regulations, industrial chain, market sales, product registration, new technology, financing and mergers and acquisitions, etc., and has built dynamic data such as oral epidemiological surveys, and shares geographical, emergency, government affairs and other data related to industrial park management departments. Integrate and access multi-source heterogeneous data and build basic databases, subject information databases, and special information databases to build an oral health industry big data center in the industrial smart park, provide accurate, active and timely data services for enterprises, park operating institutions and governments, and provide data support for decision-making in business operations and industrial development. BRIEF DESCRIPTION OF THE DRAWINGS

[0117] Figure 1 It is a step diagram of a method for constructing an industrial park big data center provided by an embodiment of the present invention;

[0118] Figure 2 It is a logical diagram of constructing an industrial park big data center provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0119] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0120] A method for constructing a big data center in an industrial park includes: data collection, data calculation, data service and data application. Among them, data collection is to automatically collect non-electrical or electrical signals from analog and digital units under test such as sensors and other devices under test through the Internet of Things platform, and send them to the host computer for analysis and processing, and finally form various physical quantities, analog quantities or digital quantities and other data; data calculation is connected with data collection, and the collected data is stored after transmission, filtering, cleaning, disambiguation and other processing; data service is supported as a service interface of data application, and all data display sources will be supported by data service; data application is the display window of the system, and the user has the most intuitive interface display and interface operation.

[0121] 1. Data collection:

[0122] The data required by the big data platform is obtained from multiple data sources through interfaces and manual methods, including offline data synchronization, real-time synchronization, data connection, task scheduling, breakpoint continuation, multi-channel transmission, cross-network routing and other methods for data access. Through visual configuration and task scheduling management, the process of data aggregation and access is controlled. Data collection specifically includes:

[0123] (1) Offline data synchronization: For departments that have not built business systems and only have electronic files or whose original business systems cannot be connected due to maintenance issues, we dynamically generate Excel template files based on the information items provided by the departments. After the departments fill in the data, we provide the ability to parse and import the data.

[0124] (2) Real-time synchronization: When business databases cannot be collected through the front-end database and need to be collected in real time, incremental data capture of the database can be achieved by parsing the database logs. This does not change the original network structure and does not require the transformation of the business system, thus minimizing the impact. RDS can capture DML, DDL, and DCL data, as well as data source information, by parsing database log files. The data collected by RDS is more comprehensive, which can better ensure data consistency and become a useful supplement for data operation behavior auditing.

[0125] (3) Data connection: Data connection management supports editing and modifying data connections. After a data connection is created, users can modify the parameters of the data connection according to their needs. When users no longer need to use a data connection, they can delete the data connection in data management.

[0126] (4) Task Scheduling: By running, stopping, resuming, and other operations on jobs, the jobs can be effectively scheduled and executed in a pre-set logical order. The system supports single-time scheduling and periodic scheduling, supports running, pausing, resuming, and terminating jobs, supports viewing the running details of jobs and task nodes in jobs, supports dependent scheduling between jobs, provides visual workflow design functions, and has built-in process design node components.

[0127] (5) Breakpoint resume: The encrypted compressed data packet is divided into segments, and each segment is uploaded / downloaded using a thread. If a network failure occurs, the system supports resuming the upload / download of the unfinished segment from the uploaded / downloaded segment. With the breakpoint resume capability, when sending a large file in an unstable network, the file is automatically split into database packets. If a high-priority task or network interruption occurs, the transmission can be temporarily stopped. When the transmission can be continued, it is continued from the last interruption without the need to resend the file.

[0128] (6) Cross-network routing: In different network segments and layers, data packets are directed forwarded through the https encrypted transmission protocol based on the directional communication between exchange node services, thus achieving cross-segment and cross-layer data transmission.

[0129] 2. Data calculation:

[0130] Extract data from data sources, process data, and store data to complete data reconstruction and meet the data format requirements of big data search applications and other data mining applications; data governance is carried out from the aspects of data application data management, system development, massive data analysis and mining, etc. Specifically including:

[0131] (1) Data attribute design:

[0132] Formulate coding rules and standardize data for the park information system, and formulate and maintain information coding and maintenance procedures;

[0133] Create data elements and related entities, describe data elements and their attributes using data modeling, provide a common vocabulary for information exchange and sharing, and support integrated business between government departments;

[0134] Create a standard file for the value range table of a data element;

[0135] Ensure the quality of data elements;

[0136] Compile the required data dictionary set to form the data level standard of application software, provide standards and compliance for data sharing, intercommunication, data collection and integration, and support SQL statements to access database data dictionary;

[0137] (2) Data model management:

[0138] Create data modeling, abstractly organize various types of data, determine the scope of the database, the organization form of the data, and transform it into a real database;

[0139] According to the conceptual data model (CDM), data entities are mapped into tables, entity attributes are mapped into fields, and entity relationships are mapped into constraints, gradually converting the gain year data model into the physical data model (PDM);

[0140] The logical data model includes the overall logical model of the design system, the cross-business model, the business information association model, the service object information association model, the service object information association model and the business system logical data model design;

[0141] (3) Metadata design:

[0142] Access the metadata of the data management system, develop an adapter package according to the adapter interface specification requirements, upload the adapter package through the adaptation function, and complete the extraction of this type of metadata;

[0143] Metadata information maintenance supports query, addition, modification, and deletion operations, and supports retaining historical version information;

[0144] Divide metadata into three categories: business metadata, technical metadata, and operational metadata;

[0145] Supports querying metadata change records, including change content and creation time;

[0146] Metadata identifies the version, including the version name, version description, and version finalization time;

[0147] Supports metadata version comparison.

[0148] (4) Data collection:

[0149] Data collection supports SDK collection of stock data and subsequent incremental data collection;

[0150] Supports summarizing and organizing the collected data and storing the database in a structured form;

[0151] Supports unstructured data storage database, retrieval and knowledge mining;

[0152] Supports converting fields in the target data into corresponding fields in the business system and supports deleting fields;

[0153] Data acquisition monitoring monitors and controls on-site operating equipment.

[0154] (5) Data backup:

[0155] The original business data of the park is stored in the buffer, extracted to the data center through data extraction tools, and standardized cleaning and conversion are performed in the area;

[0156] Load business processes and original business data into the sedimentation area for conversion, store the data in the park's original business data area, and provide data summaries for the data warehouse of basic data;

[0157] The data support area includes the main database, index library, code library, and provides databases for data management and basic support;

[0158] Supports backup and restore of system data, business models, data models, and data tables;

[0159] Collect system status at time intervals, record the time series of system status, divide the system status time series according to rules, and calculate statistics for each shard based on data characteristics;

[0160] (6) Data association:

[0161] The collected, converted and loaded internal and external data are associated according to the associated primary key to form an information view related to the park, enterprise and event.

[0162] (7) Data extraction:

[0163] Data association forms a data pool, and objects and metadata storage mapping rules are configured in the data pool. Metadata is extracted from the extraction source to the specified target metadata database, supporting incremental extraction and full extraction.

[0164] (8) Data fusion: Use data matching algorithms to compare the similarities of duplicate data collected from various campus systems, identify possible duplicate record pairs from the data, and integrate the information of duplicate record pairs through automatic fusion strategies and manual intervention to form unique and correct records.

[0165] (9) Data standard design:

[0166] Establish data entity standards and data relationship standards, continuously implement data standards and handle data that does not meet the standards. At the same time, continuously evaluate data standards and promptly correct problems with outdated data standards.

[0167] (10) Data evaluation:

[0168] Evaluate the data, discover hidden problems in the data, assist in the formulation of data standards or rules, and provide reference for the formulation of data cleaning solutions and quality plans.

[0169] (11) Data cleaning:

[0170] According to the requirements of the basic database system model, the extracted metadata is converted, cleaned, split, and summarized. The identified erroneous data is repaired automatically or manually, incomplete data is completed or enhanced, missing data is automatically filled, and non-standard data is normalized.

[0171] 3. Data services, including:

[0172] (1) Data call:

[0173] Deploy an intermediate database to realize the rapid exchange of different data on park services, park security, and park affairs;

[0174] Establish rules for process sharing, rule sharing, and service sharing to ensure that park services, park security, park affairs, etc. are interconnected at the business level.

[0175] (2) Data persistence:

[0176] Provide a unified data access mechanism to shield the restrictions of specific database system platforms on the system implementation layer;

[0177] Provide unified programming for DB2, Oracle, and SQL Server databases;

[0178] Build a bridge between business logic data objects and physical databases to reduce the difficulty of changing physical databases;

[0179] Configure the same calling method for different remote access protocols.

[0180] (3) Safety management:

[0181] Support user memory permission allocation and control mechanism;

[0182] Supports data to be encrypted, encapsulated and transmitted using a unique encryption method.

[0183] (4) Basic data services:

[0184] Provides data services including address services, encoding services, encoding conversion and address translation.

[0185] (5) Data collision service:

[0186] Provides services including specified data association and comparison analysis (collision).

[0187] (6)Holographic query service:

[0188] It supports external provision of comprehensive credit information query services for individuals and enterprises. External applications pass in specific information and query scope, and the corresponding information of the entity will be returned.

[0189] (7) Relationship-based services:

[0190] Based on data mining technology, the existing information of the three basic databases is mined and stored in the relationship graph; at the same time, the data service platform provides social relationship data of each entity based on population and legal person relationship analysis, and provides it to external applications through the data service interface;

[0191] (8) Location-based services:

[0192] The geographic information GIS system provides location-based API services, including returning GPS coordinates based on addresses, as well as location services such as proximity queries and geographic analysis.

[0193] 4. Data application:

[0194] By providing a standard unified data access mechanism, the structured and unstructured data of various departments and the Internet are uniformly aggregated and accessed; and unified cleaning, conversion, deduplication and cataloging functions are provided for the accessed data, including the management of data definition, data structure, data identification, data coding, data cataloging, responsible person, source, conversion relationship, target, quality level, dependency, security authority and related content, forming a data model that meets the unified storage requirements; using distributed databases and distributed file systems to store structured and unstructured data, establishing a high availability, high performance and scalability mechanism for information resources, and using the elastic expansion and dynamic scaling mechanism of cloud computing to achieve flexible storage of big data. Including:

[0195] (1) Data warehouse management:

[0196] A data warehouse is a structured data environment for decision support systems and online analytical application data sources, providing all types of data support including business process improvement, monitoring time, cost, quality, and control.

[0197] (2) Real-time calculation and analysis:

[0198] Provides a unified real-time computing platform to meet the requirements of business development, refined operations and monitoring.

[0199] (3) Offline calculation and analysis:

[0200] Supports inputting all known and unchanged data before the calculation starts, and the calculation results can be obtained immediately after running the calculation;

[0201] (4) Data discovery and analysis:

[0202] The classification engine scans the database and identifies columns containing potentially sensitive data. You can view and apply classification suggestions or manually classify data columns.

[0203] Mark, which permanently marks the column with a sensitivity classification label;

[0204] You can view the database classification status in the detailed report. If you need it in compliance and audit matters, you can directly print / export the report.

[0205] (5) Exploratory Data Analysis:

[0206] Starting from objective data, explore the inherent data regularity from actual data and avoid the shortcomings of traditional statistical analysis methods.

[0207] (6) Data Accuracy Service:

[0208] On the basis of the existing information system, build data channels between systems, drill and integrate current, historical and scattered business data, make full use of existing resources, activate electronic data, and improve management decision-making level;

[0209] Based on the user's behavior data, error processing is performed on the user's service data to analyze the user's service habits. When the user's service gradually stabilizes on a certain type and related items, a scheduled push service is provided to the user.

[0210] The label accurate identification service supports the construction of service label portraits of enterprises and individuals in the park, providing support for the rapid matching of government service resources and needs;

[0211] Based on the information in the service item database, the service items are classified according to various themes, and precise government services are provided in the form of themes for users who meet the requirements, including the association of service themes with tags and the maintenance of service themes;

[0212] Obtain user needs through technical algorithms, provide push methods including broadcast push, tag push, scheduled push, and associated push, and push the right service content to the right people at the right time;

[0213] By analyzing and presenting active service objects, service content and service evaluation data in multiple dimensions, it provides effective support for government departments to innovate active services and precise service models and make scientific and reasonable government service decisions; it provides analysis of service objects, access channels, visitor trajectories, specific objects, service themes, popular services, service evaluations and service needs;

[0214] Based on the historical service database, the matters are classified by park level and by the matters of each functional department. Through the peak service volume analysis of one day, one week and one year, the peak service volume of each time period is analyzed. During the peak service period, the public is actively reminded to avoid the peak service period and guide the public to handle the matters during the off-peak service period.

[0215] The user's mathematical model is calculated based on the user's behavioral characteristics, and the needs that the user may be interested in are found and recommended to the user.

[0216] This method aggregates multi-source heterogeneous data, mines implicit, unknown relationships, patterns, and trends that are potentially valuable for decision-making from large amounts of data (including text), uses machine learning to establish models for decision support based on the mined knowledge and rules, and provides relationships, tools, and processes for predictive decision support.

[0217] The embodiment of the present invention further provides an industrial park big data center system, including:

[0218] The data collection module is used to obtain the data required by the big data platform from multiple data sources, access the data, and control the process of data aggregation and access through visual configuration and task scheduling management;

[0219] The data computing module is used to extract data from the data source, process data, and store data, thereby completing data reconstruction to meet the data format requirements of big data search applications and other data mining applications;

[0220] Data service module, including data call, data persistence, security management, basic data service, data collision service, holographic query service, relationship-based service and location-based service;

[0221] The data application module provides a standard unified data access mechanism to aggregate and access structured and unstructured data from various departments and the Internet in a unified manner; it also provides unified cleaning, conversion, deduplication and cataloging functions for the accessed data to form a data model that meets the unified storage requirements; it uses distributed databases and distributed file systems to store structured and unstructured data, establishes a high availability, high performance and scalability mechanism for information resources, and uses the elastic expansion and dynamic scaling mechanism of cloud computing to achieve flexible storage of big data;

[0222] The system is specifically constructed through the method for constructing the industrial park big data center described in the above embodiment.

[0223] An embodiment of the present invention further provides a device for constructing an industrial park big data center, including: at least one memory and at least one processor;

[0224] The at least one memory is used to store a machine-readable program;

[0225] The at least one processor is used to call the machine-readable program to implement the method for constructing the industrial park big data center described in the above embodiment.

[0226] The embodiment of the present invention further provides a computer-readable medium, on which computer instructions are stored, and when the computer instructions are executed by a processor, the method for constructing the industrial park big data center described in the above embodiment is implemented. Specifically, a system or device equipped with a storage medium can be provided, on which software program codes implementing the functions of any of the above embodiments are stored, and a computer (or CPU or MPU) of the system or device is enabled to read and execute the program code stored in the storage medium.

[0227] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0228] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0229] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0230] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0231] The present invention is shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A method for constructing a big data center in an industrial park, characterized in that: include: Data collection: obtain the data required by the big data platform from multiple data sources, access the data, and manage the process of data aggregation and access through visual configuration and task scheduling management; Data computing: extracting data from data sources, processing data, and storing data, thereby completing data reconstruction to meet the data format requirements of big data search applications and other data mining applications; Data services, including data call, data persistence, security management, basic data services, data collision services, holographic query services, relationship-based services, and location-based services; Data application, by providing a standard unified data access mechanism, can centralize and access structured and unstructured data from various departments and the Internet; It also provides unified cleaning, conversion, deduplication and cataloging functions for accessed data to form a data model that meets unified storage requirements; uses distributed databases and distributed file systems to store structured and unstructured data, establishes high availability, high performance and scalability mechanisms for information resources, and uses the elastic expansion and dynamic scaling mechanisms of cloud computing to achieve flexible storage of big data.

2. The method for constructing a big data center in an industrial park according to claim 1, characterized in that: Data collection and data access methods include: Offline data synchronization: For departments that have not built business systems and only have electronic files or the original business systems cannot be connected due to maintenance issues, we dynamically generate Excel template files based on the information items provided by the departments. After the departments fill in the data, we provide the ability to parse and import the data. Real-time synchronization: When the business database cannot be collected through the front-end database and real-time collection is required, incremental data capture of the database is achieved by parsing the database logs and parsing the database log files through RDS; Data connection: Data connection management supports editing and modifying data connections. After a data connection is created, users can modify the parameters of the data connection according to their needs. When users do not need to use a data connection, they can delete the data connection in data management. Task scheduling: By running, stopping, and resuming jobs, the jobs can be effectively scheduled and executed in a pre-set logical order; Breakpoint resume: The encrypted compressed data packet is processed into pieces, and each part is uploaded / downloaded using a thread. If a network failure occurs, it supports continuing to upload / download the unfinished part from the part that has been uploaded / downloaded. Through the breakpoint resume capability, in an unstable network, when sending a large file, it is automatically split into database packets. If a high-priority task or network interruption occurs, the transmission can be temporarily stopped; when the transmission can be continued, it will continue from the last interruption without resending; Cross-network routing: In different network segments and layers, based on the directional communication between exchange node services, data packets are directed forwarded through the https encrypted transmission protocol to achieve cross-segment and cross-layer data transmission.

3. The method for constructing a big data center in an industrial park according to claim 2, characterized in that: The task scheduling supports single scheduling and periodic scheduling, supports running, pausing, resuming, and terminating jobs, supports viewing the running details of jobs and task nodes in jobs, supports dependency scheduling between jobs, provides visual workflow design functions, and has built-in process design node components.

4. The method for constructing a big data center in an industrial park according to claim 1, characterized in that: The data calculation includes: 1) Data attribute design: Formulate coding rules and standardize data for the park information system, and formulate and maintain information coding and maintenance procedures; Create data elements and related entities, describe data elements and their attributes using data modeling, provide a common vocabulary for information exchange and sharing, and support integrated business between government departments; Create a standard file for the value range table of a data element; Ensure the quality of data elements; Compile the required data dictionary set to form the data level standard of application software, provide standards and compliance for data sharing, intercommunication, data collection and integration, and support SQL statements to access database data dictionary; 2) Data model management: Create data modeling, abstractly organize various types of data, determine the scope of the database, the organization form of the data, and transform it into a real database; According to the conceptual data model, data entities are mapped into tables, entity attributes are mapped into fields, and entity relationships are mapped into constraints, gradually converting the gain year data model into a physical data model; The logical data model includes the overall logical model of the design system, the cross-business model, the business information association model, the service object information association model, the service object information association model and the business system logical data model design; 3) Metadata design: Access the metadata of the data management system, develop an adapter package according to the adapter interface specification requirements, upload the adapter package through the adaptation function, and complete the extraction of this type of metadata; Metadata information maintenance supports query, addition, modification, and deletion operations, and supports retaining historical version information; Divide metadata into three categories: business metadata, technical metadata, and operational metadata; Supports querying metadata change records, including change content and creation time; Metadata identifies the version, including the version name, version description, and version finalization time; Support metadata version comparison; 4) Data collection: Data collection supports SDK collection of stock data and subsequent incremental data collection; Supports summarizing and organizing the collected data and storing the database in a structured form; Supports unstructured data storage database, retrieval and knowledge mining; Supports converting fields in the target data into corresponding fields in the business system and supports deleting fields; Data acquisition monitoring monitors and controls on-site operating equipment. 5) Data backup: The original business data of the park is stored in the buffer, extracted to the data center through data extraction tools, and standardized cleaning and conversion are performed in the area; Load business processes and original business data into the sedimentation area for conversion, store the data in the park's original business data area, and provide data summaries for the data warehouse of basic data; The data support area includes the main database, index library, code library, and provides databases for data management and basic support; Supports backup and restoration of system data, business models, data models, and data tables; Collect system status at time intervals, record the time series of system status, divide the system status time series according to rules, and calculate statistics for each shard based on data characteristics; 6) Data association: The collected, converted and loaded internal and external data are associated according to the associated primary key to form an information view related to the park, enterprise and event; 7) Data extraction: Data association forms a data pool, and objects and metadata storage mapping rules are configured in the data pool. Metadata is extracted from the extraction source to the specified target metadata database, supporting incremental extraction and full extraction; 8) Data Fusion: Use data matching algorithms to compare the similarity of duplicate data collected from various campus systems, identify possible duplicate record pairs from the data, and integrate the information of duplicate record pairs through automatic fusion strategies and manual intervention to form unique and correct records; 9) Data standard design: Establish data entity standards and data relationship standards, continuously implement data standards and handle non-compliant data, and continuously evaluate data standards to promptly correct issues where data standards have expired; 10) Data evaluation: Evaluate the data, discover hidden problems in the data, assist in the formulation of data standards or rules, and provide reference for the formulation of data cleaning solutions and quality plans; 11) Data cleaning According to the requirements of the basic database system model, the extracted metadata is converted, cleaned, split, and summarized. The identified erroneous data is repaired automatically or manually, incomplete data is completed or enhanced, missing data is automatically filled, and non-standard data is normalized.

5. The method for constructing a big data center in an industrial park according to claim 1, characterized in that: The data call: Deploy an intermediate database to realize the rapid exchange of different data on park services, park security, and park affairs; Establish rules for process sharing, rule sharing, and service sharing to ensure that park services, park security, park affairs, etc. are interconnected at the business level. The data is persisted: Provide a unified data access mechanism to shield the restrictions of specific database system platforms on the system implementation layer; Provide unified programming for DB2, Oracle, and SQL Server databases; Build a bridge between business logic data objects and physical databases to reduce the difficulty of changing physical databases; Configure the same calling method for different remote access protocols; The security management: Support user memory permission allocation and control mechanism; Supports data encryption, encapsulation and transmission using a unique encryption method; The basic data services: Provide data services including address services, encoding services, encoding conversion and address translation; The data collision service: Provide specified data association, comparison and analysis services; The holographic query service: Support external personal and corporate comprehensive credit information query services. External applications pass in specific information and query scope, and the corresponding information of the entity will be returned; The relationship-based services: Based on data mining technology, the existing information of the three basic databases is mined and stored in the relationship graph; at the same time, the data service platform provides social relationship data of each entity based on population and legal person relationship analysis, and provides it to external applications through the data service interface; The location-based services: The geographic information GIS system provides location-based API services, including returning GPS coordinates based on addresses, as well as location services such as proximity queries and geographic analysis.

6. The method for constructing a big data center in an industrial park according to claim 1, characterized in that: The data application provides unified cleaning, conversion, deduplication and cataloging functions for the accessed data, including management of data definition, data structure, data identification, data coding, data cataloging, responsible persons, sources, conversion relationships, targets, quality levels, dependencies, and security permissions.

7. The method for constructing an industrial park big data center according to claim 1 or 6, characterized in that: The data application specifically includes: 1) Data warehouse management: A data warehouse is a structured data environment for decision support systems and online analytical application data sources, providing all types of data support including business process improvement, monitoring time, cost, quality, and control: 2) Real-time computing and analysis: Provide a unified real-time computing platform to meet the requirements of business development, refined operations and monitoring; 3) Offline calculation analysis: Supports inputting all known and unchanged data before the calculation starts, and the calculation results can be obtained immediately after running the calculation; 4) Data discovery and analysis: The classification engine scans the database and identifies columns containing potentially sensitive data, allowing you to review and apply corresponding classification suggestions or manually classify data columns; Mark, which permanently marks the column with a sensitivity classification label; View the database classification status in the detailed report, and directly print / export the report if it is required in compliance and audit matters; 5) Exploratory Data Analysis: Starting from objective data, explore the inherent data regularity from actual data; 6) Accurate data services: On the basis of the existing information system, build data channels between systems to drill and integrate current, historical and scattered business data; Based on the user's behavior data, error processing is performed on the user's service data to analyze the user's service habits. When the user's service gradually stabilizes on a certain type and related items, regular push services are provided to the user. The label accurate identification service supports the construction of service label portraits for enterprises and individuals in the park, providing support for the rapid matching of government service resources and needs; Based on the information in the service item database, the service items are classified according to various themes, and precise government services are provided in the form of themes for users who meet the requirements, including the association of service themes with tags and the maintenance of service themes; Obtain user needs through technical algorithms, provide push methods including broadcast push, tag push, scheduled push, and associated push, and push the right service content to the right people at the right time; By analyzing and presenting the active service objects, service content and service evaluation data in multiple dimensions; providing analysis including service objects, access channels, visitor trajectories, specific objects, service themes, popular services, service evaluations and service needs; Based on the historical service database, the matters are classified by park level and by the matters of each functional department. Through the peak service volume analysis of one day, one week and one year, the peak service volume of each time period is analyzed. During the peak service period, the public is actively reminded to avoid the peak service period and guide the public to handle the matters during the off-peak service period. The user's mathematical model is calculated based on the user's behavioral characteristics, and the needs that the user may be interested in are found and recommended to the user.

8. An industrial park big data center system, characterized in that: include: The data collection module is used to obtain the data required by the big data platform from multiple data sources, access the data, and control the process of data aggregation and access through visual configuration and task scheduling management; The data computing module is used to extract data from the data source, process data, and store data, thereby completing data reconstruction to meet the data format requirements of big data search applications and other data mining applications; Data service module, including data call, data persistence, security management, basic data service, data collision service, holographic query service, relationship-based service and location-based service; The data application module provides a standard unified data access mechanism to aggregate and access structured and unstructured data from various departments and the Internet. It also provides unified cleaning, conversion, deduplication and cataloging functions for accessed data to form a data model that meets unified storage requirements; it uses distributed databases and distributed file systems to store structured and unstructured data, establishes a high availability, high performance and scalability mechanism for information resources, and uses the elastic expansion and dynamic scaling mechanism of cloud computing to achieve flexible storage of big data; The system is specifically constructed through the method for constructing an industrial park big data center as described in any one of claims 1 to 7.

9. A device for constructing a big data center in an industrial park, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is used to call the machine-readable program to implement the method described in any one of claims 1 to 7.

10. A computer-readable medium, characterized in that The computer readable medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images