Intelligent integrated operation system based on data fusion and data access method thereof

Abstract

The present application relates to the field of data processing, and relates to an intelligent integrated operation system based on data fusion and a data access method thereof.The system comprises: a multi-source data integration module, configured to standardize and integrate data of a source layer, a shared layer and an analysis layer through a metadata weaving method, and generate structured data with a blood relationship graph; an intelligent service module, configured to generate a data model based on ER relationship intelligent recommendation of the structured data, perform zero-code modeling based on the data model, and generate a RESTful API interface based on the modeling result; a dynamic operation module; a data synchronization module, configured to perform synchronization operation on metadata of the multi-source data integration module through a push-pull combination mode; and a security control module.The system of the present application can realize "zero approval" of non-negative list data and "online application" of negative list data, and can provide more convenient and intelligent services.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing. More specifically, the present application relates to an intelligent integrated operation system based on data fusion and a data access method thereof. BACKGROUND

[0002] Currently, enterprises generally face serious data island, uneven data quality, low service efficiency and other outstanding problems in data management. Traditional data management mode often adopts decentralized architecture, each business system is independently constructed, and data standards are not unified, which leads to difficulty in cross-system data sharing, and seriously restricts the full play of data value. Taking the power industry as an example, there are obvious data barriers between enterprise operation management, peer benchmarking, digital audit and other core business systems, marketing data and production data are difficult to effectively penetrate, which directly affects the efficiency of enterprise decision-making and the level of business collaboration. In terms of data quality, due to the lack of unified governance mechanism, there are generally problems such as data duplication, field ambiguity, and update not in time. The statistical data of a provincial power grid company shows that among the 120,000 data fields accessed by its data center, about 23% have non-standard naming or ambiguous business meaning, which seriously affects the credibility and usability of data.

[0003] The existing data service mode has obvious efficiency bottleneck. Users often need to search across multiple systems, and the application process is complex, with an average time consumption of 3-5 working days. The data modeling process is highly dependent on technical personnel, and ordinary business personnel cannot complete it independently, which seriously restricts the agility of data application. In terms of security control, the traditional way mostly adopts static permission management, which is difficult to meet the increasingly complex data security needs, especially in the process of data sharing, there is a risk of leakage. The internal evaluation report of a certain energy enterprise points out that the response timeliness of its data service meets the standard rate of only 68%, which is far lower than the expectation value of 90% of the business department. At the same time, data changes lack effective dynamic sensing mechanism, when the data structure of the source system is adjusted, downstream applications often have data retrieval abnormalities, and the proportion of data problem work orders caused by this problem is as high as 15% on average every month.

[0004] In terms of technical implementation, the existing data platform architecture generally has the problems of single function and insufficient scalability. Most systems only provide basic metadata management functions, lacking intelligent data recommendation, automated model building, and other advanced capabilities. In terms of data operation, it mainly relies on manual statistics and report analysis, making it difficult to timely grasp the usage and value output of data assets. In particular, in terms of model update synchronization, a one-way push mechanism is commonly used, which cannot real-time perceive data changes in the source system, making it difficult to guarantee data consistency. The practice of a large manufacturing enterprise shows that the average update delay of its data model is more than 24 hours, seriously affecting the accuracy of real-time business analysis. In addition, the existing system lacks sufficient integration support for third-party analysis tools, and users need to frequently switch between different platforms to complete the entire data analysis process, which not only reduces work efficiency but also increases learning costs.

[0005] The related technical solutions in the industry focus on improving local problems and lack a systematic data integration and operation system. For example, some patent technologies focus on data quality management but do not solve the efficiency problem of data services; some solutions improve data retrieval functions but ignore the modeling and application stages. In terms of authoritative data identification, existing methods mostly use manual annotation, which is inefficient and difficult to maintain. Data security management also mostly stays at the basic permission management level, lacking fine-grained field-level control and dynamic desensitization capabilities. These technical limitations lead enterprises to combine multiple independent systems when implementing data asset operations, increasing construction costs and causing additional integration complexity.

[0006] With the development of new technologies such as artificial intelligence and big data, enterprises have higher requirements for the real-time, intelligence, and security of data operation. In data-intensive industries such as power and finance, it is urgent to establish an integrated operation system covering the entire life cycle of data to optimize the entire process from data access, governance to service, and application.

[0007] In summary, existing data management technologies have the problems of serious data silos, low data service efficiency, low intelligence, and poor security. SUMMARY

[0008] To solve the technical problems of existing data management technologies such as serious data silos, low data service efficiency, low intelligence, and poor security, the present application provides solutions in the following aspects.

[0009] In a first aspect, the present application provides an intelligent integrated operation system based on data fusion, comprising:

[0010] A multi-source data integration module for standardizing and integrating data in the source layer, shared layer, and analysis layer through a metadata weaving method to generate structured data with a blood relationship graph.

[0011] The intelligent service module is used to generate a data model by performing intelligent recommendation based on the structured data and ER relationship, perform zero-code modeling based on the data model, and generate a RESTful API interface based on the modeling results.

[0012] The dynamic operation module is used to construct a panoramic data view by aggregating the structured data and data model, and to connect the RESTful API interface to the work order management system to realize online work order flow;

[0013] The data synchronization module is used to synchronize the metadata of the multi-source data integration module through a push-pull combination method to ensure the real-time consistency between the ER relationship intelligent recommendation and the data source.

[0014] The security management module is used to implement flow control during the structured data transmission process, perform blacklist filtering on the RESTful API interface, and perform field-level desensitization at the data output end.

[0015] Its beneficial effects are as follows: The intelligent integrated operation system based on data fusion in this embodiment can realize the intelligent management closed loop of the entire life cycle of data elements, forming a complete technology chain from data access, processing to service and operation; In addition, the intelligent integrated operation system based on data fusion in this embodiment enhances capabilities in three aspects: data service, data operation and data security control, thereby realizing "zero approval" for non-negative list data and "online" application for negative list data, and can provide more convenient and intelligent services.

[0016] Preferably, the multi-source data integration module includes:

[0017] Metadata extraction unit, used to extract raw metadata from heterogeneous data sources;

[0018] A data standardization unit is used to convert the raw metadata into a unified description format, thereby achieving the standardization of metadata;

[0019] The bloodline relationship construction unit is used to build a bloodline relationship graph between data tables based on standardized metadata. The bloodline relationship graph is input into the intelligent service module as a component of the structured data.

[0020] Preferably, the intelligent service module includes:

[0021] The intelligent recommendation unit is used for data retrieval capabilities. It allows users to add keywords based on global search settings and provides quick access to popular search models.

[0022] A visual modeling unit is configured to realize zero-code construction of a data model through visual drag-and-drop according to an ER relationship recommendation scheme;

[0023] An API generation unit is configured to automatically convert the data model into a callable RESTful API interface, and to generate an API service customized according to a function of a self-service.

[0024] Preferably, the dynamic operation module comprises:

[0025] A data panoramic view unit is configured to fuse the blood relationship graph and a data model topology structure, and to assist in building a data operation view capability.

[0026] A work order management unit is configured to process problems through online feedback of an open platform upgrade capability, and to grasp a feedback progress with a user in real time.

[0027] A model import and maintenance unit is configured to provide a data model import entry and a model import template, and to realize online maintenance of an incremental data model.

[0028] A monitoring and early warning unit is configured to realize monitoring and early warning of a data API service in terms of calling, performance and activity, and to ensure running stability of the data service.

[0029] Preferably, the data synchronization module comprises:

[0030] An active push unit is configured to analyze and extract a storage model that has been fixed in a data platform, including information of a source layer and information of a shared layer, and to maintain and manage model metadata.

[0031] An active pull unit is configured to generate a model result, to perform consistency comparison with model information of the source layer and the shared layer of an intelligent integrated operation system, to refine a difference part for early warning.

[0032] A consistency checking unit is configured to, when a field changes, dynamically perceive and timely acquire change information from the source layer to the shared layer and to the analysis layer, to help a user actively respond, and to actively adjust model data acquisition logic.

[0033] Preferably, the security control module comprises:

[0034] An application flow limitation unit is configured to improve data service security, and to strengthen application access flow limitation.

[0035] A blacklist management unit is configured to add a blacklist for an intelligent integrated operation system based on data fusion.

[0036] A data watermark unit is configured to add a data watermark for a data preview page, and to display login user information.

[0037] The data desensitization unit is configured to formulate a data desensitization configuration capability for negative list data to realize self-defined desensitization content.

[0038] Preferably, the metadata weaving method comprises:

[0039] An adaptive parser is used to identify structural features of different data sources to generate original metadata with semantic annotations.

[0040] The original metadata is standardized and converted based on a unified data model to establish a field-level mapping relationship.

[0041] A blood relationship graph is constructed by a graph database to record the complete conversion link of data from the source layer to the analysis layer.

[0042] In a second aspect, the present application provides a data access method for accessing multi-source heterogeneous data into the intelligent integrated operation system based on data fusion, comprising:

[0043] Based on the data resources of the data center, high-share, high-usage, and high-cross-professional-demand core business data are screened to generate a core data source table list.

[0044] The data resource objects in the core data source table list are subjected to attribute abstraction and traceability analysis to determine the authoritative data sources in cooperation with the business department to generate an authoritative data source list.

[0045] A hierarchical business domain model directory is constructed, including a first-level business domain directory, a second-level professional directory, and a third-level model classification directory, to realize the shelving management of basic models and analysis models.

[0046] A field management mechanism is used to uniformly manage, dynamically update, and share the application of the authoritative data sources.

[0047] The data access method can practically improve the data sharing application efficiency and comprehensively improve the user data sharing application experience and efficiency by dividing the existing models into different professions, combining the authoritative data source identification, and integrating the unified basic models and analysis models of the company.

[0048] Preferably, generating the core data source table list comprises:

[0049] Statistical data usage application records and service call logs are screened to select high-frequency access data.

[0050] After summarizing and deduplicating, the source table information is output, including the source system, table name, and management department to clarify the data integration range.

[0051] Preferably, generating the authoritative data source list comprises:

[0052] Attribute classification and business meaning comparison are performed on the data resource object;

[0053] Distribution of the data in the business system is identified through data resource directory tracing;

[0054] A standardization list is formed by confirming the unique authority source in cooperation with the business department.

[0055] In summary, the beneficial effects of the present application are that the intelligent integrated operation system based on data fusion of the embodiment can realize non-negative list data "zero approval" and negative list data application "online", and can provide more convenient and intelligent services for users. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is a structural schematic diagram of an intelligent integrated operation system based on data fusion according to an embodiment of the present application;

[0057] Figure 2 is a general architecture diagram of the structure of an intelligent integrated operation system based on data fusion according to an embodiment of the present application;

[0058] Figure 3 is a security architecture diagram of the structure of an intelligent integrated operation system based on data fusion according to an embodiment of the present application;

[0059] Figure 4 is a structural schematic diagram of a multi-source data integration module according to an embodiment of the present application;

[0060] Figure 5 is a data access method flowchart according to an embodiment of the present application. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0062] The specific embodiments of the present application will be described in detail below with reference to the drawings.

[0063] Embodiment of an intelligent integrated operation system based on data fusion:

[0064] As shown in Figure 1 , the intelligent integrated operation system based on data fusion of the present application comprises:

[0065] A multi-source data integration module is configured to standardize and integrate data of a source layer, a shared layer and an analysis layer by a metadata weaving method to generate structured data with a blood relationship graph;

[0066] An intelligent service module is configured to generate a data model by performing ER relationship intelligent recommendation based on the structured data, to perform zero-code modeling based on the data model, and to generate a RESTful API interface based on a modeling result;

[0067] A dynamic operation module is configured to construct a panoramic data view by aggregating the structured data and the data model, and to realize online work order transfer by connecting the RESTful API interface to a work order management system.

[0068] A data synchronization module is configured to perform synchronization operation on metadata of the multi-source data integration module by a push-pull combination to ensure real-time consistency of the ER relationship intelligent recommendation and a data source.

[0069] A security control module is configured to perform flow control in a structured data transmission process, to perform blacklist filtering on the RESTful API interface, and to perform field-level desensitization at a data output end.

[0070] The following introduces a technical route, an application architecture, a data architecture and a security architecture of the intelligent integration and operation system based on data fusion of the embodiment.

[0071] The intelligent integration and operation system based on data fusion of the embodiment adopts a micro-service framework, and decomposes each functional module into each discrete service to realize high availability of a data asset integration and operation platform, including common services such as general storage services, application authorization services and log monitoring services, and application services such as data assembly services, data graph services and data API services. A front end adopts a VUE technical framework, a back end micro-service development is based on a Java language SpringCloud framework, micro-services are uniformly accessed to EDAS components, and key technologies involved include cloud platforms, data platforms, micro-services and the like.

[0072] The overall architecture of the intelligent integration and operation system based on data fusion of the embodiment is shown in Figure 2 .

[0073] The research and application of the intelligent integration and operation system based on data fusion of the embodiment include user data searching, application processes, data modeling, service self-construction, E-R relationship recommendation, model authorization and the like.

[0074] Data of the intelligent integration and operation system based on data fusion and data resources to be shared with other systems are stored on an RDS.

[0075] The data warehouse uses MaxCompute to store various types of data disclosed by various business systems. The source layer stores raw business data, and the shared layer stores relevant data synchronized with the data asset integration and operation platform development needs.

[0076] The sample data and formal data of the standard model are stored in the data warehouse. The standard model business data within the approved application scope is synchronized to the RDS of the intelligent integration and operation system based on data fusion. The sample data and application data of the analysis model are stored in the RDS of the intelligent integration and operation system based on data fusion. The data asset integration and operation platform has two RDS databases. One is used to store the model and application information maintained by itself, including the sample data of the standard model and the analysis model, which occupies a small amount of storage space. The other is used to store the application data of the standard model and the analysis model, which occupies a large amount of storage space.

[0077] The main analysis model data flow steps are as follows:

[0078] The main analysis model data flow steps are as follows:

[0079] In this embodiment, the blood relationship graph technology is a technology that organizes and represents entities and their relationships in the real world in a graphical way. This technology plays an important role in the field of artificial intelligence and data science, mainly used for information organization and intelligent processing. The core elements of the knowledge graph include entity (Entity), relationship (Relationship) and attribute (Attribute). Entity represents objects in the real world, such as people, places, organizations, etc.; relationship is the interaction and connection between entities.

[0080] Zero-code modeling technology breaks through the threshold of traditional modeling technology, innovatively designs visual drag-and-drop modeling interface, integrates 200+ industry general data conversion operators, and enables non-technical personnel to independently complete more than 85% of data modeling work, with modeling efficiency improved by 60%.

[0081] The intelligent integration and operation system based on data fusion of this embodiment has been successfully applied to multiple key business applications. The above modules cooperate with each other to build a complete data asset intelligent operation system. In terms of data recognition accuracy, service response speed, security protection level, etc., significant breakthroughs have been made, and the overall operation efficiency has been improved by more than 50% compared with traditional solutions.

[0082] The security architecture complies with the national information system security level protection requirements and the secondary system requirements. In view of the security risks faced, the design focuses on application security and data security, network security, terminal security and the like, to ensure the safe, reliable and stable operation of the system. The security architecture diagram is shown in Figure 3 .

[0083] RESTful API (Representational State Transfer) is a web-based distributed architecture style widely used in designing web applications, especially when transmitting data between clients and servers. RESTful APIs follow some design conventions that make them concise, scalable, and maintainable.

[0084] The intelligent integrated operation system based on data fusion can realize intelligent management of the whole life cycle of data elements, from data access, processing to service and operation, forming a complete technical chain. In addition, the intelligent integrated operation system based on data fusion improves the capabilities from data service, data operation and data security control, thereby realizing "zero approval" of non-negative list data and "online" application of negative list data, and providing more convenient and intelligent services.

[0085] The intelligent integrated operation system based on data fusion constructs a complete data asset intelligent operation system, improves the data full-link monitoring capability, improves the user's experience of using data, simplifies the data use process, enhances the data service capability, and improves the intelligent management capability of data service, which has outstanding technical advancement and practical value.

[0086] In one embodiment, as shown in Figure 4 , the multi-source data integration module includes:

[0087] A metadata extraction unit is configured to extract original metadata from the heterogeneous data sources.

[0088] A data standardization unit is configured to convert the original metadata into a unified description format, thereby realizing the standardization of the metadata.

[0089] A blood relationship construction unit is configured to establish a blood relationship graph between data tables based on the standardized metadata, and the blood relationship graph is input to the intelligent service module as a component of the structured data.

[0090] In one embodiment, the intelligent service module includes:

[0091] An intelligent recommendation unit is configured to provide a shortcut entry of a popular search model based on the global search and the addition of a keyword.

[0092] A visual modeling unit is configured to realize zero-code construction of a data model by visual drag-and-drop according to an ER relationship recommendation scheme;

[0093] An API generation unit is configured to automatically convert the data model into a callable RESTful API interface, and open a function customization generated API service.

[0094] The main functions of the intelligent service module of the embodiment are as follows:

[0095] User data search: optimize data retrieval capability, add keyword search on the basis of global search, and improve data retrieval efficiency; standard public model and internal data resource directory data resource query function, distinguish offline data and real-time data, CIM model and dimension model, facilitate users to quickly find data; model popular search function, provide a shortcut entry for popular search models; external data resource directory query function, provide external data resource query, data relationship graph, data link panorama and other functions.

[0096] Application process: optimize the model application process, functions involve adding shopping cart, applied data, my application list, my approval, improve the efficiency of the application process, enhance the optimization of user experience, functions include "zero" approval of non-negative list, online negative list approval, re-initiation and application draft box after process return, and other functions;

[0097] Data modeling: optimize data modeling experience, create "normal mode" and "advanced mode"; reduce technical threshold, realize data model "zero code" construction by visual drag-and-drop; build assembly task scheduling strategy function; optimize data assembly monitoring capability, realize data assembly whole process monitoring, strengthen data assembly process log output and readability; build model modification function, modify the description of data table and field, and perform approval process and record archiving after modification.

[0098] Service self-construction: create service self-construction capability, open service self-generation function to users to customize "API service" for ads analysis models constructed by other means.

[0099] E-R relationship recommendation: create model recommendation capability, intelligently recommend data modeling correlation according to the selected standard public model by the user.

[0100] Model authorization: create model authorization automatic function, realize data table access authorization of data center and operator algorithm tool data table and field level authorization.

[0101] In one embodiment, the dynamic operation module comprises:

[0102] a data panorama view unit configured to fuse the blood relationship graph and a data model topology structure, and assist in building a data operation view capability;

[0103] a work order management unit configured to handle problems online through an open platform upgrade capability, and grasp the feedback progress with users in real time;

[0104] a model import and maintenance unit configured to provide a data model import portal and a model import template, and realize online maintenance of incremental data models.

[0105] a monitoring and early warning unit configured to realize monitoring and early warning of data API service calls, performance, and activity, and ensure the running stability of data services.

[0106] In this embodiment, the open platform upgrade capability refers to improving or expanding the functions and interfaces of the original open platform, so that the open platform supports more business scenarios and more efficient interaction modes, especially the capabilities of online feedback, problem handling, and state synchronization with external systems or users.

[0107] The main functions of the dynamic operation module in this embodiment are as follows:

[0108] Data operation view: The data operation view capability is built, and data such as operation monthly reports, headquarter monthly reports, and daily weekly reports are counted to assist operation managers in judging the data supermarket resource usage and further improving the data operation management level.

[0109] Data service work order online: The online of provincial and local data service work orders, through the upgrade of the open platform capability, enables the data service team of the city to handle problems online, realizes online evaluation of work orders, and other related function process construction. The supermarket service record function is added on the operation side, and the feedback progress with users is grasped in real time.

[0110] Data model import and maintenance: A data model import portal and a model import template are provided, including application system, data source table, source table, standard model, directory relationship, UC relationship, ER relationship, and other information, to realize online maintenance of incremental data models of the data supermarket.

[0111] Data service monitoring: The data service monitoring capability is added, which realizes monitoring of data API service calls, performance, activity, and the like, to ensure the running stability of data services.

[0112] The dynamic operation module of this embodiment can realize operation improvement of the data middle platform through the setting of the data panorama view unit, the work order management unit, the model import and maintenance unit, and the monitoring and early warning unit.

[0113] In one embodiment, the data synchronization module comprises:

[0114] The active pushing unit maintains and manages the model metadata by analyzing and extracting the storage model that has been solidified in the data platform, including the source layer information and the shared layer information.

[0115] The active pulling unit compares the consistency of the model result and the source layer and shared layer model information of the intelligent integrated operation system, extracts and compares the difference part, and performs early warning.

[0116] The consistency checking unit is used for dynamically sensing and timely obtaining the change information from the source layer to the shared layer and then to the analysis layer when the field changes, which helps users to actively respond and actively adjust the model data extraction logic.

[0117] The main functions of the data synchronization module in the embodiment are as follows:

[0118] Obtaining the storage model: The model metadata is maintained and managed by analyzing and extracting the storage model that has been solidified in the data platform, including the source layer, the shared layer and other related information. The obtaining process supports functions such as automatic starting at a fixed time, manual starting, and self-defined rules.

[0119] Model update detection: The consistency of the model result and the source layer and shared layer model information of the supermarket itself is compared, the difference part is extracted and compared, and early warning is performed. Users can change the marked part, or automatically mark it through system settings. The comparison rule can be self-defined.

[0120] Field consistency: When the field changes, the user can dynamically sense and timely obtain the change information from the source layer to the shared layer and then to the analysis layer, which helps users to actively respond and actively adjust the model data extraction logic in the rapid iteration of data.

[0121] Change range detection: The system detects and analyzes the influence range of the model change part, including the fusion analysis model, the service API, the authorized consumption application, and forms a change influence range link diagram.

[0122] Model relationship maintenance and import: The model management function is added or optimized, including model relationship maintenance, data import, etc.

[0123] The data synchronization module of the embodiment can improve the synchronization capability of the data supermarket and the data platform. On the basis of the one-way "push" metadata of the supermarket in the platform, the process of the supermarket actively "pulling" is added, realizing "push-pull" double synchronization, actively sensing the change of the data model in the platform, and improving the data management capability and the data model quality.

[0124] In one embodiment, the security control module comprises:

[0125] An application flow limiting unit is applied to improve data service security and strengthen application access flow limiting;

[0126] A blacklist management unit is applied to add a blacklist to the intelligent integrated operation system based on data fusion;

[0127] A data watermark unit is applied to add data watermark to a data preview page and display login user information;

[0128] A data desensitization unit is applied to formulate data desensitization configuration capability for negative list data to realize custom desensitization content.

[0129] The main functions of the security control module of the embodiment are as follows:

[0130] Application flow limiting: improve data service security, strengthen application access flow limiting, and prevent malicious concurrent data service requests.

[0131] Blacklist management: by adding a blacklist, the application IP of malicious attacks can be effectively avoided to ensure the safe operation of the data supermarket data service API.

[0132] Data watermark management: add data watermark to the data preview page to display login user information, ensure data security query use, and ensure data leakage traceability.

[0133] Data desensitization management: add data desensitization configuration capability for negative list data, and users can customize desensitization content to ensure the safety of negative list data involved in data preview and data service.

[0134] The security control module of the embodiment can prevent malicious concurrent data service requests by setting an application flow limiting unit, effectively prevent malicious application IP attacks by setting a blacklist management unit, ensure data security query use and ensure data leakage traceability by setting a data watermark unit, and ensure the safety of negative list data involved in data operation and data service modules by setting a data desensitization unit. The security control module of the embodiment can improve the security control capability of the data middle platform.

[0135] In one embodiment, the metadata weaving method comprises:

[0136] S1, an adaptive parser is used to identify the structural characteristics of different data sources to generate original metadata with semantic annotations;

[0137] S2, the original metadata is standardized converted based on a unified data model to establish a field-level mapping relationship;

[0138] S3, build a blood relationship graph through a graph database, and record the complete conversion link of data from the origin layer to the analysis layer.

[0139] The data access method embodiment includes the following steps:

[0140] As shown in the figure, the data access method of the application is used to access multi-source heterogeneous data into the intelligent integrated operation system based on data fusion described in the above embodiment, which includes the following steps: Figure 5

[0141] S101, generate a core data source table list, specifically: based on the data resources of the data center, filter high sharing, high use, and high cross-professional demand core business data, and then generate a core data source table list;

[0142] S102, generate an authoritative data source list, specifically: attribute abstraction and traceability analysis are performed on the data resource objects in the core data source table list, and the authoritative data source is determined in cooperation with the business department to generate an authoritative data source list;

[0143] S103, build a hierarchical business domain model directory, specifically: build a hierarchical business domain model directory, including a first-level business domain directory, a second-level professional directory, and a third-level model classification directory, to realize the management of basic models and analysis models;

[0144] S104, use a field management mechanism to uniformly manage, dynamically update, and share the application of the authoritative data source.

[0145] After the original data resources are processed by the method of the embodiment, they can be input into the intelligent integrated operation system based on data fusion.

[0146] The data access method of the embodiment divides the existing model by profession, combines the authoritative data source identification, and integrates to form the company's unified basic model and analysis model, that is, one source of data, which effectively improves the data sharing application efficiency and improves the user data sharing application experience and efficiency.

[0147] In one embodiment, generating a core data source table list includes:

[0148] Statistical data use application records and service call logs to filter high-frequency access data;

[0149] After summarizing and deduplicating, output the source table information, including the source system, table name, and management department, to clearly define the data integration range.

[0150] ​The specific process is: based on the data resource status of the data platform, according to the data use application record and the data service call log, focusing on four types of data with high sharing degree, high use heat, high cross-professional demand and core business data, the source tables of the four types are summarized, de-duplicated, and the information of source system, source table English name, source table Chinese name, management department and the like is output to form a "core data source table list" to clearly define the source and responsibility of the data source table range.

[0151] In one embodiment, generating the authoritative data source list includes:

[0152] Attribute classification and business meaning comparison of data resource objects are performed.

[0153] The distribution of data in business systems is identified through data resource directory tracing.

[0154] The unique authoritative source is confirmed in cooperation with business departments to form a standardized list.

[0155] The specific process is: based on the "core data source table list", the attribute classification of data resource objects is abstracted and refined, the data resource object attributes are filled according to the attribute classification, and the data resource object list is formed. Through the data resource directory, the distribution of the integrated data resource objects in the business system is sorted out, and through the attribute name, description and business meaning, the source system, source table and field list of the data resource object attributes are formed, and the data authoritative source is determined in cooperation with the relevant business departments to form the "authoritative data source list".

[0156] In one embodiment, the hierarchical business domain model directory includes: a first-level directory, a second-level directory and a third-level directory, wherein the first-level directory is divided according to business domains, including customer domain, device domain, personnel domain, financial domain, material domain, project domain, comprehensive domain, safety domain, power grid domain and market domain; the second-level directory is divided according to professions, and the third-level directory distinguishes between basic models and analysis models. Thus, the data model is connected to the data supermarket for shelving and development.

[0157] In one embodiment, the field management mechanism includes:

[0158] The authoritative fields are online published, version updated and uniformly called.

[0159] When the data of the business system changes, it is synchronized to the data platform to ensure data consistency.

[0160] The specific process of field management is: adhere to the principle of "one source for multiple uses, source maintenance", and clearly data integration, sharing and data analysis applications should be based on authoritative data source, for the adjustment of the new data of the authoritative data source, according to the latest data architecture design specification, by information system source unified, synchronous update to data middle platform, realize the unified management of authoritative data source, online release, online update, online application, support the achievement of authoritative data source landing practical.

[0161] Relying on the model directory connection function of the data middle platform, 10 business domains of customer domain, equipment domain, personnel domain, financial domain, material domain, project domain, comprehensive domain, safety domain, power grid domain and market domain are constructed as the first-level directory; the corresponding professional of the model is combed as the second-level directory; the basic model and the analysis model are the third-level directory, and the connection of the data model on the data middle platform is realized.

[0162] In one embodiment, the data access method further comprises:

[0163] Based on the data service call log, the model directory and the authoritative data source list are dynamically optimized, and the data integration efficiency is improved.

[0164] In the description of the present specification, the meaning of "multiple", "several" is at least two, for example, two, three or more, etc., unless otherwise explicitly specified.

[0165] Although the present specification has shown and described several embodiments of the present application, it will be apparent to those skilled in the art that such embodiments are provided only in an exemplary manner. Those skilled in the art will think of many changes, changes and alternatives without departing from the idea and spirit of the present application. It should be understood that various alternatives to the embodiments of the present application described herein can be employed in practicing the present application.

Claims

1. An intelligent converged operations system based on data fusion, characterized by, The application relates to a multi-source data integration system, which comprises the following modules: a multi-source data integration module for standardizing and integrating data in a source layer, a shared layer and an analysis layer through a metadata weaving method to generate structured data with a blood relationship graph; an intelligent service module for generating a data model through ER relationship intelligent recommendation based on the structured data, performing zero-code modeling based on the data model, and generating a RESTful API interface based on modeling results; a dynamic operation module for constructing a panoramic data view by aggregating the structured data and the data model, and realizing online work order transfer by connecting the RESTful API interface to a work order management system; a data synchronization module for performing synchronization operation on metadata of the multi-source data integration module through a push-pull combination mode to ensure real-time consistency of the ER relationship intelligent recommendation and a data source; a security control module for implementing flow control in the structured data transmission process, performing blacklist filtering on the RESTful API interface, and performing field-level desensitization at a data output end; the dynamic operation module comprises: a data panoramic view unit for fusing the blood relationship graph and a data model topology structure to assist in building a data operation view capability; a work order management unit for online feedback processing of problems through open platform upgrade capability, and real-time mastering of feedback progress with users; a model import and maintenance unit for providing a data model import portal and a model import template to realize online maintenance of incremental data models; a monitoring and early warning unit for realizing monitoring and early warning of data API service calling, performance and activity, and guaranteeing running stability of data services; the data synchronization module comprises: an active push unit for maintaining and managing model metadata by analyzing and extracting a storage model fixed in a data platform, including source layer information and shared layer information; an active pull unit for comparing consistency of model results and source layer and shared layer model information of an intelligent integrated operation system, refining and comparing difference parts to perform early warning; a consistency checking unit for acquiring change information from a source layer to a shared layer and then to an analysis layer in a timely manner through dynamic sensing when a field changes, which helps users to actively respond and adjust model data taking logic.

2. The intelligent converged operations system based on data fusion as claimed in claim 1, wherein, the multi-source data integration module comprises: a metadata extraction unit for extracting original metadata from heterogeneous data sources; a data standardization unit for converting the original metadata into a unified description format to realize standardization of the metadata; a blood relationship construction unit for establishing a blood relationship graph between data tables based on the standardized metadata, wherein the blood relationship graph is input into the intelligent service module as a component of the structured data.

3. The intelligent converged operations system based on data fusion as claimed in claim 1, wherein, the intelligent service module comprises: an intelligent recommendation unit for data retrieval capability, setting a global retrieval basis to add a keyword, and providing a shortcut entrance of a hot search model; a visual modeling unit for realizing zero-code construction of a data model through visual drag-and-drop based on an ER relationship recommendation scheme. An API generation unit is configured to automatically convert the data model into a callable RESTful API interface, and to customize the generation of API services based on the self-service opening function.

4. The intelligent converged operations system based on data fusion of claim 1, wherein, The security management module comprises: An application traffic limiting unit is configured to improve data service security and to strengthen application access traffic limiting. A blacklist management unit is configured to add a blacklist to the intelligent integrated operation system based on data fusion. A data watermark unit is configured to add data watermark to a data preview page and to display login user information. A data desensitization unit is configured to formulate data desensitization configuration capability for negative list data to realize custom desensitization content.

5. The intelligent converged operation system based on data fusion according to any one of claims 1 to 4, characterized in that, The metadata weaving method comprises: An adaptive parser is used to identify structural features of different data sources to generate original metadata with semantic annotations. Based on a unified data model, the original metadata is standardized converted to establish a field-level mapping relationship. A graph database is used to construct a cross-system blood relationship graph to record the complete conversion link of data from the source layer to the analysis layer.

6. A data access method, characterized by, The data access method is used to access the multi-source heterogeneous data into the intelligent integrated operation system based on data fusion of any one of claims 1-5, comprising: Based on the data resources of the data middle platform, high sharing degree, high use frequency, and high cross-professional demand core business data are screened to generate a core data source table list. Attribute abstraction and traceability analysis are performed on the data resource objects in the core data source table list to determine the authoritative data source in cooperation with the business department to generate an authoritative data source list. A hierarchical business domain model directory is constructed, including a first-level business domain directory, a second-level professional directory, and a third-level model classification directory, to realize the management of basic models and analysis models. A field management mechanism is used to uniformly manage, dynamically update, and share the application of the authoritative data source.

7. The data access method of claim 6, wherein, The generation of the core data source table list comprises: Statistical data use application records and service call logs are screened to select high-frequency access data. After aggregation and deduplication, the source table information is output, including the source system, table name, and management department to clarify the data integration range.

8. The data access method of claim 6 or 7, wherein, The generation of the authoritative data source list comprises: Attribute classification and business meaning comparison are performed on the data resource objects. Data resource directory traceability is used to identify the distribution of data in the business system. In cooperation with the business department, the unique authoritative source is confirmed to form a standardized list.

Images