Informationized multi-service data integration method, system, equipment and medium

Abstract

The invention relates to an informatization multi-service data integration method, system and device and a medium, and belongs to the technical field of service data processing. The integration method comprises the steps that a target data source is collected and initialized; structured data and semi-structured data are regularly and automatically extracted from the target data source to a source-attached data source through an ETL (Extract-Transform-Load) engine; performing digital semantic analysis on the unstructured data in the target data source, and performing partial structured processing according to a digital semantic analysis result to obtain partial structured data; taking structured data, semi-structured data and partial structured data in the source data source as a to-be-integrated data source; and performing data integration on the data in the to-be-integrated data source, synchronizing the integrated data to the short-term data center in real time, performing data aggregation, obtaining the aggregated data, and storing the aggregated data to the comprehensive data center. According to the method, data integration is effectively realized in an enterprise multi-source heterogeneous data environment, and the data management and decision-making requirements of an enterprise can be better supported.

Description

Technical Field

[0001] The present application relates to the field of business data processing technology, and in particular to an information-based multi-business data integration method, system, device and medium. Background Art

[0002] With the rapid development of information technology, a large amount of business data has been accumulated within enterprises. These data are not only huge in quantity, but also diverse in types, including structured data (such as database tables), semi-structured data (such as XML, JSON files) and unstructured data (such as documents, images, audio, video). In order to improve the operational efficiency and decision-making quality of enterprises, it is crucial to effectively integrate and manage these data. However, common data integration methods have many problems in practical applications.

[0003] First, the data of many enterprises are distributed in different business systems and storage media, forming multiple data islands. The lack of effective interconnection between these data islands makes it difficult to fully utilize the data, affecting the overall data analysis capabilities and decision-making efficiency of the enterprise. Although the traditional point-to-point integration method can achieve data exchange, since each system needs to be connected to other systems separately, the integration process is very complicated and requires a lot of development and maintenance work. In addition, every time the system is upgraded or changed, all related interfaces may need to be readjusted, further increasing the cost and risk of integration.

[0004] Secondly, in a multi-source heterogeneous data environment, the quality and consistency of data are often difficult to guarantee. Data in different systems may have problems such as inconsistent formats, redundancy, and errors. If these problems are not effectively handled, they will directly affect the subsequent data analysis results. At the same time, with the increase in data volume and the improvement in data value, data security and privacy protection become particularly important. However, existing data integration solutions often lack a sound data security management mechanism and cannot effectively prevent data leakage and abuse, which brings potential risks to enterprises.

[0005] Furthermore, many existing data integration solutions are difficult to support real-time data processing and dynamic decision-making needs. Especially in the big data environment, traditional batch processing methods can no longer meet rapidly changing business needs. Enterprises need more flexible and efficient data integration solutions. Traditional data integration methods usually tightly bind data processing logic with business systems, resulting in a high degree of coupling between systems. Once a system is changed, it may have a chain reaction on other related systems, increasing the vulnerability of the system and the difficulty of maintenance.

[0006] Therefore, how to effectively realize data integration in the enterprise's multi-source heterogeneous data environment to better support the enterprise's data management and decision-making needs is an urgent problem that needs to be solved. Summary of the invention

[0007] In order to achieve effective integration and management of multi-source heterogeneous data, the present application provides an information-based multi-service data integration method, system, device and medium.

[0008] In a first aspect, the present application provides an information-based multi-service data integration method, which adopts the following technical solution: An information-based multi-service data integration method, the integration method comprising: Collect and initialize target data sources; the target data sources include structured data sources, semi-structured data sources and unstructured data sources; Automatically extracting structured data and semi-structured data from the target data source to the source data source through the ETL engine; Performing digital semantic analysis on the unstructured data in the target data source, and performing partial structuring processing according to the digital semantic analysis results to obtain partial structured data; Using the structured data and semi-structured data in the source data source and the partially structured data as data sources to be integrated; Integrate the data in the data source to be integrated to obtain integrated data; Building a short-term data center, and synchronizing the integrated data to the short-term data center in real time; Aggregating the integrated data of the short-term data center to obtain aggregated data; The aggregated data is stored in an integrated data center.

[0009] By adopting the above technical solutions, a multi-dimensional integration platform for structured, semi-structured and unstructured data sources is built, achieving comprehensive, real-time and accurate management of data. Under this integrated framework, the interconnection and sharing of various types of data can be seamlessly implemented; at the same time, the dynamic update, centralized management and secure storage of data provide solid technical support for the business decision-making of enterprises, improve data quality and reduce operation and maintenance costs.

[0010] Optionally, the step of performing data integration on the data in the to-be-integrated data source to obtain integrated data includes: Generate a corresponding data integration task according to the data source to be integrated; Build a data integration model and load integration processing rules; wherein the integration processing rules include data cleaning rules, data review rules and data integration processing logic; According to the data integration task, data is collected from the data source to be integrated and input into the data integration model to obtain integrated data.

[0011] By adopting the above technical solutions, effective integration of multi-source heterogeneous data is achieved, which not only simplifies the data integration process and improves work efficiency, but also enhances the consistency and accuracy of the data, ensuring the efficiency and reliability of the integration process, thereby providing strong data support for enterprises and promoting the improvement of decision-making quality and operational efficiency.

[0012] Optionally, the step of aggregating the integrated data of the short-term data center to obtain aggregated data includes: Using the integrated data of the short-term data center as the data source to be aggregated; Generate a corresponding data aggregation task according to the data source to be aggregated; Build a data aggregation model and load data aggregation logic; wherein the data aggregation logic includes data governance logic, data lineage processing logic, data watermark processing logic, data fingerprint processing logic and data annotation processing logic; According to the data aggregation task, data is collected from the data source to be aggregated and input into the data aggregation model to obtain aggregated data.

[0013] By adopting the above technical solutions, not only can the centralized management and optimization of data be achieved, but also the traceability and security of data can be improved through a series of data processing logics (such as data governance, data lineage, data watermark, data fingerprint and data labeling). This approach significantly enhances the availability and value of data, provides enterprises with more reliable data support, promotes the depth and breadth of data analysis, and ultimately helps enterprises make better decisions and improve operational efficiency and competitiveness.

[0014] Optionally, after building a short-term data center and synchronizing the integrated data to the short-term data center in real time, the method further includes: Receive business demand information sent by the application layer; Acquire hot data corresponding to the business demand information from the target data source according to the business demand information; Building a thermal data center and loading the thermal data; The hot data of the hot data center is stored in the short-term data center.

[0015] By adopting the above technical solution, we can accurately obtain the hot data in the target data source according to the business demand information of the application layer, build and load the hot data center and then store it in the short-term data center, so as to achieve agile response to business needs and efficient data processing. This technical solution not only improves the flexibility and efficiency of data management, but also enhances the real-time and practicality of data, which helps the business system to maintain efficient operation in a dynamic environment.

[0016] Optionally, the integrated data center includes a static data center, a dynamic data center and an unstructured data center. The static data center is used to store structured business data that does not need to be changed later, the dynamic data center is used to store structured business data that needs to be changed in real time later, and the unstructured data center is used to store unstructured business data.

[0017] In a second aspect, the present application provides an information-based multi-service data integration system, which adopts the following technical solutions: An information-based multi-service data integration system, the integration system comprising: A data collection module, used to collect and initialize a target data source; the target data source includes a structured data source, a semi-structured data source and an unstructured data source; A data extraction module, used to automatically extract structured data and semi-structured data from the target data source to a source data source through an ETL engine at a fixed time; A structured processing module, used for performing digital semantic analysis on the unstructured data in the target data source, and performing partial structured processing according to the digital semantic analysis result to obtain partial structured data; A module for determining data to be integrated, used for taking the structured data and semi-structured data in the source data source and the part of the structured data as the data source to be integrated; A data integration module, used to integrate the data in the data source to be integrated to obtain integrated data; A short-term data synchronization module is used to build a short-term data center and synchronize the integrated data to the short-term data center in real time; A data aggregation module, used for aggregating the integrated data of the short-term data center to obtain aggregated data; The data storage module is used to store the aggregated data in the integrated data center.

[0018] Optionally, the data integration module includes: An integration task generating unit, used to generate a corresponding data integration task according to the data source to be integrated; An integration model building unit, which builds a data integration model and loads integration processing rules; wherein the integration processing rules include data cleaning rules, data review rules and data integration processing logic; The integrated data generating unit is used to collect data from the data source to be integrated according to the data integration task and input the data into the data integration model to obtain integrated data.

[0019] Optionally, the data aggregation module includes: a to-be-integrated data determining unit, configured to use the integrated data of the short-term data center as a source of to-be-aggregated data; An aggregation task generating unit, used for generating a corresponding data aggregation task according to the data source to be aggregated; A convergence model construction unit, used to construct a data convergence model and load data convergence logic; wherein the data convergence logic includes data governance logic, data lineage processing logic, data watermark processing logic, data fingerprint processing logic and data annotation processing logic; The aggregated data generating unit is used to collect data from the data sources to be aggregated according to the data aggregation task and input the data into the data aggregation model to obtain aggregated data.

[0020] In a third aspect, the present application provides a computer device, which adopts the following technical solution: A computer device comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method as described in the first aspect.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium stores a computer program that can be loaded by a processor and execute any one of the methods in the first aspect.

[0022] To summarize, the present application includes at least one of the following beneficial technical effects: The data integration method of the present application solves the problems faced by enterprises in the process of informatization construction, such as insufficient overall planning, difficulty in integration, inadequate security control, and difficulty in data concentration, interconnection and sharing. It can not only empower the existing business of the enterprise, but also create good conditions for future innovation and development. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a first flow chart of an information-based multi-business data integration system according to one of the embodiments of the present application.

[0024] Figure 2 It is a second flow chart of an information-based multi-business data integration system according to one of the embodiments of the present application.

[0025] Figure 3 It is a third flow chart of an information-based multi-business data integration system according to one of the embodiments of the present application.

[0026] Figure 4 It is a fourth process diagram of an information-based multi-business data integration system of one of the embodiments of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of this application more clear, the following Figure 1-4 It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0028] The embodiment of the present application discloses an information-based multi-service data integration method.

[0029] Reference Figure 1 , an information-based multi-service data integration method, the integration method includes, Step S101, collecting and initializing target data sources; The target data sources include structured data sources, semi-structured data sources and unstructured data sources; Specifically, identify various data sources in the enterprise system, including structured data (such as database tables), semi-structured data (such as XML, JSON files) and unstructured data (such as documents, images, audio, video). For example, by scanning the enterprise's database system and file storage system, all types of target data sources can be identified, and by fully understanding the distribution of data sources, a basic guarantee can be provided for subsequent data integration.

[0030] Step S102, extracting structured data and semi-structured data from the target data source to the source data source automatically and regularly through the ETL engine; Specifically, ETL (Extract, Transform, Load) tools can be used to automatically and regularly extract and transfer data from identified target data sources, convert it into a form suitable for analysis, and then load it into the source data source; by maintaining data consistency and quality, the effective integration of multi-source data is ensured, and the efficiency and accuracy of data acquisition are improved.

[0031] The source data source refers to an intermediate data storage set up during the data integration process to reduce the impact on the original business system and improve data processing efficiency. By synchronizing the data that requires complex operations to the source data source first, you can avoid executing a large number of queries or data conversion tasks directly in the production environment, thereby reducing the performance impact on the business system.

[0032] Step S103, performing digital semantic analysis on the unstructured data in the target data source, and performing partial structuring processing according to the digital semantic analysis result to obtain partial structured data; Specifically, for unstructured data, such as text files, pictures, audio or video, natural language processing (NLP) or other related technologies are used to understand the content, and then based on the results of the understanding, attempts are made to convert it into a structured format that is easier to manage and analyze.

[0033] For text files, natural language processing (NLP) technology can be used; for images and videos, computer vision technology can be used; and for audio, speech recognition technology can be used. By generating a digital semantic analysis result containing key information extraction, classification, summary, etc., this result is an understanding and analysis of the original unstructured data content.

[0034] It should be noted that partial structured processing based on the results of digital semantic analysis will produce a partially structured data set; this data set retains some of the original characteristics of the original unstructured data, and also includes structured information extracted after semantic understanding. By associating the two, it not only maintains the richness of the original data, but also facilitates subsequent data processing and analysis, and can provide a more comprehensive data perspective to help companies discover potential patterns and trends, thereby making more accurate business decisions.

[0035] Step S104, taking the structured data, semi-structured data and part of the structured data in the source data source as the data source to be integrated; Among them, different types of data are managed in a unified manner to form a complete set of data sets to be integrated.

[0036] Step S105, integrating the data in the data source to be integrated to obtain integrated data; Among them, according to predefined integration rules and processing logic, data redundancy and contradiction are eliminated, and data from different sources are merged to form a unified view to ensure the integrity, consistency and accuracy of the data and support comprehensive data analysis.

[0037] Step S106, building a short-term data center, and synchronizing the integrated data to the short-term data center in real time; Among them, setting up a temporary storage space to store the latest version of integrated data currently in use can not only support real-time data processing needs and improve the real-time availability of data, but also support dynamic decision-making and reduce the pressure on the main database.

[0038] Step S107, aggregating the integrated data of the short-term data center to obtain aggregated data; Among them, further aggregation and processing of integrated data can help generate strategically significant analysis results to support high-level decision-making; Step S108, storing the aggregated data in the integrated data center.

[0039] Specifically, the processed data will eventually be stored in a long-term data warehouse and allocated to corresponding sub-databases according to their nature to facilitate historical analysis and tracking.

[0040] In some embodiments, the integrated data center includes a static data center, a dynamic data center, and an unstructured data center; wherein the static data center is used to store structured business data that does not need to be changed later, the dynamic data center is used to store structured business data that needs to be changed in real time later, and the unstructured data center is used to store all file-type data, such as text, images, audio, video and other unstructured business data. It should be noted that the structured data associated with the unstructured data will be stored in the static data center or the dynamic data center according to specific business needs. In this way, orderly management and efficient access to data are ensured, which can support the long-term data analysis and decision-making needs of enterprises.

[0041] In the above implementation, a multi-dimensional integration platform of structured, semi-structured and unstructured data sources is constructed to achieve comprehensive, real-time and accurate management of data. Under this integrated framework, the interconnection and sharing of various types of data can be seamlessly executed; at the same time, the dynamic update, centralized management and secure storage of data provide solid technical support for the business decision-making of the enterprise, and improve the data quality and reduce the operation and maintenance costs. The data integration method of the present application solves the problems faced by enterprises in the process of informatization construction, such as insufficient overall planning, difficulty in integration, inadequate security control, and difficulty in data concentration, interconnection and sharing. It can not only empower the existing business of the enterprise, but also create good conditions for future innovation and development.

[0042] As some specific implementations of step S103, in the process of unstructured data processing, digital semantic analysis results can be generated through the following four implementations: first, users can manually upload original files, and the platform background will automatically perform digital semantic analysis; or import original files into the platform by physical copying, and the background will also perform semantic analysis; in addition, users can also choose to let the platform perform semantic analysis immediately while uploading original files; finally, if users have performed preliminary semantic analysis on files, they can directly upload these processed files, and the platform will automatically generate the final semantic analysis results based on the existing information. These four methods flexibly adapt to the needs of different scenarios and ensure that unstructured data can be efficiently and accurately converted into usable information.

[0043] Reference Figure 2 As an implementation of step S105, the step of integrating the data in the data source to be integrated to obtain the integrated data includes: Step S201, generating corresponding data integration tasks according to the data source to be integrated; Among them, data integration tasks refer to a series of operations and workflows defined in the data integration process to achieve data processing goals. The main purpose of generating data integration tasks is to collect data from different sources in an orderly manner to ensure that the data can be processed according to the predetermined process. Through systematic and standardized data processing processes, data integration efficiency can be improved and manual intervention and errors can be reduced.

[0044] Step S202, building a data integration model and loading integration processing rules; Among them, the integrated processing rules include data cleaning rules, data review rules and data integration processing logic; In one embodiment of the present application, the data integration model is an algorithm and structural framework for implementing data integration from multiple data sources, which operates on the input data through a series of processing rules to obtain integrated data with consistency and integrity. The integration processing rules include data cleaning, data review, and data processing logic, which are the key to ensuring data quality and consistency.

[0045] Specifically, data cleaning rules include missing value filling, outlier processing, etc.; data audit rules are used to verify the accuracy, validity and consistency of data; data integration processing logic defines how to extract, transform and load data from different data sources.

[0046] For example, suppose two data sources are to be integrated, one is a customer basic information table, and the other is a customer transaction record table; the data cleaning rules may include removing blank fields and duplicate records; the data review rules may include verifying whether the transaction amount is reasonable; the data integration processing logic may define how to associate customer IDs for data integration.

[0047] It can be understood that by building a data integration model and loading the corresponding integration processing rules, data quality can be ensured, data redundancy can be reduced, and the consistency and availability of data between different systems and platforms can be improved.

[0048] Step S203: According to the data integration task, data is collected from the data source to be integrated and input into the data integration model to obtain integrated data.

[0049] Specifically, according to the defined data integration tasks, data is collected from the preset data sources to be integrated, and then input into the data integration model for processing. Through the automated data collection process, the efficiency and accuracy of data integration can be improved, ensuring that all relevant data can enter the data integration model for processing on time and in quantity, thereby generating high-quality integrated data in a timely manner.

[0050] In the above implementation, through the effective integration of multi-source heterogeneous data, not only the data integration process is simplified and work efficiency is improved, but also the consistency and accuracy of the data are enhanced, ensuring the efficiency and reliability of the integration process, thereby providing strong data support for the enterprise and promoting the improvement of decision-making quality and operational efficiency.

[0051] Reference Figure 3 As an implementation method of step S107, the step of aggregating the integrated data of the short-term data center to obtain the aggregated data includes: Step S301, taking the integrated data of the short-term data center as the data source to be aggregated; Among them, after completing the data integration, the integrated data is stored in the short-term data center. These data are regarded as new data sources, namely "target data sources to be aggregated", so as to provide input for the next data aggregation step.

[0052] Step S302, generating a corresponding data aggregation task according to the data source to be aggregated; Among them, based on the understanding and analysis of the integrated data of the short-term data center, a data aggregation task is set to achieve the predetermined data aggregation goal. The data aggregation task includes clarifying which data needs to be aggregated, how to process the data, and when to perform these tasks, so as to ensure that each link is carried out in an orderly manner, avoid omissions or repeated processing, and thus improve the accuracy and effectiveness of data aggregation.

[0053] Step S303, constructing a data aggregation model and loading data aggregation logic; Among them, the data aggregation logic includes data governance logic, data lineage processing logic, data watermark processing logic, data fingerprint processing logic and data annotation processing logic; Specifically, a data aggregation model that meets specific needs can be built, which specifies how to process business data from short-term data centers, including data cleaning rules for removing duplicate records, data lineage logic for marking data sources, data watermarking for adding timestamps and user IDs to each data item, data fingerprint processing for calculating hash values ​​to verify data integrity, and data labeling logic for automatically tagging business data with product categories.

[0054] In some embodiments, the data aggregation logic is described as follows: Data governance logic: defines how to ensure data quality, consistency, and compliance; for example, setting data cleaning rules, data quality inspection standards, etc. Data lineage processing logic: Track the entire process of data from source to final use to ensure that the source and historical changes of data can be traced; Data watermark processing logic: adding invisible identifiers to data to ensure data ownership and integrity; Data fingerprint processing logic: Calculate the unique identifier of the data to detect whether the data has been tampered with or copied; Data annotation processing logic: automatically add labels according to the content of the data, such as classification, clustering, etc., to facilitate subsequent retrieval and analysis.

[0055] It can be understood that by building a detailed data aggregation model and loading various data processing logics, the high quality, traceability, security and ease of use of the data can be ensured, thereby improving the value and credibility of the data.

[0056] Step S304: According to the data aggregation task, data is collected from the data source to be aggregated and input into the data aggregation model to obtain aggregated data.

[0057] Among them, according to the defined data aggregation tasks, the required data is extracted from the target data source to be aggregated in the short-term data center, and processed according to the preset data aggregation model and logic, and finally the aggregated data is generated, thereby ensuring that the conversion process from the original data to the final aggregated data is orderly and efficient, making the aggregated data more suitable for subsequent analysis and application.

[0058] In the above implementation, not only the centralized management and optimization of data is achieved, but also the traceability and security of data are improved through a series of data processing logics (such as data governance, data lineage, data watermark, data fingerprint and data labeling). This method significantly enhances the availability and value of data, provides more reliable data support for enterprises, promotes the depth and breadth of data analysis, and ultimately helps enterprises make better decisions and improve operational efficiency and competitiveness.

[0059] Reference Figure 4 As a further implementation of the data integration method, after building a short-term data center and synchronizing the integrated data to the short-term data center in real time, the method further includes: Step S401, receiving business demand information sent by the application layer; Among them, business demand information refers to the data request made by the application layer according to the needs of runtime. The application layer can be a business system, user interface or other upper-layer application, which sends requests to the data center through the interface. These requests reflect the type and quantity of data required for the current system operation.

[0060] Step S402, acquiring hot data corresponding to the business demand information from the target data source according to the business demand information; The target data source is the database or data warehouse of various data sources in the enterprise system, and hot data refers to data that is highly active and frequently accessed in the current business stage. Through business demand information, these high-value data can be extracted in a targeted manner to improve data processing and response efficiency.

[0061] Step S403, constructing a hot data center and loading hot data; Specifically, the hot data center is a small data storage unit dedicated to storing and managing hot data; after receiving the hot data, it is quickly loaded into the hot data center, which uses high-performance storage devices and cache technology to achieve fast data reading and writing and provide fast query response capabilities.

[0062] Step S404: storing the hot data in the hot data center in the short-term data center.

[0063] In one of the embodiments of the present application, the short-term data center is mainly used to store short-life cycle data that is recently active and strategically important; after the hot data is processed, it is transferred to the short-term data center for subsequent business analysis and operations, providing basic support for subsequent data analysis and decision-making.

[0064] In the above implementation, the hot data in the target data source is accurately obtained according to the business demand information of the application layer, and the process of building and loading the hot data center and then storing it in the short-term data center is realized, so as to achieve agile response to business needs and efficient data processing. This technical solution not only improves the flexibility and efficiency of data management, but also enhances the real-time and practicality of data, which helps the business system to maintain efficient operation in a dynamic environment.

[0065] This application has established unified data integration specifications and designed flexible data integration logic by building a data integration platform, achieving interconnection and decoupling of shared data from the source production system. This decoupling mechanism ensures that changes to the source system or data model will not affect other upstream and downstream business systems. In this way, not only the flexibility and independence of the system are improved, but also the risks brought by changes are significantly reduced, and the maintenance and management of the system are simplified. Ultimately, enterprises can manage and utilize various types of business data more efficiently, support faster decision-making and higher operational efficiency, and thus gain an advantage in the fierce market competition.

[0066] The embodiment of the present application also discloses an information-based multi-service data integration system.

[0067] An information-based multi-service data integration system, the integration system includes: A data collection module is used to collect and initialize target data sources; the target data sources include structured data sources, semi-structured data sources and unstructured data sources; The data extraction module is used to automatically extract structured data and semi-structured data from the target data source to the source data source through the ETL engine at a fixed time; The structured processing module is used to perform digital semantic analysis on the unstructured data in the target data source, and perform partial structured processing according to the digital semantic analysis results to obtain partial structured data; A module for determining data to be integrated, used to take structured data, semi-structured data and part of structured data in the source data source as data sources to be integrated; A data integration module is used to integrate the data in the data source to be integrated to obtain integrated data; Short-term data synchronization module, used to build a short-term data center and synchronize integrated data to the short-term data center in real time; A data aggregation module is used to aggregate the integrated data of the short-term data center to obtain aggregated data; The data storage module is used to store the aggregated data in the integrated data center.

[0068] The information-based multi-business data integration system of the present application can provide enterprises with comprehensive and accurate information on various types of business data, reduce the possibility of data redundancy and errors, and achieve seamless integration and aggregation of business data. At the same time, it can also support the decoupled application of business systems and related data, especially interconnection and sharing, and provide automated data processing and analysis capabilities as well as data-driven decision support capabilities, thereby providing a solid technical guarantee for the development, innovation, cost reduction and efficiency improvement of enterprises.

[0069] As an implementation of the data integration module, the data integration module includes: An integration task generating unit, used to generate corresponding data integration tasks according to the data source to be integrated; An integration model building unit builds a data integration model and loads integration processing rules; wherein the integration processing rules include data cleaning rules, data review rules and data integration processing logic; The integrated data generation unit is used to collect data from the data source to be integrated according to the data integration task and input the data into the data integration model to obtain integrated data.

[0070] In the above implementation, by realizing the effective integration of multi-source heterogeneous data, not only the data integration process is simplified and work efficiency is improved, but also the consistency and accuracy of the data are enhanced, ensuring the efficiency and reliability of the integration process, thereby providing strong data support for the enterprise and promoting the improvement of decision-making quality and operational efficiency.

[0071] As an implementation of the data aggregation module, the data aggregation module includes: A to-be-integrated data determination unit, used to use the integrated data of the short-term data center as the data source to be aggregated; The aggregation task generating unit is used to generate corresponding data aggregation tasks according to the data source to be aggregated; The aggregation model building unit is used to build a data aggregation model and load data aggregation logic; wherein the data aggregation logic includes data governance logic, data lineage processing logic, data watermark processing logic, data fingerprint processing logic and data annotation processing logic; The aggregated data generating unit is used to collect data from the data source to be aggregated according to the data aggregation task and input the data into the data aggregation model to obtain the aggregated data.

[0072] In the above implementation, not only the centralized management and optimization of data is achieved, but also the traceability and security of data are improved through a series of data processing logics (such as data governance, data lineage, data watermark, data fingerprint and data labeling). This method significantly enhances the availability and value of data, provides more reliable data support for enterprises, promotes the depth and breadth of data analysis, and ultimately helps enterprises make better decisions and improve operational efficiency and competitiveness.

[0073] In the actual application process, this application can efficiently integrate all business data (including structured, semi-structured and unstructured data) within the unit through the integration platform to improve R&D and production efficiency and achieve cost reduction and efficiency improvement. At the same time, the integration platform provides integration rules, data processing logic, data models and core functions, and realizes core applications such as unified data application exports, data application design, permission verification, security management and data aggregation. This idea breaks the traditional data island model, enhances the correlation and integration of data, and improves the level of data management and application. It provides technical support for enterprises to improve decision-making efficiency, improve data quality, support data application process, diversification and diversification, strengthen data security, enhance data sharing, support real-time data analysis, and reduce operating costs, thereby enabling enterprises to gain operating advantages in the fiercely competitive market.

[0074] An information-based multi-service data integration system of an embodiment of the present application can implement any of the above-mentioned information-based multi-service data integration methods, and the specific working process of each module in the information-based multi-service data integration system can refer to the corresponding process in the above-mentioned method embodiment.

[0075] In the several embodiments provided in this application, it should be understood that the provided methods and systems can be implemented in other ways. For example, the system embodiments described above are only illustrative; for example, the division of a certain module is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0076] The embodiment of the present application also discloses a computer device.

[0077] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, an information-based multi-service data integration method as described above is implemented.

[0078] The embodiment of the present application also discloses a computer-readable storage medium.

[0079] A computer-readable storage medium stores a computer program that can be loaded by a processor and execute any one of the above-mentioned information-based multi-business data integration methods.

[0080] Among them, computer-readable storage media can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus or device; the program code contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0081] It should be noted that in the above embodiments, the description of each embodiment has different emphases, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0082] The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any feature disclosed in this specification (including the abstract and drawings), unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

Claims

1. An information-based multi-service data integration method, characterized in that: The integration method comprises: Collect and initialize target data sources; The target data source includes a structured data source, a semi-structured data source and an unstructured data source; Automatically extracting structured data and semi-structured data from the target data source to the source data source through the ETL engine; Performing digital semantic analysis on the unstructured data in the target data source, and performing partial structuring processing according to the digital semantic analysis results to obtain partial structured data; Using the structured data and semi-structured data in the source data source and the partially structured data as data sources to be integrated; Integrate the data in the data source to be integrated to obtain integrated data; Building a short-term data center, and synchronizing the integrated data to the short-term data center in real time; Aggregating the integrated data of the short-term data center to obtain aggregated data; The aggregated data is stored in an integrated data center.

2. The information-based multi-service data integration method according to claim 1, characterized in that: The step of integrating the data in the data source to be integrated to obtain integrated data includes: Generate a corresponding data integration task according to the data source to be integrated; Build a data integration model and load integration processing rules; wherein the integration processing rules include data cleaning rules, data review rules and data integration processing logic; According to the data integration task, data is collected from the data source to be integrated and input into the data integration model to obtain integrated data.

3. The information-based multi-service data integration method according to claim 1, characterized in that: The step of aggregating the integrated data of the short-term data center to obtain aggregated data includes: Using the integrated data of the short-term data center as the data source to be aggregated; Generate a corresponding data aggregation task according to the data source to be aggregated; Build a data aggregation model and load data aggregation logic; wherein the data aggregation logic includes data governance logic, data lineage processing logic, data watermark processing logic, data fingerprint processing logic and data annotation processing logic; According to the data aggregation task, data is collected from the data source to be aggregated and input into the data aggregation model to obtain aggregated data.

4. An information-based multi-service data integration method according to any one of claims 1 to 3, characterized in that: After the step of building a short-term data center and synchronizing the integrated data to the short-term data center in real time, the following steps are also included: Receive business demand information sent by the application layer; Acquire hot data corresponding to the business demand information from the target data source according to the business demand information; Building a thermal data center and loading the thermal data; The hot data of the hot data center is stored in the short-term data center.

5. The information-based multi-service data integration method according to claim 1, characterized in that: The integrated data center includes a static data center, a dynamic data center and an unstructured data center. The static data center is used to store structured business data that does not need to be changed later, the dynamic data center is used to store structured business data that needs to be changed in real time later, and the unstructured data center is used to store unstructured business data.

6. An information-based multi-service data integration system, characterized in that: The integrated system comprises: A data collection module, used to collect and initialize target data sources; The target data source includes a structured data source, a semi-structured data source and an unstructured data source; A data extraction module, used to automatically extract structured data and semi-structured data from the target data source to a source data source through an ETL engine at a fixed time; A structured processing module, used for performing digital semantic analysis on the unstructured data in the target data source, and performing partial structured processing according to the digital semantic analysis result to obtain partial structured data; A module for determining data to be integrated, used for taking the structured data and semi-structured data in the source data source and the part of the structured data as the data source to be integrated; A data integration module, used to integrate the data in the data source to be integrated to obtain integrated data; A short-term data synchronization module is used to build a short-term data center and synchronize the integrated data to the short-term data center in real time; A data aggregation module, used for aggregating the integrated data of the short-term data center to obtain aggregated data; The data storage module is used to store the aggregated data in the integrated data center.

7. The information-based multi-service data integration system according to claim 6 is characterized in that: The data integration module includes: An integration task generating unit, used to generate a corresponding data integration task according to the data source to be integrated; An integration model building unit, which builds a data integration model and loads integration processing rules; wherein the integration processing rules include data cleaning rules, data review rules and data integration processing logic; The integrated data generating unit is used to collect data from the data source to be integrated according to the data integration task and input the data into the data integration model to obtain integrated data.

8. The information-based multi-service data integration system according to claim 6 is characterized in that: The data aggregation module includes: a to-be-integrated data determining unit, configured to use the integrated data of the short-term data center as a source of to-be-aggregated data; An aggregation task generating unit, used for generating a corresponding data aggregation task according to the data source to be aggregated; A convergence model construction unit, used to construct a data convergence model and load data convergence logic; wherein the data convergence logic includes data governance logic, data lineage processing logic, data watermark processing logic, data fingerprint processing logic and data annotation processing logic; The aggregated data generating unit is used to collect data from the data sources to be aggregated according to the data aggregation task and input the data into the data aggregation model to obtain aggregated data.

9. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 5 when executing the program.

10. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 5.

Images