A data integration service intelligent management method

By performing data consistency and accuracy detection, sensitivity data identification and resource optimization configuration in data integration services, the problems of data inconsistency, security risks and low resource allocation in the existing technology are solved, and high-quality Data integration and resource optimization ensure the effective utilization of data assets.

CN118885476BActive Publication Date: 2024-11-29SHANDONG LINGRAN INTELLIGENT TECH CO LTD

Patent Information

Application Number
CN202411337197.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-25
Publication Date
2024-11-29
Estimated Expiration
2044-09-25

AI Technical Summary

Technical Problem

The existing data integration management methods do not perform data quality verification before data integration, which may lead to inconsistent or inaccurate data, affect business decisions, and lack sensitive data identification and processing, pose security risks, and fail to effectively utilize data assets.

Method used

An intelligent management method for data integration services is proposed, including data consistency detection, accuracy detection, sensitive data identification, data integration storage, resource optimization configuration and data classification annotation to ensure data quality, security and resource configuration.

Benefits of technology

Ensure the quality of data integration by detecting data consistency and accuracy, identifying and processing sensitive data to ensure security, optimizing resource allocation and reducing costs, and achieving effective utilization of data assets through data value assessment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118885476B_ABST
    Figure CN118885476B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of data integration management, and specifically discloses a data integration service intelligent management method. The present invention detects the consistency and accuracy of data from each data source before data integration and performs corresponding processing, thereby ensuring the data quality after data integration, which is conducive to better providing reliable data support for enterprise business decisions; identifies sensitive data from each data source before data integration, and performs data desensitization, thereby ensuring the security of data during the integration process; evaluates the priority of allocating computing resources to transmit data from the data center according to the data volume and data update frequency, which is conducive to optimizing the configuration of data integration processes and resources and reducing the cost of data integration; evaluates the value of data according to the use of data access and downloading after data integration, and then classifies and labels it, which is conducive to enterprises to better understand and use data, explore the potential value of data, and realize the effective use of data assets.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the field of data integration management, and in particular to a data integration service intelligent management method. Background Art

[0002] In today's digital age, various business departments within enterprises and organizations, as well as external partners, generate massive amounts of data, which may be stored in various data sources such as different databases, data warehouses, and file systems. The types and formats of data sources are highly heterogeneous. In order to optimize business processes and improve operational efficiency, enterprises need to integrate data from different business systems. As the importance of data analysis and data-driven decision-making becomes increasingly prominent, enterprises need to quickly and accurately obtain data from scattered data sources for analysis, which requires efficient management of data integration to ensure the timeliness and accuracy of data.

[0003] However, the existing data integration management methods still have some limitations and shortcomings in practical applications.

[0004] For example, the existing Chinese patent with publication number CN106408448A discloses an intelligent business integration and management system, including a management server, a data integration server, a standardized switch, multiple business servers and a data storage device, wherein the business server is connected to the data integration server via a standardized switch; the data integration server is connected to the management server, and the management server is connected to the data storage device. The advantages and positive effects of the intelligent business integration and management system provided by the invention are: by adopting a new structure, data integration between business management agencies at all levels is realized, so that business management agencies at all levels in the power grid business system are no longer isolated from each other, thereby effectively enhancing business collaboration and effectively improving business efficiency.

[0005] However, the above patent has the following problems: 1. Before integrating the data from each data source, the quality of the data, such as the consistency and accuracy of the data, is not verified, and thus the quality of the data after data integration cannot be guaranteed, which will not be conducive to better providing reliable data support for enterprise business decisions, and may also lead to business conflicts and decision-making errors due to inconsistent or incorrect data.

[0006] 2. In the process of integrating data from various data sources, the transmission and storage of sensitive information may be involved. The above patent lacks the identification and processing of sensitive data before data integration.

[0007] 3. The above-mentioned patents did not further sort out, classify and label the data after data integration. If the value of the data is evaluated in combination with the usage of the data and then classified and labeled, it will be detrimental to the company's better understanding and use of data, mining the potential value of data, and realizing the effective use of data assets. Summary of the invention

[0008] In response to the above problems, the present invention proposes a data integration service intelligent management method, and the specific technical solution is as follows: A data integration service intelligent management method, comprising the following steps: Step 1, data source data consistency detection and processing: Obtain the basic information of each data entity in each data source to be integrated, where the basic information includes storage format, data content, association rules, update time, and data capacity, and compare the basic information of each data entity that is the same between different data sources to be integrated, determine whether the basic information of each data entity in each data source to be integrated meets the consistency requirements, and perform corresponding processing.

[0009] Step 2: Data source data accuracy detection and processing: Based on the basic information of each data entity in each data source to be integrated, determine whether each data type data entity in each data source to be integrated meets the accuracy requirements, and perform corresponding processing.

[0010] Step 3: Identify and process sensitive data in data sources: Identify sensitive data entities in each data source to be integrated and perform corresponding processing.

[0011] Step 4: Data integration and storage of data sources: Integrate the data from each data source and store it in the data center.

[0012] Step 5. Optimal configuration of data integration resources: Obtain the amount of data integrated into the data center from each data source and the data update frequency of each data source during the historical period, analyze the priority coefficient of the data center's allocation of computing resources to transmit data from each data source, and provide feedback.

[0013] Step 6. Data source data classification and labeling: Obtain the access information and download information of each data entity in the data center during the evaluation period, analyze the value index of each data entity in the data center, and further classify and label each data entity in the data center.

[0014] Compared with the prior art, the data integration service intelligent management method described in the present invention has the following beneficial effects: 1. The present invention detects the consistency and accuracy of the data from each data source and performs corresponding processing before integrating the data from each data source, thereby ensuring the data quality after data integration, which is conducive to better providing reliable data support for enterprise business decisions and avoiding business conflicts and decision-making errors caused by inconsistent or erroneous data.

[0015] 2. The present invention identifies the sensitive data of each data source before integrating the data from each data source, and desensitizes the data, thereby ensuring the security of the data during the integration process and protecting the privacy information of enterprises and users.

[0016] 3. The present invention obtains the amount of data integrated into the data center by each data source and the data update frequency of each data source, and evaluates the priority of the data center in allocating computing resources to transmit data from each data source, which is beneficial to optimizing the configuration of data integration processes and resources and reducing the cost of data integration.

[0017] 4. The present invention evaluates the value of data according to the usage of data access and download after data integration, and then classifies and labels the data, which is conducive to enterprises to better understand and use data, explore the potential value of data, and realize the effective use of data assets. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0019] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0020] Figure 2 This is a schematic diagram of the data content consistency analysis process in step one of the present invention.

[0021] Figure 3 It is a block diagram of the composition of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] See also Figure 1 and Figure 3 As shown, a data integration service intelligent management method provided by the present invention includes the following steps: Step 1, data source data consistency detection processing: obtain the basic information of each data entity in each data source to be integrated, wherein the basic information includes storage format, data content, association rules, update time, and data capacity, and compare the basic information of the same data entities between different data sources to be integrated, determine whether the basic information of each data entity in each data source to be integrated meets the consistency requirements, and perform corresponding processing.

[0024] As a preferred solution, the specific analysis process of step 1 includes: comparing the data entities in the data sources to be integrated with each other, and screening out the same data entities between the different data sources to be integrated.

[0025] It should be noted that the same data entity between different data sources means that the data entity appears in other data sources in addition to its own data source.

[0026] In a specific embodiment, data of various enterprise application platform systems to be integrated are obtained and recorded as data of various data sources to be integrated.

[0027] It should be noted that the data from various data sources to be integrated include internal enterprise data and external enterprise data, which are scattered across various business departments of the enterprise.

[0028] Obtain the storage format of each data entity in each data source to be integrated, and compare the storage formats of the same data entities between different data sources to be integrated. If the storage formats of the same data entity between two different data sources to be integrated are inconsistent, the storage formats of the data entity of the two data sources to be integrated do not meet the consistency requirements. Then, determine whether the storage formats of each data entity in each data source to be integrated meet the consistency requirements. If they do not meet the consistency requirements, intercept them, mark the interception reason type, and return to the original data source.

[0029] In a specific embodiment, the storage format of a certain date in a certain data source is 2023-09-20, while the storage format of the corresponding date in another data source is 2023 / 09 / 20, which may cause consistency problems during data integration.

[0030] It should be noted that if the storage format of a data entity of a data source to be integrated does not meet the consistency requirements, the interception reason type marked will be "inconsistent storage format".

[0031] As a preferred option, see Figure 2 As shown, the specific analysis process of step one also includes: A1, data type acquisition: acquiring the data type of each data entity in each data source to be integrated, and obtaining the data type of each data entity that is the same between different data sources to be integrated. If the data type of a certain data entity that is the same between two different data sources to be integrated is a single item of numerical data, execute A2; if the data type of a certain data entity that is the same between two different data sources to be integrated is a numerical data set, execute A3; if the data type of a certain data entity that is the same between two different data sources to be integrated is character data, execute A4; if the data type of a certain data entity that is the same between two different data sources to be integrated is a data table, execute A5.

[0032] A2. Numerical single data consistency analysis: compare the values ​​of the same data entity between the two different data sources to be integrated. If the values ​​of the same data entity between the two different data sources to be integrated are different, the data content of the data entity of the two data sources to be integrated does not meet the consistency requirements.

[0033] In a specific embodiment, when comparing the sales quantity of the same product in two different data sources, if one data source records 100 pieces and the other data source records 120 pieces, the data contents are inconsistent.

[0034] A3. Consistency analysis of numerical data sets: obtain the mean and variance corresponding to the same data entity between the two different data sources to be integrated and compare them one by one. If the mean and variance of the same data entity between the two different data sources to be integrated are the same, then the data content of the data entity of the two data sources to be integrated meets the consistency requirements; otherwise, it does not meet the consistency requirements.

[0035] In a specific embodiment, when comparing the same monthly sales data set in two different data sources, if the mean of the monthly sales data set in one data source is 500,000 and the variance is 100,000, while the mean of the monthly sales data set in the other data source is 400,000 and the variance is 50,000, then the data content is inconsistent.

[0036] It should be noted that, it is also possible to obtain the distribution histogram shapes corresponding to the same data entity between the two different data sources to be integrated and compare them with each other. If the distribution histogram shapes of the same data entity between the two different data sources to be integrated are similar, then the data content of the data entity of the two data sources to be integrated meets the consistency requirements. Otherwise, it does not meet the consistency requirements. For example, when comparing the sales data of products in different regions in two different data sources, the sales data of products in different regions in one data source show a normal distribution, while the sales data of products in different regions in the other data source show a skewed distribution, then the data contents are inconsistent.

[0037] A4. Character data consistency analysis: compare the character strings of the data entity that are the same between the two different data sources to be integrated one by one. If the character strings of the data entity that are the same between the two different data sources to be integrated are the same, the data content of the data entity of the two data sources to be integrated meets the consistency requirements. Otherwise, it does not meet the consistency requirements.

[0038] In a specific embodiment, when comparing the customer's name information in two different data sources, if one data source records the name as "Zhang San" and the other data source records the name as "Zhang Shan", then the data content is inconsistent.

[0039] A5. Data table consistency analysis: The number of fields, field names, field types, field lengths, and constraints of the data entity that are the same between the two different data sources to be integrated are compared one by one. If the number of fields, field names, field types, field lengths, and constraints of the data entity that are the same between the two different data sources to be integrated are the same, then the data content of the data entity of the two data sources to be integrated meets the consistency requirements; otherwise, it does not meet the consistency requirements.

[0040] In a specific embodiment, if the customer information table in one data source has a "customer_name" field, and the corresponding field in the corresponding table in another data source is "client_name", there is inconsistency in the data content.

[0041] In a specific embodiment, the data type of the "age" field in one data source is an integer type, while the data type of the "age" field in another data source is a character type, and the data content is inconsistent.

[0042] In a specific embodiment, the constraint condition includes whether it is allowed to be empty, whether it is unique, etc.

[0043] Then determine whether the data content of each data entity in each data source to be integrated meets the consistency requirements. If it does not meet the consistency requirements, intercept it, mark the type of interception reason, and return it to the original data source.

[0044] It should be noted that if the data content of a data entity of a data source to be integrated does not meet the consistency requirements, the interception reason type marked is "inconsistent data content".

[0045] As a preferred solution, the specific analysis process of step one also includes: obtaining the association rules of each data entity in each data source to be integrated, comparing the association rules of the same data entities between different data sources to be integrated; if the association rules of the same data entity between two different data sources to be integrated are inconsistent, the association rules of the data entity of the two data sources to be integrated do not meet the consistency requirements, and then determine whether the association rules of the data entities in each data source to be integrated meet the consistency requirements; if they do not meet the consistency requirements, interception is performed, and the interception reason type is marked, and the data is returned to the original data source.

[0046] In a specific embodiment, according to business rules, each order in the e-commerce data should correspond to a valid customer and a valid product. If there is an order without corresponding customer or product information, or the corresponding relationship does not conform to the business logic, it means that the data is inconsistent, that is, the association rules are inconsistent.

[0047] It should be noted that if the association rule of a data entity of a data source to be integrated does not meet the consistency requirements, the interception reason type marked is "inconsistent association rules".

[0048] Obtain the data capacity of each data entity in each data source to be integrated, and compare the data capacity of the same data entities between different data sources to be integrated. If the data capacity of the same data entity between two different data sources to be integrated is different, the data capacity of the data entity of the two data sources to be integrated does not meet the consistency requirements. Then determine whether the data capacity of each data entity in each data source to be integrated meets the consistency requirements. If it does not meet the consistency requirements, intercept it, mark the interception reason type, and return to the original data source.

[0049] It should be noted that if the data capacity of a data entity of a data source to be integrated does not meet the consistency requirements, the interception reason type marked will be "inconsistent data capacity".

[0050] It should be noted that data capacity refers to the size of the storage space occupied by the data. If the difference is large, there may be problems such as data loss or duplicate storage.

[0051] As a preferred solution, the specific analysis process of step one also includes: obtaining the update time of each data entity in each data source to be integrated, comparing the update time of each identical data entity between different data sources to be integrated; if the interval between the update time of a certain identical data entity between two different data sources to be integrated is greater than a set threshold, the update time of the data entity of the two data sources to be integrated does not meet the consistency requirement, and then determining whether the update time of each data entity in each data source to be integrated meets the consistency requirement; if it does not meet the consistency requirement, interception is performed, and the interception reason type is marked, and the data is returned to the original data source.

[0052] It should be noted that if the data in one data source is the latest updated, while the data in another data source is updated earlier, data inconsistency may occur. For example, in a real-time inventory management system, the inventory data of different warehouses are updated at different times, and it may happen that the inventory data of one warehouse has been updated while the data of other warehouses has not yet been synchronized.

[0053] It should be noted that if the update time of a data entity of a data source to be integrated does not meet the consistency requirements, the interception reason type marked will be "inconsistent update time".

[0054] Step 2: Data source data accuracy detection and processing: Based on the basic information of each data entity in each data source to be integrated, determine whether each data type data entity in each data source to be integrated meets the accuracy requirements, and perform corresponding processing.

[0055] As a preferred solution, the specific analysis process of step 2 is: according to the basic information of each data entity in each data source to be integrated, the basic information of each data type data entity in each data source to be integrated is screened to obtain the value of each data type data entity in each data source to be integrated, and the historical value range of each data type data entity in each data source to be integrated stored in the database is extracted; if the value of a data type data entity in a data source to be integrated exceeds the historical value range of its data entity, then the data type data entity in the data source to be integrated does not meet the accuracy requirements, is intercepted, and the interception reason type is marked, and returned to the original data source.

[0056] It should be noted that if a data type data entity of a data source to be integrated does not meet the accuracy requirements, the interception reason type marked will be "inaccurate".

[0057] In this embodiment, the present invention detects the consistency and accuracy of the data from each data source and performs corresponding processing before integrating the data from each data source, thereby ensuring the data quality after data integration, which is conducive to better providing reliable data support for enterprise business decisions and avoiding business conflicts and decision-making errors caused by inconsistent or erroneous data.

[0058] Step 3: Identify and process sensitive data in data sources: Identify sensitive data entities in each data source to be integrated and perform corresponding processing.

[0059] As a preferred solution, the specific analysis process of step three is: using keyword extraction technology to obtain each keyword of each data entity in each data source to be integrated, and comparing it with the sensitive vocabulary library stored in the database; if a keyword of a data entity belongs to the sensitive vocabulary library, then the data entity is recorded as a sensitive data entity, and the sensitive data entities in each data source to be integrated are counted, and data desensitization processing is performed.

[0060] It should be noted that data desensitization refers to the desensitization management of key control data through the use of technical means such as removing identification columns, converting semi-identification columns, data replacement, and data reordering.

[0061] In this embodiment, the present invention identifies sensitive data of each data source before integrating the data from each data source, and desensitizes the data, thereby ensuring the security of the data during the integration process and protecting the privacy information of the enterprise and the user.

[0062] Step 4: Data integration and storage of data sources: Integrate the data from each data source and store it in the data center.

[0063] Step 5. Optimal configuration of data integration resources: Obtain the amount of data integrated into the data center from each data source and the data update frequency of each data source during the historical period, analyze the priority coefficient of the data center's allocation of computing resources to transmit data from each data source, and provide feedback.

[0064] As a preferred solution, the specific analysis process of step 5 is as follows: set the duration of the historical period, obtain the amount of data integrated into the data center by each data source during the historical period and the data update frequency of each data source, and record them as , Indicates The number of the data source. , by analyzing the formula Get the priority coefficient of allocating computing resources to the data center to transmit data from each data source ,in They represent the weights of the preset data volume and data update frequency respectively, , Indicates the number of data sources and feeds back to the data center.

[0065] It should be noted that the computing resources for transmitting data between the data center and various data sources refer to the computing power-related resources used to perform operations such as data extraction, conversion, and loading in the process of data flowing from various data sources to the data center and flowing and processing within the data center.

[0066] In this embodiment, the present invention obtains the amount of data integrated into the data center by each data source and the data update frequency of each data source, and evaluates the priority of the data center in allocating computing resources to transmit data from each data source, which is beneficial to optimizing the configuration of data integration processes and resources and reducing the cost of data integration.

[0067] Step 6. Data source data classification and labeling: Obtain the access information and download information of each data entity in the data center during the evaluation period, analyze the value index of each data entity in the data center, and further classify and label each data entity in the data center.

[0068] As a preferred solution, the specific analysis process of step 6 includes: setting the duration of the evaluation cycle, retrieving the access information and download information of each data entity in the data center during the evaluation cycle from the data center, obtaining the number of visits, number of visitors, and access frequency of each data entity in the data center during the evaluation cycle, and recording them as , Indicates The number of the data entity, , and obtain the download times, downloading numbers, and downloading frequency of each data entity in the data center during the evaluation period, and record them as .

[0069] By analyzing the formula Get the value index of each data entity in the data center ,in and They represent the preset thresholds of the number of visits, the number of visitors, and the frequency of visits, and the thresholds of the number of downloads, the number of downloaders, and the frequency of downloads, respectively. Respectively represent the preset access and download weights, .

[0070] As a preferred solution, the specific analysis process of step six also includes: extracting the value index range corresponding to each importance level of the data entities stored in the database, screening the importance level of each data entity in the data center according to the value index of each data entity in the data center, classifying each data entity in the data center according to the same importance level, and marking them.

[0071] In this embodiment, the present invention evaluates the value of data based on the usage of data such as access and download after data integration, and then classifies and labels the data, which is conducive to enterprises to better understand and use data, explore the potential value of data, and realize the effective use of data assets.

[0072] The above contents are merely examples and explanations of the concept of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the concept of the invention or exceed the scope defined by the present invention, they shall all fall within the protection scope of the present invention.

Claims

1. A data integration service intelligent management method, characterized in that: The steps include: Step 1: Data consistency detection and processing of data sources: obtain the basic information of each data entity in each data source to be integrated, where the basic information includes storage format, data content, association rules, update time, and data capacity, and compare the basic information of the same data entities between different data sources to be integrated, determine whether the basic information of each data entity in each data source to be integrated meets the consistency requirements, and perform corresponding processing; Step 2: Data source data accuracy detection and processing: Based on the basic information of each data entity in each data source to be integrated, determine whether each data type data entity in each data source to be integrated meets the accuracy requirements, and perform corresponding processing; Step 3: Identify and process sensitive data in data sources: Identify sensitive data entities in each data source to be integrated and process them accordingly; Step 4: Data source data integration and storage: Integrate the data from each data source and store it in the data center; Step 5: Optimize the configuration of data integration resources: Obtain the amount of data integrated into the data center by each data source and the data update frequency of each data source in the historical period, analyze the priority coefficient of the data center to allocate computing resources to transmit data from each data source, and provide feedback; Step 6. Data source data classification and labeling: Obtain the access information and download information of each data entity in the data center during the evaluation period, analyze the value index of each data entity in the data center, and further classify and label each data entity in the data center.

2. A data integration service intelligent management method according to claim 1, characterized in that: The specific analysis process of step 1 includes: Compare the data entities in the data sources to be integrated with each other, and screen out the same data entities between the different data sources to be integrated; Obtain the storage format of each data entity in each data source to be integrated, and compare the storage formats of the same data entities between different data sources to be integrated. If the storage formats of the same data entity between two different data sources to be integrated are inconsistent, the storage formats of the data entity of the two data sources to be integrated do not meet the consistency requirements. Then, determine whether the storage formats of each data entity in each data source to be integrated meet the consistency requirements. If they do not meet the consistency requirements, intercept them, mark the interception reason type, and return to the original data source.

3. The method for intelligent management of data integration services according to claim 1, characterized in that: The specific analysis process of step one also includes: A1. Data type acquisition: acquire the data type of each data entity in each data source to be integrated, and obtain the data type of each data entity that is the same between different data sources to be integrated. If the data type of a certain data entity that is the same between two different data sources to be integrated is single data of numerical type, execute A2; if the data type of a certain data entity that is the same between two different data sources to be integrated is a set of numerical data, execute A3; if the data type of a certain data entity that is the same between two different data sources to be integrated is character data, execute A4; if the data type of a certain data entity that is the same between two different data sources to be integrated is a data table, execute A5; A2. Consistency analysis of numerical single data: compare the values ​​of the same data entity between the two different data sources to be integrated. If the values ​​of the same data entity between the two different data sources to be integrated are different, the data content of the data entity of the two data sources to be integrated does not meet the consistency requirements. A3. Consistency analysis of numerical data sets: obtain the mean and variance corresponding to the same data entity between the two different data sources to be integrated and compare them one by one. If the mean and variance of the same data entity between the two different data sources to be integrated are the same, the data content of the data entity of the two data sources to be integrated meets the consistency requirements, otherwise, it does not meet the consistency requirements; A4. Character data consistency analysis: compare the character strings of the data entity that is the same between the two different data sources to be integrated one by one. If the character strings of the data entity that is the same between the two different data sources to be integrated are the same, the data content of the data entity of the two data sources to be integrated meets the consistency requirement. Otherwise, it does not meet the consistency requirement. A5. Data table consistency analysis: compare the number of fields, field names, field types, field lengths, and constraints of the data entity that is the same between the two different data sources to be integrated. If the number of fields, field names, field types, field lengths, and constraints of the data entity that is the same between the two different data sources to be integrated are the same, then the data content of the data entity of the two data sources to be integrated meets the consistency requirements; otherwise, it does not meet the consistency requirements. Then determine whether the data content of each data entity in each data source to be integrated meets the consistency requirements. If it does not meet the consistency requirements, intercept it, mark the type of interception reason, and return it to the original data source.

4. The data integration service intelligent management method according to claim 1, characterized in that: The specific analysis process of step one also includes: Obtain the association rules of each data entity in each data source to be integrated, and compare the association rules of the same data entities between different data sources to be integrated. If the association rules of the same data entity between two different data sources to be integrated are inconsistent, the association rules of the data entity of the two data sources to be integrated do not meet the consistency requirements. Then, determine whether the association rules of each data entity in each data source to be integrated meet the consistency requirements. If they do not meet the consistency requirements, intercept them, mark the interception reason type, and return to the original data source; Obtain the data capacity of each data entity in each data source to be integrated, and compare the data capacity of the same data entities between different data sources to be integrated. If the data capacity of the same data entity between two different data sources to be integrated is different, the data capacity of the data entity of the two data sources to be integrated does not meet the consistency requirements. Then determine whether the data capacity of each data entity in each data source to be integrated meets the consistency requirements. If it does not meet the consistency requirements, intercept it, mark the interception reason type, and return to the original data source.

5. The data integration service intelligent management method according to claim 1, characterized in that: The specific analysis process of step one also includes: Get the update time of each data entity in each data source to be integrated, and compare the update time of the same data entities between different data sources to be integrated. If the interval between the update time of the same data entity between two different data sources to be integrated is greater than the set threshold, the update time of the data entity of the two data sources to be integrated does not meet the consistency requirement. Then determine whether the update time of each data entity in each data source to be integrated meets the consistency requirement. If it does not meet the consistency requirement, intercept it, mark the interception reason type, and return to the original data source.

6. A data integration service intelligent management method according to claim 1, characterized in that: The specific analysis process of step 2 is as follows: According to the basic information of each data entity in each data source to be integrated, the basic information of each data type data entity in each data source to be integrated is screened to obtain the value of each data type data entity in each data source to be integrated, and the historical value range of each data type data entity in each data source to be integrated stored in the database is extracted. If the value of a data type data entity in a data source to be integrated exceeds the historical value range of its data entity, the data type data entity in the data source to be integrated does not meet the accuracy requirements and is intercepted, and the interception reason type is marked and returned to the original data source.

7. A data integration service intelligent management method according to claim 1, characterized in that: The specific analysis process of step three is as follows: Keyword extraction technology is used to obtain the keywords of each data entity in each data source to be integrated, and compared with the sensitive vocabulary library stored in the database. If a keyword of a data entity belongs to the sensitive vocabulary library, the data entity is recorded as a sensitive data entity. The sensitive data entities in each data source to be integrated are counted, and data desensitization is performed.

8. The data integration service intelligent management method according to claim 1, characterized in that: The specific analysis process of step 5 is as follows: Set the duration of the historical period, obtain the amount of data integrated into the data center by each data source during the historical period and the data update frequency of each data source, and record them as , Indicates The number of the data source. , by analyzing the formula Get the priority coefficient of allocating computing resources to the data center to transmit data from each data source ,in They represent the weights of the preset data volume and data update frequency respectively, , Indicates the number of data sources and feeds back to the data center.

9. A data integration service intelligent management method according to claim 1, characterized in that: The specific analysis process of step six includes: Set the duration of the evaluation cycle, retrieve the access information and download information of each data entity in the data center during the evaluation cycle from the data center, obtain the number of visits, number of visitors, and access frequency of each data entity in the data center during the evaluation cycle, and record them as , Indicates The number of the data entity, , and obtain the download times, downloading numbers, and downloading frequency of each data entity in the data center during the evaluation period, and record them as ; By analyzing the formula Get the value index of each data entity in the data center ,in and They represent the preset thresholds of the number of visits, the number of visitors, and the frequency of visits, and the thresholds of the number of downloads, the number of downloaders, and the frequency of downloads, respectively. Respectively represent the preset access and download weights, .

10. A data integration service intelligent management method according to claim 9, characterized in that: The specific analysis process of step six also includes: Extract the value index range corresponding to each importance level of the data entities stored in the database, filter the importance level of each data entity in the data center according to the value index of each data entity in the data center, classify each data entity in the data center according to the same importance level, and mark them.

Citation Information

Patent Citations

  • Intelligent business integration and management system

    CN106408448A

  • System and method for business performance management

    CA2339063A1

  • Data asset unified management system based on enterprise data

    CN113434487A

Cited By

  • Municipal road construction section data integrated management method and system based on distributed database

    CN120067112A

  • Method and System for Data Integration Management of Municipal Road Construction Sections Based on Distributed Database

    CN120067112B