Multi-source data management method and device

Abstract

Embodiments of the present application provide a multi-source data management method and device. The method comprises: uniformly managing model data of different sources in a form of metadata; wherein the metadata comprises a plurality of metadata fields; in the case that a plurality of model data have an association relationship under a certain metadata field, determining source data from the plurality of model data; and performing consistency management on non-source data according to the source data. Through the embodiments of the present application, the problem that data standards need to be defined in advance and fine data standards are difficult to manage in related technologies can be solved, and the effect of reducing the complexity of data consistency management is achieved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to a multi-source data management method and apparatus. Background Technology

[0002] When a data management system has multiple data sources, there are differences in the management methods and data generation time of the source systems due to the different sources of data. Therefore, a unified and standardized data standard is required when managing the consistency of data from different sources.

[0003] Existing data management systems, before implementing multi-source data consistency management, define data from different sources as different data models. Then, for each data model, the attributes included in the model are specifically defined. During data management, standards must be defined in advance, and these standards must be refined to the attribute level; that is, personalized data standards are defined for a specific attribute of a specific model. Then, based on multiple personalized data standards, data from different sources are verified and managed one by one, ultimately forming a closed loop of data inspection and modification. This approach has the following drawbacks:

[0004] (1) Due to the diversity of data types, the dispersion of model attributes and the variety of data standards, data management needs to be refined to the point that data with different attributes are managed separately according to standards. In this way, although the data standards are refined, the management cost is high. Moreover, when the system is implemented, due to the complexity of the structure, it is difficult to use a general device to complete the definition of data standards. It is impossible to adopt a unified and automated approach to simultaneously manage rules and verify data from multiple sources.

[0005] (2) Data from different sources needs to be verified based on data standards. When the data source management agencies are parallel agencies, the determination of responsibility for maintaining and managing data standards, as well as the determination of responsibility for modification after verification and inspection of each source of data, are difficult due to the independent management of data standards for each source of data. This presents challenges in identifying data standards and data quality responsibilities and in data governance. Summary of the Invention

[0006] This invention provides a multi-source data management method and apparatus to at least solve the problem in related technologies that data standards need to be defined in advance and that fine data standards are difficult to manage.

[0007] According to an embodiment of the present invention, a multi-source data management method is provided, comprising: performing structurally unified management of model data from different sources in the form of metadata; wherein the metadata includes multiple metadata fields; when multiple model data have a correlation under a certain metadata field, determining the source data from the multiple model data; and performing consistency management on non-source data based on the source data.

[0008] In one exemplary embodiment, the metadata field includes at least one of the following: model attribute, configuration model, associated model, associated model attribute, consistency benchmark, data source, data verification method, verification rule, and data source.

[0009] In one exemplary embodiment, determining source data from the plurality of model data includes: determining the source data and the non-source data from the plurality of model data based on the source data source field of the metadata, wherein the source data source field records the data source of each of the model data.

[0010] In an exemplary embodiment, before performing consistency management on non-source data, the method further includes: performing a first data verification on the source data using preset rules, wherein the preset rules are recorded in the verification rule field of the metadata.

[0011] In one exemplary embodiment, consistency management of non-source data includes: performing a second data verification on the non-source data based on the source data that has passed the first data verification, so as to perform consistency management on the non-source data.

[0012] In an exemplary embodiment, performing a second data verification on the non-source data includes: defining a reference sequence based on the source data that has passed the first data verification, wherein each piece of data in the reference sequence is the same as the source data; and comparing each piece of non-source data with the data in the reference sequence according to the reference sequence to perform the second data verification.

[0013] In one exemplary embodiment, comparing each non-source data with the data in the reference sequence includes: for any non-source data, if the non-source data is included in the data range of the reference sequence, determining that the non-source data passes the second data verification; if one or more data in the non-source data are not included in the data range of the reference sequence, determining that the non-source data fails the second data verification.

[0014] In one exemplary embodiment, after determining that the non-source data has failed the second data verification, the method further includes: generating an inconsistency report for the non-source data that has failed the second data verification.

[0015] According to another embodiment of the present invention, a multi-source data management device is provided, comprising: a first management module, configured to perform structural unified management of model data from different sources in the form of metadata; wherein the metadata includes multiple metadata fields; a determination module, configured to determine the source data from the multiple model data when multiple model data have a correlation under a certain metadata field; and a second management module, configured to perform consistency management of non-source data based on the source data.

[0016] According to yet another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0017] According to yet another embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0018] Through the above embodiments of the present invention, by using metadata to uniformly describe and manage data structures from different sources, all model data with consistent relationships are clearly recorded, facilitating consistency management of non-source data. Furthermore, by managing the consistency of non-source data based on source data, no predefined standards are required, enabling data consistency management between model data even without standards. Therefore, this solves the problem in related technologies where predefined data standards are necessary and the management of fine-grained data standards is difficult, thus reducing the complexity of data consistency management. Attached Figure Description

[0019] Figure 1 This is a hardware structure block diagram of a computer terminal running the multi-source data management method according to an embodiment of the present invention;

[0020] Figure 2 This is a flowchart of a multi-source data management method according to an embodiment of the present invention;

[0021] Figure 3 This is a structural block diagram of a multi-source data management device according to an embodiment of the present invention;

[0022] Figure 4 This is a flowchart of a multi-source data consistency management method according to an embodiment of the present invention;

[0023] Figure 5 This is a schematic diagram of the inter-model association topology according to an embodiment of the present invention. Detailed Implementation

[0024] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0026] The methods and embodiments provided in this application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal running the multi-source data management method according to an embodiment of the present invention. Figure 1 As shown, a computer terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor or a programmable gate array (FPGA) or similar processing device) and a memory 104 for storing data are also shown. The computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0027] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the multi-source data management method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0028] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0029] This embodiment provides a multi-source data management method running on the aforementioned computer terminal. Figure 2 This is a flowchart of a multi-source data management method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0030] Step S202: Perform unified structural management of model data from different sources in the form of metadata; wherein, the metadata includes multiple metadata fields.

[0031] Step S204: When multiple model data have a relationship under a certain metadata field, determine the source data from the multiple model data.

[0032] In this embodiment, it is clearly stated that the source system of the source model is responsible for maintaining the data. By clarifying the source of the data in the metadata, a consistent assessment benchmark for multi-source data and the identification of data quality responsibility are realized, thereby ensuring continuous and effective data governance.

[0033] Step S206: Based on the source data, perform consistency management on the non-source data.

[0034] In one embodiment, the metadata fields include at least one of the following: model attributes, configuration model, associated model, associated model attributes, consistency benchmark, data source, data verification method, verification rules, and data source.

[0035] In step S204 of this embodiment, the method includes: determining the source data and the non-source data from the multiple model data according to the data source header field of the metadata, wherein the data source header field records the data source of each model data.

[0036] Before step S206 in this embodiment, the method further includes: performing a first data verification on the source data using preset rules, wherein the preset rules are recorded in the verification rule field of the metadata.

[0037] In step S206 of this embodiment, the method includes: performing a second data verification on the non-source data based on the source data that has passed the first data verification, so as to perform consistency management on the non-source data.

[0038] In one embodiment, performing a second data verification on the non-source data includes: defining a reference sequence based on the source data that has passed the first data verification, wherein each piece of data in the reference sequence is the same as the source data; and comparing each piece of non-source data with the data in the reference sequence according to the reference sequence to perform the second data verification.

[0039] In one embodiment, comparing each non-source data with the data in the reference sequence includes: for any non-source data, if the non-source data is included in the data range of the reference sequence, determining that the non-source data passes the second data verification; if one or more of the non-source data are not included in the data range of the reference sequence, determining that the non-source data fails the second data verification.

[0040] In one embodiment, after determining that the non-source data has failed the second data verification, the method further includes: generating an inconsistency report for the non-source data that has failed the second data verification.

[0041] Through the above steps and embodiments of the present invention, by using metadata to uniformly describe and manage data structures from different sources, all model data with consistent relationships are clearly recorded, facilitating consistency management of non-source data. Furthermore, by managing the consistency of non-source data based on source data, no predefined standards are required, enabling data consistency management between model data even without standards. Therefore, this solves the problem in related technologies where predefined data standards are necessary and managing fine-grained data standards is difficult, thus reducing the complexity of data consistency management.

[0042] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory (ROM / RAM), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0043] This embodiment also provides a multi-source data management device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0044] Figure 3 This is a structural block diagram of a multi-source data management device according to an embodiment of the present invention, such as... Figure 3 As shown, the device includes: a first management module 10, a determination module 20, and a second management module 30.

[0045] The first management module 10 is used to perform unified structural management of model data from different sources in the form of metadata; wherein, the metadata includes multiple metadata fields;

[0046] The determination module 20 is used to determine the source data from the multiple model data when there is a correlation between multiple model data under a certain metadata field;

[0047] The second management module 30 is used to perform consistency management on non-source data based on the source data.

[0048] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0049] Existing data management systems, taking configuration data management systems as an example, mainly include: a configuration module, which includes configuration model management and configuration object management; further, configuration model management includes configuration model maintenance, configuration attribute management, and validation rule maintenance.

[0050] Specifically, the configuration model maintenance is used to define data of the same type as the same model and to specify all attribute information included in the model.

[0051] Configuration attribute management is used to set model attributes of the configuration model, including defining attribute categories, attribute value ranges, etc.

[0052] The data management system can also be configured by selecting validation rules based on data type through the drop-down list of "Data Rules" on the front-end page.

[0053] Validation rule maintenance is used to maintain and manage validation rules. Validation rules serve as the basis for data auditing. Through validation rules, specific data standards for specific attributes of a particular data model are defined to ensure data accuracy. Users can select different rule categories to customize specific data validation rules.

[0054] Based on the above configuration data management system, existing multi-source data consistency management methods include:

[0055] Step S1: Differentiate between different types (sources) of data and establish multiple personalized data standards;

[0056] Step S2: Define multiple personalized data standards under different model attributes;

[0057] Step S3: Verify the data from different sources one by one according to the content defined by multiple standards.

[0058] Due to the diversity of data types and standards, the aforementioned multi-source data consistency management methods require separate management of different data types when managing data standards. A universal, automated approach cannot achieve unified management of multi-source data standards.

[0059] This invention provides a method for managing multi-source data consistency. Based on the business processes of each type of data, it clarifies that the data at the source of the business process shall prevail. There is no need to define standards for different types of data separately. A general device is used to complete data consistency assessment, data rule verification and inconsistency report generation, so as to complete the unified (consistency) management of multi-source data in a general and automated manner.

[0060] In the multi-source data consistency management method of this embodiment, data benchmark management is performed on metadata, verification rules are defined according to metadata, verification rules are associated with model attributes, and the associated multi-source data is verified periodically to generate a data inconsistency report.

[0061] Figure 4 This is a flowchart of a multi-source data consistency management method according to an embodiment of the present invention, such as... Figure 4 The method includes the following steps:

[0062] Step S402: Manage model data through metadata structure.

[0063] The content managed through metadata structure includes data models, attributes, sources, rules, etc. Through metadata, the model data at the source of the business process can be clearly identified, as well as all model data with consistent relationships.

[0064] The following five data models will be used as examples to further describe the above-mentioned multi-source data management method.

[0065] Specifically, the five data models include: data center module data model, rack data model, server data model, switch data model, and logical server data model. The data for each of the five models comes from five systems, A through E. Specifically, the data source for the data center module data model is system A; the data source for the rack data model is system B; the data source for the server data model is system C; the data source for the switch data model is system D; and the data source for the logical server data model is system E.

[0066] In this embodiment, the common business attributes in the metadata of the five models are: data center module name, rack number, and server serial number. Since the five models share the same business attribute values, they are interconnected. The specific topology diagram of the relationships between the models is as follows: Figure 5 As shown.

[0067] For the above five data models, the metadata structure is designed as follows:

[0068] Metadata includes, but is not limited to, the following fields: model attributes, configuration model, associated model, associated model attributes, consistency benchmark, data source, model name, data validation method, validation rules, and data source.

[0069] Based on the above metadata design, the metadata for the five models—"Data Center Module," "Rack," "Server," "Switch," and "Logical Server"—is stored as shown in Table 1.

[0070] Table 1

[0071]

[0072] As can be seen from Table 1, "Data Center Module Name" is a business attribute present in all five models, and it originates from five different systems.

[0073] Taking "Data Center Module Name" as an example, this method illustrates the design of metadata fields:

[0074] (1) Under the consistency benchmark field of metadata, namely the “consistency benchmark” column in Table 1, the “data center module name” of the “data center module” model is defined as “benchmark”, and the “**data center module where it is located” of the other 4 models is defined as “comply”. That is, under the “data center module name” associated field, the model data of the other 4 models are based on the “data center module” model for data verification.

[0075] (2) Under the data source field of the metadata, namely the “Data Source” column in Table 1, the source of “Data Center Module Name” of the “Data Center Module” model is the model itself, and the “Data Source” of the other 4 models is “Data Center Module”.

[0076] (3) Under the data verification method field of the metadata, namely the “data verification method” column in Table 1, the “data room module name” of the “data room module” model is “get rule”, that is, the verification is performed according to the rules defined in the “verification rules”; the “data verification method” of the other 4 models is “get data”, that is, the source attribute data is retrieved according to the “data source” (model) + “verification rules” (attributes) and then verified.

[0077] Step S404: Implement the "data verification method" defined in the metadata for each model data.

[0078] Specifically, the data validation methods do not include "rule retrieval" and "data retrieval". "Rule retrieval" is to verify the accuracy of data through regular expression rules, while "data retrieval" is to define a "reference sequence" to ensure that the range of values ​​of the associated data should be within the referenced model attribute values.

[0079] Regular expressions, or regular expressions, are patterns used to match and process text. Composed of specific characters and symbols, they can be used to check, search, replace, and extract specific patterns from strings. They can be used in various programming languages ​​and text editors to process string data. Regular expressions can be used to validate input against specific formatting requirements, filter and extract data, and perform text replacement and transformation operations. By defining specific patterns, text data can be processed more flexibly and efficiently.

[0080] The "reference sequence" is defined as follows: The attribute values ​​of the data source model are defined as a referenced baseline value sequence. Then, through unified rule validation, the range of attribute values ​​associated with this "reference sequence" rule is clarified, thereby ensuring that the attribute values ​​of the data source model serve as the evaluation benchmark for consistency management of other data models. In the example above, this method defines the "Data Center Module Name" attribute value of the "Data Center Module" model as a referenced baseline value sequence. This definition is saved as a "reference sequence" rule. Subsequently, the attribute values ​​of other models are compared with the attribute values ​​in this reference sequence. If they do not exceed the range, the model's data is considered to meet the consistency conditions; otherwise, it is considered not to meet the consistency conditions, and an inconsistency report must be generated.

[0081] In this embodiment, the "Consistency Baseline" of the model attribute (metadata field) can be Baseline, Compliance, or Empty. Through explicit model attribute classification, a multi-source data consistency baseline is established, that is, the source data model is used as the baseline to verify non-source data models. For the same business attribute in multiple models, the "Consistency Baseline" must not be empty. If it is the source of the business process, the value is "Baseline"; if it is not the source data, the value is "Compliance".

[0082] Step S406: Configure all data in the same business process model to be associated with the verification rules defined based on the source data of that business process.

[0083] The "reference sequence" rule defined in step S404 establishes a data consistency assessment benchmark for data with the same business attribute from multiple sources. All non-source models with this business attribute are associated with the source models that have this business attribute. In the example above, this method associates the attribute values ​​of "**the data center module where the rack, server, switch, and logical server are located" with the "reference sequence" defined for the "data center module name" of the "data center module," meaning that the "data center module name" is used as the value benchmark.

[0084] Step S408: Perform data verification periodically, based on the benchmark defined by the verification rules, i.e., using the source data as the standard, to achieve consistency verification of multi-source data.

[0085] For multi-model, multi-attribute data values ​​aggregated from multiple data sources, all validation rules, including "reference sequences," are used to periodically verify the data. For data error results, inconsistency reports are generated, thereby completing the closed loop of multi-source data consistency management.

[0086] Through the above embodiments of the present invention, a metadata structure definition for multi-source data is provided. This defined metadata structure can clearly identify the model data at the source of the business process and all model data with consistency relationships. Based on the value of the "consistency benchmark" field in the metadata, the attribute values ​​of the data source model are determined as benchmark values. Each attribute value of the data source model is defined as a sequence of benchmark values ​​referenced by other non-source models. Based on this sequence of benchmark values, data consistency assessment can be performed on the attribute values ​​of other non-source models, thereby completing the multi-source data consistency verification.

[0087] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0088] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0089] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0090] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0091] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0092] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0093] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-source data management method, characterized in that, include: The model data from different sources is managed in a unified structure in the form of metadata; wherein, the metadata includes multiple metadata fields, and the model data includes at least one of data center module data, rack data, server data, switch data, and logical server data; When multiple model data have a relationship under a certain metadata field, the source data is determined from the multiple model data; The source data is subjected to a first data verification using preset rules, wherein the preset rules are recorded in the verification rule field of the metadata; Based on the source data, perform consistency management on the non-source data; The consistency management of non-source data includes: performing a second data verification on the non-source data based on the source data that has passed the first data verification, so as to manage the consistency of the non-source data.

2. The method according to claim 1, characterized in that, in, The metadata fields include at least one of the following: model attributes, configuration model, associated model, associated model attributes, consistency benchmark, data source, data verification method, verification rules, and data source.

3. The method according to claim 1, characterized in that, The source data is determined from the various model data, including: Based on the data source header field of the metadata, the source data and the non-source data are determined from the various model data, wherein the data source header field records the data source of each model data.

4. The method according to claim 1, characterized in that, A second data verification is performed on the non-source data, including: Based on the source data that has passed the first data verification, a reference sequence is defined, wherein each data in the reference sequence is the same as the source data; According to the reference sequence, each non-source data is compared with the data in the reference sequence to perform the second data verification.

5. The method according to claim 4, characterized in that, Each non-source data point is compared with the data in the reference sequence, including: For any non-source data, if the non-source data is included in the data range of the reference sequence, it is determined that the non-source data passes the second data verification. If one or more data points in the non-source data are not included in the data range of the reference sequence, it is determined that the non-source data has failed the second data verification.

6. The method according to claim 5, characterized in that, After determining that the non-source data failed the second data verification, the method further includes: For non-source data that fails the second data verification, an inconsistency report is generated.

7. A multi-source data management device, characterized in that, include: The first management module is used to perform unified structural management of model data from different sources in the form of metadata; wherein, the metadata includes multiple metadata fields, and the model data includes at least one of data center module data, rack data, server data, switch data, and logical server data; The determination module is used to determine the source data from the multiple model data when there is a correlation between the multiple model data under a certain metadata field; The second management module is used to perform consistency management on non-source data based on the source data; The multi-source data management device is further configured to perform a first data verification on the source data using preset rules, wherein the preset rules are recorded in the verification rule field of the metadata. The second management module is further configured to perform a second data verification on the non-source data based on the source data that has passed the first data verification, so as to perform consistency management on the non-source data.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is configured to perform the method described in any one of claims 1 to 6 when executed.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method as described in any one of claims 1 to 6.

Images