Data system building method and device of multi-source heterogeneous data system and electronic equipment

Abstract

The invention discloses a data system building method and device for a multi-source heterogeneous data system and electronic equipment, and relates to the field of data processing, and the method comprises the steps: determining a data item object corresponding to original business data based on a preset data item metadata building rule; determining data source configuration information, an index calculation model, service attribute information and a verification rule corresponding to the data item object; and based on the data source configuration information, the index calculation model, the service attribute information and the verification rule, executing data generation and quality verification operation, performing standardized data output on the data item object, and determining standardized data matched with the data item object. According to the embodiment provided by the invention, a data standardization system based on the data item object is established, the data standard of each service is unified to a great extent, the standardization of a multi-source heterogeneous data interface is realized, the decoupling cost and difficulty of the service and the data are reduced, and cross-platform, cross-system and cross-organization data sharing and collaboration are realized.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a method, apparatus and electronic device for constructing a data system for a multi-source heterogeneous data system. Background Technology

[0002] Currently, data architecture serves as a strategic blueprint for organizational data resource management. By systematically planning the rules for data storage, flow, and use, it enables data governance throughout the entire lifecycle, thereby effectively supporting the coordinated development of enterprise business objectives and technological implementation.

[0003] With social progress and technological development, the construction of existing mining data mainly focuses on the business implementation level, lacking planning for data architecture. This results in poor overall interconnectivity, difficulty in expansion, inconsistent data standards across different businesses, and incompatible interface protocols, making it impossible to call data across systems. Furthermore, the boundaries between business and data are blurred in various business systems, limiting the full potential of data. Moreover, decoupling business and data is costly and difficult. Therefore, there is an urgent need to build a new generation of data architecture system that covers data standard system, processing flow, distributed storage architecture, and data analysis platform. Summary of the Invention

[0004] This application provides a method, apparatus, and electronic device for building a data system for a multi-source heterogeneous data system. The embodiments provided by this application solve the technical problems in the prior art, such as inconsistent data standards and incompatible interface protocols among various businesses, which prevent data from being called across systems. In addition, the boundaries between business and data are blurred in various business systems, which limits the utilization of data capabilities. Furthermore, the decoupling cost and difficulty between business and data are high. The embodiments provided by this application establish a data standardization system based on data item objects, which greatly unifies the data standards of various businesses, standardizes the interfaces of multi-source heterogeneous data, reduces the decoupling cost and difficulty between business and data, and enables cross-platform, cross-system, and cross-organizational data sharing and collaboration.

[0005] In a first aspect, this application provides a method for constructing a data architecture for a multi-source heterogeneous data system, the method comprising: Acquire raw business data from multiple heterogeneous business systems; Based on preset data item metadata construction rules, a data item object corresponding to the original business data is determined, wherein the data item object is used to standardize and encapsulate the definition, collection, storage and service rules of the business data; Determine the data source configuration information, indicator calculation model, business attribute information, and verification rules corresponding to the data item object; Based on the data source configuration information, the indicator calculation model, the business attribute information, and the verification rules, data generation and quality verification operations are performed to standardize the data item objects and determine the standardized data that matches the data item objects, so as to realize the construction of a data system for external multi-source heterogeneous data.

[0006] In one feasible implementation, determining the data item object corresponding to the original business data based on preset data item metadata construction rules includes: Determine the business domain and business function type corresponding to the original business data; Based on the business domain, the business function type, and the preset data item metadata construction rules, determine the metadata template corresponding to the original business data; Based on the metadata template, a data item object is created for the original business data, containing at least one of the following basic attributes: factory, material, identity category, professional category, and data category.

[0007] In one feasible implementation, the data source configuration information corresponding to the data item object is determined by the following method: Based on the login information of the preset data source, determine the data source configuration information corresponding to the data item object.

[0008] In one feasible implementation, the indicator calculation model corresponding to the data item object is determined by the following method: Based on the preset calculation logic defined by preset business rules, a parameter index calculation model library for executing calculations of multi-source heterogeneous data is determined. The preset calculation logic includes the calculation logic of input parameters, calculation formulas and output results. Based on the parameter index calculation model library and the data item object, determine the index calculation model corresponding to the data item object.

[0009] In one feasible implementation, the step of performing data generation and quality verification operations based on the data source configuration information, the indicator calculation model, the business attribute information, and the verification rules, and outputting standardized data for the data item object to determine standardized data compatible with the data item object includes: The indicator calculation model is triggered based on a preset collection frequency; Based on the data source connection information, source data is obtained from the corresponding multiple heterogeneous business systems; Based on the preset calculation logic defined in the indicator calculation model, the source data is calculated to generate the target data value of the data item object; Based on business attribute information and the preset upper and lower limits defined in the verification rules, the target data value is verified for compliance, so as to achieve standardized data output for the data item object and determine the standardized data that is compatible with the data item object.

[0010] In one feasible implementation, the method further includes: Based on the business logic relationships between each data item object, a data knowledge graph is constructed, wherein the data knowledge graph is used to describe the inheritance, reference, or association relationships between the data item objects; In response to a target business query request, data association analysis is performed on the target business based on the data knowledge graph and the standardized data.

[0011] In one feasible implementation, the step of responding to a target business query request and performing data association analysis on the target business based on the data knowledge graph and the standardized data includes: Retrieve the target data item query request from the target business query request; Based on the data knowledge graph, at least one associated data item is identified that is related to the target data item; Obtain the target historical data of the target data item and the at least one associated data item; Based on the target's historical data, perform data correlation analysis on the target business and generate a data analysis report.

[0012] In a second aspect, this application provides a data architecture construction apparatus for a multi-source heterogeneous data system, the apparatus comprising: The acquisition module is used to acquire raw business data from multiple heterogeneous business systems. The first determining module is used to determine the data item object corresponding to the original business data based on the preset data item metadata construction rules, wherein the data item object is used to standardize and encapsulate the definition, collection, storage and service rules of the business data; The second determining module is used to determine the data source configuration information, indicator calculation model, business attribute information and verification rules corresponding to the data item object; The third determination module is used to perform data generation and quality verification operations based on the data source configuration information, the indicator calculation model, the business attribute information, and the verification rules, to standardize the data item object and output standardized data that is compatible with the data item object, so as to realize the construction of a data system for external multi-source heterogeneous data.

[0013] In a third aspect, this application provides an electronic device, including a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus, and the machine-readable instructions are executed by the processor to perform the steps of the data architecture construction method for a multi-source heterogeneous data system as described above.

[0014] In a fourth aspect of this application, an embodiment of this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the data architecture construction method for a multi-source heterogeneous data system as described above.

[0015] Compared with the prior art, the data system construction method, apparatus, and electronic device for multi-source heterogeneous data systems provided in this application can acquire the original business data of multiple heterogeneous business systems, and then determine the data item objects corresponding to the original business data based on preset data item metadata construction rules. The data item objects are used to standardize and encapsulate the definition, collection, storage, and service rules of the business data. Next, the data source configuration information, indicator calculation model, business attribute information, and verification rules corresponding to the data item objects are determined. Finally, based on the data source configuration information, indicator calculation model, business attribute information, and verification rules, data generation and quality verification operations are performed to output standardized data to the data item objects, determining standardized data that matches the data item objects. This achieves the construction of a data system for external multi-source heterogeneous data. The embodiments provided in this application establish a data standardization system based on data item objects, greatly unifying the data standards of various businesses, standardizing multi-source heterogeneous data interfaces, reducing the decoupling cost and difficulty of business and data, and realizing cross-platform, cross-system, and cross-organizational data sharing and collaboration. Attached Figure Description

[0016] Figure 1 A flowchart illustrating a data architecture construction method for a multi-source heterogeneous data system provided in an embodiment of this application is shown. Figure 2 This invention provides a structural block diagram of a data architecture construction method and apparatus for a multi-source heterogeneous data system according to an embodiment of this application. Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown.

[0017] Figure 2 and Figure 3 The correspondence between the figure labels and figure titles in the accompanying drawings is as follows: Data architecture building device for a multi-source heterogeneous data system; 210 acquisition module; 220 first determination module; 230 second determination module; 240 third determination module; 300 electronic device; 310 processor; 320 memory; 330 bus. Detailed Implementation

[0018] To better understand the technical solutions provided in the embodiments of this specification, the technical solutions of the embodiments of this specification will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of this specification and the specific features in the embodiments are detailed descriptions of the technical solutions of the embodiments of this specification, rather than limitations on the technical solutions of this specification. In the absence of conflict, the embodiments of this specification and the technical features in the embodiments can be combined with each other.

[0019] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The term "two or more" includes two or more cases.

[0020] First, the applicable application scenarios of this application will be introduced. The embodiments provided in this application are applicable to the field of data processing, and in particular, they relate to a data system construction method, apparatus and electronic equipment for a multi-source heterogeneous data system.

[0021] Currently, the construction of existing mining data mainly focuses on the business implementation level, lacking planning for data architecture. The overall interconnectivity is poor, expansion is difficult, data standards are not unified across businesses, and interface protocols are incompatible, resulting in data being unable to be called across systems. The boundaries between business and data are blurred in various business systems, which limits the utilization of data capabilities. Furthermore, the cost and difficulty of decoupling business and data are high. Therefore, there is an urgent need to build a new generation of data architecture system that covers data standard system, processing flow, distributed storage architecture, and data analysis platform.

[0022] Based on this, embodiments of this application provide a method, apparatus, and electronic device for building a data system for a multi-source heterogeneous data system. The embodiments provided by this application solve the technical problems in the prior art where data standards for various businesses are not unified and interface protocols are incompatible, resulting in data being unable to be called across systems. In various business systems, the boundaries between business and data are blurred, which limits the utilization of data capabilities. Furthermore, the cost and difficulty of decoupling business and data are high. The embodiments provided by this application establish a data standardization system based on data item objects, which greatly unifies the data standards of various businesses, standardizes multi-source heterogeneous data interfaces, reduces the cost and difficulty of decoupling business and data, and enables cross-platform, cross-system, and cross-organizational data sharing and collaboration.

[0023] Figure 1 A flowchart illustrating a data architecture construction method for a multi-source heterogeneous data system provided in an embodiment of this application is shown. Figure 1 As shown, the data architecture construction method for a multi-source heterogeneous data system includes the following steps: S101. Obtain the original business data from multiple heterogeneous business systems.

[0024] In this step, the embodiments provided in this application first acquire and collect the original business data of the target mine scattered in various corners (i.e., multiple heterogeneous business systems), and then, based on the data processing of the original business data of the above-mentioned multiple heterogeneous business systems, standardized and normalized data is generated to generate standardized original business data.

[0025] It is understood that the various heterogeneous business system categories provided in the embodiments of this application specifically include production execution systems, equipment management systems, and geographic information systems (GIS).

[0026] The data system construction method of the multi-source heterogeneous data system provided in this application can be applied to different application scenarios and usage conditions. The embodiments provided in this application can be specifically applied to the application scenario of data system construction in mines.

[0027] S102. Based on the preset data item metadata construction rules, determine the data item objects corresponding to the original business data. The data item objects are used to standardize and encapsulate the definition, collection, storage and service rules of the business data.

[0028] In this step, in the embodiments provided in this application, after determining the original business data after standardization processing, a data item object corresponding to the original business data is created based on the pre-designed preset data item metadata construction rules. The data item object specifically includes basic attribute data such as factory, material, functional unit, identity category, professional classification and data category.

[0029] For example, based on preset data item metadata construction rules, the data item objects corresponding to the original business data are determined, including: Determine the business domain and business function type corresponding to the original business data; based on the business domain, business function type, and preset data item metadata construction rules, determine the metadata template corresponding to the original business data; based on the metadata template, create data item objects for the original business data that contain at least one of the following basic attributes: factory, material, identity category, professional classification, and data classification.

[0030] It should be noted that, in order to achieve standardization of data definition, the embodiments provided in this application will preset data item metadata construction rules. The metadata template library under these rules stores standard metadata templates for different business domains and business function types. Each template is actually a structured framework that defines which attribute fields must be included in a certain type of original business data and the constraints of these fields. The system uses the "business domain" and "functional unit" determined in the previous step as index keys to automatically retrieve and call the most matching metadata template corresponding to the original business data from the preset data item metadata construction rules. If no completely matching template is found, the default metadata template of its parent business domain can be called, or the administrator can copy and modify it based on a similar template, thereby ensuring the flexibility and full coverage of the data template.

[0031] After invoking the metadata template, this application instantiates a data item object with at least one basic attribute and assigns the attribute structure defined in the template to the data item object. Subsequently, it automatically assigns values ​​to each attribute of the data item object based on the specific information of the raw business data being processed. For example, it identifies the physical factory or organization to which the data belongs, such as a factory; identifies the material object associated with the data, such as "raw ore" or "concentrate powder"; identifies the identity category of the data to distinguish whether the data is a planned value, actual value, or predicted value; identifies the professional classification of the data to divide the data type from a technical perspective, such as "geological data", "production data", "energy consumption data", and "safety data"; and classifies the data to divide the data granularity or type from a management perspective, such as "indicator data" and "detailed data".

[0032] For example, in the above, the "business domain" of a piece of ore weight data from a belt scale should be determined as "production", and its "business function type" can be further refined to "grinding workshop" or "measuring unit"; the "business domain" of a piece of equipment vibration data should be "equipment", and its "business function type" can be determined as "crusher" or "ball mill". This classification process can be automatically completed through the preset data item metadata construction rule "business-system-data mapping table", or configured by the implementers during system initialization.

[0033] Here, the embodiments provided in this application can establish a mapping relationship between business and data, quickly build a cross-domain data integration and sharing mechanism, and enhance information sharing and collaboration between mining businesses.

[0034] S103. Determine the data source configuration information, indicator calculation model, business attribute information, and verification rules corresponding to the data item object.

[0035] In this step, after determining the data item object, the embodiments provided in this application need to construct different data source configuration information, indicator calculation model, business attribute information and verification rules corresponding to the data item object. Then, based on the determined data source configuration information, indicator calculation model, business attribute information and verification rules, the data item object is analyzed and processed, and corresponding standardized data is output to help mine-related businesses quickly build a full life cycle data governance system.

[0036] Understandably, this application is able to build flexible computing models based on data item objects to meet the needs of complex business scenarios and personalized scenarios in different mines.

[0037] For example, the data source configuration information corresponding to the data item object is determined in the following way: Based on the login information of the preset data source, determine the data source configuration information corresponding to the data item object.

[0038] It should be noted that the login information of the preset data source in the embodiments provided in this application may specifically include the data source name, data source type, driver type, username, password, and data source connection information, etc.

[0039] For example, the indicator calculation model corresponding to the data item object is determined in the following way: Based on the preset calculation logic defined by the preset business rules, a parameter index calculation model library for calculating multi-source heterogeneous data is determined. The preset calculation logic includes the calculation logic of input parameters, calculation formulas and output results. Based on the parameter index calculation model library and data item objects, an index calculation model corresponding to the data item object is determined.

[0040] It should be noted that the indicator calculation model in the embodiments provided in this application mainly provides a customized model for indicator data items. It is necessary to first obtain information such as model code, model name, model category and data table name, and then input the data information through the user, configure the corresponding indicator calculation model library according to the preset calculation logic defined by the preset business rules, and then bind the data item object to the corresponding indicator calculation model in the indicator calculation model library.

[0041] It is understood that the business attribute information provided in the embodiments of this application includes, but is not limited to, completion attribute information and planning attribute information. The attribute information includes, but is not limited to, information such as the factory, material, identity category, professional classification, data classification, unit of measurement, shift type, upper limit, and lower limit to which the data item object belongs; the planning attribute information includes, but is not limited to, information such as factory, planning dimension, responsible role, and calculation logic. Here, the planning dimension mainly includes daily, monthly, and yearly planning dimensions.

[0042] The following example demonstrates the process of determining the output of the index calculation model and calculating standardized data: The embodiments provided in this application require determining the calculation process of the concentrate completion quantity model: 1) Automatically collect the moisture content yw of raw ore entering the mill for each production series from multiple heterogeneous business systems, namely the belt metering system; collect the moisture information mc of the corresponding batch from the quality inspection system; and input the moisture content of the belt scale and the moisture information of the quality inspection system into the dry content model of the belt scale to obtain the dry content yd of raw ore entering the mill, where yd=yw*(1-mc).

[0043] 2) Obtain the amount of ore transported by rail from the rail transport metering system based on information such as factory, material, storage location, and production date. Combine this with the dry amount of raw ore entering the mill (yd) from step 1) and input it into the feed rate model represented by the index calculation model to obtain the ore feed rate for different functional units.

[0044] 3) Obtain the concentrate grade jp, raw ore grade yp and tailings grade wp of the corresponding batch in the quality inspection system, input them into the index calculation model, and calculate the beneficiation ratio r in the beneficiation ratio model, r=(jp-wp) / (yp-wp)+k, where k is the adjustment coefficient.

[0045] 4) Input the results of steps 2) and 3) into the concentrate quantity calculation model represented by the index calculation model. The final calculation result is the concentrate completion quantity.

[0046] S104. Based on the data source configuration information, indicator calculation model, business attribute information and verification rules, perform data generation and quality verification operations, standardize the data output of data item objects, and determine the standardized data that matches the data item objects, so as to realize the construction of a data system for external multi-source heterogeneous data.

[0047] In this step, the embodiment provided in this application specifically triggers the indicator calculation model based on a preset collection frequency; based on the data source connection information, source data is obtained from multiple corresponding heterogeneous business systems; based on the preset calculation logic defined in the indicator calculation model, the source data is calculated to generate the target data value of the data item object; based on the preset upper and lower limits defined in the business attribute information and verification rules, the target data value is verified for compliance, so as to realize the standardized data output of the data item object and determine the standardized data that is compatible with the data item object.

[0048] In this step, the embodiment provided in this application is to periodically configure the collection method according to the data item object, set a preset collection frequency to trigger data collection, model calculation and storage, and obtain source data from multiple heterogeneous business systems based on the preset calculation logic defined in the indicator calculation model. Then, the target data value required in the data item object can be automatically or manually entered, and the target data value is verified within a preset range. After the verification is passed, it is stored in the target database.

[0049] It is understood that the verification rules in the embodiments provided in this application can be used to perform range verification on the target data value. Specifically, it can be as follows: determine whether the target data value is within a preset value range; if it exceeds the preset value range, generate a data anomaly prompt message; and trigger a manual review process based on the data anomaly prompt message.

[0050] For example, the method further includes: constructing a data knowledge graph based on the business logic relationships between each data item object, wherein the data knowledge graph is used to describe the inheritance, reference, or association relationships between data item objects; and performing data association analysis on the target business based on the data knowledge graph and standardized data in response to a target business query request.

[0051] It is understood that the data knowledge graph provided in the embodiments of this application supports business personnel to conduct self-service data exploration and analysis, thereby maximizing the value of mine management data.

[0052] It should be noted that the embodiments provided in this application construct data logical relationships, innovate the system mechanism of data item objects, decouple data from business and establish logical relationships, forming a data-driven business mapping framework. Furthermore, this application clarifies the definition of data, calculation logic, storage granularity, hierarchical relationships and management dimensions, etc., to achieve cross-platform, cross-system and cross-organizational data sharing and collaboration.

[0053] For example, in response to a target business query request, based on a data knowledge graph and standardized data with range validation, data association analysis is performed on the target business, including: Obtain the target data item query request from the target business query request; based on the data knowledge graph, determine at least one associated data item related to the target data item; obtain the target historical data of the target data item and at least one associated data item; based on the target historical data, perform data association analysis and traceability services for the target business, and generate a data analysis report.

[0054] It is understood that the data analysis report provided in the embodiments of this application is specifically a data insight report that includes analysis of data change trends and related impacts.

[0055] The data analysis reports provided in this application can be customized and used according to different business needs and application environments.

[0056] It should be noted that the embodiments provided in this application can also calculate the value contribution of a data item based on the frequency of its invocation, its data quality score, and the business value it supports. Based on the value contribution, an enterprise data asset catalog can be constructed, and the data item objects in the catalog can be prioritized and visualized.

[0057] Furthermore, the embodiments provided in this application can also be based on a distributed storage architecture to elastically expand the storage of standardized data items, and based on read-write separation and master-slave synchronization mechanisms, improve data access efficiency and system reliability.

[0058] Compared with the prior art, the data system construction method for multi-source heterogeneous data systems provided in this application can acquire the original business data of multiple heterogeneous business systems, and then determine the data item objects corresponding to the original business data based on preset data item metadata construction rules. The data item objects are used to standardize and encapsulate the definition, collection, storage and service rules of business data. Next, the data source configuration information, indicator calculation model, business attribute information and verification rules corresponding to the data item objects are determined. Finally, based on the data source configuration information, indicator calculation model, business attribute information and verification rules, data generation and quality verification operations are performed to output standardized data to the data item objects and determine the standardized data that matches the data item objects, so as to realize the construction of a data system for external multi-source heterogeneous data. The embodiments provided in this application establish a data standardization system based on data item objects, which greatly unifies the data standards of various businesses, realizes the standardization of multi-source heterogeneous data interfaces, reduces the decoupling cost and difficulty of business and data, and realizes cross-platform, cross-system and cross-organization data sharing and collaboration.

[0059] The embodiments provided in this application can realize the standardization, service-orientation, and value maximization of mining data, providing a replicable technical paradigm for the digital transformation of traditional mining.

[0060] Figure 2 This is a structural block diagram of a data architecture construction device for a multi-source heterogeneous data system provided in an embodiment of this application. Figure 2 As shown, the data architecture construction device 200 for a multi-source heterogeneous data system includes: The acquisition module 210 is used to acquire raw business data from multiple heterogeneous business systems.

[0061] The first determining module 220 is used to determine the data item object corresponding to the original business data based on the preset data item metadata construction rules. The data item object is used to standardize and encapsulate the definition, collection, storage and service rules of the business data.

[0062] The second determining module 230 is used to determine the data source configuration information, indicator calculation model, business attribute information, and verification rules corresponding to the data item object.

[0063] The third determination module 240 is used to perform data generation and quality verification operations based on data source configuration information, indicator calculation model, business attribute information and verification rules, to standardize data output for data item objects, and to determine standardized data that is compatible with data item objects, so as to realize the construction of a data system for external multi-source heterogeneous data.

[0064] For example, the first determining module 220 is specifically used for: Determine the business domain and business function type corresponding to the original business data.

[0065] Based on the business domain, business function type, and preset data item metadata construction rules, determine the metadata template corresponding to the original business data.

[0066] Based on the metadata template, create data item objects for the original business data that contain at least one of the following basic attributes: factory, material, identity category, professional category, and data category.

[0067] For example, the data source configuration information corresponding to the data item object is determined in the following way: Based on the login information of the preset data source, determine the data source configuration information corresponding to the data item object.

[0068] For example, the indicator calculation model corresponding to the data item object is determined in the following way: Based on the preset calculation logic defined by the preset business rules, a parameter index calculation model library for calculating multi-source heterogeneous data is determined. The preset calculation logic includes the calculation logic of input parameters, calculation formulas and output results.

[0069] Based on the parameter index calculation model library and data item objects, determine the index calculation model corresponding to the data item objects.

[0070] For example, the third determining module 240 is specifically used for: The indicator calculation model is triggered based on the preset collection frequency.

[0071] Based on the data source connection information, source data is obtained from multiple corresponding heterogeneous business systems.

[0072] Based on the preset calculation logic defined in the indicator calculation model, the source data is calculated to generate the target data value of the data item object.

[0073] Based on the business attribute information and the preset upper and lower limits defined in the verification rules, the target data value is verified for compliance, so as to achieve standardized data output of data item objects and determine the standardized data that is compatible with the data item objects.

[0074] For example, a data knowledge graph is constructed based on the business logic relationships between each data item object, whereby the data knowledge graph is used to describe the inheritance, reference, or association relationships between data item objects.

[0075] In response to the target business query request, data correlation analysis is performed on the target business based on data knowledge graph and standardized data.

[0076] For example, retrieve the target data item query request from the target business query request.

[0077] Based on the data knowledge graph, identify at least one associated data item that is related to the target data item.

[0078] Obtain the target historical data of the target data item and at least one associated data item.

[0079] Based on the target's historical data, perform data correlation analysis on the target business and generate a data analysis report.

[0080] Please see Figure 3 , Figure 3 This application provides a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 3 As shown, the electronic device 300 includes a processor 310, a memory 330, and a bus 330.

[0081] Memory 330 stores machine-readable instructions executable by processor 310. When electronic device 300 is running, processor 310 and memory 330 communicate via bus 330. When the machine-readable instructions are executed by processor 310, they can perform the operations described above. Figure 1The steps of the data system construction method for the multi-source heterogeneous data system in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0082] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of the data system construction method for the multi-source heterogeneous data system in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0083] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0084] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0085] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-readable program code.

[0086] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0087] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0088] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0089] This application also provides a computer program product, which includes computer software instructions. When the computer software instructions are run on a processing device, they cause the processing device to execute a process for constructing a data architecture method for a multi-source heterogeneous data system.

[0090] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0091] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0092] In the several embodiments provided in this application, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0093] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0094] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0095] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0096] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

[0097] Although preferred embodiments have been described in this specification, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this specification.

[0098] Obviously, those skilled in the art can make various modifications and variations to this specification without departing from its spirit and scope. Therefore, if such modifications and variations fall within the scope of the claims and their equivalents, this specification is also intended to include such modifications and variations.

Claims

1. A method for constructing a data architecture for a multi-source heterogeneous data system, characterized in that, The method for constructing the data system for multi-source heterogeneous data includes: Acquire raw business data from multiple heterogeneous business systems; Based on preset data item metadata construction rules, a data item object corresponding to the original business data is determined, wherein the data item object is used to standardize and encapsulate the definition, collection, storage and service rules of the business data; Determine the data source configuration information, indicator calculation model, business attribute information, and verification rules corresponding to the data item object; Based on the data source configuration information, the indicator calculation model, the business attribute information, and the verification rules, data generation and quality verification operations are performed to standardize the data item objects and determine the standardized data that matches the data item objects, so as to realize the construction of a data system for external multi-source heterogeneous data.

2. The data architecture construction method for a multi-source heterogeneous data system according to claim 1, characterized in that, The process of determining the data item object corresponding to the original business data based on preset data item metadata construction rules includes: Determine the business domain and business function type corresponding to the original business data; Based on the business domain, the business function type, and the preset data item metadata construction rules, determine the metadata template corresponding to the original business data; Based on the metadata template, a data item object is created for the original business data, containing at least one of the following basic attributes: factory, material, identity category, professional category, and data category.

3. The data architecture construction method for a multi-source heterogeneous data system according to claim 1, characterized in that, The data source configuration information corresponding to the data item object is determined in the following way: Based on the login information of the preset data source, determine the data source configuration information corresponding to the data item object.

4. The data architecture construction method for a multi-source heterogeneous data system according to claim 1, characterized in that, The indicator calculation model corresponding to the data item object is determined in the following way: Based on the preset calculation logic defined by preset business rules, a parameter index calculation model library for executing calculations of multi-source heterogeneous data is determined. The preset calculation logic includes the calculation logic of input parameters, calculation formulas and output results. Based on the parameter index calculation model library and the data item object, determine the index calculation model corresponding to the data item object.

5. The data architecture construction method for a multi-source heterogeneous data system according to claim 4, characterized in that, The process of performing data generation and quality verification operations based on the data source configuration information, the indicator calculation model, the business attribute information, and the verification rules, and outputting standardized data for the data item object to determine standardized data compatible with the data item object includes: The indicator calculation model is triggered based on a preset collection frequency; Based on the data source connection information, source data is obtained from the corresponding multiple heterogeneous business systems; Based on the preset calculation logic defined in the indicator calculation model, the source data is calculated to generate the target data value of the data item object; Based on business attribute information and the preset upper and lower limits defined in the verification rules, the target data value is verified for compliance, so as to achieve standardized data output for the data item object and determine the standardized data that is compatible with the data item object.

6. The data architecture construction method for a multi-source heterogeneous data system according to claim 1, characterized in that, The method further includes: Based on the business logic relationships between each data item object, a data knowledge graph is constructed, wherein the data knowledge graph is used to describe the inheritance, reference, or association relationships between the data item objects; In response to a target business query request, data association analysis is performed on the target business based on the data knowledge graph and the standardized data.

7. The data architecture construction method for a multi-source heterogeneous data system according to claim 6, characterized in that, In response to a target business query request, the data association analysis performed on the target business based on the data knowledge graph and the standardized data includes: Retrieve the target data item query request from the target business query request; Based on the data knowledge graph, at least one associated data item is identified that is related to the target data item; Obtain the target historical data of the target data item and the at least one associated data item; Based on the target's historical data, perform data correlation analysis on the target business and generate a data analysis report.

8. A data architecture construction device for a multi-source heterogeneous data system, characterized in that, The data architecture construction device for the multi-source heterogeneous data system includes: The acquisition module is used to acquire raw business data from multiple heterogeneous business systems. The first determining module is used to determine the data item object corresponding to the original business data based on the preset data item metadata construction rules, wherein the data item object is used to standardize and encapsulate the definition, collection, storage and service rules of the business data; The second determining module is used to determine the data source configuration information, indicator calculation model, business attribute information and verification rules corresponding to the data item object; The third determination module is used to perform data generation and quality verification operations based on the data source configuration information, the indicator calculation model, the business attribute information, and the verification rules, to standardize the data item object and output standardized data that is compatible with the data item object, so as to realize the construction of a data system for external multi-source heterogeneous data.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the data architecture construction method for a multi-source heterogeneous data system as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the steps of the data architecture construction method for a multi-source heterogeneous data system as described in any one of claims 1-7.

Images