A data processing method and system based on a data middle platform

Abstract

The application provides a data processing method and system based on a data middle platform. The method is applied to a pre-constructed data middle platform. The method comprises the following steps: first, receiving external data, wherein the external data comprises data from at least one data source; then, extracting data entities from the external data, wherein the data entities are the same entities in the data from different data sources; and finally, updating master data according to the data entities, wherein the master data represents the recognized shared data among the multiple data sources. In this way, the consistent data entities in different data sources are extracted by the data middle platform, the associated data of the same data entities in different data sources is stored and shared, the shared data domain among different data sources can be formed based on the data middle platform, and the automatic update of the master data is realized. In this way, the manual maintenance cost can be saved, the data from multiple data sources can be efficiently interconnected and communicated, and the use requirements at the data application level can be met.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and system based on a data middle platform. Background Art

[0002] When processing financial market data, the challenge of integrating multiple data sources is unavoidable. Currently, data from different data sources is stored relatively independently, mostly stored directly according to the data source's format. However, data from different sources may have certain business relevance, necessitating data association and utilization at the data application level. However, each data source is an independent system with its own internal encoding and labeling system, making direct association impossible. Therefore, data from these different sources needs to be connected and interconnected, forming a shared data domain to enable flexible utilization at the data application level.

[0003] In existing technologies, data comparison between two data sources or systems is achieved by setting up a code mapping table. However, this method can only interconnect data from a limited number of data sources or systems, making its application scope limited. Furthermore, the initialization and subsequent maintenance of the code mapping table require significant manual effort, resulting in high costs and uncertain timeliness, making it unsuitable for the use of financial data. Summary of the Invention

[0004] In view of this, the present application provides a data processing method and system based on a data middle platform, aiming to efficiently realize the interconnection and interoperability of data from multiple data sources, thereby meeting the usage requirements at the data application level.

[0005] In a first aspect, the present application provides a data processing method based on a data middle platform, the method comprising:

[0006] The data center receives external data, where the external data includes data from at least one data source;

[0007] The data middle platform extracts a data entity from the external data, where the data entity is the same entity in data from different data sources;

[0008] The data middle platform updates the master data according to the data entity, where the master data represents recognized shared data among multiple data sources.

[0009] Optionally, before the data center receives external data, the method further includes:

[0010] The data center receives standard master data in order to determine the data template. The standard master data is pre-compiled related data from different data sources.

[0011] The data middle platform determines the master data model based on the data template, and the master data model is used to manage the basic attribute information of the master data and standardize the fields, attributes and processing rules of the master data.

[0012] Optionally, the corresponding data of the data entity includes first data and second data, the first data includes a first field, the second data includes a second field, the first data and the second data come from different data sources, and the data middle platform updates the master data according to the data entity, including:

[0013] The data center determines whether the first data and the second data are related data;

[0014] If so, the data center determines whether the first field and the second field are the same;

[0015] If they are the same, the data center stores the first field;

[0016] If they are not the same, the data center determines and stores the first master data based on the first field and the second field.

[0017] Optionally, the master data model has a standard code and a personalized code, and determining and storing the first master data according to the first field and the second field includes:

[0018] The data center generates a third field according to the standard code, and the third field serves as the identity identification of the first master data;

[0019] The data center processes the first field and the second field according to the personality code to obtain corresponding fourth and fifth fields to distinguish and identify the data sources from which the first field and the second field come;

[0020] The data middle platform obtains and stores the first master data based on the third field, the fourth field and the fifth field.

[0021] Optionally, before the data center determines whether the first data and the second data are related data, the process further includes:

[0022] The data center determines whether the first data and the second master data are related data, and the second master data is the existing master data in the data center;

[0023] If so, the data center determines whether the second master data contains the first field;

[0024] If it does not exist, the first field is not saved;

[0025] If so, the data center determines whether the corresponding field value in the second master data is equal to the field value corresponding to the first field;

[0026] If they are not equal, update the corresponding field value in the second master data.

[0027] Optionally, the method further includes:

[0028] The data center receives a display instruction;

[0029] The data middle platform dynamically combines the master data according to the display instruction and displays it.

[0030] Optionally, the method further includes:

[0031] The data middle station receives the sharing instruction;

[0032] The data middle platform extracts the target master data to be shared according to the sharing instruction;

[0033] The data middle platform outputs the target master data through an external interface.

[0034] In a second aspect, the present application provides a data middle platform, which includes a data entry component, an offline task component, an asset component, a data lake, and an interface component;

[0035] The data entry component is used to enter standard master data. The standard master data is used to determine the master data model so as to standardize the basic attribute information of the master data and the fields, attributes and processing rules of the master data. The master data is recognized and shared among multiple data sources.

[0036] The offline task component is used to receive external data and update the master data according to the external data;

[0037] The data lake is used to realize shared storage of the master data;

[0038] The asset component is used to display the master data in the data lake;

[0039] The interface component is used to realize the sharing and output of the master data.

[0040] In a third aspect, the present application provides a data processing system based on a data middle platform, characterized in that the system is applied to the data middle platform described in the second aspect above, and the system includes: an external data receiving unit, a data entity extraction unit, and a master data update unit;

[0041] The external data receiving unit is configured to receive external data, wherein the external data includes data from at least one data source;

[0042] The data entity extraction unit is configured to extract a data entity from the external data, wherein the data entity is the same entity in data from different data sources;

[0043] The master data updating unit is configured to update master data according to the data entity, where the master data represents recognized shared data among multiple data sources.

[0044] Optionally, the system further comprises: a standard master data entry module, a master data model determination unit;

[0045] The standard master data entry module is used to receive standard master data in order to determine the data template. The standard master data is pre-compiled related data from different data sources;

[0046] The master data model determination unit is used to determine a master data model according to the data template. The master data model is used to manage basic attribute information of the master data and standardize fields, attributes and processing rules of the master data.

[0047] The present application provides a data processing method and system based on a data middle platform. The method is applied to a pre-built data middle platform, and the method includes: first, receiving external data, and the external data includes data from at least one data source. Then, extracting data entities from the external data, and the data entity is the same entity in the data from different data sources. Finally, updating the master data according to the data entity, and the master data represents the recognized shared data between multiple data sources. In this way, consistent data entities from different data sources are extracted through the data middle platform, and the associated data of the same data entity in different data sources are stored and shared, so that a shared data domain between different data sources can be formed based on the data middle platform, and the master data can be automatically updated. In this way, the cost of manual maintenance can be saved, and the interconnection and interoperability of data from multiple data sources can be efficiently realized, thereby meeting the usage requirements at the data application level. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the following briefly introduces the drawings required for use in the embodiment or the prior art description. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 A flow chart of a data processing method based on a data middle platform provided in an embodiment of the present application;

[0050] Figure 2A schematic diagram of a data processing method based on a data middle platform provided in an embodiment of the present application;

[0051] Figure 3 A flow chart of a data processing method based on a data middle platform provided in an embodiment of the present application;

[0052] Figure 4 A structural diagram of a data processing system based on a data middle platform provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] As mentioned earlier, when analyzing financial market data, aggregating and storing multiple data sources presents a significant challenge. Data from different data sources or systems exhibit correlations, but much of this data is stored independently by source, unable to communicate with each other. This hinders flexible sharing and use at the data source application level. Currently, a common method for cross-data access involves establishing a code mapping table between the two systems, mapping and storing the code and data from the two systems to create shared characteristics. However, this method requires manual intervention and maintenance, resulting in high labor costs and limited applicability. Furthermore, this mapping storage approach is inherently inflexible. Interconnecting a large number of data sources or systems is not only costly but also inefficient, failing to meet the demands of data application requirements.

[0054] In view of this, the present application provides a data processing method and system based on a data middle platform. The method is applied to a pre-built data middle platform, and the method includes: first, receiving external data, and the external data includes data from at least one data source. Then, extracting data entities from the external data, and the data entity is the same entity in the data from different data sources. Finally, updating the master data according to the data entity, and the master data represents the recognized shared data between multiple data sources. In this way, consistent data entities from different data sources are extracted through the data middle platform, and the associated data of the same data entity in different data sources are stored and shared, so that a shared data domain between different data sources can be formed based on the data middle platform, and the master data can be automatically updated. In this way, the cost of manual maintenance can be saved, and the interconnection and interoperability of data from multiple data sources can be efficiently realized, thereby meeting the usage requirements at the data application level.

[0055] In order to help those skilled in the art better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0056] The embodiment of the present application can be applied to a pre-built data middle platform, which includes: a data entry component, an offline task component, an asset component, a data lake and an interface component.

[0057] Data entry component, used to enter standard master data.

[0058] The offline task component is used to receive external data and update the master data based on the external data.

[0059] Data lake, used to achieve shared storage of master data;

[0060] Asset component, used to display master data in the data lake;

[0061] Interface component, used to realize the sharing and output of master data.

[0062] The data middle platform is a platform that integrates data resource management, data processing, and data monitoring and analysis, designed to help enterprises better manage and utilize data. The data middle platform contains a data lake (containing multiple data repositories) that integrates data from various sources and converts it into understandable and actionable formats. Using data technology, the data middle platform collects, calculates, stores, and processes massive amounts of data, while also standardizing standards and calibers. This unified data is then standardized and stored.

[0063] Among them, standard master data is used to determine the master data model in order to standardize the basic attribute information of the master data and the fields, attributes and processing rules of the master data.

[0064] Master data is the commonly recognized shared data across multiple data sources. At the application level, master data is the fundamental, standardized data shared by multiple data systems or data sources within the context of multi-system integration. Common master data includes: suppliers, customers, materials, personnel, departments, and projects.

[0065] The master data model is the primary data information model shared between various data sources or data systems. The master data model is the foundation of master data management. A complete, scalable, and relatively stable master data model is crucial for master data management.

[0066] Shared storage is a storage method used to share data storage space among multiple computers or servers. By pooling storage resources, multiple computers or data systems can simultaneously access and manipulate the same data. This is typically achieved using specialized storage devices such as storage area networks (SANs) or network-attached storage (NAS). These storage devices provide high-speed, reliable data access and transmission services to meet the needs of multiple computers or data systems for simultaneous reading and writing data, offering advantages such as flexibility, scalability, and high availability.

[0067] Optionally, when the data middle platform receives a data sharing instruction, the interface component can be specifically used to: output the master data to be shared through an API path or a data interface.

[0068] Optionally, when the data center receives a data display instruction, the asset component can dynamically extract the required master data, combine them, and display them.

[0069] See also Figure 1 , Figure 1 A flow chart of a data processing method based on a data middle platform provided in an embodiment of the present application, the method comprising:

[0070] S101: The data center receives external data.

[0071] The external data includes data from at least one data source and may include interrelated data from multiple data sources or data systems.

[0072] Optionally, the external data can be collected through a main data collection program or obtained through manual entry, which does not affect the normal implementation of the embodiment of the present application.

[0073] Optionally, before the data center receives external data, the method further includes: the data center receives standard master data to determine a data template. The standard master data is pre-compiled, related data from different data sources. The data center determines a master data model based on the data template. The master data model is used to manage basic attribute information of the master data and standardize the fields, attributes, and processing rules of the master data.

[0074] The data template is used to provide a form of presentation for data, and may include information such as data fields and data types that the user needs to input.

[0075] S102: The data center extracts data entities from external data.

[0076] A data entity is the same entity in data from different data sources. An entity is an object or thing with a unique identifier that can be distinguished from other objects. In the context of data management, entities are often used to represent real-world objects or concepts in databases or knowledge graphs.

[0077] Optionally, one or more entities described by data in different data sources or data systems may be extracted, and entities corresponding to data associations in different data sources or data systems may be used as data entities.

[0078] S103: The data center updates the master data based on the data entity.

[0079] Master data refers to the recognized shared data among multiple data sources.

[0080] Optionally, the corresponding data of the data entity may include first data and second data, and the first data and the second data come from different data sources. The first data includes a first field, and the second data includes a second field. Specifically, the data middle station updates the master data according to the data entity, which may include: the data middle station determines whether the first data and the second data are related data. If so, the data middle station determines whether the first field and the second field are the same. If they are the same, the data middle station stores the first field. If they are not the same, the data middle station determines and stores the first master data based on the first field and the second field.

[0081] Optionally, the master data model has standard coding and personalized coding.

[0082] Specifically, determining and storing the first master data based on the first field and the second field includes: the data center generates a third field based on the standard code, and the third field serves as the identity of the first master data. The data center processes the first field and the second field based on the personalized code to obtain corresponding fourth and fifth fields to distinguish and identify the data sources of the first and second fields. The data center obtains and stores the first master data based on the third field, the fourth field, and the fifth field.

[0083] Optionally, before the data center determines whether the first data and the second data are related data, the process further includes: determining whether the first data and the second master data are related data, where the second master data is existing master data in the data center. If so, determining whether the first field exists in the second master data. If not, not saving the first field. If so, determining whether the corresponding field value in the second master data is equal to the field value corresponding to the first field. If not, updating the corresponding field value in the second master data.

[0084] Optionally, the method may further include: after the data middle platform receives the display instruction, dynamically combining the stored master data according to the display instruction and displaying it.

[0085] Optionally, the method may further include: after the data middle platform receives the sharing instruction, the data middle platform extracts the target master data to be shared according to the sharing instruction, and outputs the target master data through an external interface.

[0086] The embodiment of this application uses the data center to extract consistent data entities from different data sources, store and share the associated data of the same data entity in different data sources, and form a shared data domain between different data sources based on the data center, realizing the automatic update of master data. This can save manual maintenance costs and efficiently realize the interconnection and interoperability of data from multiple data sources, thereby meeting the usage requirements at the data application level.

[0087] The above describes the data processing method based on the data center provided by the embodiment of the present application. Figure 2 The method is illustrated by using a specific application scenario in FIG.

[0088] See also Figure 3 , Figure 3 A schematic diagram of a data processing method based on a data middle platform provided in an embodiment of the present application, the method comprising:

[0089] S201: Input master data c into the data platform through the entry platform, so that the data platform generates rules.

[0090] The rules are used to complete the processing of master data. For example, the master data c may be personnel master data, and the rules may include rules for personnel information changes, personnel transfers, and department field changes.

[0091] Optionally, the entry platform is a data entry component of the data platform. The entry platform can store the master data c in the data lake of the data platform to achieve shared storage and use of the master data.

[0092] S202: Receive data A, data B, and data N through the offline task component of the data platform.

[0093] Data A, Data B, and Data N are data from different data systems or data sources. Data A includes fields a, b, and c. Data B includes field c and other fields. Data N includes field a and other fields.

[0094] S203: Update the master data c according to the associated fields in data A, data B, and data N.

[0095] Among them, data A and data B are related data, and the related field is field c. Data A and data N are related data, and the related field is field a.

[0096] Optionally, the master data model in the data platform has both standard and personalized encodings. Standard encoding is the preferred encoding method for associating master data with all relational data. Personalized encoding is the encoding method used to associate data within a single data system or data source with master data. In the process of creating new systems and upgrading and renovating systems, the standard encoding should be gradually replaced with personalized encoding, ultimately achieving universal master data access using the standard encoding for master data association across all systems.

[0097] Optionally, two encoding methods in the master data model may be used to store associated fields in data A, data B, and data N into master data c.

[0098] For example, if data A and data B are related data and the related field is field c, if field c in data A is the same as field c in data B, both are name, age, or other fields, and the field values ​​are the same, field c is used as the standard code. If field c in data A is different from field c in data B, for example, if field c is the id field, the value of field c in data A is a1, and the value of field c in data B is b9, a new value, such as sf0001, is generated using the standard code to identify the master data, and personalized codes are used to generate the Aid and Bid fields to store field values ​​in different data systems.

[0099] like Figure 2 As shown, the fields in the final master data c include: master ID, data A field c, data B field c, data N field a, and data A field a. Among them, the master ID is a standard code, and data A field c, data B field c, data N field a, and data A field a are personalized codes.

[0100] S204: The updated master data c is output to the outside through the API component of the data platform.

[0101] Optionally, in the subsequent use of the master data, the standard encoding is preferably used. Therefore, when the master data c is output externally through the API, the master ID is used as the identifier of the master data c.

[0102] S205: Display master data through the asset management component of the data platform.

[0103] Optionally, the asset management component can dynamically combine master data in the data lake in the data platform for display.

[0104] Optionally, step S204 and step S205 may be implemented after receiving a specific instruction, which is not specifically limited in the embodiment of the present application.

[0105] The embodiment of the present application utilizes the data lake in the data center to store shared data from different data systems or data sources, and continuously expands the amount of shared data through programs, thereby realizing automated iterative updates of the master data program stored in the data center, improving production efficiency, establishing connections between all data, and facilitating shared use at the data application level.

[0106] The above are some specific implementations of the data processing method based on the data middle platform provided in the embodiment of this application. Based on this, this application also provides a corresponding system. The system provided by the embodiment of this application will be introduced from the perspective of functional modularization.

[0107] See also Figure 4 The structure diagram of the data processing system 300 based on the data middle platform is shown, and the system 300 includes an external data receiving unit 301, a data entity extraction unit 302 and a master data updating unit 303.

[0108] The external data receiving unit 301 is configured to receive external data, where the external data includes data from at least one data source.

[0109] A data entity extraction unit 302 is used to extract data entities from external data, where the data entity is the same entity in data from different data sources;

[0110] The master data updating unit 303 is used to update the master data according to the data entity. The master data represents the recognized shared data among multiple data sources.

[0111] Optionally, the system further includes: a standard master data entry module and a master data model determination unit.

[0112] The standard master data entry module is used to receive standard master data in order to determine the data template. The standard master data is pre-compiled related data from different data sources.

[0113] The master data model determining unit is used to determine the master data model according to the data template. The master data model is used to manage the basic attribute information of the master data and standardize the fields, attributes and processing rules of the master data.

[0114] The “first” and “second” in the names such as “first” and “second” (if any) mentioned in the embodiments of this application are only used as name identifiers and do not mean the first or second in order.

[0115] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment methods can be implemented by means of software plus a general hardware platform. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product. The computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or certain parts of the embodiments of the present application.

[0116] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiment. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Ordinary technicians in this field can understand and implement it without making any creative efforts.

[0117] The above description is merely an exemplary embodiment of the present application and is not intended to limit the scope of protection of the present application.

Claims

1. A data processing method based on a data middle station, characterized in that: The method comprises: The data center receives external data, where the external data includes data from at least one data source; The data middle platform extracts a data entity from the external data, where the data entity is the same entity in data from different data sources; The data middle platform updates master data according to the data entity, where the master data represents recognized shared data among multiple data sources; Before the data center receives external data, the method further includes: The data center receives standard master data in order to determine the data template. The standard master data is pre-compiled related data from different data sources. The data center determines a master data model based on the data template, and the master data model is used to manage the basic attribute information of the master data and standardize the fields, attributes and processing rules of the master data; The corresponding data of the data entity includes first data and second data, the first data includes a first field, the second data includes a second field, the first data and the second data come from different data sources, and the data middle platform updates the master data according to the data entity, including: The data center determines whether the first data and the second data are related data; If so, the data center determines whether the first field and the second field are the same; If they are the same, the data center stores the first field; If they are not the same, the data center determines and stores the first master data according to the first field and the second field; The master data model has a standard code and a personalized code, and determining and storing the first master data according to the first field and the second field includes: The data center generates a third field according to the standard code, and the third field serves as the identity identification of the first master data; The data center processes the first field and the second field according to the personality code to obtain corresponding fourth and fifth fields to distinguish and identify the data sources from which the first field and the second field come; The data middle platform obtains and stores the first master data based on the third field, the fourth field and the fifth field.

2. The method according to claim 1, characterized in that Before the data center determines whether the first data and the second data are related data, the method further includes: The data center determines whether the first data and the second master data are related data, and the second master data is the existing master data in the data center; If so, the data center determines whether the second master data contains the first field; If it does not exist, the first field is not saved; If so, the data center determines whether the corresponding field value in the second master data is equal to the field value corresponding to the first field; If they are not equal, update the corresponding field value in the second master data.

3. The method according to claim 1, characterized in that: The method further comprises: The data center receives a display instruction; The data middle platform dynamically combines the master data according to the display instruction and displays it.

4. The method according to claim 1, characterized in that: The method further comprises: The data middle station receives the sharing instruction; The data middle platform extracts the target master data to be shared according to the sharing instruction; The data middle platform outputs the target master data through an external interface.

5. A data middle platform, characterized in that: The data middle platform includes data entry components, offline task components, asset components, data lake and interface components; The data entry component is used to enter standard master data. The standard master data is used to determine the master data model so as to standardize the basic attribute information of the master data and the fields, attributes and processing rules of the master data. The master data is recognized and shared among multiple data sources. The offline task component is used to receive external data, extract data entities from the external data, and update the master data according to the data entities; The corresponding data of the data entity includes first data and second data, the first data includes a first field, the second data includes a second field, the first data and the second data come from different data sources, and the updating of the master data according to the data entity is specifically used to: determining whether the first data and the second data are associated data; If so, determining whether the first field and the second field are the same; If they are the same, store the first field; If they are not the same, determining and storing the first master data according to the first field and the second field; The master data model has a standard code and a personalized code, and the first master data is determined and stored according to the first field and the second field, specifically for: generating a third field according to the standard code, the third field serving as the identity identification of the first master data; Processing the first field and the second field according to the personality code to obtain corresponding fourth and fifth fields to distinguish and identify data sources from which the first field and the second field come; According to the third field, the fourth field and the fifth field, the first master data is obtained and stored; The data lake is used to realize shared storage of the master data; The asset component is used to display the master data in the data lake; The interface component is used to realize the sharing and output of the master data.

6. A data processing system based on a data middle platform, characterized in that: The system is applied to the data middle platform according to claim 4, and the system includes: an external data receiving unit, a data entity extraction unit and a master data updating unit; The external data receiving unit is configured to receive external data, wherein the external data includes data from at least one data source; The data entity extraction unit is configured to extract a data entity from the external data, wherein the data entity is the same entity in data from different data sources; The master data updating unit is configured to update master data according to the data entity, wherein the master data represents recognized shared data among multiple data sources; The system further comprises: a standard master data entry module and a master data model determination unit; The standard master data entry module is used to receive standard master data in order to determine the data template. The standard master data is pre-compiled related data from different data sources; The master data model determination unit is used to determine a master data model according to the data template, wherein the master data model is used to manage basic attribute information of the master data and standardize fields, attributes and processing rules of the master data; The corresponding data of the data entity includes first data and second data, the first data includes a first field, the second data includes a second field, the first data and the second data come from different data sources, and the data middle platform updates the master data according to the data entity, including: The data center determines whether the first data and the second data are related data; If so, the data center determines whether the first field and the second field are the same; If they are the same, the data center stores the first field; If they are not the same, the data center determines and stores the first master data according to the first field and the second field; The master data model has a standard code and a personalized code, and determining and storing the first master data according to the first field and the second field includes: The data center generates a third field according to the standard code, and the third field serves as the identity identification of the first master data; The data center processes the first field and the second field according to the personality code to obtain corresponding fourth and fifth fields to distinguish and identify the data sources from which the first field and the second field come; The data middle platform obtains and stores the first master data based on the third field, the fourth field and the fifth field.

Images