A data mart management method, apparatus, electronic device, and storage medium

Abstract

This application discloses a data mart management method, apparatus, electronic device, and storage medium. The method includes: receiving data to be processed from a source system; storing the data to be processed in a predetermined table structure to obtain a standard-format data table; processing the data to be processed sequentially at multiple levels according to a hierarchical order based on the standard-format data table to obtain a processing result table of the data to be processed; and sending the processing result table of the data to be processed to a downstream system. The method of this application can standardize the creation and storage format of data tables, directly process the standardized data, reduce the application and subsequent maintenance difficulty of data tables in data marts, and improve the management efficiency of data marts.

Description

Technical Field

[0001] This application relates to the field of big data processing, and in particular to a data mart management method, apparatus, electronic device, and storage medium. Background Technology

[0002] A data mart, also known as a data marketplace, caters to the needs of specific departments or users by storing data in a multi-dimensional manner. This includes defining dimensions, metrics to be calculated, and the hierarchy of dimensions. Data marts can generate data cubes tailored to decision analysis requirements. The goal of a data mart is to enable downstream systems to quickly, accurately, and securely extract target data. A data mart needs to possess reasonable data storage, efficient data processing, strict access control, and rapid problem-solving capabilities to provide high-quality services to downstream systems.

[0003] Existing data marts are relatively small in scale, and their management primarily relies on hierarchical construction using Hadoop clusters. This approach lacks standardized methods for creating and storing data tables. As the number of source systems increases, the varying data formats and encodings provided by different source systems complicate the application and maintenance of data tables within the data mart, thus reducing management efficiency. Summary of the Invention

[0004] This application provides a data mart management method, apparatus, electronic device, and storage medium, which can improve data processing efficiency, standardize the creation and storage format of data tables in data marts, reduce the application and subsequent maintenance difficulty of data tables in data marts, and improve the management efficiency of data marts.

[0005] In a first aspect, embodiments of this application provide a data mart management method, the method comprising:

[0006] Receive data to be processed from the source system;

[0007] The data to be processed is stored in a predetermined table structure to obtain a data table in a standard format;

[0008] Based on the standard format data table, the data to be processed is processed sequentially at multiple levels according to hierarchical order to obtain the processing result table of the data to be processed; and the processing result table of the data to be processed is sent to the downstream system.

[0009] Secondly, embodiments of this application also provide a data mart management device, the device comprising:

[0010] The receiving module is used to receive data to be processed sent by the source system;

[0011] The storage module is used to store the data to be processed into a predetermined table structure to obtain a data table in a standard format;

[0012] The processing module is used to process the data to be processed sequentially at multiple levels according to the hierarchical order based on the data table in the standard format, to obtain a processing result table of the data to be processed; and to send the processing result table of the data to be processed to the downstream system.

[0013] Thirdly, embodiments of this application also provide an electronic device, the electronic device comprising:

[0014] One or more processors;

[0015] Memory, used to store one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the data mart management method provided in any embodiment of this application.

[0017] Fourthly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data mart management method as provided in any embodiment of this application.

[0018] In this embodiment, the system receives data to be processed from a source system; stores the data to be processed in a predetermined table structure to obtain a standard format data table; processes the data to be processed sequentially at multiple levels according to the hierarchical order based on the standard format data table to obtain a processing result table; and sends the processing result table to a downstream system. In other words, this embodiment can standardize the creation method and storage format of data tables, directly process the standardized data, reduce the application and subsequent maintenance difficulty of data tables in data marts, and improve the management efficiency of data marts. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a first flowchart of the data mart management method provided in the embodiments of this application;

[0021] Figure 2 This is a second flowchart of the data mart management method provided in the embodiments of this application;

[0022] Figure 3 This is a flowchart of the data tracing method provided in the embodiments of this application;

[0023] Figure 4 This is a schematic diagram of the data mart management device provided in the embodiments of this application;

[0024] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present application, not the entire structure.

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] Figure 1This is a first flowchart of the data mart management method provided in this embodiment. The method of this embodiment can improve data processing efficiency, standardize the creation and storage format of data tables in the data mart, reduce the application and subsequent maintenance difficulty of data tables in the data mart, and improve the management efficiency of the data mart. This method can be executed by the data mart management device in this embodiment, which can be integrated into an electronic device, which can be a server. The method can be implemented using software and / or hardware. The data mart management method provided in this embodiment specifically includes the following steps:

[0029] Step 101: Receive the data to be processed sent by the source system.

[0030] The source system is the application system that provides raw data to the data mart. The data to be processed is the raw data that needs to be managed, determined according to user requirements. Specifically, the source system stores data resources from different sources and in different formats, and the data mart can receive data to be processed from the source system in various formats and from different sources. After receiving the diverse data to be processed, the data mart can clean and process it according to specific business scenarios, providing basic business-oriented data to meet the personalized needs of professional user groups.

[0031] Step 102: Store the data to be processed into a predetermined table structure to obtain a standard format data table.

[0032] The table structure is a predefined data format used by the server to uniformly manage the data to be processed. A standard format data table is a data table consisting of data to be processed, and it has the same predefined table structure format.

[0033] In one optional implementation, the server can receive an application system selection instruction sent by the user and determine the target application system selected by the user based on the system selection instruction. After determining the target application system, the server can obtain the system information of the target application system, construct a physical model of the target application system based on the various fields of the system information, and further determine the table structure based on the physical model.

[0034] In one optional implementation, before the server stores the data to be processed into a predetermined table structure, it can receive the user's data entry method, which includes active entry or passive entry. When the data entry method is active entry, the server obtains the data entry application form and reviews it. When the review result of the data entry application form is approved, the data to be processed is stored into the predetermined table structure to obtain a standard format data table.

[0035] Step 103: Based on the standard format data table, process the data to be processed sequentially at multiple levels according to the hierarchical order to obtain the processing result table of the data to be processed; and send the processing result table of the data to be processed to the downstream system.

[0036] Downstream systems are those that directly provide services to the data mart, i.e., systems that submit data processing and storage requests to the data mart. The processing result table is a data table obtained by the server processing a standard-format data table based on the requests from the downstream systems.

[0037] In one optional implementation, after obtaining a standard-formatted data table, the server stores the standard-formatted data table in the raw data layer according to a preset lifecycle and sends the standard-formatted data table to the basic data layer. Based on pre-configured data management information, the server processes the standard-formatted data table in the basic data layer to obtain a basic data table. The basic data table is then sent to the general model layer. The general model layer determines the personalized fields of the basic data table and performs personalized filtering on the data in the basic data table based on these personalized fields to obtain a candidate data table. The candidate data table is then sent to the interface layer. After determining the candidate data table, the server can send pre-determined test data to downstream systems, enabling the downstream systems to test the candidate data table based on the test data. The server receives the test results from the downstream systems. If the test result is successful, the candidate data table is determined as the processing result table. If the test result is unsuccessful, a change instruction is sent to the downstream systems, causing the downstream systems to respond to the change instruction and modify the requirement information until the test result is successful, at which point the candidate data table is determined as the processing result table. The processing result table is then sent to the downstream systems.

[0038] In this embodiment, the technical solution involves receiving data to be processed from a source system; storing the data to be processed in a predetermined table structure to obtain a standard-format data table; processing the data to be processed sequentially at multiple levels according to the hierarchical order based on the standard-format data table to obtain a processing result table; and sending the processing result table to the downstream system. This embodiment's technical solution can standardize the creation and storage format of data tables, directly process the standardized data, reduce the application and subsequent maintenance difficulty of data tables in data marts, and improve the management efficiency of data marts.

[0039] Figure 2 This is a second flowchart of the data mart management method provided in the embodiments of this application, such as... Figure 2 As shown, the method mainly includes the following steps:

[0040] Step 201: Receive the data to be processed sent by the source system.

[0041] The source system is the application system that provides the raw data to the data mart. The data to be processed is the raw data that needs to be managed, as determined by user requirements.

[0042] In one alternative implementation, after receiving the data to be processed, the server also needs to perform the following steps A1-A3:

[0043] Step A1: Receive the data to be processed sent by the user and enter it into the database.

[0044] The data import methods include active import and passive import. Passive import refers to the import method in which functional data is automatically loaded into the original data layer (Operational Data Store, ODS) on a regular basis, such as log table data and core system master table data. Active import refers to the import method initiated by the source system, which automatically loads the data into the original data layer according to a certain process.

[0045] In one optional implementation, the server can receive the data to be processed from the source system and the method for storing the data to be processed sent by the user, and store the data to be processed in the database according to the method for storing the data to be processed.

[0046] Step A2: When the data to be processed is entered into the database through active entry, obtain the data entry application form and review it.

[0047] The data entry application form, provided by the source system, includes information such as the reason for the application, the data lifecycle, and rule configurations. Specifically, when the data to be processed is entered into the database through proactive entry, the source system provides the application form to the server. After receiving the application form from the source system, the server forwards it to the reviewers in the data management department. The reviewers will assess the reason for entry, data lifecycle, and other factors to obtain the review results. When the data to be processed is entered into the database through reactive entry, the server can directly classify and store the data.

[0048] Step A3: When the review result of the inbound application form is "approved", execute the step of storing the data to be processed into a predetermined table structure.

[0049] Specifically, when the review result of the inbound application form is "approved," it means that the server can store the data to be processed into a predetermined table structure. In an optional implementation, when the review result of the inbound application form is "approved," the source system can name the table composed of the data to be processed according to a predetermined table naming format. Further, the named table of data to be processed is stored into the predetermined table structure according to a predetermined process.

[0050] For example, the source system names the table containing the data to be processed according to the format "Province-System Name-File Name-Full Data Identifier-Data Date". For example, "00-ORISYS-TABLE-2G-20220101", where "00" represents the whole country, "ORISYS" is the name of the source system, "TABLE" is the file name, "2G" represents the full data, and "20220101" represents the date.

[0051] In the above steps, the server can store the data to be processed in the database according to different storage methods, which improves the efficiency of data mart management.

[0052] Step 202: Store the data to be processed into a predetermined table structure to obtain a standard format data table.

[0053] Step 203: Store standard format data tables in the raw data layer according to the preset lifecycle, and send the standard format data tables to the basic data layer.

[0054] The lifecycle refers to the duration for which the server stores data to be processed. Preset lifecycles are typically 1 year, 3 years, 5 years, or permanent storage. When the storage period for data to be processed exceeds the preset lifecycle, the server can send the data to the user. The user will then determine whether the data can be permanently deleted based on its current value. If the user determines that the data can be permanently deleted, the server receives the deletion command and permanently deletes the data. Otherwise, the server can continue storing the data.

[0055] The raw data layer, also known as the operational data storage layer, is a subject-oriented, integrated, variable, and detailed collection of data reflecting current data values. It serves to meet the comprehensive, integrated, and operational processing needs of enterprises. The raw data layer typically includes two types of data: one for storing data currently being loaded, and the other for storing processed historical data. The raw data layer can act as an isolation layer between business systems and data marts. Generally, data marts have very complex data sources, stored in different geographical locations, different databases, and different applications. Extracting data from these source systems is not an easy task. Therefore, the raw data layer stores data directly extracted from the source systems, ensuring that the data structure and logical relationships between data are largely consistent with the downstream systems. This significantly reduces the complexity of data transformation during the extraction process and lowers the query pressure on downstream systems.

[0056] In this embodiment of the solution, optionally, generating the table structure includes the following steps B1-B2:

[0057] Step B1: Receive the application system selection instruction sent by the user, and determine the target application system in response to the application system selection instruction.

[0058] In this context, an application system can be a business system, a function within a business system, or even a specific page within a business system. Specifically, when a user triggers an application system, the server can receive the user's application system selection instruction and determine the target application system from among multiple application systems based on that instruction.

[0059] Step B2: Obtain the system information of the target application system, construct a physical model of the target application system based on the various fields of the system information, and determine the table structure based on the physical model.

[0060] The system information includes the fields of the target application system and the implemented business logic. The physical model refers to a model that provides the basic elements needed for the initial design of the system, as well as the relationships between these elements.

[0061] In one optional implementation, after obtaining the system information of the target application system, the server first establishes a logical model based on the individual characters in the system information. The logical model includes at least: business modules, business objects, entity names, full English names, Chinese names, connection types, and descriptive information. After obtaining the logical model, a corresponding set of fields is added to it. Each field includes at least: the field's English name, sequence number, whether it is a primary key, whether it is nullable, whether it is an auto-incrementing column, its business meaning, data type, data length, and whether it is sensitive information. After determining the logical model, a suitable database is selected based on specific business requirements, and the logical model is converted into a physical model according to the database's format.

[0062] In this embodiment, optionally, after obtaining the physical model, the server can send it to the project team's reviewers. The reviewers can evaluate the physical model, including checking for omissions, accuracy of sensitive information labeling, and completeness of additional information, among other things. If the review fails, the server can maintain the logical model based on the reviewers' feedback. For example, if a user needs to modify a field, the server can use field search to obtain all logical models in the application system containing that field; modifying the same field only needs to be done once to automatically update other models. If the review passes, the physical model is sent to the project management personnel. The project management personnel can assess the impact of the physical model on other related systems, including whether it involves core tables and the number of tables. If the assessment fails, the logical model continues to be maintained until the assessment passes. If the assessment passes, the physical model is designated as a table structure.

[0063] In the above steps, the server can use fields as the smallest management unit and standardize the data table according to the field level, which reduces the difficulty of maintaining the data mart in the later stage.

[0064] Step 204: Based on the pre-configured data management information, process the standard format data table in the basic data layer to obtain the basic data table; send the basic data table to the general model layer.

[0065] Data management information includes cleaning rules and validation rules. Data processing includes data cleaning and data validation. The BaseDataStorage (BDS) layer is responsible for cleaning and validating the standard-format data tables stored in the original data layer. Data cleaning is a process of re-examining and validating data, aiming to remove duplicate information, correct existing errors, and provide data consistency.

[0066] In one optional implementation, the server can perform data cleaning on the standard-format data table according to the cleaning rules configured by the technical personnel of the source system when adding the data, removing outliers. Then, based on business requirements and judgment conditions such as thresholds configured by the technical personnel, the standard-format data table is validated to obtain a basic data table that meets the business scenario.

[0067] Step 205: Determine the personalized fields of the basic data table in the general model layer, perform personalized filtering on the data in the basic data table based on the personalized fields, and obtain the candidate data table; send the candidate data table to the interface layer.

[0068] Personalized fields are fields determined by the server based on the needs of downstream systems. The General Data Storage (GDS) layer is used to process the basic data tables obtained from the basic data layer according to the needs of downstream systems and to visualize the processing process, which includes data filtering, concatenation, summarization, and special value transformation.

[0069] Specifically, data filtering includes filtering by field and by value. For example, filtering data from a basic data table to select personalized fields or values ​​that are specific to a personalized field. The purpose of concatenating the basic data tables is to create a wide table, thereby obtaining a common data table. Summarizing the basic data tables includes: summarizing data information level by level according to organizational hierarchy, currency, business, etc. Special transformations include converting sliced ​​tables to zippered tables to improve the storage and retrieval efficiency of the data tables.

[0070] In this solution, during data processing at the general model layer, the processed tables are categorized into three types based on their functions: temporary tables, intermediate tables, and candidate data tables. These three types of tables are named during processing, allowing the server to distinguish them using table name prefixes. The server will automatically clean up temporary tables daily by default, while the cleaning schedule for intermediate tables is determined by technical personnel and can be freely set. Candidate data tables are then sent to the interface layer (Interface, I).

[0071] Step 206: Obtain the processing result table at the interface layer based on the candidate data table.

[0072] The interface layer, which is also the output layer of the data mart, is primarily responsible for providing services to downstream systems. Specifically, after the server identifies candidate data tables, downstream systems request the data mart's service interface from the server via a work order. The server can then send the purpose, frequency, timeliness, and priority of the data included in the work order to reviewers. If the review fails, the work order is returned to the downstream system. If approved, the server receives the candidate data table sent by the general model layer. Upon receiving the candidate data table, the interface layer can also filter common field data based on the personalized needs of the downstream systems.

[0073] After obtaining the candidate data table, the server can determine whether the downstream system needs test data based on the downstream system's requirements. If the downstream system needs test data, then the following steps C1-C3 are executed:

[0074] Step C1: Send predetermined test data to the downstream system, enabling the downstream system to test the candidate data table based on the test data.

[0075] When a downstream system needs test data, it can request it from the server. The server can then forward this request to an auditor. If the requested test data table structure differs from the production environment's table structure, the auditor can return the request to the downstream system via the server. If the request is approved, the server can send the anonymized test data to the downstream system for testing and obtain the test results.

[0076] Step C2: Receive the test results sent by the downstream system. When the test result is passed, determine the candidate data table as the processing result table.

[0077] If the test result is that the test passes, it means that the candidate data table meets the requirements of the downstream system. Furthermore, the candidate data table can be determined as the processing result table.

[0078] Step C3: When the test result is unsuccessful, a change instruction is sent to the downstream system, causing the downstream system to respond to the change instruction and modify the requirement information.

[0079] If the test result is "test failed", it means that the candidate data table does not meet the requirements of the downstream system. The server can then re-formulate the data filtering rules based on the test result until the test result is "test passed".

[0080] In the above steps, when the downstream system needs to test the processing result table, the server can send test data to the downstream system and obtain a processing result table that meets the needs of the downstream system based on the test results, thereby improving the user experience.

[0081] Step 207: Send the processing result table to the downstream system.

[0082] The data mart management method provided in this application embodiment receives data to be processed from a source system. The data to be processed is stored in a pre-determined table structure to obtain a standard-format data table. The standard-format data table is stored in the original data layer according to a preset lifecycle and then sent to the basic data layer. Based on pre-configured data management information, the standard-format data table is processed in the basic data layer to obtain a basic data table; this basic data table is then sent to the general model layer. The general model layer determines the personalized fields of the basic data table and performs personalized filtering on the data in the basic data table based on these personalized fields to obtain a candidate data table; this candidate data table is then sent to the interface layer. Based on the candidate data table, a processing result table is obtained at the interface layer. The processing result table is then sent to the downstream system. The technical solution of this embodiment can standardize the creation method and storage format of data tables, and directly process the standardized data hierarchically according to the hierarchical order, reducing the application and subsequent maintenance difficulty of data tables in the data mart and improving the management efficiency of the data mart.

[0083] Figure 3 This is a flowchart of the data tracing method provided in the embodiments of this application, such as... Figure 3 As shown, the method mainly includes the following steps:

[0084] Step 301: Receive the data tracing instruction sent by the downstream system.

[0085] Data tracing instructions include tracing conditions. Tracing conditions are the names of the tracing targets determined by the user based on their needs, such as table names, field names, and job names. Data tracing represents the origin of data and its movement between databases. It describes the original data before the target data is derived and the evolution process, used to record workflow evolution, annotation information, and experimental procedures. Data tracing instructions are commands sent by users to the server through downstream systems to retrieve data.

[0086] In one alternative implementation, when a user needs to modify or query a piece of data, a process, a data table, or a field in the data mart, the user can send a data tracing instruction to the server through a downstream system, and the server can receive the data tracing instruction sent by the downstream system.

[0087] Step 302: In response to the data tracing instruction, trace the processing result table based on the tracing conditions and the names of each data link in the pre-determined processing result table to obtain the tracing result.

[0088] Specifically, the phased processing of data is encapsulated into individual jobs. Multiple jobs exist in each of the raw data layer, basic data layer, general model layer, and interface layer. Jobs within the same layer are interdependent and linked together to form job chains. A job chain encompasses the complete data processing process within a particular layer. By scheduling job chains, dependencies between jobs can be configured, forming data links. Each data link has a corresponding name, which includes the key fields of that data link. When a user needs to extend the length of a field in the processing result table, the data mart must also synchronously extend the length of the relevant field; otherwise, data inaccuracies and missing data will occur during data loading. Furthermore, fields may be renamed during processing, making it impossible to retrieve all data tables requiring investigation and updates by field name alone. However, the processing result table can be traced back to its source by the names of the various data links within it.

[0089] In this embodiment, before tracing the processing result table based on tracing conditions and the names of each data link in the pre-determined processing result table, it is necessary to determine the name of each data link. Determining the name of the data link includes: obtaining the generation information of the processing result table, and determining the name of each data link in the processing result table based on the generation information of the processing result table and the pre-determined naming format.

[0090] The generated information includes at least the data source information for the processing result table, the application level of the processing result table, the name of the source system, the time when the processing result table was obtained, and the job number of the processing result table. Specifically, each node in the data link is defined in the format "Application Name:Job Chain Name:Job Name", and the data link is generated in reverse order by connecting nodes with ">", ending with "|!|!|!". The application name is named "Application System_Processing Level", the job chain name is named "Processing Level_Source System_Date", and the job name is named "Processing Level_Data Table Name_Job Number".

[0091] For example, the user needs to increase the length of field 'a' in the processing result table, and the name of the data link in the processing result table is "SYS_I:I_ORISYS_20220101:I_TABLE_OUT3_00>SYS_GDS:GDS_ORISYS_20220101:GDS_TABLE_OUT2_00>SYS_BDS:BDS_ORISYS_20220101:BDS_TABLE_OUT1_00>SYS_ODS:ODS_ORISYS_20220101:ODS_TABLE_00>|!|!|!". Assuming this field is loaded into the data table TABLE of the original data layer ODS, the content before the keyword ":ODS_TABLE_00>|!|!|!" in the job chain is extracted using regular expressions. This indicates that the data table TABLE involves data links to other levels. Then, the content without ":" between ":BDS_" and "_00>" is extracted to obtain the basic data table in the basic data layer BDS. Similarly, the names of data tables involved in other levels can be obtained.

[0092] This data tracing method can also be used to troubleshoot changes in downstream system requirements. For example, if a data mart needs to change its data source and reselect the source system, the entire data chain needs to be adjusted. If the processing result table provided to the interface layer I of the downstream system is TABLE_OUT3, the content between "I_TABLE_OUT3_00>" and ">|!|!|!" in the data chain can be extracted using regular expressions to obtain the processing chain involved in the processing result table TABLE_OUT3. Then, the content without ":" between ":GDS_" and "_00>" can be extracted to obtain the table name of the candidate processing table in the general model layer. Similarly, the data table names involved in other layers can be obtained.

[0093] In the above steps, the server can standardize the naming of data links, effectively improve the management of data links, and quickly trace the source of data when anomalies occur, thereby improving the data management efficiency of the data mart.

[0094] Step 303: Send the traceability results to the downstream system.

[0095] The data tracing method provided in this embodiment receives a data tracing instruction sent by a downstream system. The data tracing instruction includes tracing conditions. In response to the data tracing instruction, the processing result table is traced based on the tracing conditions and the names of each data link in a pre-determined processing result table to obtain the tracing result. The tracing result is then sent to the downstream system. This embodiment's technical solution effectively improves data link management by standardizing data link naming and enables rapid data tracing in case of anomalies, thus improving the accuracy and efficiency of data tracing. When updates to upstream system data or changes in downstream system requirements affect the data mart, the data mart can quickly respond and complete maintenance, further improving the user experience.

[0096] Figure 4 This is a schematic diagram of the structure of the data mart management device provided in the embodiments of this application. The embodiments of this application provide a data mart management device, the device comprising:

[0097] The receiving module 401 is used to receive data to be processed sent by the source system;

[0098] Storage module 402 is used to store the data to be processed into a predetermined table structure to obtain a data table in a standard format;

[0099] The processing module 403 is used to process the data to be processed in a hierarchical order at multiple levels based on the data table in the standard format, to obtain a processing result table of the data to be processed; and to send the processing result table of the data to be processed to the downstream system.

[0100] Optionally, before receiving the data to be processed from the source system, the storage module 402 is specifically used to: receive an application system selection instruction sent by the user, and determine the target application system in response to the application system selection instruction;

[0101] Obtain the system information of the target application system;

[0102] A physical model of the target application system is constructed based on each field of the system information, and the table structure is determined based on the physical model.

[0103] After receiving the data to be processed sent by the source system, the processing module 403 is specifically used to: receive the data to be processed sent by the user and specify the data entry method, wherein the data entry method includes active data entry or passive data entry.

[0104] When the data to be processed is entered into the database via active entry, the data entry application form is obtained and reviewed.

[0105] When the review result of the inbound application form is "approved", the step of storing the data to be processed into a predetermined table structure is executed.

[0106] Optionally, the hierarchy includes a raw data layer, a basic data layer, a general model layer, and an interface layer. The processing module 403 is further configured to: store the data table in the standard format in the raw data layer according to a preset lifecycle, and send the data table in the standard format to the basic data layer.

[0107] Based on pre-configured data management information, the data table in the standard format is processed in the basic data layer to obtain a basic data table; the basic data table is then sent to the general model layer.

[0108] The personalized fields of the basic data table are determined in the general model layer, and the data in the basic data table is filtered in a personalized way according to the personalized fields to obtain a candidate data table; the candidate data table is sent to the interface layer.

[0109] Optionally, the processing module 403 is further configured to: obtain the processing result table at the interface layer based on the candidate data table.

[0110] Send predetermined test data to the downstream system, so that the downstream system can test the candidate data table based on the test data;

[0111] Receive the test results sent by the downstream system, and when the test results are passed, determine the candidate data table as the processing result table;

[0112] When the test result is unsuccessful, a change instruction is sent to the downstream system, causing the downstream system to respond by changing the requirement information in accordance with the change instruction.

[0113] Optionally, the processing module 403 is further configured to: receive a data tracing instruction sent by the downstream system; wherein the data tracing instruction includes tracing conditions;

[0114] In response to the data tracing instruction, the processing result table is traced based on the tracing conditions and the names of each data link in the pre-determined processing result table to obtain the tracing result;

[0115] The tracing results are sent to the downstream system.

[0116] Optionally, before sending the processing result table to the downstream system, the processing module 403 is further configured to: obtain the generation information of the processing result table, wherein the generation information includes at least the data source information of the processing result table, the application level of the processing result table, the name of the source system, the time when the processing result table was obtained, and the job number of the processing result table.

[0117] The names of each data link in the processing result table are determined based on the generation information of the processing result table and the predetermined naming format.

[0118] The data mart management device provided in this application embodiment can execute the data mart management method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of executing the method.

[0119] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, with reference to... Figure 5 It shows a schematic diagram of the structure of a computer system 12 suitable for implementing electronic devices according to embodiments of the present application. Figure 5 The illustrated electronic device is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this application. Components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and a bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0120] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (SA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced Interface SA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PC Interface) bus.

[0121] Electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0122] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 5Not shown; usually referred to as a "hard drive"). Although Figure 5 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0123] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of this application.

[0124] Electronic device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with electronic device 12, and / or with any device that enables electronic device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (interface / O) interface 22. Furthermore, in this embodiment, electronic device 12 and display 24 are not separate entities, but are embedded in a mirror, so that when the display surface of display 24 is not displayed, the display surface of display 24 and the mirror surface visually blend together. Moreover, electronic device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of electronic device 12 via bus 18. It should be understood that although... Figure 5 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RA interface D systems, tape drives, and data backup storage systems.

[0125] The processing unit 16 executes various functional applications and data mart management by running programs stored in the system memory 28. For example, it implements a data mart management method provided in the embodiments of this application: acquiring the current value of the disconnector at each time point in the current cycle; sending the current value of the disconnector at each time point and the time point to a pre-trained neural network model to obtain the disconnector's open / closed state in the current cycle; and determining whether the disconnector's open / closed state is normal based on the open / closed state.

[0126] This application provides a computer-readable storage medium storing a computer program. When executed by a processor, the program implements a data mart management method as provided in all embodiments of this application: acquiring the current value of a disconnector at various time points within the current cycle; sending the current value of the disconnector at each time point and the time points themselves to a pre-trained neural network model to obtain the disconnector's open / closed state in the current cycle; and determining whether the disconnector's open / closed state is normal based on the open / closed state. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of a computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0127] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0128] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0129] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0130] Note that the above description is merely a preferred embodiment and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments. Many other equivalent embodiments may be included without departing from the concept of this application, and the scope of this application is determined by the scope of the appended claims.

Claims

1. A data mart management method, characterized in that, The method includes: Receive data to be processed from the source system; The data to be processed is stored in a predetermined table structure to obtain a data table in a standard format; Based on the standard format data table, the data to be processed is processed sequentially at multiple levels in hierarchical order to obtain a processing result table of the data to be processed; and the processing result table of the data to be processed is sent to the downstream system. Before sending the processing result table to the downstream system, the method further includes: Obtain the generation information of the processing result table, wherein the generation information includes at least the data source information of the processing result table, the application level of the processing result table, the name of the source system, the time when the processing result table was obtained, and the job number of the processing result table; The names of each data link in the processing result table are determined based on the generation information of the processing result table and the predetermined naming format; The method further includes: Receive data tracing instructions sent by the downstream system; wherein the data tracing instructions include tracing conditions; In response to the data tracing instruction, the processing result table is traced based on the tracing conditions and the names of each data link in the pre-determined processing result table to obtain the tracing result; The tracing results are sent to the downstream system.

2. The method according to claim 1, characterized in that, Before receiving the data to be processed from the source system, the method further includes: Receive an application system selection instruction sent by the user, and determine the target application system in response to the application system selection instruction; Obtain the system information of the target application system; A physical model of the target application system is constructed based on each field of the system information, and the table structure is determined based on the physical model.

3. The method according to claim 1, characterized in that, After receiving the data to be processed from the source system, the method further includes: The method for receiving the data to be processed sent by the user for entering the database includes active database entry or passive database entry. When the data to be processed is entered into the database via active entry, the data entry application form is obtained and reviewed. When the review result of the inbound application form is "approved", the step of storing the data to be processed into a predetermined table structure is executed.

4. The method according to claim 1, characterized in that, The hierarchy includes a raw data layer, a basic data layer, a general model layer, and an interface layer. Based on the standard format data table, the data to be processed is processed sequentially at multiple layers according to the hierarchical order to obtain a processing result table of the data to be processed, including: The standard format data table is stored in the raw data layer according to a preset lifecycle, and the standard format data table is sent to the basic data layer; Based on pre-configured data management information, the data table in the standard format is processed in the basic data layer to obtain a basic data table; the basic data table is then sent to the general model layer. The personalized fields of the basic data table are determined in the general model layer, and the data in the basic data table is filtered in a personalized way according to the personalized fields to obtain a candidate data table; the candidate data table is sent to the interface layer. The processing result table is obtained at the interface layer based on the candidate data table.

5. The method according to claim 4, characterized in that, The processing result table is obtained at the interface layer based on the candidate data table, including: Send predetermined test data to the downstream system, so that the downstream system can test the candidate data table based on the test data; Receive the test results sent by the downstream system, and when the test results are passed, determine the candidate data table as the processing result table; When the test result is unsuccessful, a change instruction is sent to the downstream system, causing the downstream system to respond by changing the requirement information in accordance with the change instruction.

6. A data mart management device, characterized in that, The device includes: The receiving module is used to receive data to be processed sent by the source system; The storage module is used to store the data to be processed into a predetermined table structure to obtain a data table in a standard format; The processing module is used to process the data to be processed sequentially at multiple levels according to the hierarchical order based on the data table in the standard format, to obtain a processing result table of the data to be processed; and to send the processing result table of the data to be processed to the downstream system. Before sending the processing result table to the downstream system, the processing module is further configured to: Obtain the generation information of the processing result table, wherein the generation information includes at least the data source information of the processing result table, the application level of the processing result table, the name of the source system, the time when the processing result table was obtained, and the job number of the processing result table; The names of each data link in the processing result table are determined based on the generation information of the processing result table and the predetermined naming format; The processing module is also used for: Receive data tracing instructions sent by the downstream system; wherein the data tracing instructions include tracing conditions; In response to the data tracing instruction, the processing result table is traced based on the tracing conditions and the names of each data link in the pre-determined processing result table to obtain the tracing result; The tracing results are sent to the downstream system.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the data mart management method as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the data mart management method as described in any one of claims 1-5.

Images