Data storage method, device and equipment, medium and product
By directly filling in data in the user operation interface and using preset scheduling tools and processing strategies, the problem of data entry flexibility and efficiency caused by template dependence in the existing technology is solved, and efficient data entry and display is achieved.
Patent Information
- Application Number
- CN202411080183.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-12
AI Technical Summary
The existing data entry method relies on templates, resulting in data entry errors and inconsistent formats, reducing the flexibility and efficiency of data entry.
By directly filling in the indicators in the user operation interface and writing them to the preset relational database, importing them into the data warehouse tool using the preset scheduling tool to form an indicator source table, using the indicator processing strategy to generate a detailed list, and finally writing it to the columnized database Manage the system and avoid template parsing process.
It reduces data input errors, reduces dependence on templates, improves the flexibility and efficiency of data entry, ensures data integrity and consistency, and improves the efficiency of data display.
Smart Images

Figure CN118939717A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing, and in particular to a data storage method, device, equipment, medium and product. Background Art
[0002] Data warehousing refers to the process of importing data from external data sources into an internal database or data warehouse. The goal of data warehousing is to ensure the integrity, consistency, and availability of the data for subsequent data analysis and business applications.
[0003] When the existing data entry methods involve indicators that need to be manually reported, it is usually necessary to download and fill in the reporting indicators into the corresponding data entry template, and then upload it. The system then parses the template and enters the data into the database. During the template parsing process, it is easy for the template parsing to fail due to problems such as data input errors and inconsistent formats, which limits the flexibility of data entry. In addition, the upload and parsing process of the data entry template usually takes a long time, resulting in reduced efficiency of data entry. Summary of the invention
[0004] The present application provides a data warehousing method, device, equipment, medium and product to solve the technical problems of poor data warehousing flexibility and low data warehousing efficiency caused by high dependence on templates in existing data warehousing technical solutions.
[0005] In a first aspect, the present application provides a data storage method, the method comprising:
[0006] In response to receiving the reporting indicators input by the user through the operation interface, the reporting indicators are written into a preset relational database to form an original indicator table;
[0007] The original indicator table is imported into a preset data warehouse tool using a preset scheduling tool to form an indicator source table, wherein the structure and data contained in the indicator source table are consistent with the structure and data contained in the original indicator table;
[0008] Using a preset indicator processing strategy, the indicator source table is processed to obtain an indicator detail table;
[0009] The indicator detail table is written into a column-based database management system, so that the data in the indicator detail table that meets the display condition is displayed when the indicator display condition is met.
[0010] In a possible design, the use of a preset scheduling tool to import the original indicator table into a preset data warehouse tool to form an indicator source table includes:
[0011] Use preset scheduling tools to create data import tasks;
[0012] Based on the data import task, the original indicator table is imported into a preset data warehouse tool to form an indicator source table.
[0013] In a possible design, in response to receiving the reporting indicator input by the user through the operation interface, before writing the reporting indicator into a preset relational database to form an original indicator table, the method further includes:
[0014] Get the pre-configured reporting indicator type;
[0015] The metadata information of the reported indicator is written into the indicator metadata information table based on the reported indicator type, and the indicator metadata information table is located in the preset relational database.
[0016] In a possible design, the operation interface is an operation interface corresponding to a performance measurement system, and the preset indicator processing strategy is adopted to process the indicator source table to obtain an indicator detail table, including:
[0017] Adopting a preset indicator processing strategy, the indicator source table is split according to the organization dimension and the employee dimension to form an organization dimension indicator source table and an employee dimension indicator source table;
[0018] Generate an institution dimension indicator detail table based on the metadata information of each indicator in the institution dimension indicator source table, the time information and batch information of the corresponding data import task;
[0019] Generate an employee dimension indicator detail table based on the metadata information of each indicator in the employee dimension indicator source table, the time information and batch information of the corresponding data import task.
[0020] In a possible design, the original indicator table, the indicator source table and the indicator detail table are all in high-level table form;
[0021] Before writing the indicator detail table into the column-based database management system, the method further includes:
[0022] Convert the indicator details table from tall table format to wide table format.
[0023] In a possible design, after writing the indicator detail table into a column-based database management system, the method further includes:
[0024] If the indicator update conditions in the indicator details table are met, the preset table engine is used to update the indicators in the indicator details table based on the negative value offset mechanism.
[0025] In one possible design, the method further includes:
[0026] In response to receiving the performance indicator display request, reading the organization dimension indicator detail table and the employee dimension indicator detail table from the column database management system;
[0027] The data satisfying the display conditions in the organization dimension indicator detail table and the employee dimension indicator detail table are sent to the terminal device so that the terminal device can display them.
[0028] In a second aspect, the present application provides a data storage device, the device comprising:
[0029] A writing module, configured to write the reporting indicators into a preset relational database in response to receiving the reporting indicators input by the user through the operation interface to form an original indicator table;
[0030] An import module, used to import the original indicator table into a preset data warehouse tool using a preset scheduling tool to form an indicator source table, wherein the structure and data contained in the indicator source table are consistent with the structure and data contained in the original indicator table;
[0031] A processing module, used to process the indicator source table using a preset indicator processing strategy to obtain an indicator detail table;
[0032] The writing module is also used to write the indicator detail table into a column database management system, so that when the indicator display conditions are met, the data in the indicator detail table that meets the display conditions are sent to the terminal device, so that the terminal device can display it.
[0033] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;
[0034] The memory stores computer-executable instructions;
[0035] The processor executes the computer-executable instructions stored in the memory to implement the method according to the first aspect.
[0036] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.
[0037] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.
[0038] The data warehousing method, device, equipment, medium and product provided by the present application, in response to receiving the reporting indicators input by the user through the operation interface, write the reporting indicators into a preset relational database to form an original indicator table; use a preset scheduling tool to import the original indicator table into a preset data warehouse tool to form an indicator source table, the structure and data contained in the indicator source table are consistent with the structure and data contained in the original indicator table; use a preset indicator processing strategy to process the indicator source table to obtain an indicator detail table; write the indicator detail table into a columnar database management system to display the data in the indicator detail table that meets the display conditions when the indicator display conditions are met. By directly reporting the input indicators in the user operation interface and then writing the reported indicators into the preset relational database, there is no need to download, fill in and upload the data warehousing template, reducing data input errors and format inconsistencies caused by cumbersome manual operations, reducing dependence on data warehousing templates, and thus improving the flexibility of data warehousing. In addition, by using the preset scheduling tools, data warehouse tools and indicator processing strategies, the indicator source table and indicator detail table are obtained, and the column-based database management system is used to store the data, which can avoid the complex template parsing process and improve the efficiency of data storage. And when querying, the column-based database management system can quickly query the indicator detail table data that needs to be displayed, and then display the indicator detail table data efficiently. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0040] Figure 1 An application scenario diagram of the data storage method provided in this application;
[0041] Figure 2 A flowchart of a data storage method provided in one embodiment of the present application;
[0042] Figure 3 A flowchart of a data storage method provided in another embodiment of the present application;
[0043] Figure 4 A schematic diagram of the structure of a data storage device provided in one embodiment of the present application;
[0044] Figure 5 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application.
[0045] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0046] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0047] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved in the technical solution of this application are in compliance with the relevant laws and regulations and do not violate public order and good morals.
[0048] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.
[0049] In order to clearly understand the technical solution of the present application, the solution of the prior art is first introduced in detail.
[0050] At present, the methods used to enter data into the database are mainly carried out manually, and these methods have many limitations and shortcomings. First of all, most of these methods need to rely on templates for data entry. The templates are usually Excel templates generated by the system and contain the fields that need to be entered into the database. After filling in the data, the Excel template needs to be uploaded to the system. After the system parses the template, the data is imported into the database. However, during the template parsing process, human errors are prone to occur during the data filling process, for example, the format, font, border and other style settings of the cells in the Excel template are changed. This will cause the system to fail to parse the template, thereby limiting the flexibility of data entry. In addition, the upload and parsing process of the data entry template is usually time-consuming, which will reduce the efficiency of data entry.
[0051] Therefore, in the face of technical problems in the prior art, in order to improve the flexibility of data entry and reduce the dependence on the data entry template, the data can be directly reported in the user operation interface and then written into the preset relational database, without downloading, filling in and uploading the data entry template, reducing the problems of data input errors and format inconsistency caused by cumbersome manual operations. In addition, in order to improve the efficiency of data entry, the column-based database management system has efficient performance in processing large-scale data and complex queries, and the data is stored in the column-based database management system. Before storing the data, the original indicator table corresponding to the reported indicator needs to be processed. Therefore, in order to facilitate the processing of the original indicator table, the preset scheduling tool can be used to import the original indicator table into the preset data warehouse tool to form an indicator source table consistent with the original indicator table. After processing the indicator source table, the processed indicator detail table corresponding to the indicator source table is directly written into the column-based database management system. The whole process has a high degree of automation, which reduces the steps of manual operation and further improves the flexibility and efficiency of data entry.
[0052] Figure 1 This is an application scenario diagram that can implement the data storage method provided by this application, such as Figure 1 As shown, the scene diagram corresponding to the data warehousing method provided by the present application includes: a terminal device 101 and a server 102. Among them, the terminal device 101 includes a client corresponding to the data warehousing method, and has an operation interface on the client, and the user fills in the indicator data on this operation interface. And the reported data is written into the original indicator table corresponding to the preset relational database in the server 102. The server 102 includes a preset scheduling tool for importing the data of the original indicator table into the indicator source table; the server 102 will also process the indicator source table according to the preset indicator processing strategy, and obtain the indicator detailed table. The data of the indicator detailed table will be stored in the server 102. The server 104 returns the data of the indicator detailed table that meets the indicator display conditions to the terminal device 101 for display.
[0053] The server is a form of data storage device, and the data storage device can also be in other forms, which are not limited in this embodiment. The data storage device is a form of electronic equipment.
[0054] A specific application scenario of this solution may be a warehousing scenario for performance indicators. It is understandable that in this embodiment, the specific application scenario is not limited.
[0055] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0056] Figure 2 A flowchart of a data storage method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the execution subject of this embodiment is a data storage device, which can be implemented by a computer program. It can also be implemented by a medium storing relevant computer programs, such as a USB flash drive and / or a CD, or it can also be implemented by a physical device integrated or installed with relevant computer programs, such as a chip or a data storage device. The data storage device can be a server, a server cluster or other electronic equipment. The data storage method provided in this embodiment includes the following steps:
[0057] Step 201: In response to receiving the reporting indicators input by the user through the operation interface, the reporting indicators are written into a preset relational database to form an original indicator table.
[0058] The operation interface refers to the interface that allows users to interact with the database storage system. The operation interface can be located on the client corresponding to the database storage system. The operations performed by the user in the operation interface, such as filling in information and clicking buttons, will trigger the corresponding functions and processing logic of the system.
[0059] The reported indicators refer to the data corresponding to the indicators filled in by the user in the operation interface. The indicators can be indicators defined by business personnel. It is understandable that these user-reported indicators usually refer to indicators that cannot be automatically obtained by the system and can only be obtained by manual reporting by the user.
[0060] The original indicator table refers to a table used to store the indicators reported by the user.
[0061] The preset relational database refers to a predefined database based on a relational model. The relational database organizes and stores data through tables, each table consists of rows and columns, rows represent records, and columns represent fields. Optionally, the preset relational database can be MySQL.
[0062] Specifically, in this embodiment, the user can fill in the data corresponding to the indicator in the operation interface through various forms of input controls such as text boxes, drop-down menus, check boxes, etc. For example, the user can fill in the corresponding data for specific business indicators such as sales volume and number of customers; then, these filled-in indicator data will be written into the preset relational database, and the preset relational database will use the original indicator table to store these data.
[0063] It is understandable that the type of indicators and the reporting format filled in by the user in the operation interface are not limited in this embodiment.
[0064] Step 202: Use a preset scheduling tool to import the original indicator table into a preset data warehouse tool to form an indicator source table, the structure of the indicator source table and the data contained therein are consistent with the structure of the original indicator table and the data contained therein.
[0065] The preset scheduling tool is a predefined tool with a table scheduling function. It is understandable that the preset scheduling tool is not limited in this embodiment.
[0066] The preset data warehouse tool refers to a predefined software system for building, managing and operating a data warehouse. Optionally, the preset data warehouse tool may be Hive, which is not limited in this embodiment. The preset data warehouse tool is used to provide efficient data query and analysis capabilities. The preset data warehouse tool can extract data from a variety of data sources such as relational databases and file systems, clean, convert and load the data into the data warehouse, and process the data efficiently.
[0067] Among them, the indicator source table refers to a table that temporarily stores the data in the original indicator table and has the same structure and data as the original indicator table. The purpose of the indicator source table is to provide an intermediate storage area during the data loading and conversion process, temporarily storing the data extracted from the original indicator table for subsequent data processing operations, which can ensure the integrity and consistency of the data.
[0068] Specifically, in this embodiment, for the original indicator table in the preset relational database, using the preset scheduling tool, the corresponding scheduling tasks can be set according to the specific business scenario, and the data in the original indicator table is imported into the indicator source table defined in the data warehouse tool based on the scheduling task. The structure and data of the indicator source table are exactly the same as the original indicator table, and it is used to temporarily store the data extracted from the original indicator table.
[0069] It is understandable that, depending on different business scenarios, the selection of scheduling tools and the setting of scheduling tasks are not limited in this embodiment.
[0070] Step 203: Using a preset indicator processing strategy, the indicator source table is processed to obtain an indicator detail table.
[0071] Among them, the indicator processing strategy refers to the specific process, logic and other aspects of the strategy for processing the indicator source table using the data warehouse tool. The indicator processing strategy can be data cleaning, data type conversion, data grouping and index creation. The processed indicator data can better support the needs of complex data query and analysis. This embodiment does not limit the indicator processing strategy.
[0072] Among them, the indicator detail table refers to a table used to store detailed data after data processing such as cleaning, conversion, and loading. The indicator detail table contains detailed information of the original indicator, such as detailed records of business activities, and has undergone a structured and optimized process to support efficient data query and analysis.
[0073] Specifically, according to business needs and data characteristics, during the data processing process, an indicator processing strategy containing a series of rules and operations is pre-defined to clean, convert, aggregate and optimize the indicator data in the indicator table source table, and save the processed data in the indicator detail table.
[0074] Step 204: writing the indicator detail table into the column-based database management system, so as to display the data in the indicator detail table that meets the display condition when the indicator display condition is met.
[0075] Among them, the column-based database management system refers to a database system in which data is stored in the form of columns. The column-based storage method has significant performance advantages when processing analytical queries. Optionally, the column-based database management system can be ClickHouse. This embodiment does not limit the type of column-based database management system.
[0076] Among them, the indicator display conditions are the conditions for displaying the data in the indicator details table. Specifically, they can be the conditions used to determine when, how, and under what circumstances to display the data in the indicator details table during data analysis and result generation. The indicator display conditions help ensure the accuracy, relevance, and comprehensibility of data display, allowing users to interpret and use data more effectively.
[0077] Specifically, the indicator display condition can be set according to the time range, for example, only the indicator data within a specific time range is displayed; the indicator display condition can be set according to the threshold, for example, only the indicator data exceeding or below the threshold is displayed. It can be understood that the setting method of the indicator display condition is not limited in this embodiment.
[0078] Specifically, first select a column-based database management system that suits your business needs; then use a custom script to load the data in the indicator details table into the column-based database; then define the indicator display conditions according to the specific business scenario, and configure corresponding query filtering rules based on the display conditions to filter out indicator data that meets the indicator display conditions from the indicator details table; finally, display the filtered indicator data.
[0079] The data warehousing method provided in this embodiment, in response to receiving the reporting indicators input by the user through the operation interface, writes the reporting indicators into a preset relational database to form an original indicator table; uses a preset scheduling tool to import the original indicator table into a preset data warehouse tool to form an indicator source table, the structure of the indicator source table and the data contained in it are consistent with the structure and data contained in the original indicator table; uses a preset indicator processing strategy to process the indicator source table to obtain an indicator detail table; writes the indicator detail table into a columnar database management system to display the data in the indicator detail table that meets the display conditions when the indicator display conditions are met. By directly reporting the input indicators in the user operation interface and writing these indicator data into the preset relational database, there is no need to download, fill in and upload the data warehousing template, thereby reducing problems such as data input errors and format inconsistencies caused by cumbersome manual operations, reducing dependence on data warehousing templates, and improving the flexibility of data warehousing. In addition, through the preset scheduling tools, data warehouse tools and indicator processing strategies, the indicator source table and indicator detail table are generated, and the column-based database management system is used to store the data, which can avoid the complex template parsing process and improve the efficiency of data storage. During the query process, the column-based database management system can quickly retrieve the required indicator detail table data and efficiently display these indicator detail table data.
[0080] As an optional implementation, based on the above embodiment, a preset scheduling tool is used to import the original indicator table into a preset data warehouse tool to form an indicator source table, including:
[0081] Use preset scheduling tools to create data import tasks;
[0082] Based on the data import task, the original indicator table is imported into the preset data warehouse tool to form the indicator source table.
[0083] The data import task refers to the task of importing data from the source system to the target system. Data import tasks usually need to be executed on a scheduled or quantitative basis, and can be automated through preset scheduling tools.
[0084] Optionally, in this embodiment, in the preset data warehouse tool, an indicator source table can be set, and the structure of the indicator source table is consistent with the original indicator table. Exemplarily, Hive can be selected as the preset data warehouse tool, and the data import task can be set to execute data import every 15 minutes to import the data in the original indicator table into the indicator source table in Hive.
[0085] Among them, Hive refers to a Hadoop-based data warehouse tool for efficient data query and analysis.
[0086] It is understandable that the preset scheduling tool and the preset data warehouse tool are not limited in this embodiment.
[0087] The data warehousing method provided in this embodiment uses a preset scheduling tool to import the original indicator table into a preset data warehouse tool to form an indicator source table, including: using the preset scheduling tool to create a data import task; based on the data import task, importing the original indicator table into the preset data warehouse tool to form an indicator source table. Using the preset scheduling tool to create a data import task, and based on using the data warehouse tool to transfer data from the original indicator table to the indicator source table, it can ensure that the task is accurately executed according to the time point and sequence preset in the data import task.
[0088] As an optional implementation, based on the above embodiment, in response to receiving the reporting indicator input by the user through the operation interface, before writing the reporting indicator into the preset relational database to form the original indicator table, it also includes:
[0089] Get the pre-configured reporting indicator type;
[0090] Based on the reported indicator type, the metadata information of the reported indicator is written into the indicator metadata information table, and the indicator metadata information table is located in a preset relational database.
[0091] The pre-configured reporting indicator type refers to the type of indicator that needs to be manually reported and is defined by business personnel. This embodiment does not limit the reporting indicator type.
[0092] Metadata information refers to data that describes and defines the attributes and characteristics of the indicator itself. It provides detailed information about the indicator so that users can understand, manage and use these indicators. Optionally, metadata information can include indicator name, indicator description, indicator data type and indicator data source, etc.
[0093] The indicator metadata information table refers to a table storing metadata information related to the reported indicator. In this embodiment, there are two indicator metadata information tables, namely the indicator business metadata information table and the indicator technology metadata information table, which store indicator metadata information related to the business and indicator metadata information related to the technology respectively.
[0094] Specifically, the business personnel will first define a series of types of reporting indicators that need to be manually reported later, and record the metadata information related to the reporting indicator types in the indicator business metadata information table and the indicator technical metadata information table of the preset relational database according to the types of reporting indicators. In this embodiment, the indicator business metadata information table can record metadata information such as the caliber description information of the indicator, the business sector to which the indicator belongs, the department to which the indicator belongs, the storage location information of the indicator, and the unit information of the indicator; the indicator technical metadata information table can record metadata information related to indicator processing and calculation, such as data update frequency, data start date, data current date, data storage table, data storage column, whether to support aggregation by organization, calculation formula, etc.
[0095] The data warehousing method provided in this embodiment, in response to receiving the reporting indicator input by the user through the operation interface, writes the reporting indicator into the preset relational database to form the original indicator table, and further includes: obtaining the pre-configured reporting indicator type; based on the reporting indicator type, writing the metadata information of the reporting indicator into the indicator metadata information table, the indicator metadata information table is located in the preset relational database. Writing the metadata information of the reporting indicator into the indicator metadata information table and storing it in the preset relational database helps to maintain the indicator-related information and ensure that each indicator has a clear definition and attributes.
[0096] As an optional implementation, based on the above embodiment, the operation interface is an operation interface corresponding to the performance measurement system, and a preset indicator processing strategy is adopted to process the indicator source table to obtain an indicator detail table, including:
[0097] Adopting the preset indicator processing strategy, the indicator source table is split according to the organization dimension and the employee dimension to form the organization dimension indicator source table and the employee dimension indicator source table;
[0098] Generate an institution dimension indicator detail table based on the metadata information of each indicator in the institution dimension indicator source table, the time information and batch information of the corresponding data import task;
[0099] Generate an employee dimension indicator detail table based on the metadata information of each indicator in the employee dimension indicator source table, the time information and batch information of the corresponding data import task.
[0100] Among them, the performance measurement system refers to a system used to measure the performance of an organization, team or individual. It helps organizations understand their business operations by collecting, analyzing and reporting various performance indicators and data.
[0101] The preset indicator processing strategy refers to a predefined rule and method for processing indicator data. In this embodiment, the preset indicator processing strategy refers to dividing the indicator data according to the organization dimension and the employee dimension.
[0102] Specifically, first, the indicator data in the indicator source table are divided according to the dimensions corresponding to the indicator data. For example, for the indicator data related to the organization information, the organization dimension indicator source table is used to store these indicator data; for the indicator data related to the employee information, the employee dimension indicator source table is used to store these indicator data. Afterwards, for the organization dimension indicator source table and the employee dimension indicator source table, the preset scheduling tool can be used to configure the corresponding data import task, and in the preset data warehouse tool Hive, the organization dimension indicator detail table and the employee dimension indicator detail table are set, and then further combined with the indicator metadata information in the indicator metadata information table, the data in the organization dimension indicator source table and the employee dimension indicator source table are imported into the organization dimension indicator detail table and the employee dimension indicator detail table respectively. Since the preset scheduling tool is used to configure the corresponding data import task and the metadata information is combined in the process of importing data from the indicator source table to the indicator detail table, the indicator detail tables of these two dimensions further refine and supplement the information compared with the indicator source table.
[0103] Optionally, compared with the indicator source table, the indicator details table in the organization dimension indicator source table and the employee dimension indicator source table may include detailed information such as the column information stored in the indicator metadata information table for each indicator, the import date and import batch information of the indicator in the indicator scheduling task, etc.
[0104] The data warehousing method provided in this embodiment has an operation interface corresponding to the operating system of the performance measurement system, and uses a preset indicator processing strategy to process the indicator source table to obtain an indicator detail table, including: using a preset indicator processing strategy to split the indicator source table according to the organization dimension and the employee dimension to form an organization dimension indicator source table and an employee dimension indicator source table; generating an organization dimension indicator detail table based on the metadata information of each indicator in the organization dimension indicator source table, the time information and batch information of the corresponding data import task; generating an employee dimension indicator detail table based on the metadata information of each indicator in the employee dimension indicator source table, the time information and batch information of the corresponding data import task. By using the preset indicator processing strategy, the indicator source table is split according to the organization dimension and the employee dimension to generate the indicator detail tables of the organization dimension and the employee dimension, which optimizes the data organization and storage structure, and can more accurately manage and analyze indicator data of different dimensions.
[0105] As an optional implementation, based on the above embodiment, the original indicator table, the indicator source table and the indicator detail table are all in high-level table form;
[0106] Before writing the indicator detail table into the column-based database management system, the method further includes:
[0107] Convert the indicator details table from tall table format to wide table format.
[0108] Among them, the high-level table format means that the data is stored in a vertical manner. In this form, each row usually represents an observation or record, and the columns represent different attributes or variables. The high-level table format can realize the efficient addition of new observations or records without large-scale modifications to the table structure. For example, in the original indicator table in the high-level table format, a new observation or record represented by a row of data can be an indicator data newly reported by the user, and the different attributes or variables represented by the columns of the row of data can be the user's reporting timestamp, indicator name, and the value corresponding to the indicator.
[0109] Among them, the wide table format means that the data is stored in a horizontal manner. In this form, each row usually represents an observation object, and the columns represent the different attributes of the object. The data structure of the wide table format is more intuitive and can achieve efficient data query. For example, in the indicator details table in the wide table format, an observation object represented by a row of data can be a user who fills in an indicator, and the different attributes of the object represented by the columns of the row of data can be all the indicators filled in by the user.
[0110] Specifically, since the number of reported indicators is uncertain each time the user reports an indicator, the original indicator table in high-level table format can adapt to this uncertainty well, so that the data length of each row can be kept consistent, which is convenient for the subsequent insertion and update of indicator data. Similarly, for the indicator source table and the indicator detail table, they also need to be stored in high-level table format. In addition, since the column-based database management system stores data of the same column type together when storing data, when querying, only the data of the relevant columns needs to be read, which greatly reduces I / O operations and improves query performance. Therefore, it is necessary to convert the indicator detail table in high-level table format into a wide table format, so that the data storage format is more in line with the storage characteristics of the column-based database management system, and give full play to its advantages in analytical query and batch processing.
[0111] In the data storage method provided by this embodiment, the original indicator table, the indicator source table and the indicator detail table are all in high table format; before writing the indicator detail table into the column-based database management system, the method also includes: converting the indicator detail table from the high table format to the wide table format. By storing the data of the original indicator table, the indicator source table and the indicator detail table in the high table format, the data can be updated flexibly to cope with the uncertainty when users fill in the indicators, thereby improving the flexibility of data storage; the subsequent conversion of the indicator detail table into a wide table format will be more in line with the storage characteristics of the column-based database management system, facilitating efficient data query, thereby improving the efficiency of data storage.
[0112] As an optional implementation, based on the above embodiment, after the indicator detail table is written into the column database management system, the method further includes:
[0113] If the indicator update conditions in the indicator details table are met, the preset table engine is used to update the indicators in the indicator details table based on the negative value offset mechanism.
[0114] Among them, the indicator update condition refers to the conditions for updating the corresponding data of the previously reported indicators.
[0115] The preset table engine is the core component used to manage and operate data tables in a column-based database management system. The table engine determines the data storage method, index mechanism, query processing method, and data read and write performance.
[0116] Among them, the negative value offset mechanism refers to a technology used for data aggregation and processing in the database, especially when processing incremental data and transactions. This mechanism is mainly used in aggregation operations to offset the previous positive value by introducing a negative value, thereby achieving the effect of updating or deleting data.
[0117] Specifically, in the present embodiment, the column-based database management system may be a ClickHouse database, and the preset table engine may be a SummingMergeTree. When the indicator update condition is met, that is, when the indicator data that has been synchronized to Clickhouse is modified, the negative value offset mechanism of SummingMergeTree is used to update the data. Exemplarily, when the indicator data that has been synchronized to ClickHouse is modified for the Nth time, the indicator data value submitted for the N-1th time may first be multiplied by -1 and inserted into the indicator detail table in the form of a wide table to offset the indicator data value synchronized for the N-1th time; and then the indicator data value submitted for the Nth time may be inserted into the indicator detail table in the form of a wide table to update the indicator data. In this way, the limitations of the ClickHouse column-based database management system in data modification can be overcome, and the purpose of updating indicator data can be achieved.
[0118] Among them, the ClickHouse database is a high-performance columnar database designed for real-time analysis and large-scale data processing; SummingMergeTree is a table engine in the ClickHouse database, which is specifically used for efficient data aggregation operations.
[0119] The data warehousing method provided in this embodiment, after writing the indicator detail table into the column-based database management system, the method further includes: if the indicator update condition in the indicator detail table is met, the preset table engine is used to update the indicator in the indicator detail table based on the negative value offset mechanism. By using a column-based database management system and a table engine, combined with a negative value offset mechanism to implement data updates, the limitations of ClickHouse in data modification are overcome, frequent full-table update operations are avoided, and changes in indicator data can be responded to more quickly, thereby improving the efficiency and flexibility of data warehousing.
[0120] As an optional implementation manner, based on the above embodiment, the method further includes:
[0121] In response to receiving the performance indicator display request, reading the organization dimension indicator detail table and the employee dimension indicator detail table from the column database management system;
[0122] The data that meets the display conditions in the organization dimension indicator detail table and the employee dimension indicator detail table are sent to the terminal device so that the terminal device can display it.
[0123] For example, the performance indicator display request may be issued periodically by setting a timed task, or by monitoring a specific user event, etc. It is understandable that the method of requesting the performance indicator display is not limited in this embodiment.
[0124] In this embodiment, the user can issue a performance indicator display request in the business broadcast module of the performance measurement system. For example, the request can be triggered by clicking a button, etc. After that, the system can receive and respond to the performance indicator display request by monitoring specific user events, such as button click events, and then read the data in the organization dimension indicator detail table and the employee dimension indicator detail table from the column-based database management system ClickHouse, and then display them in the business broadcast module of the performance measurement system.
[0125] The data storage method provided in this embodiment also includes: in response to receiving a performance indicator display request, reading an organization dimension indicator detail table and an employee dimension indicator detail table from a column-based database management system; sending the data in the organization dimension indicator detail table and the employee dimension indicator detail table that meet the display conditions to a terminal device so that the terminal device can display it. By storing indicator data according to different dimensions, the system can display data more efficiently and clearly. In addition, the way indicator data is classified according to dimensions can also be adjusted according to different business scenarios, which improves the flexibility of data storage and thus improves the efficiency of data storage.
[0126] Figure 3 A flowchart of a data storage method provided in another embodiment of the present application is shown in FIG. Figure 3 As shown, the data storage method provided in this embodiment includes the following steps:
[0127] Step 301: receiving the indicator type that needs to be reported defined by the business personnel, and saving the metadata information of the indicator type in the indicator metadata information table of the preset relational database.
[0128] Step 302, receiving the user's operation of filling in data for the selected indicators in the operation interface, obtaining the filled-in indicators and writing them into a preset relational database to form an original indicator table.
[0129] Step 303, using a preset scheduling tool to import the original indicator table into a preset data warehouse tool to form an indicator source table, the structure of the indicator source table and the data contained therein are consistent with the structure of the original indicator table and the data contained therein.
[0130] Among them, the original indicator table and the indicator source table are both in high-level table format.
[0131] Step 304, using the preset scheduling tool, combined with the indicator metadata information table and the data scheduling task, the indicator source table is imported into the institution dimension indicator detail table and the employee dimension indicator detail table according to the institution dimension and the employee dimension.
[0132] Among them, the source tables for institution dimension indicators and employee dimension indicators are both in high-level table format.
[0133] Step 305, using a preset indicator processing strategy, converting the organization dimension indicator detail table and the employee dimension indicator detail table into a wide table format.
[0134] Step 306, write the wide table format of the organization dimension indicator detail table and the employee dimension indicator detail table into the column-based database management system.
[0135] Step 307: If the indicator update condition in the indicator details table is met, the preset table engine is used to update the indicator in the indicator details table based on the negative value offset mechanism.
[0136] Step 308: When the indicator display condition is met, information of the indicator detail table is read from the column database management system for display.
[0137] In this embodiment, the implementation method and technical effects of steps 301 to 308 are similar to the implementation methods of the corresponding solutions in the above embodiments, and will not be repeated here.
[0138] Figure 4 The data storage device 40 provided in this embodiment includes: a writing module 41, an importing module 42 and a processing module 43.
[0139] Among them, the writing module 41 is used to write the reported indicators into a preset relational database in response to receiving the reported indicators input by the user through the operation interface to form an original indicator table; the importing module 42 is used to import the original indicator table into a preset data warehouse tool using a preset scheduling tool to form an indicator source table, and the structure of the indicator source table and the data contained therein are consistent with the structure and the data contained therein of the original indicator table; the processing module 43 is used to process the indicator source table using a preset indicator processing strategy to obtain an indicator details table; the writing module 41 is also used to write the indicator details table into a columnar database management system, so as to send the data in the indicator details table that meets the display conditions to the terminal device when the indicator display conditions are met, so that the terminal device can display it.
[0140] Optionally, when the preset scheduling tool is used to import the original indicator table into the preset data warehouse tool to form an indicator source table, the import module 42 is specifically used to: create a data import task using the preset scheduling tool; and import the original indicator table into the preset data warehouse tool based on the data import task to form an indicator source table.
[0141] Optionally, the data storage device provided in this embodiment further includes an acquisition module.
[0142] The acquisition module is used to acquire the pre-configured reporting indicator type before the writing module writes the reporting indicator into the preset relational database in response to receiving the reporting indicator input by the user through the operation interface to form the original indicator table;
[0143] The writing module is also used to write the metadata information of the reported indicators into the indicator metadata information table based on the reported indicator type, and the indicator metadata information table is located in a preset relational database.
[0144] Optionally, the operation interface is an operation interface corresponding to the performance measurement system.
[0145] Correspondingly, when the processing module 43 adopts the preset indicator processing strategy to process the indicator source table to obtain the indicator details table, it is specifically used to: adopt the preset indicator processing strategy to split the indicator source table according to the organization dimension and the employee dimension to form an organization dimension indicator source table and an employee dimension indicator source table; generate an organization dimension indicator details table based on the metadata information of each indicator in the organization dimension indicator source table, the time information and batch information of the corresponding data import task; generate an employee dimension indicator details table based on the metadata information of each indicator in the employee dimension indicator source table, the time information and batch information of the corresponding data import task.
[0146] Optionally, the original indicator table, the indicator source table and the indicator detail table are all in high-level table format;
[0147] Correspondingly, the data storage device provided in this embodiment also includes a conversion module.
[0148] Among them, the conversion module is used to convert the indicator detail table from a high table format to a wide table format before the writing module writes the indicator detail table into the column-based database management system.
[0149] Optionally, the data storage device provided in this embodiment also includes an update module.
[0150] Optionally, after the writing module writes the indicator details table into the column-based database management system, the update module uses a preset table engine to update the indicators in the indicator details table based on a negative value offset mechanism if the indicator update conditions in the indicator details table are met.
[0151] Optionally, the data storage device provided in this embodiment further includes a reading module and a sending module.
[0152] Among them, the reading module is used to read the organization dimension indicator detail table and the employee dimension indicator detail table from the column database management system in response to receiving a performance indicator display request; the sending module is used to send the data in the organization dimension indicator detail table and the employee dimension indicator detail table that meet the display conditions to the terminal device so that the terminal device can display it.
[0153] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the electronic device 50 provided in this embodiment includes: a processor 51 and a memory 52.
[0154] The memory 52 is used to store computer-executable instructions; the processor 51 is used to execute the computer-executable instructions stored in the memory 52 to implement the data storage method provided by any of the above embodiments. For details, please refer to the relevant description in the above method embodiments, and no further details will be given here.
[0155] The program may include program codes, and the program codes include computer-executable instructions. The memory 52 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.
[0156] In this embodiment, the memory 52 is connected to the processor 51 via a bus. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0157] The embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the data storage method provided in any of the above embodiments. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device.
[0158] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the data warehousing method provided in any one of the above embodiments.
[0159] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.
[0160] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
[0161] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.
[0162] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.
[0163] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not specifically stated, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not specifically stated, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.
[0164] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.
[0165] In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For the part that is not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered as the scope recorded in this specification. Those skilled in the art will easily think of other implementation schemes of the present application after considering the specification and practicing the invention disclosed here. This application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the technical field that are not disclosed in this application. The description and embodiments are only regarded as exemplary, and the true scope and spirit of the present application are indicated by the following claims.
[0166] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A method for storing data. It is characterized in that The method comprises: In response to receiving the reporting indicators input by the user through the operation interface, the reporting indicators are written into a preset relational database to form an original indicator table; The original indicator table is imported into a preset data warehouse tool using a preset scheduling tool to form an indicator source table, wherein the structure and data contained in the indicator source table are consistent with the structure and data contained in the original indicator table; Using a preset indicator processing strategy, the indicator source table is processed to obtain an indicator detail table; The indicator detail table is written into a column-based database management system, so that the data in the indicator detail table that meets the display condition is displayed when the indicator display condition is met.
2. The method according to claim 1, It is characterized in that The method of using a preset scheduling tool to import the original indicator table into a preset data warehouse tool to form an indicator source table includes: Use preset scheduling tools to create data import tasks; Based on the data import task, the original indicator table is imported into a preset data warehouse tool to form an indicator source table.
3. The method according to claim 1, It is characterized in that In response to receiving the reporting indicator input by the user through the operation interface, before writing the reporting indicator into the preset relational database to form the original indicator table, the method further includes: Get the pre-configured reporting indicator type; The metadata information of the reported indicator is written into the indicator metadata information table based on the reported indicator type, and the indicator metadata information table is located in the preset relational database.
4. The method according to claim 2, It is characterized in that The operation interface is an operation interface corresponding to the performance measurement system, and the preset indicator processing strategy is adopted to process the indicator source table to obtain an indicator detail table, including: Adopting a preset indicator processing strategy, the indicator source table is split according to the organization dimension and the employee dimension to form an organization dimension indicator source table and an employee dimension indicator source table; Generate an institution dimension indicator detail table based on the metadata information of each indicator in the institution dimension indicator source table, the time information and batch information of the corresponding data import task; Generate an employee dimension indicator detail table based on the metadata information of each indicator in the employee dimension indicator source table, the time information and batch information of the corresponding data import task.
5. The method according to claim 1, It is characterized in that The original indicator table, the indicator source table and the indicator detail table are all in high-level table form; Before writing the indicator detail table into the column-based database management system, the method further includes: Convert the indicator details table from tall table format to wide table format.
6. The method according to claim 1, It is characterized in that After writing the indicator detail table into the column-based database management system, the method further includes: If the indicator update conditions in the indicator details table are met, the preset table engine is used to update the indicators in the indicator details table based on the negative value offset mechanism.
7. The method according to claim 4, It is characterized in that The method further comprises: In response to receiving the performance indicator display request, reading the organization dimension indicator detail table and the employee dimension indicator detail table from the column database management system; The data satisfying the display conditions in the organization dimension indicator detail table and the employee dimension indicator detail table are sent to the terminal device so that the terminal device can display them.
8. A data storage device, It is characterized in that The device comprises: A writing module, configured to write the reporting indicators into a preset relational database in response to receiving the reporting indicators input by the user through the operation interface to form an original indicator table; An import module, used to import the original indicator table into a preset data warehouse tool using a preset scheduling tool to form an indicator source table, wherein the structure and data contained in the indicator source table are consistent with the structure and data contained in the original indicator table; A processing module, used to process the indicator source table using a preset indicator processing strategy to obtain an indicator detail table; The writing module is also used to write the indicator detail table into a column database management system, so that when the indicator display conditions are met, the data in the indicator detail table that meets the display conditions are sent to the terminal device, so that the terminal device can display it.
9. An electronic device, It is characterized in that include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.
11. A computer program product, It is characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when being executed by a processor.
Citation Information
Patent Citations
Self-defined visual configuration method for data filling pages
CN108776702A
Data table conversion method and device
CN112507017A
Data processing method and device based on data warehouse, equipment and storage medium
CN114218218A
Data filling method, system and equipment based on index system and storage medium
CN115700580A
Standardized label production method and system based on column storage database technology and storage medium
CN118245484A
