Business data management method and system, electronic equipment and storage medium

Abstract

The invention discloses a business data management method and system, electronic equipment and a storage medium. The method comprises the following steps: constructing a preset business index library; wherein the preset business index library comprises a first storage unit and a second storage unit; obtaining to-be-stored business data; wherein the business data to be stored comprises index metadata and index numerical data, and the index metadata corresponds to the index numerical data; storing the index metadata to the first storage unit, storing the index numerical data to the second storage unit, and constructing to obtain a target business index library; and when it is determined that the query index request is received, querying from the target business index library according to the query index request to obtain target business data. According to the embodiment of the invention, the management of the business data can be realized, and the query efficiency and query performance of the business data are effectively improved. The method can be widely applied to the technical field of big data.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a business data management method, system, electronic device and storage medium. Background Art

[0002] As the business scale of some enterprises expands, their demand for insights into business data is also growing. Business databases, such as indicator libraries, as the basis for data insights, need to be effectively managed and quickly respond to complex query requests. Business data management has become a key link in data-driven decision-making. In related technologies, business data management has low query efficiency and poor query performance when facing massive, multi-dimensional business data.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a business data management method, system, electronic device and storage medium, which can realize the management of business data and effectively improve the query efficiency and query performance of business data.

[0005] To achieve the above object, an embodiment of the present application provides a business data management method, the method comprising the following steps:

[0006] Constructing a preset business indicator library; wherein the preset business indicator library includes a first storage unit and a second storage unit;

[0007] Acquire the business data to be stored; wherein the business data to be stored includes indicator metadata and indicator numerical data, and the indicator metadata and the indicator numerical data correspond to each other;

[0008] The indicator metadata is stored in the first storage unit, and the indicator numerical data is stored in the second storage unit, so as to construct a target business indicator library;

[0009] When it is determined that a query indicator request is received, target business data is obtained by querying from the target business indicator library according to the query indicator request.

[0010] In some embodiments, the step of building a preset business indicator library includes:

[0011] Constructing the first storage unit by using a preset relational database;

[0012] Constructing the second storage unit by using a preset column database;

[0013] The preset business indicator library is constructed according to the first storage unit and the second storage unit.

[0014] In some embodiments, constructing the first storage unit by using a preset relational database includes:

[0015] Construct a preset indicator table according to preset indicator information;

[0016] Construct a preset dimension table based on preset dimension information;

[0017] Constructing a dimension dictionary table according to the preset dimension table;

[0018] Constructing an indicator dimension association table according to the preset indicator table and the preset dimension table;

[0019] The first storage unit is constructed by constructing the preset indicator table, the preset dimension table, the dimension dictionary table and the indicator dimension association table.

[0020] In some embodiments, the step of constructing the second storage unit by using a preset column database includes:

[0021] Constructing a preset partition key through preset dimension data; wherein the preset dimension data includes a time dimension or a geographic dimension;

[0022] Constructing a preset data column according to the preset dimension information and the preset indicator information;

[0023] An indicator value table is constructed according to the preset partition key and the preset data column, so as to obtain the second storage unit through the indicator value table.

[0024] In some embodiments, when determining that a query indicator request is received, querying and obtaining target business data from the target business indicator library according to the query indicator request includes:

[0025] Extracting target indicator information and target dimension information from the first storage unit according to the query indicator request;

[0026] The target business data is obtained by querying from the second storage unit according to the target indicator information and the target dimension information.

[0027] In some embodiments, after executing the step of building a preset business indicator library, the method further includes:

[0028] Get metadata modification instructions;

[0029] Performing preset management operations on the first storage unit according to the metadata modification instruction; wherein the preset management operations include indicator definition operations, dimension configuration settings, and association relationship settings.

[0030] In some embodiments, after performing the step of storing the indicator metadata in the first storage unit and storing the indicator numerical data in the second storage unit to construct a target business indicator library, the method further includes:

[0031] Determine the hotspot query data in the target business indicator library according to the data history query information;

[0032] The hotspot query data is stored in a preset cache database.

[0033] To achieve the above object, another aspect of the embodiment of the present application provides a business data management system, the system comprising:

[0034] The first module is used to construct a preset business indicator library; wherein the preset business indicator library includes a first storage unit and a second storage unit;

[0035] The second module is used to obtain the business data to be stored; wherein the business data to be stored includes indicator metadata and indicator numerical data, and the indicator metadata corresponds to the indicator numerical data;

[0036] The third module is used to store the indicator metadata into the first storage unit, and store the indicator numerical data into the second storage unit, so as to construct a target business indicator library;

[0037] The fourth module is used to query and obtain target business data from the target business indicator library according to the query indicator request when it is determined that a query indicator request is received.

[0038] To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising:

[0039] at least one processor;

[0040] at least one memory for storing at least one program;

[0041] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0042] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0043] The embodiments of the present application include at least the following beneficial effects: The present application provides a business data management method, system, electronic device and storage medium, and the scheme is to construct a preset business indicator library including a first storage unit and a second storage unit. Accordingly, the embodiment of the present invention obtains the business data to be stored, including indicator metadata and indicator numerical data. Then, the embodiment of the present invention stores the obtained indicator metadata in the first storage unit, and stores the indicator numerical data in the second storage unit to construct a target business indicator library. Finally, when it is determined that a query indicator request is received, the embodiment of the present invention obtains the target business data from the target business indicator library according to the query indicator request, and realizes the management of business data. It is easy to understand that the embodiment of the present invention realizes the read-write separation of related business data by constructing a preset business indicator library, storing the indicator metadata in the first storage unit, and storing the indicator numerical data in the second storage unit, constructs a target business indicator library, and obtains the target business data from the target business indicator library through the corresponding query indicator request, which effectively improves the query efficiency and query performance of business data. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of a business data management method provided by an embodiment of the present invention;

[0045] Figure 2 is a flow chart of building a preset business indicator library provided by an embodiment of the present invention;

[0046] Figure 3 is a flow chart of constructing a first storage unit by using a preset relational database provided by an embodiment of the present invention;

[0047] Figure 4 is a flow chart of constructing a second storage unit by using a preset column database provided by an embodiment of the present invention;

[0048] Figure 5 It is a flowchart of obtaining target business data from a target business indicator library according to the query indicator request when it is determined that a query indicator request is received, provided by an embodiment of the present invention;

[0049] Figure 6 is a flow chart of managing a first storage unit provided by an embodiment of the present invention;

[0050] Figure 7 is a flow chart of caching hotspot query data provided by an embodiment of the present invention;

[0051] Figure 8 It is a schematic diagram of the architecture of the business data management method provided by an embodiment of the present invention;

[0052] Fig. 9It is a structural diagram of a business data management system provided in an embodiment of the present application;

[0053] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0055] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0056] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0058] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0059] Indicator: Generally refers to quantifiable data, which is the result of statistical calculations and is used to measure the development level of things, such as sales, profits, user activity, etc.

[0060] Dimension: generally refers to a certain characteristic of a phenomenon, and is a measurement statistic from a certain perspective, such as product, region, time, etc. Dimensions can be used to compare indicators in many aspects and find trends.

[0061] MySQL database: is a relational database that can be used to store, manage and process large amounts of structured data.

[0062] ClickHouse database: is a column storage database, mainly used for online analytical processing queries. It can generate analytical data reports in real time using SQL queries, can efficiently handle workloads that require aggregation and analysis of large amounts of data, and provide high-performance data query capabilities.

[0063] As the business scale of some enterprises expands, their demand for insights into business data is also growing. Business databases, such as indicator libraries, as the basis for data insights, need to be effectively managed and quickly respond to complex query requests. Business data management has become a key link in data-driven decision-making. In related technologies, business data management has low query efficiency and poor query performance when facing massive, multi-dimensional business data. For example, in the current indicator library management, there are many indicator dimensions and many dimension dictionary values. At the same time, there are complex associations between dimensions, such as provinces, cities, districts, counties, and towns. At the same time, the real-time requirements of indicators are high, generally minute, second, or millisecond data, and the data scale is also increased from the time dimension. In addition, the amount of indicator data is large. Due to the large number of dimensions and the large number of dimension dictionary values, the amount of data combined in each dimension is large, generally above the GB or TB level. Among them, the hot indicator data requires high concurrency, short query time, and high query efficiency.

[0064] In view of this, a business data management method, system, electronic device and storage medium are provided in an embodiment of the present application. The scheme constructs a preset business indicator library including a first storage unit and a second storage unit. Accordingly, an embodiment of the present invention obtains business data to be stored, including indicator metadata and indicator numerical data. Then, an embodiment of the present invention stores the obtained indicator metadata in the first storage unit, and stores the indicator numerical data in the second storage unit to construct a target business indicator library. Finally, when it is determined that a query indicator request is received, an embodiment of the present invention obtains the target business data by querying from the target business indicator library according to the query indicator request, thereby realizing the management of business data and effectively improving the query efficiency and query performance of business data.

[0065] The business data management method provided in the embodiment of the present application relates to the field of big data technology. The business data management method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the business data management method, etc., but is not limited to the above forms.

[0066] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0067] Figure 1 is an optional flowchart of the business data management method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S110 to S140.

[0068] Step S110: constructing a preset business indicator library, wherein the preset business indicator library includes a first storage unit and a second storage unit.

[0069] Step S120: Acquire the business data to be stored, wherein the business data to be stored includes indicator metadata and indicator numerical data, and the indicator metadata and the indicator numerical data correspond to each other.

[0070] Step S130: storing the indicator metadata in the first storage unit, and storing the indicator numerical data in the second storage unit, so as to construct a target business indicator library.

[0071] Step S140: When it is determined that a query indicator request is received, the target business data is obtained from the target business indicator library according to the query indicator request.

[0072] During the working process of this specific embodiment, the embodiment of the present invention first constructs a preset business indicator library. Specifically, the preset business indicator library in the embodiment of the present invention includes a first storage unit and a second storage unit. Among them, the first storage unit in the embodiment of the present invention is used to store write data, and the second storage unit is used to store read data, so as to construct a preset business indicator library with read-write separation. Through the read-write separation architecture, the query efficiency of business data is improved. Next, the embodiment of the present invention obtains the business data to be stored. Specifically, the business data to be stored in the embodiment of the present invention includes indicator metadata and indicator numerical data, and the indicator metadata corresponds to the indicator numerical data. Among them, the indicator metadata in the embodiment of the present invention refers to the relevant metadata information of the business data, such as indicator definition, preset dimension information and dimension association relationship. Correspondingly, the indicator numerical data in the embodiment of the present invention refers to the numerical data of each business indicator under each dimension. Further, the embodiment of the present invention stores the acquired indicator metadata in the first storage unit, and stores the indicator numerical data in the second storage unit, so as to construct a target business indicator library. Specifically, the embodiment of the present invention stores write data, i.e., indicator metadata, through a first storage unit, and stores read data, i.e., indicator numerical data, through a second storage unit, thereby performing read-write separation storage on the business data to be stored, thereby optimizing storage efficiency. Finally, when a query indicator request is received, the embodiment of the present invention obtains target business data from a target business indicator library in accordance with the query indicator request. Specifically, when it is necessary to perform operations such as data analysis or visualization processing on the relevant business data, the data in the target business indicator library is queried through the corresponding query indicator request, for example, the data of a certain indicator in different geographical dimensions is queried to obtain the corresponding target business data. The embodiment of the present invention can optimize storage efficiency and improve query efficiency and query performance of business data by separating read and write data.

[0073] Reference Figure 2 In some embodiments of the present invention, building a preset business indicator library includes but is not limited to the following steps:

[0074] Step S210: constructing a first storage unit through a preset relational database.

[0075] Step S220: constructing a second storage unit by using a preset column database.

[0076] Step S230: constructing a preset business indicator library according to the first storage unit and the second storage unit.

[0077] In this specific embodiment, the embodiment of the present invention first constructs a first storage unit through a preset relational database, and constructs a second storage unit through a preset column database, so as to obtain a preset business indicator library according to the first storage unit and the second storage unit. Specifically, the preset relational database in the embodiment of the present invention refers to a database established on a relational database model, and the data in the database is processed by means of the concept and method of set algebra, such as a MySQL database, an Oracle database, etc. Correspondingly, the preset column database in the embodiment of the present invention refers to a database that stores data in columns, such as a ClickHouse database, a SAP HANA database, a ParAccel database, etc. Exemplarily, the embodiment of the present invention constructs a first storage unit through a MySQL database to store indicator metadata in business data, and constructs a second storage unit through a ClickHouse database to store indicator numerical data in business data, thereby realizing the separate storage of read data and write data, wherein the indicator metadata, such as indicator definition, preset dimension information, dimension association relationship, etc., is stored through a relational database, and the actual indicator numerical data is stored in a column-based storage manner through a column-based database, thereby optimizing query performance and query efficiency.

[0078] Reference Figure 3 In some embodiments of the present invention, the first storage unit is constructed by presetting a relational database, including but not limited to the following steps:

[0079] Step S310: constructing a preset indicator table according to the preset indicator information.

[0080] Step S320: construct a preset dimension table according to the preset dimension information.

[0081] Step S330: construct a dimension dictionary table according to the preset dimension table.

[0082] Step S340: construct an indicator-dimension association table according to the preset indicator table and the preset dimension table.

[0083] Step S350: constructing a first storage unit through a preset indicator table, a preset dimension table, a dimension dictionary table and an indicator dimension association table.

[0084] In this specific embodiment, the embodiment of the present invention first constructs a preset indicator table according to the preset indicator information. Specifically, the preset indicator information in the embodiment of the present invention refers to the definition information of the indicator. Accordingly, the preset indicator table refers to a data table storing the definition information of the relevant indicators, which includes an automatically incremented indicator unique identifier (Metric_ID), an indicator name (Metric_Name), and an indicator description information (Metric_Description). Exemplarily, the code example for constructing the preset indicator table in the embodiment of the present invention is as follows: CREATE TABLE Metrics_Definition(

[0085] Metric_ID INT AUTO_INCREMENT PRIMARY KEY,

[0086] Metric_Name VARCHAR(255)NOT NULL,

[0087] Metric_Description TEXT);

[0088] Next, the embodiment of the present invention constructs a preset dimension table based on the preset dimension information. Specifically, the preset dimension information in the embodiment of the present invention refers to the definition information of each dimension, such as geographic dimension (province, city), time dimension (year, month, day), etc. Correspondingly, the embodiment of the present invention constructs a preset dimension table to store relevant dimension definition information. Among them, the preset dimension table in the embodiment of the present invention includes an automatically incremented dimension unique identifier (Dimension_ID), a dimension name (Dimension_Name), a dimension type (Dimension_Type), and a dimension description (Dimension_Description). Exemplarily, the code example for constructing the preset dimension table in the embodiment of the present invention is as follows:

[0089] CREATE TABLE Dimensions(

[0090] Dimension_ID INT AUTO_INCREMENT PRIMARY KEY,

[0091] Dimension_Name VARCHAR(255)NOT NULL,

[0092] Dimension_Type VARCHAR(128),

[0093] Dimension_Description TEXT);

[0094] Furthermore, an embodiment of the present invention constructs a dimension dictionary table based on a preset dimension table. Specifically, a dimension dictionary table in an embodiment of the present invention refers to a data table for storing dimension value information (Dimension Values). Among them, a dimension value is a specific example associated with a corresponding dimension. For example, when the dimension is divided by geography, the dimension value under the "province" dimension can be the corresponding "city". Accordingly, the dimension dictionary table in an embodiment of the present invention includes an automatically incremented dimension value unique identifier (Dimension_Values_ID), a dimension unique identifier associated with the dimension (Dimension_ID, as a foreign key), and a dimension value (Dimension_Values). Exemplarily, the code for constructing a dimension dictionary table in an embodiment of the present invention is as follows:

[0095] CREATE TABLE Dimension_Values(

[0096] Dimension_Value_ID INT AUTO_INCREMENT PRIMARY KEY,

[0097] Dimension_ID INT NOT NULL,Value VARCHAR(255)NOT NULL,

[0098] FOREIGN KEY(Dimension_ID)REFERENCESDimensions(Dimension_ID));

[0099] Furthermore, the embodiment of the present invention constructs an indicator dimension relationship table based on a preset indicator table and a preset dimension table. Specifically, the indicator dimension relationship table in the embodiment of the present invention refers to a data table used to establish a corresponding relationship between indicators and dimensions. Among them, the indicator dimension relationship table in the embodiment of the present invention defines a composite primary key and two foreign key constraints to ensure the uniqueness and integrity of the data, thereby improving the efficiency of managing and querying the relationship between indicators and dimensions. Exemplarily, the code for constructing the indicator dimension association table in the embodiment of the present invention is as follows:

[0100] CREATE TABLE Metric_Dimensions(

[0101] Metric_ID INT NOT NULL,

[0102] Dimension_ID INT NOT NULL,

[0103] Dimension_Order INT,

[0104] PRIMARY KEY(Metric_ID,Dimension_ID),

[0105] FOREIGN KEY(Metric_ID)REFERENCES Metrics_Definition(Metric_ID),FOREIGN KEY(Dimension_ID)REFERENCESDimensions(Dimension_ID));

[0106] Finally, the embodiment of the present invention constructs a first storage unit by constructing the preset indicator table, preset dimension table, dimension dictionary table and indicator dimension association table. Specifically, the embodiment of the present invention constructs a preset indicator table, a preset dimension table, a dimension dictionary table and an indicator dimension association table in a MySQL database (a preset relational database) to store indicator metadata in business data, such as indicator definition, preset dimension information, and dimension association relationship, to construct a first storage unit.

[0107] Reference Figure 4 In some embodiments of the present invention, the second storage unit is constructed by presetting a column-based database, including but not limited to the following steps:

[0108] Step S410: constructing a preset partition key through preset dimension data, wherein the preset dimension data includes a time dimension or a geographic dimension.

[0109] Step S420: construct a preset data column according to the preset dimension information and the preset indicator information.

[0110] Step S430: construct an indicator value table according to the preset partition key and the preset data column, so as to obtain a second storage unit through the indicator value table.

[0111] In this specific embodiment, the embodiment of the present invention first constructs a preset partition key through preset dimension data, and constructs a preset number series according to the preset dimension information and the preset indicator information, and then constructs an indicator value table according to the preset partition key and the preset data column, so as to construct a second storage unit through the indicator value table. Specifically, the embodiment of the present invention creates a table for each indicator, and corresponds to each dimension and metric value of the indicator through the column of the table. Among them, the embodiment of the present invention constructs a preset partition key through different preset dimension data, including time dimension or geographical dimension, to dynamically partition the data, thereby optimizing the query performance. For example, the embodiment of the present invention partitions and stores the business data in dimensions such as time and region, and defines a data column as a partition key through the preset dimension data to store the data in different partitions, thereby optimizing the query performance of the business data. Then, the embodiment of the present invention constructs a data column according to the corresponding preset indicator information, such as the indicator unique identifier (Metric_ID), for storing the indicator unique identifier (Metric_ID) of the indicator data. At the same time, the embodiment of the present invention constructs the corresponding data column according to the corresponding preset dimension information, such as each dimension value information. Furthermore, the embodiment of the present invention combines the preset partition key and each preset data column to create a table storing the indicator values ​​of each indicator, thereby constructing a second storage unit. Exemplarily, the code for constructing the indicator value table in the embodiment of the present invention is as follows: CREATE TABLE IF NOT EXISTS clickhouse.metric_values(

[0112] date Date, / / Partition key

[0113] metric_id INT,

[0114] dimension1_value String, / / The value of the first dimension. The specific name is determined by the dimension table dimension2_value String, / / The value of the second dimension. The specific name is determined by the dimension table Dimension3_value String, / / The value of the third dimension, and so on

[0115] value Float64 / / index value)ENGINE=MergeTree() ORDER BY(dimension1_value,dimension2_value,date);

[0116] Reference Figure 5 In some embodiments of the present invention, when it is determined that a query indicator request is received, querying and obtaining target business data from a target business indicator library according to the query indicator request includes but is not limited to the following steps:

[0117] Step S510: extracting target indicator information and target dimension information from the first storage unit according to the query indicator request.

[0118] Step S520: Query and obtain target business data from the second storage unit according to the target indicator information and the target dimension information.

[0119] In this specific embodiment, the embodiment of the present invention first obtains the target indicator information and target dimension information from the first storage unit according to the query indication request. Specifically, the query indication request in the embodiment of the present invention refers to a request to query the corresponding business data, such as querying the sum of indicators of a province within a specific date range. Accordingly, after receiving the query indicator request, the embodiment of the present invention first determines the corresponding target indicator information and target dimension information from the first storage unit. For example, a specific indicator name (Metric_Name), dimension name (Dimension_Name), and corresponding dimension values ​​(such as province and city) are retrieved from several associated tables through SQL query instructions. Exemplarily, the code for extracting the target indication information and target dimension information from the first storage unit in the embodiment of the present invention is as follows:

[0120] SELECT md.Metric_Name,

[0121] d.Dimension_Name,

[0122] dv.Value AS Province,

[0123] dv2.Value AS City

[0124] FROM Metrics_Definition md

[0125] JOIN Metric_Dimensions mdim ON md.Metric_ID=mdim.Metric_ID

[0126] JOIN Dimensions d ON mdim.Dimension_ID=d.Dimension_ID

[0127] JOIN Dimension_Values ​​dv ON d.Dimension_ID=dv.Dimension_ID

[0128] JOIN Dimension_Values ​​dv2 ON dv.Dimension_Value_ID=dv2.Dimension_Value_ID

[0129] WHERE md.Metric_Name = 'Target indicator';

[0130] Next, the embodiment of the present invention obtains the target business data from the second storage unit based on the target indicator information and the target dimension information. Specifically, the embodiment of the present invention queries the indicator numerical data of all records in the corresponding range based on the indicator and dimension related information (such as the corresponding province, city, date, etc.) obtained by querying the first storage unit, thereby extracting the target business data. Exemplarily, the code for obtaining the target business data from the second storage unit in the embodiment of the present invention is as follows:

[0131] SELECT sum(value)FROM clickshouse.metric_values ​​WHERE dimension1_value='queried province'AND dimension2_value='queried city'AND date BETWEEN 'start date'AND 'end date';

[0132] Reference Figure 6 In some embodiments of the present invention, after executing the construction of the preset business indicator library, the business data management method provided by the embodiment of the present invention further includes but is not limited to the following steps:

[0133] Step S610: Obtain metadata modification instructions.

[0134] Step S620: Performing a preset management operation on the first storage unit according to the metadata modification instruction, wherein the preset management operation includes an indicator definition operation, a dimension configuration setting, and an association relationship setting.

[0135] In this specific embodiment, the embodiment of the present invention obtains a metadata modification instruction to perform a preset management operation on the first storage unit according to the metadata modification instruction. Specifically, when it is necessary to manage the indicator metadata, the embodiment of the present invention manages the indicator metadata through the corresponding metadata modification instruction, such as the addition, deletion, modification and query processing of the indicator metadata. Accordingly, the preset management operation performed on the first storage unit in the embodiment of the present invention includes indicator definition operation, dimension configuration setting and association relationship setting. Among them, the indicator definition operation refers to setting the preset indicator information, such as adding or deleting indicators. In addition, dimension configuration refers to configuring dimension data, such as modifying the dimension and the corresponding dimension value. Correspondingly, the association relationship setting refers to setting the association relationship between the dimension and the indicator. In addition, the management of indicator metadata in the embodiment of the present invention also includes indicator version management, indicator derivative management and indicator lineage management. Among them, when the indicator caliber changes, the embodiment of the present invention updates the indicator metadata according to the new indicator version, that is, uses the new indicator version to manage the indicator. In addition, the indicator management in the embodiment of the present invention generates derived indicators through simple four arithmetic operations based on the atomic indicator. Accordingly, the embodiment of the present invention can manage the derived indicators, such as deleting and modifying the association relationship. At the same time, the embodiment of the present invention manages the indicator lineage relationship according to the corresponding metadata modification instruction. It is easy to understand that the embodiment of the present invention improves the efficiency of business data management by providing indicator metadata management to facilitate addition, deletion, modification and query of indicator metadata.

[0136] Reference Figure 7 In some embodiments of the present invention, after storing the indicator metadata in the first storage unit and storing the indicator numerical data in the second storage unit to construct a target business indicator library, the business data management method provided by the embodiment of the present invention further includes but is not limited to the following steps:

[0137] Step S710: Determine hotspot query data in the target business indicator library according to the data history query information.

[0138] Step S720: storing the hotspot query data into a preset cache database.

[0139] In this specific embodiment, the embodiment of the present invention first determines the hot query data in the target business indicator library according to the data history query information, and then stores the hot query data in the preset cache database. Specifically, the historical query information in the embodiment of the present invention refers to the relevant data information of each business data in the target business indicator library being accessed and queried, such as the number of accesses or access frequency within a certain time period. Correspondingly, the hot query data in the embodiment of the present invention refers to the hot data with a high access frequency or number in the target business indicator library. For example, the embodiment of the present invention queries the data history query information to determine whether the access frequency or number of the corresponding business data is greater than the preset access threshold. When it is determined that the access comments or access times of a certain business data are greater than the preset access threshold, this marks the business data as hot query data. Correspondingly, the embodiment of the present invention caches the determined hot query data in the preset cache database. Among them, the preset cache database in the embodiment of the present invention is a data structure storage system based on memory, such as a Redis cache database. For example, the embodiment of the present invention stores the hot query data in the target business database through the Redis cache database, thereby constructing a cache mechanism for managing hot data through the Redis cache service, improving the query response speed, and supporting 5000+ concurrent millisecond data query responses at the scale of 100 million data.

[0140] It should be noted that in some embodiments of the present invention, the embodiments of the present invention obtain the business data to be stored from the original system through a preset data synchronization layer, and synchronize it to the first storage unit and the second storage unit. Among them, the embodiments of the present invention receive corresponding query index requests through the API interface of the query interface layer or the corresponding query interface, so that the front-end and back-end or other services can query the index data. Correspondingly, the embodiments of the present invention interact through the corresponding front-end display layer to display the target business data obtained by the query and to perform data query. In addition, the embodiments of the present invention access self-service reports and self-service analysis through the front-end display layer, thereby realizing the automatic generation of data reports and data dashboards.

[0141] Below, in conjunction with specific business data management scenarios / specific application examples, the solution of the embodiment of the present invention is described in detail and explained:

[0142] For example, Figure 8 As shown, Figure 8It is a schematic diagram of the architecture of the business data management method provided by an embodiment of the present invention. Specifically, the embodiment of the present invention constructs a preset business indicator library including a first storage unit (MySQL database) and a second storage unit (ClickHouse database). Among them, in the embodiment of the present invention, an indicator definition table, a dimension table, a dimension dictionary table, and an indicator dimension association table are provided in the MySQL database to store indicator metadata. In addition, in the embodiment of the present invention, the ClickHouse database is provided with an indicator value table to store indicator values. Correspondingly, the embodiment of the present invention obtains business data to be stored, including indicator metadata and indicator value data. For example, the embodiment of the present invention obtains the business data to be stored by obtaining from various source data systems, such as HDFS, Hive, Kafka, and API data sources. Then, the embodiment of the present invention stores the obtained indicator metadata in the first storage unit, and stores the indicator value data in the second storage unit to construct a target business indicator library. Finally, when it is determined that a query indicator request is received, the embodiment of the present invention queries the target business data from the target business indicator library according to the query indicator request. For example, when data analysis, report generation, visual dashboard or monitoring and early warning are required, a query indicator request of a response is sent to the target business indicator library. Then, the embodiment of the present invention obtains the corresponding target indicator information and target dimension information from the MySQL database according to the corresponding query indicator request, and then obtains the corresponding target business data from the ClickHouse database. Among them, the embodiment of the present invention monitors the key business indicators in real time through the monitoring and early warning module to issue an early warning according to the preset threshold. In addition, in some embodiments of the present invention, the facility data backup and recovery strategy is used to protect data security, and the system scalability is considered to facilitate the addition of new indicators or dimensions, as well as the flexible configuration and management of complex dimensional relationships.

[0143] It is easy to understand that the respective advantages of MySQL and ClickHouse in the embodiment of the present invention realize the separate storage and query of metadata and numerical data. At the same time, the embodiment of the present invention significantly improves the query efficiency of large amounts of data through the column storage structure of ClickHouse. Correspondingly, the embodiment of the present invention partitions the data according to dimensions such as time and region to further optimize query performance. In addition, the embodiment of the present invention effectively improves query efficiency through technical means such as pre-calculation, indexing, and query caching. At the same time, the embodiment of the present invention ensures the real-time and accuracy of data and reduces manual intervention by automating the data extraction, conversion, and loading processes. In addition, the embodiment of the present invention effectively improves query efficiency through pre-calculation, indexing, query caching, etc. For example, an embodiment of the present invention uses appropriate indexing and partitioning strategies in ClickHouse to optimize query performance.

[0144] See also Fig. 9 The embodiment of the present application also provides a business data management system, which can implement the above-mentioned business data management method, and the system includes:

[0145] The first module 810 is used to construct a preset business indicator library, wherein the preset business indicator library includes a first storage unit and a second storage unit.

[0146] The second module 820 is used to obtain the business data to be stored, wherein the business data to be stored includes indicator metadata and indicator numerical data, and the indicator metadata and the indicator numerical data correspond to each other.

[0147] The third module 830 is used to store the indicator metadata into the first storage unit and store the indicator numerical data into the second storage unit to construct a target business indicator library.

[0148] The fourth module 840 is used to query and obtain target business data from the target business indicator library according to the query indicator request when it is determined that a query indicator request is received.

[0149] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0150] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned business data management method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0151] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0152] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0153] The processor 910 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0154] The memory 920 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 920, and the processor 910 calls and executes the service data management method of the embodiment of this application;

[0155] Input / output interface 930, used to implement information input and output;

[0156] Communication interface 940, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0157] bus 950 , which transmits information between the various components of the device (e.g., processor 910 , memory 920 , input / output interface 930 , and communication interface 940 );

[0158] The processor 910 , the memory 920 , the input / output interface 930 , and the communication interface 940 are connected to each other in communication within the device via a bus 950 .

[0159] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned business data management method is implemented.

[0160] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0161] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0162] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0163] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0164] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0166] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0167] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0168] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0169] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0170] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0171] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0172] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A business data management method, characterized in that: The method comprises the following steps: Constructing a preset business indicator library; wherein the preset business indicator library includes a first storage unit and a second storage unit; Acquire the business data to be stored; wherein the business data to be stored includes indicator metadata and indicator numerical data, and the indicator metadata and the indicator numerical data correspond to each other; The indicator metadata is stored in the first storage unit, and the indicator numerical data is stored in the second storage unit, so as to construct a target business indicator library; When it is determined that a query indicator request is received, target business data is obtained by querying from the target business indicator library according to the query indicator request.

2. The method according to claim 1, characterized in that: The construction of the preset business indicator library includes: Constructing the first storage unit by using a preset relational database; Constructing the second storage unit by using a preset column database; The preset business indicator library is constructed according to the first storage unit and the second storage unit.

3. The method according to claim 2, characterized in that The step of constructing the first storage unit by using a preset relational database includes: Construct a preset indicator table according to preset indicator information; Construct a preset dimension table based on preset dimension information; Constructing a dimension dictionary table according to the preset dimension table; Constructing an indicator dimension association table according to the preset indicator table and the preset dimension table; The first storage unit is constructed by constructing the preset indicator table, the preset dimension table, the dimension dictionary table and the indicator dimension association table.

4. The method according to claim 3, characterized in that The step of constructing the second storage unit by using a preset column database includes: Constructing a preset partition key through preset dimension data; wherein the preset dimension data includes a time dimension or a geographic dimension; Constructing a preset data column according to the preset dimension information and the preset indicator information; An indicator value table is constructed according to the preset partition key and the preset data column, so as to obtain the second storage unit through the indicator value table.

5. The method according to claim 1, characterized in that When it is determined that a query indicator request is received, querying the target business indicator library to obtain target business data according to the query indicator request includes: Extracting target indicator information and target dimension information from the first storage unit according to the query indicator request; The target business data is obtained by querying from the second storage unit according to the target indicator information and the target dimension information.

6. The method according to claim 1, characterized in that After executing the step of building the preset business indicator library, the method further includes: Get metadata modification instructions; Performing preset management operations on the first storage unit according to the metadata modification instruction; wherein the preset management operations include indicator definition operations, dimension configuration settings, and association relationship settings.

7. The method according to claim 1, characterized in that After executing the step of storing the indicator metadata in the first storage unit and storing the indicator numerical data in the second storage unit to construct a target business indicator library, the method further includes: Determine the hotspot query data in the target business indicator library according to the data history query information; The hotspot query data is stored in a preset cache database.

8. A business data management system, characterized in that: The system comprises: The first module is used to construct a preset business indicator library; wherein the preset business indicator library includes a first storage unit and a second storage unit; The second module is used to obtain the business data to be stored; wherein the business data to be stored includes indicator metadata and indicator numerical data, and the indicator metadata corresponds to the indicator numerical data; The third module is used to store the indicator metadata into the first storage unit, and store the indicator numerical data into the second storage unit, so as to construct a target business indicator library; The fourth module is used to query and obtain target business data from the target business indicator library according to the query indicator request when it is determined that a query indicator request is received.

9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images