Index storage method and query method, device, equipment and medium

Abstract

The application provides a storage method and a query method, a device, equipment and a medium for an index, wherein the storage method for the index comprises the following steps: generating a derived real-time index based on an atomic real-time index; determining a business line identifier and an index storage strategy associated with the derived real-time index; determining a target storage sub-library from a real-time index library based on the business line identifier; and storing the atomic real-time index and the derived real-time index into the target storage sub-library based on the index storage strategy. The application can shorten the query time of the real-time index and improve the query efficiency of the real-time index.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method for storing and querying indicators, an apparatus, device, and medium. Background Technology

[0002] Data metrics (also known as indicators) are aggregated results obtained through statistical analysis of data. They quantify the achievements of business operations and management, making business objectives describable, measurable, and decomposable, thus giving data real value. Therefore, companies can make indicator-based decisions for their business operations.

[0003] Currently, the types of metrics include: offline metrics and real-time metrics. For example... Figure 1a As shown, the management process of offline metrics involves using an offline computing engine (such as Spark) to clean and process a batch of data accumulated in the data warehouse tables to generate offline metrics. After that, the offline metrics are stored in a wide table structure in a database that supports column storage (such as Hive, ClickHouse, and Impala). Figure 1a The medium-width table includes three types of fields: primary key, dimensions, and metrics. The primary key is user ID, the dimensions are time and gender, and the metrics are access duration, access count, and others. Because column-based storage stores data from a specific column of offline metrics together, subsequent reading of data from a specific column is more efficient than a full read. For example, when querying the access count column, the metrics database only needs to return the access count data, without needing to read the entire stored metrics, thus improving query speed and efficiency.

[0004] like Figure 1b As shown, the management process of real-time metrics involves using real-time computing engines such as Flink to clean and process data from message queues to generate real-time metrics. These metrics are then stored in a database (such as Kafka) using a wide table structure. However, the generation process for real-time metrics differs from that of offline metrics. Specifically, real-time metrics are calculated for each incoming real-time data record. Therefore, columnar storage is not feasible in the storage implementation because it requires accumulating a batch of data before merging identical fields from different records. Consequently, when a user queries a real-time metric, the real-time computing engine needs to read all data records related to the queried metric and then filter out the target field from all the read data to obtain the requested real-time metric. This method of querying real-time metrics is time-consuming, resulting in low query efficiency. Summary of the Invention

[0005] This application provides a method, apparatus, device, and medium for storing and querying indicators, which can shorten the query time of real-time indicators and improve the query efficiency of real-time indicators.

[0006] Firstly, this application provides a method for storing indicators, including:

[0007] Derived real-time indicators are generated based on atomic real-time indicators.

[0008] Determine the service line identifier and indicator storage strategy associated with the derived real-time indicator;

[0009] Based on the business line identifier, the target storage sub-library is determined from the real-time indicator library;

[0010] Based on the aforementioned indicator storage strategy, the atomic real-time indicators and the derived real-time indicators are stored in the target storage sub-library.

[0011] In some optional implementations, storing the atomic real-time metrics and the derived real-time metrics in the target storage sub-library based on the metric storage strategy includes:

[0012] Obtain the partition flow table corresponding to the aforementioned indicator storage strategy;

[0013] Determine the conditional expression corresponding to the partition field in the partitioned flow table;

[0014] Based on the pushdown mechanism of the real-time computing engine, the conditional expression is pushed down to the programming language to code conversion module of the real-time computing engine, so that the conditional expression can be converted into storage rule code that the real-time computing engine can recognize through the programming language to code conversion module;

[0015] Based on the storage rule code, the atomic real-time indicators and the derived real-time indicators are stored in the target storage sub-library.

[0016] In some optional implementations, the target storage sub-library includes at least one storage space, and the step of storing the atomic real-time metrics and the derived real-time metrics into the target storage sub-library based on the storage rule code includes:

[0017] Based on the storage rule code, determine the first target storage space corresponding to the atomic real-time indicator and the second target storage space corresponding to the derived real-time indicator;

[0018] The atomic real-time metrics are stored in the first target storage space, and the derived real-time metrics are stored in the corresponding second target storage space.

[0019] Some alternative implementations also include:

[0020] The partition flow table and the mapping relationship between the atomic real-time metrics and the first target storage space are stored as the metadata of the atomic real-time metrics.

[0021] The partitioned flow table, as well as the mapping relationship between the derived real-time metrics and the second target storage space, are stored as metadata for the derived real-time metrics.

[0022] In some optional implementations, when the indicator storage strategy is the first storage strategy, obtaining the partition flow table corresponding to the indicator storage strategy includes:

[0023] Obtain the first partition flow table corresponding to the first storage policy;

[0024] The first partition flow table includes at least a first partition field and a first conditional expression corresponding to the first partition field. The first conditional expression is determined based on the atomic real-time indicator identifier, the derived real-time indicator identifier, and the preset storage space. The preset storage space is at least two partitions in the target storage sub-database.

[0025] In some optional implementations, the first conditional expression is: when the condition satisfies the atomic real-time indicator identifier, the result is returned as a first preset storage space; when the condition satisfies the derived real-time indicator identifier, the result is returned as a second preset storage space; when all conditions are not satisfied, the result is returned as a third preset storage space and the process ends.

[0026] In some optional implementations, when the indicator storage strategy is the second storage strategy, obtaining the partition flow table corresponding to the indicator storage strategy includes:

[0027] Obtain the second partition flow table corresponding to the second storage strategy;

[0028] The second partition flow table includes at least a second partition field and a second condition expression corresponding to the second partition field. The second condition expression is determined based on a first modulo value obtained by hashing and modulo operations on the atomic real-time indicator identifier, a second modulo value obtained by hashing and modulo operations on the derived real-time indicator identifier, and a preset storage space. The preset storage space consists of at least two partitions in the target storage sub-database.

[0029] In some optional implementations, the second conditional expression is: when the condition is satisfied and the modulus is equal to the first value, the result is returned as the first preset storage space; when the condition is satisfied and the modulus is equal to the second value, the result is returned as the second preset storage space; when none of the conditions are satisfied, the result is returned as the third preset storage space and the process ends.

[0030] In some optional implementations, when the indicator storage strategy is a third storage strategy, obtaining the partition flow table corresponding to the indicator storage strategy includes:

[0031] Obtain the third partition flow table corresponding to the third storage strategy;

[0032] The third partition flow table includes at least a third partition field and a third condition expression corresponding to the third partition field. The third condition expression is determined based on a first modulo value obtained by hashing and modulo operations on the atomic real-time indicator identifier, a second modulo value obtained by hashing and modulo operations on the derived real-time data indicator identifier, a preset storage space, and a preset flow value of the preset storage space. The preset storage space consists of at least two partitions in the target storage sub-database.

[0033] In some optional implementations, the third condition expression is as follows: when the condition is satisfied that the modulus is equal to the first value, the result is the first preset storage space and the preset flow value of the first preset storage space; when the condition is satisfied that the modulus is equal to the second value, the result is the second preset storage space and the preset flow value of the second preset storage space; when all conditions are not satisfied, the result is the third preset storage space and the preset flow value of the third preset storage space, and the process ends.

[0034] In some optional implementations, before storing the atomic real-time metrics and the derived real-time metrics into the target storage sub-library based on the storage rule code, the method further includes:

[0035] When the storage rule code corresponds to the third storage strategy, the current traffic value of each preset storage space is detected;

[0036] In response to any current traffic value of the preset storage space being greater than the preset traffic value, the preset traffic value of the preset storage space is expanded.

[0037] In some optional implementations, storing the atomic real-time metrics and the derived real-time metrics in the target storage sub-library includes:

[0038] Obtain the data storage structure corresponding to the target storage sub-library;

[0039] According to the data storage structure, the atomic real-time indicators and the derived real-time indicators are stored in the target storage sub-library.

[0040] In some optional implementations, the generation of derived real-time metrics based on atomic real-time metrics includes:

[0041] Obtain the processing parameters corresponding to the real-time atomic indicators;

[0042] The derived real-time index is generated based on the atomic real-time index and the processing parameters.

[0043] Secondly, this application provides a method for querying indicators, including:

[0044] In response to a user's real-time metric query request, obtain the identifier of the real-time metric to be queried;

[0045] Based on the identifier of the real-time indicator to be queried, determine the target storage location of the real-time indicator to be queried in the real-time indicator database.

[0046] Based on the target storage location, the real-time indicator to be queried is obtained from the real-time indicator library and displayed to the user.

[0047] In some optional implementations, determining the target storage location of the real-time indicator to be queried in the real-time indicator database based on the identifier of the real-time indicator to be queried includes:

[0048] Determine the metadata corresponding to the real-time indicator identifier to be queried;

[0049] Based on the metadata, the target storage location of the real-time indicator to be queried in the real-time indicator database is determined.

[0050] In some optional implementations, determining the target storage location of the real-time metric to be queried in the real-time metric database based on the metadata includes:

[0051] Based on the aforementioned metadata, the target partition flow table is determined;

[0052] The conditional expressions corresponding to the partition fields in the target partition flow table are converted into code to transform the conditional expressions into storage rule codes that can be recognized by the real-time computing engine.

[0053] Based on the storage rule code, the target storage location of the real-time indicator to be queried in the real-time indicator library is determined.

[0054] In some optional implementations, determining the target storage location of the real-time metric to be queried in the real-time metric database based on the metadata includes:

[0055] Based on the metadata, determine the mapping relationship between the real-time metric to be queried and the storage space;

[0056] Based on the mapping relationship between the real-time indicator to be queried and the storage space, the target storage location of the real-time indicator to be queried in the real-time indicator library is determined.

[0057] In some optional implementations, obtaining the real-time indicator to be queried from the real-time indicator library based on the target storage location includes:

[0058] Based on the target storage location, multiple discrete target fields are obtained from the real-time indicator library;

[0059] The real-time metric to be queried is obtained based on the multiple target discrete fields.

[0060] Thirdly, this application provides a storage device for indicators, comprising:

[0061] The indicator generation module is used to generate derived real-time indicators based on atomic real-time indicators.

[0062] The first determining module is used to determine the business line identifier and indicator storage strategy associated with the derived real-time indicator;

[0063] The second determining module is used to determine the target storage sub-library from the real-time indicator library based on the business line identifier;

[0064] The indicator storage module is used to store the atomic real-time indicators and the derived real-time indicators into the target storage sub-library based on the indicator storage strategy.

[0065] In some optional implementations, the indicator storage module includes:

[0066] The acquisition unit is used to acquire the partition flow table corresponding to the indicator storage strategy;

[0067] The first determining unit is used to determine the conditional expression corresponding to the partition field in the partition flow table;

[0068] The conversion unit is used to push down the conditional expression to the programming language to code conversion module of the real-time computing engine based on the push-down mechanism of the real-time computing engine, so that the conditional expression can be converted into stored rule code that the real-time computing engine can recognize through the programming language to code conversion module.

[0069] A storage unit is used to store the atomic real-time indicators and the derived real-time indicators into the target storage sub-library based on the storage rule code.

[0070] In some alternative implementations, the target storage sub-library includes at least one storage space, the storage unit being specifically used for:

[0071] Based on the storage rule code, a first target storage space corresponding to the atomic real-time indicator and a second target storage space corresponding to the derived real-time indicator are determined; the atomic real-time indicator is stored in the first target storage space, and the derived real-time indicator is stored in the corresponding second target storage space.

[0072] In some alternative implementations, the storage unit is further used for:

[0073] The partition flow table and the mapping relationship between the atomic real-time metrics and the first target storage space are stored as the metadata of the atomic real-time metrics.

[0074] The partitioned flow table, as well as the mapping relationship between the derived real-time metrics and the second target storage space, are stored as metadata for the derived real-time metrics.

[0075] In some optional implementations, when the indicator storage strategy is the first storage strategy, the acquisition unit is specifically used for:

[0076] Obtain the first partition flow table corresponding to the first storage policy;

[0077] The first partition flow table includes at least a first partition field and a first conditional expression corresponding to the first partition field. The first conditional expression is determined based on the atomic real-time indicator identifier, the derived real-time indicator identifier, and the preset storage space. The preset storage space is at least two partitions in the target storage sub-database.

[0078] In some optional implementations, the first conditional expression is: when the condition satisfies the atomic real-time indicator identifier, the result is returned as a first preset storage space; when the condition satisfies the derived real-time indicator identifier, the result is returned as a second preset storage space; when all conditions are not satisfied, the result is returned as a third preset storage space and the process ends.

[0079] In some optional implementations, when the indicator storage strategy is the second storage strategy, the acquisition unit is specifically used for:

[0080] Obtain the second partition flow table corresponding to the second storage strategy;

[0081] The second partition flow table includes at least a second partition field and a second condition expression corresponding to the second partition field. The second condition expression is determined based on a first modulo value obtained by hashing and modulo operations on the atomic real-time indicator identifier, a second modulo value obtained by hashing and modulo operations on the derived real-time indicator identifier, and a preset storage space. The preset storage space consists of at least two partitions in the target storage sub-database.

[0082] In some optional implementations, the second conditional expression is: when the condition is satisfied and the modulus is equal to the first value, the result is returned as the first preset storage space; when the condition is satisfied and the modulus is equal to the second value, the result is returned as the second preset storage space; when none of the conditions are satisfied, the result is returned as the third preset storage space and the process ends.

[0083] In some optional implementations, when the indicator storage strategy is a third storage strategy, the acquisition unit is specifically used for:

[0084] Obtain the third partition flow table corresponding to the third storage strategy;

[0085] The third partition flow table includes at least a third partition field and a third condition expression corresponding to the third partition field. The third condition expression is determined based on a first modulo value obtained by hashing and modulo operations on the atomic real-time indicator identifier, a second modulo value obtained by hashing and modulo operations on the derived real-time data indicator identifier, a preset storage space, and a preset flow value of the preset storage space. The preset storage space consists of at least two partitions in the target storage sub-database.

[0086] In some optional implementations, the third condition expression is as follows: when the condition is satisfied that the modulus is equal to the first value, the result is the first preset storage space and the preset flow value of the first preset storage space; when the condition is satisfied that the modulus is equal to the second value, the result is the second preset storage space and the preset flow value of the second preset storage space; when all conditions are not satisfied, the result is the third preset storage space and the preset flow value of the third preset storage space, and the process ends.

[0087] In some optional implementations, the indicator storage module further includes:

[0088] The detection unit is configured to detect the current traffic value of each preset storage space in response to the storage rule code corresponding to the third storage strategy;

[0089] An expansion unit is configured to expand the preset storage space to accommodate a preset traffic value in response to any current traffic value of the preset storage space being greater than the preset traffic value.

[0090] In some optional implementations, the indicator storage module is specifically used to: obtain the data storage structure corresponding to the target storage sub-library; and store the atomic real-time indicator and the derived real-time indicator into the target storage sub-library according to the data storage structure.

[0091] In some optional implementations, the indicator generation module is specifically used to: obtain the processing parameters corresponding to the atomic real-time indicator; and generate the derived real-time indicator based on the atomic real-time indicator and the processing parameters.

[0092] Fourthly, this application provides a query device for indicators, comprising:

[0093] The identifier acquisition module is used to obtain the identifier of the real-time indicator to be queried in response to the real-time indicator query operation sent by the user.

[0094] The third determining module is used to determine the target storage location of the real-time indicator to be queried in the real-time indicator database based on the identifier of the real-time indicator to be queried.

[0095] The indicator acquisition module is used to acquire the real-time indicator to be queried from the real-time indicator library based on the target storage location, and display the real-time indicator to be queried to the user.

[0096] In some alternative implementations, the third determining module includes:

[0097] The second determining unit is used to determine the metadata corresponding to the real-time indicator identifier to be queried;

[0098] The third determining unit is used to determine the target storage location of the real-time indicator to be queried in the real-time indicator database based on the metadata.

[0099] In some optional implementations, the third determining unit is specifically used to: determine the target partition flow table based on the metadata; perform code conversion on the condition expression corresponding to the partition field in the target partition flow table to convert the condition expression into a storage rule code recognizable by the real-time computing engine; and determine the target storage location of the real-time indicator to be queried in the real-time indicator library based on the storage rule code.

[0100] In some optional implementations, the third determining unit is specifically used to: determine the mapping relationship between the real-time indicator to be queried and the storage space based on the metadata; and determine the target storage location of the real-time indicator to be queried in the real-time indicator library according to the mapping relationship between the real-time indicator to be queried and the storage space.

[0101] In some optional implementations, the indicator acquisition module is specifically used to: acquire multiple discrete target fields from the real-time indicator library based on the target storage location; and obtain the real-time indicator to be queried based on the multiple discrete target fields.

[0102] Fifthly, this application provides an electronic device, comprising:

[0103] A processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to perform a method for storing indicators as described in the first aspect embodiment, or a method for querying indicators as described in the second aspect embodiment.

[0104] In a sixth aspect, this application provides a computer-readable storage medium for storing a computer program that causes a computer to perform a method for storing indicators as described in the first aspect embodiment, or a method for querying indicators as described in the second aspect embodiment.

[0105] In a seventh aspect, this application provides a computer program product containing program instructions that, when executed on an electronic device, cause the electronic device to perform the index storage method as described in the first aspect embodiment, or the index query method as described in the second aspect embodiment.

[0106] The technical solutions disclosed in the embodiments of this application have at least the following beneficial effects:

[0107] This application generates derived real-time indicators based on atomic real-time indicators, then determines the business line identifier and indicator storage strategy associated with the derived real-time indicators. Next, based on the business line identifier, a target storage sub-database is determined from the real-time indicator database. Finally, based on the indicator storage strategy, the atomic and derived real-time indicators are stored in the target storage sub-database. This method, by storing atomic and derived real-time indicators in the real-time indicator database based on the business line identifier associated with the derived indicators and the indicator storage strategy, shortens the query time and improves the query efficiency of real-time indicators. Attached Figure Description

[0108] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0109] Figure 1a This is a schematic diagram illustrating the storage of offline indicators in related technologies;

[0110] Figure 1b This is a schematic diagram illustrating the storage of real-time indicators in related technologies;

[0111] Figure 2 This is a flowchart illustrating a method for storing indicators provided in an embodiment of this application;

[0112] Figure 3This is a schematic diagram illustrating the generation of derived real-time indicators based on atomic real-time indicators and processing parameters, provided in an embodiment of this application.

[0113] Figure 4 This is a flowchart illustrating another method for storing indicators provided in an embodiment of this application;

[0114] Figure 5 This is a schematic diagram illustrating the storage of real-time metrics provided in an embodiment of this application;

[0115] Figure 6 This is a schematic diagram of a query method for an indicator provided in an embodiment of this application;

[0116] Figure 7 This is a schematic diagram of a visual query interface provided in an embodiment of this application;

[0117] Figure 8 This is a schematic block diagram of an index storage device provided in an embodiment of this application;

[0118] Figure 9 This is a schematic block diagram of an indicator query device provided in an embodiment of this application;

[0119] Figure 10 This is a schematic block diagram of an electronic device provided in an embodiment of this application;

[0120] Figure 11 This is a schematic block diagram of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0121] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0122] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0123] Currently, when users query real-time metrics, the real-time computing engine needs to read all data records related to the metric being queried, and then filter or select the target field from all the read data to obtain the real-time metric requested by the user. However, this method of querying real-time metrics is time-consuming, resulting in low efficiency.

[0124] To address the aforementioned technical problems, the inventive concept of this application is as follows: by determining the business line identifier associated with real-time indicators and the indicator storage strategy, a target storage sub-library is determined from the real-time indicator library based on the business line identifier, and atomic real-time indicators and derived real-time indicators are stored in the target storage sub-library according to the indicator storage strategy. This reduces the query time for real-time indicators and improves the query efficiency when users query real-time indicators.

[0125] Before providing a detailed description of the embodiments of this application, the nouns and terms used in the embodiments of this application will be explained first. The nouns and terms used in the embodiments of this application shall be interpreted as follows:

[0126] A metric is data used to measure a specific attribute of a target and is used for data statistics. For example, in a music context, the number of songs a user listens to each day would be defined as a metric: daily music listens.

[0127] Spark is an open-source project by Apache, a fast and general-purpose computing engine designed for large-scale data processing. Spark is similar to Hadoop's MapReduce computing framework, but it offers advantages such as scalability, in-memory computing, and the ability to directly read data of any format from Hadoop, resulting in higher efficiency and lower latency during batch processing.

[0128] Hadoop is a distributed system infrastructure that allows users to develop distributed programs without understanding the underlying details of distributed systems, fully leveraging the power of clusters for high-speed computation and storage. The core design of the Hadoop framework is HDFS and MapReduce. HDFS provides storage for massive amounts of data, while MapReduce provides computation for massive amounts of data.

[0129] HDFS: Hadoop Distributed File System. HDFS is a distributed file system designed to run on commodity hardware. It is a highly fault-tolerant system that provides high-throughput data access.

[0130] MapReduce is a distributed computing framework that integrates user-written business logic code and built-in default components into a complete distributed computing program that runs concurrently on a Hadoop cluster.

[0131] Dimension: refers to the dimension of data statistics, such as statistics by day, statistics by hour, etc.

[0132] Column storage (also known as columnar storage): a database storage method. In general, each record in a database is stored together, while column storage stores a column of data in a record together, resulting in better performance when reading data from a specific column.

[0133] Flink is an open-source stream processing framework and a real-time computing engine. Flink executes arbitrary streaming data programs in a data-parallel and pipelined manner. Flink's pipelined runtime system can execute both batch and stream processing programs. Furthermore, Flink's runtime itself also supports the execution of iterative algorithms.

[0134] Kafka is an open-source stream processing platform, a high-throughput distributed publish-subscribe messaging system. Kafka aims to unify online and offline message processing through Hadoop's parallel loading mechanism, and also to provide real-time messaging through clusters. Kafka features high throughput, supports message partitioning via Kafka server and consumer clusters, and supports Hadoop's parallel data loading capabilities.

[0135] Batch and Stream Integration: Develop a single codebase for data processing that can handle both offline and real-time data. This means that a single data model and a single SQL statement can simultaneously access batch (offline) and streaming (real-time) data, providing a unified query interface for data applications. This enables minute-level pure real-time data analysis or batch-stream fusion analysis, supporting enterprise monitoring of real-time data and the integrated analysis of historical and real-time data.

[0136] Atomic metrics: These are metrics that cannot be further divided in the business definition, and are terms with clear business meanings, such as login count or number of songs listened to. In other words, atomic metrics refer to basic metrics without dimensions.

[0137] Partitioned Stream Tables: This refers to the integration of offline data partitioning techniques with real-time stream tables to achieve stream table partitioning. Here, the stream table is a table mapped from real-time streaming data, while partitioning refers to offline partitioning techniques, such as directory-level partitioning using Hive.

[0138] SQL (Structured Query Language) is a special-purpose programming language used for database querying and programming. It is used to store data and to query, update, and manage relational database systems. It is also the file extension for database script files.

[0139] Calcite is an open-source SQL parsing tool that can parse various SQL statements into an Abstract Syntax Tree (AST). By manipulating the AST, the algorithms and relationships expressed in the SQL can be reflected in the specific code.

[0140] AST: is a tree-like representation of the abstract syntactic structure of source code, where each node in the tree represents a structure in the source code.

[0141] In Kafka, every message must be written to a topic. A topic is the logical unit of a message, analogous to a mailbox in a post office. It can be considered a message queue with categories; for example, an order system might have an "order" topic, an "inventory" topic, and so on. Each topic can be divided into multiple partitions, and each partition is an ordered sequence of messages.

[0142] A partition, or message partition, is a structure where a topic is typically divided into multiple partitions. Each partition is an ordered, immutable collection of messages, distributed across different brokers. Within a topic, messages within each partition are unordered, but messages within a partition are ordered. Kafka uses partitions to achieve message load balancing and improve throughput.

[0143] Metadata: also known as intermediary data or relay data, is data that describes data, mainly information that describes data attributes, used to support functions such as indicating storage location, historical data, resource lookup, and file records.

[0144] After introducing some concepts involved in the embodiments of this application, the following describes in detail, with reference to the accompanying drawings, an index storage method and query method, apparatus, device and medium provided by the embodiments of this application.

[0145] Figure 2 This is a flowchart illustrating a method for storing metrics according to an embodiment of this application. This embodiment is applicable to real-time metric storage scenarios, and the metric storage method can be executed by a metric storage device. The metric storage device can consist of hardware and / or software and can be integrated into an electronic device. In this embodiment, the electronic device can be any hardware device with data processing capabilities, such as smartphones, tablets, PDAs, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and wearable devices, etc. This application does not specifically limit the type of electronic device.

[0146] like Figure 2 As shown, the method may include the following steps:

[0147] S101 generates derived real-time metrics based on atomic real-time metrics.

[0148] Optionally, this application may obtain multiple atomic metrics from an upstream data source as atomic real-time metrics. Then, at least one derived real-time metric is generated based on each atomic real-time metric. The term "multiple" can be understood as two or more.

[0149] The aforementioned atomic real-time metrics refer to atomic metrics created based on actual business operations to statistically analyze the status of that business activity. When creating atomic metrics associated with business activities, this can be achieved using the atomic metric configuration function provided by electronic devices. Specifically, users can create atomic metrics in the metric factory through a visual metric configuration interface, or they can create atomic metrics in the metric factory by entering SQL statements. This application does not impose any restrictions on the creation of atomic metrics.

[0150] For example, assuming the actual business is music business, the atomic metrics associated with the music business activities can be selected as the number of songs listened to, the number of logins, and the listening time, etc.

[0151] Furthermore, after creating the atomic metrics, this application can obtain upstream data from the upstream data source and use the upstream data to obtain the atomic metrics. Optionally, obtaining atomic metrics based on upstream data specifically involves filtering atomic metrics from the upstream data.

[0152] The aforementioned upstream data source can be understood as a device or apparatus that provides metric data; correspondingly, the upstream data provided by the upstream data source can be log data, etc. Log data can be, but is not limited to, server-side logs, business traffic logs, and client-side logs, etc.

[0153] In some optional embodiments, when a user creates an atomic real-time indicator, a derived real-time indicator corresponding to the atomic real-time indicator may also be created. Here, a derived real-time indicator refers to a derived indicator that can reflect the status of the business activity.

[0154] Furthermore, when creating derived real-time indicators corresponding to atomic real-time indicators, users can check or select processing parameters in the visual creation interface. These processing parameters can be understood as the parameter information used to process and clean the atomic real-time indicators to generate the derived real-time indicators.

[0155] In this application, the above processing parameters include: time parameters and at least one modifier.

[0156] The time parameter mentioned above can be understood as a time period, which is used to determine the time range that needs to be statistically analyzed, such as one natural day or three natural days.

[0157] The aforementioned modifiers can be understood as being used to limit the scope of business activities, such as male, female, online, and offline.

[0158] Therefore, this application generates at least one derived real-time indicator based on each atomic real-time indicator. It can first obtain the processing parameters corresponding to each atomic real-time indicator, and then generate at least one derived real-time indicator based on each atomic real-time indicator and the processing parameters.

[0159] In some optional embodiments, such as Figure 3 shown, at least one derived real-time metric is generated based on each atomic real-time metric and the processing parameters corresponding to the atomic real-time metric. Optionally, it is implemented according to the metric processing logic of one atomic real-time metric + time parameter + one or more modifiers. That is, a derived real-time metric can include one or more modifiers, and a derived real-time metric uniquely belongs to one atomic real-time metric.

[0160] Correspondingly, the SQL corresponding to the above metric processing logic is expressed as <Group by time, modifier 1,..., modifier k>. Where k is a positive integer greater than or equal to 1.

[0161] Exemplarily, assume that a certain atomic real-time metric is the number of logins, the time parameter corresponding to the number of logins is 1 day, and the modifier is male. Then, based on the metric processing logic of the above atomic real-time metric + time parameter + one or more modifiers, the derived real-time metric generated is the number of male logins per day.

[0162] For another example, assume that a certain atomic real-time metric is the number of songs listened to, the time parameter corresponding to the number of songs listened to is one week, and the modifiers are female and post-90s. Then, based on the metric processing logic of the above atomic real-time metric + time parameter + one or more modifiers, the derived real-time metric generated is the number of songs listened to by post-90s females per week.

[0163] It can be understood that the derived real-time metrics in this application are composed of three elements: atomic real-time metrics, time periods, and modifiers, and are used to reflect the numerical performance of atomic real-time metrics under specific times and business conditions, and to reflect the business status of a certain business activity. For example, the number of songs listened to by males per day, the number of logins by females per week, etc.

[0164] S102. Determine the business line identifier and metric storage policy associated with the derived real-time metric.

[0165] Among them, the business line can be understood as the business for a certain type of product. For example, XX Music is a business line for music products. For another example, XX Mailbox is a business line for information transmission products.

[0166] In this application, the business line identifier can be understood as being used to uniquely identify the corresponding business line. For example, the business line name or business line ID, etc. This application does not impose any restrictions on this.

[0167] Before generating derived real-time metrics based on atomic real-time metrics, users can create derived real-time metrics that reflect the status of a business activity using the derived metric creation function provided by electronic devices, in addition to creating derived real-time metrics that reflect the status of that business activity. They can also create derived real-time metrics and configure corresponding metric storage strategies so that the generated derived real-time metrics can be stored in the real-time metric library based on the configured metric storage strategy.

[0168] The storage strategy for derived real-time metrics can be understood as the storage rules or methods for storing derived real-time metrics in the actual underlying storage of the real-time metric library. This application does not impose any restrictions on the storage strategy for derived real-time metrics. Specifically, when the real-time metric library is Kafka, the real-time base layer storage of the aforementioned real-time metric library is a Kafka topic.

[0169] In this application, creating derived real-time metrics specifically includes configuring business logic and configuring basic information. Configuring business logic involves configuring the time period, modifiers, and associated atomic real-time metrics. Configuring basic information includes configuring the data layer to which the metric belongs, the business process, the derived real-time metric identifier, and the descriptive information of the derived real-time metric.

[0170] The aforementioned business process refers to the business process from which the derived real-time metrics belong, used to determine the type of business activity.

[0171] The aforementioned derived real-time indicator identifier can be understood as a unique identifier for the corresponding derived real-time indicator. In this application, the derived real-time indicator identifier can be the name of the derived real-time indicator. The name of the derived real-time indicator can be a Chinese name and / or an English name.

[0172] Therefore, this application can obtain the configuration information of derived real-time indicators from the indicator management module, and obtain the business line identifier and indicator storage strategy associated with the derived real-time indicators from the above configuration information.

[0173] The aforementioned indicator management module can be understood as a device or process responsible for registering and managing the metadata, atomic indicators, and derived indicators of indicators. The metadata of an indicator refers to all data related to that indicator, such as the indicator's definition, calculation logic, and storage location—information describing the indicator's attributes.

[0174] Considering that the metrics include both offline and real-time metrics, the atomic metrics in the above-mentioned metric management module are atomic real-time metrics and / or atomic offline metrics, and the corresponding derived metrics are derived real-time metrics and / or derived offline metrics.

[0175] S103, based on the business line identifier, determines the target storage sub-database from the real-time metric database.

[0176] S104, based on the indicator storage strategy, stores the atomic real-time indicators and derived real-time indicators into the target storage sub-library.

[0177] In this application, the real-time metrics library can be understood as a real-time metrics warehouse, and the real-time metrics warehouse can be Kafka, a Kafka cluster, or other real-time databases. This application does not impose any restrictions on this.

[0178] Considering that each derived real-time indicator is associated with a different business line—for example, one derived real-time indicator may be associated with business line XX, while another derived real-time indicator may be associated with business line YY, etc.—this approach is necessary.

[0179] Therefore, this application optionally divides the real-time indicator library into multiple sub-libraries that do not share data by using business lines associated with derived real-time indicators, so that one business line can correspond to one storage sub-library. This allows atomic real-time indicators and derived real-time indicators corresponding to the same business line to be stored in the same storage sub-library according to the indicator storage strategy. This achieves partitioned storage of real-time indicators, so when querying real-time indicators later, the metadata of the real-time indicator can be used to determine which storage sub-library the real-time indicator is stored in, and the desired real-time indicator can be quickly retrieved from that storage sub-library. This reduces the amount of data read and improves the query speed for real-time indicators.

[0180] In this application, when the real-time metric library is Kafka, the multiple sub-libraries into which Kafka is divided can be selected as dividing Kafka into Kafka1, Kafka2, Kafkan, etc., where n is a positive integer greater than 1.

[0181] In some optional embodiments, since the dimensional information of derived real-time metrics corresponding to the same business line differs, such as differences in time periods and / or modifiers, this application may optionally split the storage sub-database corresponding to a business line into multiple storage spaces according to the dimensions of the real-time metrics. For example, a certain storage sub-database KafkaXX may be split into Topic1, Topic2, and Topic3, etc. Then, the derived real-time metrics are stored in the corresponding storage spaces according to the dimensions, thereby distributing the derived real-time metrics to different storage spaces of the corresponding sub-databases in the real-time metric database according to the dimensions, in order to achieve the purpose of columnar storage for real-time metrics.

[0182] Considering that atomic real-time metrics lack dimensional information, this application, when splitting a sub-database corresponding to a business line into multiple storage spaces according to dimensions, also allocates an additional storage space without dimensions. Furthermore, the atomic real-time metrics corresponding to that business line can be stored in this dimensionless storage space. This facilitates users' subsequent querying of atomic real-time metrics associated with any derived real-time metric.

[0183] In some optional embodiments, when storing atomic real-time metrics and derived real-time metrics, this application first obtains the data storage structure corresponding to the target storage sub-database. Then, according to the data storage structure, the atomic real-time metrics and derived real-time metrics are stored in the target storage sub-database. The advantage of this setup is that, by optimizing the real-time metric storage structure, real-time metrics with the same statistical dimensions can be stored in a single storage space within the corresponding target storage sub-database, thereby improving the speed of subsequent real-time metric queries for users.

[0184] Optionally, the data storage structure in this application may be as shown in Table 1 below:

[0185] Table 1

[0186]

[0187]

[0188] As shown in Table 1, real-time metrics with the same statistical dimensions are stored in a storage space of the corresponding target storage sub-database. Specifically, the real-time metrics and corresponding dimensions are all stored in Value in map form, and the record is written to the Topic of the storage space corresponding to the partition in JSON format.

[0189] Considering that after storing both atomic and derived real-time metrics, users may only need to query any real-time derived metric, this application may optionally store only the derived real-time metrics and not the atomic real-time metrics corresponding to them, in order to reduce data storage volume and lower data storage costs.

[0190] This application provides a method for storing metrics. It involves generating derived real-time metrics from atomic real-time metrics, determining the business line identifier associated with the derived real-time metrics and the metric storage strategy, then determining a target storage sub-database from the real-time metric database based on the business line identifier, and finally storing the atomic and derived real-time metrics in the target storage sub-database based on the metric storage strategy. This application stores atomic and derived real-time metrics in the real-time metric database based on the business line identifier associated with the derived metrics and the metric storage strategy, thereby shortening the query time and improving the query efficiency of real-time metrics.

[0191] Based on the above embodiments, the following is combined with Figure 4 The following section provides further explanation of how the atomic real-time index and the derived real-time index are stored in the target storage sub-library in this application.

[0192] like Figure 4 As shown, the above S104 may include the following steps:

[0193] S104-1, Obtain the partition flow table corresponding to the indicator storage strategy.

[0194] In this application, the partitioned flow table is a single flow table mapped from multiple storage spaces and exposed externally as a single flow table. Therefore, subsequent use, reading, and writing of this partitioned flow table all utilize its identification information. The identification information of the partitioned flow table can be understood as uniquely identifying the corresponding partitioned flow table. For example, the partitioned flow table name, etc., are not limited in this application.

[0195] In some optional embodiments, when configuring atomic real-time metrics and derived real-time metrics, the aforementioned real-time metrics are also configured as flow tables. Therefore, this application will automatically perform the partition flow table creation operation based on the user's configuration information, and set the corresponding metric storage strategy for the partition fields in the created partition flow tables.

[0196] In the creation of partitioned flow tables and the setting of indicator storage strategies for partitioned fields in partitioned flow tables, users may choose to implement this using generalized SQL syntax, and this application does not impose any restrictions on this.

[0197] For example, the partitioned flow table created is partitioned flow table A, and the SQL statement for creating partitioned flow table A can be:

[0198]

[0199]

[0200] Here, `id` represents the identity information of partition flow table A, and its data type is integer; `name` represents the name information of partition flow table A, and its data type is string; `action` represents the partition field of partition flow table A, which can be a real-time metric identifier; `partition by action` represents the partitioning logic, that is, the metric storage strategy corresponding to the partition field, and this partitioning logic can be implemented through SQL conditional expressions (CASE WHEN syntax). For example, the partitioning logic of the above partition flow table A is: `when action = play then topic1 when action = view then topic2 else topic3 ​​end`. This exemplary partitioning logic can be understood as follows: when `action = 'play'`, it is written to Kafka topic1; when `action = 'view'`, it is written to Kafka topic2; otherwise, it is written to topic3.

[0201] In the above kafka.bootstrap=127.0.0.1,xxx=xxx, kafka represents the metric library where the metrics are stored, bootstrap=127.0.0.1 represents the actual storage location information of the metrics, and xxx=xxx represents other storage parameters.

[0202] Therefore, after generating the derived real-time metrics, this application can obtain the partition flow table corresponding to the metric storage strategy based on the metric storage strategy associated with the derived real-time metrics.

[0203] In this application, the index storage strategy can be a first storage strategy, a second storage strategy, or a third storage strategy. Specifically, the first storage strategy is a specified storage space (Topic) strategy; the second storage strategy is a hash modulo strategy; and the third storage strategy is a dynamic storage space Topic strategy.

[0204] Accordingly, obtaining the partition flow table corresponding to the storage strategy of this metric can include the following situations:

[0205] The first scenario,

[0206] When the storage strategy associated with the derived real-time metric is determined to be the first storage strategy, this application obtains the partition flow table corresponding to the metric storage strategy, specifically, it obtains the first partition flow table corresponding to the first storage strategy.

[0207] The first partition flow table includes at least a first partition field and a first condition expression corresponding to the first partition field. The first condition expression is determined based on the atomic real-time indicator identifier, the derived real-time indicator identifier, and the preset storage space. The preset storage space consists of at least two partitions in the target storage sub-database.

[0208] In some optional embodiments, when the first partition flow table is established, the first conditional expression in the SQL language can be selected as: when indexName in('metric name 1', 'metric name 2') then topic1 when xxx then topic2 else topic3 ​​end. Here, indexName represents the real-time metric name of the index. topic1, topic2, and topic3 ​​represent preset storage spaces.

[0209] The first conditional expression above can be understood as follows: when the condition satisfies the atomic real-time indicator, the result is returned to the first preset storage space; when the condition satisfies the derived real-time indicator, the result is returned to the second preset storage space; when none of the conditions are satisfied, the result is returned to the third preset storage space and the process ends.

[0210] In other words, this application directly specifies which storage space the real-time indicator name should be mapped to through the above expression, thus realizing the configuration of indicator storage strategy in an exhaustive manner.

[0211] The second scenario,

[0212] When the indicator storage strategy associated with the derived real-time indicator is determined to be the second storage strategy, this application obtains the partition flow table corresponding to the indicator storage strategy, specifically, it obtains the second partition flow table corresponding to the second storage strategy.

[0213] The second partition flow table includes at least a second partition field and a second condition expression corresponding to the second partition field. The second condition expression is determined based on the first modulo value obtained by hashing and modulo operations on the atomic real-time indicator identifier, the second modulo value obtained by hashing and modulo operations on the derived real-time indicator identifier, and a preset storage space. The preset storage space consists of at least two partitions in the target storage sub-library.

[0214] For example, when the second partition flow table is created, the second conditional expression in the SQL language is: `when hash(indexName)%3=0then topic1 when hash(indexName)%3=1then topic2 elsetopic3 ​​end`. Here, `indexName` represents the real-time metric name of the index. `topic1`, `topic2`, and `topic3` represent the preset storage space.

[0215] The second condition above can be understood as follows: First, each real-time metric identifier (indexName) is hashed using Java's String hashcode function to generate a hash value of type int. Then, a modulo operation is performed on the hash value corresponding to each real-time metric identifier to obtain the modulo value for each real-time metric. It is worth noting that in this application, the number of modulo operations should be equal to the configured number of preset storage space topics, and the configured number of preset storage space topics is an adjustable parameter.

[0216] Furthermore, when the modulus value is 0, the real-time indicator corresponding to that modulus value is assigned to topic0; when the modulus value is 1, the real-time indicator corresponding to that modulus value is assigned to topic1, and so on, storing all real-time indicators in their respective preset storage spaces. Specifically, all real-time indicators are either atomic real-time indicators or derived real-time indicators.

[0217] In other words, the second conditional expression returns the result in the first preset storage space when the modulus of the condition is equal to the first value; it returns the result in the second preset storage space when the modulus of the condition is equal to the second value; and it returns the result in the third preset storage space and ends when none of the conditions are met.

[0218] The advantage of this setup is that, according to the principle of random storage, each preset storage space topic will be randomly assigned roughly the same number of real-time metrics, thus achieving a uniform distribution.

[0219] For example, assuming three preset storage space topics are registered and there are 15 real-time metrics, after hashing and modulo 3 operations on the names of the 15 real-time metrics, five different real-time metrics will be allocated in each preset storage space topic. This will make the storage of real-time metric data more even when the amount of real-time metric data is large.

[0220] The third scenario,

[0221] When the indicator storage strategy associated with the derived real-time indicator is determined to be the third storage strategy, this application obtains the partition flow table corresponding to the indicator storage strategy, specifically the third partition flow table corresponding to the third storage strategy.

[0222] The third partition flow table includes at least a third partition field and a third condition expression corresponding to the third partition field. The third condition expression is determined based on the first modulo value obtained by hashing and modulo operations on the atomic real-time indicator identifier, the second modulo value obtained by hashing and modulo operations on the derived real-time data indicator identifier, the preset storage space, and the preset flow value of the preset storage space. The preset storage space consists of at least two partitions in the target storage sub-database.

[0223] For example, when the third partition flow table is created, the third conditional expression in SQL is: `when hash(indexName) % 3 = 0 then dynamics(topic1, '10k') when hash(indexName) % 3 = 1 then dynamics(topic2, '10k') else dynamics(topic3, '10k') end`. Here, each preset storage space is allocated 10 kilobytes (KB) of traffic. `indexName` represents the real-time metric name of the index. `topic1`, `topic2`, and `topic3` represent the preset storage spaces.

[0224] The second storage strategy described above can only evenly distribute real-time metrics across different preset storage space topics. However, during real-time use, when the traffic of a certain type of real-time metric is uneven—for example, out of 100 real-time metrics, one metric has a traffic of 100 items / second, while the other 99 real-time metrics each have a traffic of 1 item / second—if the real-time metrics are evenly distributed across the preset storage space topics in the real-time metric library, data skew may occur, or even the real-time metrics may fail to be stored in their corresponding preset storage space topics.

[0225] Therefore, this application detects the traffic of each preset storage space topic based on the second storage strategy. When the traffic of any preset storage space topic exceeds the pre-allocated traffic size, an automatic expansion method is used to expand the traffic value of the preset storage space topic that exceeds the preset traffic size, in order to solve the problems of real-time indicator data storage skew and inability to store data.

[0226] In some optional embodiments, when expanding the traffic of a preset storage space Topic that exceeds a preset traffic size, the expansion can be performed according to a preset expansion rule. In this application, the preset expansion rule can be any strategy or algorithm capable of expanding traffic, and this application does not impose any restrictions on it. For example, the preset expansion rule can be to expand the traffic by the same amount based on the preset traffic size, or it can be to expand the traffic by a preset multiple. The preset multiple can be 2, 3, or 4, etc., and this application does not impose any restrictions on it.

[0227] In other words, the third condition expression can be understood as follows: when the condition is satisfied and the modulus is equal to the first value, the result returned is the first preset storage space and the preset flow rate of the first preset storage space; when the condition is satisfied and the modulus is equal to the second value, the result returned is the second preset storage space and the preset flow rate of the second preset storage space; when all conditions are not satisfied, the result returned is the third preset storage space and the preset flow rate of the third preset storage space and the process ends.

[0228] In some optional embodiments, considering that the partition field in the partitioned flow table may be updated, such as by adding a field, this application preferably adopts the second or third storage strategy described above to store the real-time metrics corresponding to the newly added field in a preset storage space Topic. This allows the target storage space Topic for storing the real-time metrics of different fields to be determined based on the metadata of the real-time metric when querying any real-time metric. Furthermore, the real-time metric to be queried can be read from the target storage space Topic, thereby reducing the amount of data read during metric queries and improving the query speed of real-time metrics.

[0229] S104-2, Determine the conditional expression corresponding to the partition field in the partitioned flow table.

[0230] After obtaining the partition flow table, this application can retrieve the conditional expression corresponding to the partition field in the partition flow table from the SQL statement code that creates the partition flow table.

[0231] Considering that the partitioned flow table can be a first partitioned flow table, a second partitioned flow table, or a third partitioned flow table, the conditional expression determined in this embodiment is either a first conditional expression corresponding to the first partitioned flow table, or a second conditional expression corresponding to the second partitioned flow table, or a third conditional expression corresponding to the third partitioned flow table.

[0232] S104-3, based on the pushdown mechanism of the real-time computing engine, pushes down the conditional expression to the programming language to code conversion module of the real-time computing engine, so that the conditional expression can be converted into stored rule code that the real-time computing engine can recognize.

[0233] The real-time computing engine is specifically Fink.

[0234] The aforementioned programming language to code module is specifically an implementation of SQL to code conversion in the Flink open-source software. For example, the programming language to code module is Flink's Kafka sink, etc. This application does not impose any restrictions on it.

[0235] When users store atomic real-time metrics and derived real-time metrics to the target storage sub-database, they can send storage instructions to the electronic device using SQL statements, or they can trigger the real-time metric storage operation through a visual interface to send SQL statement storage instructions to the electronic device. Considering that electronic devices cannot understand the SQL statements for real-time metric storage input by the user, it is necessary to push down the conditional expressions from the obtained partition flow table to the programming language to code conversion module of the real-time computing engine Flink. This module then converts the conditional expressions into storage rule code that the real-time computing engine Flink can recognize.

[0236] For example, suppose the real-time metrics to be stored are (1, "a", play) and (2, "b", view). The user inputs an SQL statement indicating the storage of real-time metrics: `Insert into A select id, name, action from data`, to instruct that (1, "a", play) and (2, "b", view) be stored in partition flow table A. Then, the electronic device, through SQL syntax parsing, obtains the user-input SQL statement as writing to partition flow table A, and retrieves the directory information of partition flow table A from the metric management module. If, based on the directory information of partition flow table A, it is determined that partition flow table A is a flow table, and based on the flow table information registered by the user, it is determined that partition flow table A has a partition field, and this partition field is the action field (`action`), then the conditional expression corresponding to this partition field is obtained: `when action = play then topic1 when action = view then topic2 else topic3 ​​end`. Subsequently, the real-time computing engine Flink, based on the pushdown mechanism, pushes down the conditional expression `when action = play then topic1 when action = view then topic2 else topic3 ​​end` to the Kafka sink of Flink's Fink, so that the Kafka sink of Flink's Fink can convert the above conditional expression into storage rule code that Flink can recognize.

[0237] S104-4, based on the storage rule code, stores the atomic real-time indicators and derived real-time indicators to the target storage sub-library.

[0238] After obtaining the storage rule code for the real-time metrics, this application can analyze and determine the partition fields corresponding to the atomic real-time metrics and derived real-time metrics to be stored. Then, based on the partition fields corresponding to the atomic real-time metrics, the atomic real-time metrics are stored in the target storage sub-database, and based on the partition fields corresponding to the derived real-time metrics, the derived real-time metrics are stored in the target storage sub-database.

[0239] In this application, when storing atomic real-time metrics and derived real-time metrics to the target storage sub-library, storage can be performed using the storage methods supported by the real-time metrics library's own Software Development Kit (SDK). For details of the storage process, please refer to [link to relevant documentation]. Figure 5 As shown. In other words, this application can realize the distribution and storage of real-time indicators into different preset storage spaces (Topics) of the target storage sub-library in the real-time indicator library.

[0240] For example, assuming the real-time metric database is Kafka, and the real-time metric to be stored is (id=1, name='a', action=play), then we can determine that the partition field action=play in this real-time metric. Based on the storage rule code, we determine the preset storage space Topic1 in the target storage sub-database corresponding to play. Therefore, the above real-time metric 'a' can be written to different topics directly using the functionality supported by the Kafka SDK, as shown in the following manner using ProducerRecord.<String,String> The `record = new ProducerRecord<>(topic,"hello,Kafka!")` method stores the real-time metric 'a' in the preset storage space `Topic1`.

[0241] In other words, this application stores atomic real-time indicators and derived real-time indicators in a target storage sub-library, specifically including: determining the first target storage space corresponding to the atomic real-time indicators and the second target storage space corresponding to the derived real-time indicators based on the storage rule code; and then storing the atomic real-time indicators in the first target storage space and storing the derived real-time indicators in the corresponding second target storage space.

[0242] Considering that the storage rule code may correspond to the rule code of the third storage strategy, this application may optionally include the following steps before storing the atomic real-time indicators and derived real-time indicators in the target storage sub-database: determining whether the storage rule code corresponds to the third storage strategy; detecting the current traffic value of each preset storage space when the storage rule code corresponds to the third storage strategy; and expanding the preset traffic value of the preset storage space when the current traffic value of any preset storage space is greater than the preset traffic value. This ensures that each real-time indicator can be correctly stored in its corresponding preset storage space without data skew issues.

[0243] In some optional embodiments, after storing the atomic real-time metrics and derived real-time metrics in the target storage sub-database, this application optionally stores the partition flow table and the mapping relationship between the atomic real-time metrics and the first target storage space as metadata of the atomic real-time metrics; and stores the partition flow table and the mapping relationship between the derived real-time metrics and the second target storage space as metadata of the derived real-time metrics. Specifically, when storing the metadata of the atomic real-time metrics and the metadata of the derived real-time metrics, this application stores the metadata of the atomic real-time metrics and the metadata of the derived real-time metrics in the metric management module. Therefore, when a user queries a real-time metric, the metadata of the real-time metric to be queried can be obtained from the metric management module, and the specific storage location of the real-time metric to be queried in the real-time metric database can be determined based on the obtained metadata. Then, the real-time metric to be queried can be quickly obtained from the real-time metric database based on the specific storage location, thereby improving the query efficiency of real-time metrics.

[0244] This application provides a method for storing metrics. It involves generating derived real-time metrics from atomic real-time metrics, determining the business line identifier associated with the derived real-time metrics and the metric storage strategy, then determining a target storage sub-database from the real-time metric database based on the business line identifier, and finally storing the atomic and derived real-time metrics in the target storage sub-database based on the metric storage strategy. This application stores atomic and derived real-time metrics in the real-time metric database based on the business line identifier associated with the derived metrics and the metric storage strategy, thereby shortening the query time and improving the query efficiency of real-time metrics.

[0245] Based on the foregoing Figures 2 to 5 In addition to the method for storing the indicators shown, this application also provides a method for querying the indicators, see [link to relevant documentation]. Figure 6 .

[0246] like Figure 6 As shown, the query method for this indicator may include the following steps:

[0247] S301, in response to a user's real-time indicator query operation, obtains the identifier of the real-time indicator to be queried.

[0248] Optionally, users can enter or select the real-time indicator identifiers to be queried through the visual query interface provided by the electronic device, such as... Figure 7 As shown. Figure 7 The display includes at least one selectable real-time indicator and a search box. Alternatively, if the user is proficient in SQL, they can also use SQL to perform real-time indicator queries, etc. This application does not impose any restrictions on the real-time indicator query operations sent by the user.

[0249] When a real-time metric query request is received from a user, the query request is parsed to determine the identifier of the real-time metric that the user needs to query.

[0250] It is understood that the aforementioned real-time indicator identifier is the name of the real-time indicator to be used, or other identity information that can uniquely identify the real-time indicator, and this application does not impose any restrictions on this.

[0251] S302, based on the identifier of the real-time indicator to be queried, determine the target storage location of the real-time indicator to be queried in the real-time indicator database.

[0252] Specifically, the target storage location is the preset storage space Topic location of any sub-library in the real-time indicator library.

[0253] Because the indicator management module in this application registers and manages the metadata of all indicators, including at least the storage location of the indicator and information describing the indicator attributes such as the partition flow table identifier corresponding to the indicator, this application can obtain the metadata corresponding to the real-time indicator to be queried from the indicator management module based on the real-time indicator identifier to be queried. Furthermore, based on the metadata, the target storage location of the real-time indicator to be queried in the real-time indicator database can be determined.

[0254] In some optional embodiments, determining the target storage location of the real-time metric to be queried in the real-time metric database based on metadata may include the following scenarios:

[0255] In the first scenario, the target partition flow table is determined based on metadata. Then, the conditional expressions corresponding to the partition fields in the target partition flow table are converted into storage rule codes recognizable by the real-time computing engine. Finally, based on these storage rule codes, the target storage location of the real-time metric to be queried in the real-time metric library is determined.

[0256] The process of converting conditional expressions into recognizable storage rule codes for the real-time computing engine is the same as the method of converting conditional expressions into recognizable storage rule codes for the real-time computing engine in the aforementioned metric storage method, and will not be elaborated on here.

[0257] Scenario 2: Based on metadata, determine the mapping relationship between the real-time metric to be queried and the storage space. Then, based on the mapping relationship between the real-time metric to be queried and the storage space, determine the target storage location of the real-time metric in the real-time metric database.

[0258] The storage space can be understood as any of the multiple preset storage spaces into which the sub-libraries in the real-time indicator library are split.

[0259] S303 retrieves the real-time indicator to be queried from the real-time indicator library based on the target storage location and displays the real-time indicator to be queried to the user.

[0260] Optionally, once the target storage location of the real-time indicator to be queried is determined, this application can retrieve the real-time indicator from the real-time indicator database based on the target storage location. Then, the real-time indicator to be queried is displayed to the user through a visual interface, so that the user can use the real-time indicator.

[0261] In some alternative embodiments, considering that real-time metrics are stored in a distributed manner within the storage space, this application, when retrieving the real-time metric to be queried from the real-time metric library based on the target storage location, also retrieves multiple discrete target fields from the real-time metric library based on the target storage location. Then, the real-time metric to be queried is obtained based on these multiple discrete target fields.

[0262] In other words, when this application displays the real-time indicator to be queried to the user, it needs to restore the fragmented real-time indicator data to its original form before storage so that the user can intuitively and clearly view the complete real-time indicator to be queried.

[0263] Considering that metrics include both real-time and offline metrics, and with the increasing demand for integrated batch and stream development, users often expect real-time and offline metrics to be queried in the same way, meaning they want queries for both to be completed using a single SQL statement and logic. Therefore, this application, in addition to retrieving the corresponding real-time metric from the real-time metric library based on a defined target storage location, can also meet users' needs for querying offline metrics. Specifically, when a user queries any metric, which may be a real-time or offline metric, the query operation can include two parts: the first part is the metric data generation part. This part first retrieves the metadata corresponding to the metric identifier from the metric management module based on the metric identifier queried by the user, and then determines which storage spaces the metric is stored in. For example, when the metric is a real-time metric, it determines which topics it is stored on. When the metric is an offline metric, it determines which Hive instances it is stored in. Then, a synchronization task synchronizes the metric to a higher-performance key / value database. For example, assuming the synchronous task is a real-time task, a Flink task is generated to query messages in the corresponding real-time metric database within a pre-defined storage space (e.g., Kafka topic messages). Then, the key from the message is used as the primary key in the key / value database. The corresponding dimension information is flattened into multiple fields, and the metric to be queried is retrieved from the map using the identifier of the metric to be queried (e.g., the name of the metric to be queried). If a record stores multiple metric data, it is converted into multiple rows according to the identifier of the metric data, and the flattened metric to be queried is written to the key / value database.

[0264] The second part is the query service section. Specifically, after the user's query indicator data is generated in the key / value database, the user can be provided with the real-time and / or offline indicators to be queried through the key / value database.

[0265] The key / value database mentioned above can be a non-relational distributed database such as HBase or Redis, and this application does not impose any restrictions on it.

[0266] It should be noted that the Key / value database in this application includes an offline indicator database and a real-time indicator database. That is, the Key / value database is a large database, while the offline indicator database and the real-time indicator database are two smaller databases within the large database, and the difference between the indicator data in the offline indicator database and the real-time indicator database lies in their different dimensions.

[0267] In this application, when the above-mentioned synchronization task is a real-time task, the SQL language for generating the Flink task can be as follows:

[0268] INSERT INTO Habse1

[0269] SELECT

[0270] --Dimension field,

[0271] dimensionMap['userid']as userid,

[0272] --Indicator field,

[0273] indexMap['Daily Active Users'] as ua,

[0274] FROM A

[0275] WHERE action in('play')

[0276] --partition

[0277] This involves optimizing SQL syntax. Flink's pushdown feature is used to push the WHERE condition to the Kafka source. The Kafka source is also modified, and the pushed-down WHERE condition is analyzed. Metadata of the partitioned flow table A is obtained through the metadata corresponding to the queried metric identifier. If the partition field of partitioned flow table A is determined to be `action`, and the WHERE condition contains a filter condition for `action`, then `action = play` in the WHERE condition is checked to determine which topics `play` should map to. Analysis using the `CASE WHEN` syntax reveals that `play` in partitioned flow table A maps to `Topic1`. Then, the metric management module is called to obtain the actual Kafka address where the queried metric is stored based on the metadata corresponding to the queried metric identifier, and the actual Kafka query operation is performed. If the pushed-down WHERE condition does not contain partition information, but table A is still a partitioned flow table, then all partitions are read to ensure that no queried metric data is missed. In this case, performance optimization is not performed, and the data read traffic is always less than the total metric read traffic.

[0278] In some optional embodiments, considering that some real-time indicators may be rarely queried by users after being stored, this application, after storing real-time indicators in the real-time indicator library, detects which real-time indicators have not been queried by users for a preset period of time, or which real-time indicators have been queried by users less than a preset number of times within the preset period of time. If it is determined that any real-time indicator has not been queried by users for a preset period of time, or has been queried by users less than a preset number of times within the preset period of time, an offline reminder message for that real-time indicator is output to the user to remind the user to stop managing the real-time indicator, thereby reducing the resource cost of data storage. Considering that users may not receive the above offline reminder message in a timely manner, this application, while outputting the offline reminder message for the real-time indicator to the user, also stores the above-mentioned unused and / or less-than-preset-number-of-use data as metadata of the real-time indicator in the indicator management module. Thus, the user can remove real-time indicators with low usage based on the indicator metadata managed in the indicator management module, thereby improving the utilization value of the indicators.

[0279] The testing can be conducted periodically, and the testing cycle can be flexibly set according to actual testing needs. For example, if a stronger real-time requirement is needed, the testing cycle can be set shorter, such as 5 hours or one day. If a weaker real-time requirement is needed, the testing cycle can be set longer, such as one week or two weeks. This application does not impose any restrictions in this regard.

[0280] The preset duration can be flexibly set according to the actual needs of querying the indicators, and this application does not impose any restrictions on it. For example, the preset duration can be 1 day or 7 days, etc.

[0281] This application provides a method for querying indicators. Based on a user's real-time indicator query operation, the method obtains the identifier of the real-time indicator to be queried. Then, based on the identifier, it determines the target storage location of the real-time indicator in a real-time indicator database. Finally, it retrieves the real-time indicator from the database based on the target storage location and displays it to the user. This shortens the query time for real-time indicators and improves query efficiency.

[0282] The following is a reference to the appendix. Figure 8 The present application describes a storage device for an index based on an embodiment. Figure 8 This is a schematic block diagram of a storage device for an index provided in an embodiment of this application. For example... Figure 9 As shown, the storage device 400 for the indicator includes: an indicator generation module 410, a first determination module 420, a second determination module 430, and an indicator storage module 440.

[0283] Among them, the indicator generation module 410 is used to generate derived real-time indicators based on atomic real-time indicators;

[0284] The first determining module 420 is used to determine the business line identifier and indicator storage strategy associated with the derived real-time indicator;

[0285] The second determining module 430 is used to determine the target storage sub-library from the real-time indicator library based on the business line identifier;

[0286] The indicator storage module 440 is used to store the atomic real-time indicator and the derived real-time indicator into the target storage sub-library based on the indicator storage strategy.

[0287] In one optional implementation of this application embodiment, the indicator storage module 440 includes:

[0288] The acquisition unit is used to acquire the partition flow table corresponding to the indicator storage strategy;

[0289] The first determining unit is used to determine the conditional expression corresponding to the partition field in the partition flow table;

[0290] The conversion unit is used to push down the conditional expression to the programming language to code conversion module of the real-time computing engine based on the push-down mechanism of the real-time computing engine, so that the conditional expression can be converted into stored rule code that the real-time computing engine can recognize through the programming language to code conversion module.

[0291] A storage unit is used to store the atomic real-time indicators and the derived real-time indicators into the target storage sub-library based on the storage rule code.

[0292] In one optional implementation of this application, the target storage sub-library includes at least one storage space, and the storage unit is specifically used for:

[0293] Based on the storage rule code, a first target storage space corresponding to the atomic real-time indicator and a second target storage space corresponding to the derived real-time indicator are determined; the atomic real-time indicator is stored in the first target storage space, and the derived real-time indicator is stored in the corresponding second target storage space.

[0294] In an optional implementation of this application embodiment, the storage unit is further used for:

[0295] The partition flow table and the mapping relationship between the atomic real-time metrics and the first target storage space are stored as the metadata of the atomic real-time metrics.

[0296] The partitioned flow table, as well as the mapping relationship between the derived real-time metrics and the second target storage space, are stored as metadata for the derived real-time metrics.

[0297] In an optional implementation of this application embodiment, when the indicator storage strategy is a first storage strategy, the acquisition unit is specifically used for:

[0298] Obtain the first partition flow table corresponding to the first storage policy;

[0299] The first partition flow table includes at least a first partition field and a first conditional expression corresponding to the first partition field. The first conditional expression is determined based on the atomic real-time indicator identifier, the derived real-time indicator identifier, and the preset storage space. The preset storage space is at least two partitions in the target storage sub-database.

[0300] In one optional implementation of this application, the first conditional expression is: when the condition satisfies the atomic real-time indicator identifier, the result is returned as a first preset storage space; when the condition satisfies the derived real-time indicator identifier, the result is returned as a second preset storage space; when all conditions are not satisfied, the result is returned as a third preset storage space and the process ends.

[0301] In an optional implementation of this application embodiment, when the indicator storage strategy is the second storage strategy, the acquisition unit is specifically used for:

[0302] Obtain the second partition flow table corresponding to the second storage strategy;

[0303] The second partition flow table includes at least a second partition field and a second condition expression corresponding to the second partition field. The second condition expression is determined based on a first modulo value obtained by hashing and modulo operations on the atomic real-time indicator identifier, a second modulo value obtained by hashing and modulo operations on the derived real-time indicator identifier, and a preset storage space. The preset storage space consists of at least two partitions in the target storage sub-database.

[0304] In one optional implementation of this application, the second conditional expression is: when the condition satisfies that the modulus is equal to the first numerical value, the result is returned as the first preset storage space; when the condition satisfies that the modulus is equal to the second numerical value, the result is returned as the second preset storage space; when all conditions are not satisfied, the result is returned as the third preset storage space and the process ends.

[0305] In an optional implementation of this application embodiment, when the indicator storage strategy is a third storage strategy, the acquisition unit is specifically used for:

[0306] Obtain the third partition flow table corresponding to the third storage strategy;

[0307] The third partition flow table includes at least a third partition field and a third condition expression corresponding to the third partition field. The third condition expression is determined based on a first modulo value obtained by hashing and modulo operations on the atomic real-time indicator identifier, a second modulo value obtained by hashing and modulo operations on the derived real-time data indicator identifier, a preset storage space, and a preset flow value of the preset storage space. The preset storage space consists of at least two partitions in the target storage sub-database.

[0308] In one optional implementation of this application, the third condition expression is as follows: when the condition satisfies that the modulus is equal to the first value, the result is a first preset storage space and a preset traffic value of the first preset storage space; when the condition satisfies that the modulus is equal to the second value, the result is a second preset storage space and a preset traffic value of the second preset storage space; when all conditions are not satisfied, the result is a third preset storage space and a preset traffic value of the third preset storage space, and the process ends.

[0309] In one optional implementation of this application embodiment, the indicator storage module 440 further includes:

[0310] The detection unit is configured to detect the current traffic value of each preset storage space in response to the storage rule code corresponding to the third storage strategy;

[0311] An expansion unit is configured to expand the preset storage space to accommodate a preset traffic value in response to any current traffic value of the preset storage space being greater than the preset traffic value.

[0312] In one optional implementation of this application, the indicator storage module 440 is specifically used to: obtain the data storage structure corresponding to the target storage sub-library; and store the atomic real-time indicator and the derived real-time indicator into the target storage sub-library according to the data storage structure.

[0313] In one optional implementation of this application, the indicator generation module is specifically used to: obtain the processing parameters corresponding to the atomic real-time indicator; and generate the derived real-time indicator based on the atomic real-time indicator and the processing parameters.

[0314] This application provides a storage device for indicators. It generates derived real-time indicators based on atomic real-time indicators, then determines the business line identifier associated with the derived real-time indicators and the indicator storage strategy. Next, based on the business line identifier, it determines a target storage sub-database from a real-time indicator database. Finally, based on the indicator storage strategy, it stores the atomic and derived real-time indicators in the target storage sub-database. This application stores atomic and derived real-time indicators in a real-time indicator database based on the business line identifier associated with the derived indicators and the indicator storage strategy, thereby shortening the query time and improving the query efficiency of real-time indicators.

[0315] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 8 The device 400 shown can perform Figure 2 The corresponding method embodiments, and the foregoing and other operations and / or functions of each module in device 400 are respectively implemented to achieve Figure 2 For the sake of brevity, the corresponding processes in each method are not described in detail here.

[0316] The apparatus 400 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0317] The following is a reference to the appendix. Figure 9 This application describes a query device for an indicator proposed in its embodiments. Figure 9 This is a schematic block diagram of an indicator query device provided in an embodiment of this application. For example... Figure 10 As shown, the query device 500 for this indicator includes: an identifier acquisition module 510, a third determination module 520, and an indicator acquisition module 530.

[0318] The identifier acquisition module 510 is used to obtain the identifier of the real-time indicator to be queried in response to the real-time indicator query operation sent by the user.

[0319] The third determining module 520 is used to determine the target storage location of the real-time indicator to be queried in the real-time indicator database based on the identifier of the real-time indicator to be queried.

[0320] The indicator acquisition module 530 is used to acquire the real-time indicator to be queried from the real-time indicator library based on the target storage location, and display the real-time indicator to be queried to the user.

[0321] In one optional implementation of this application embodiment, the third determining module 520 includes:

[0322] The second determining unit is used to determine the metadata corresponding to the real-time indicator identifier to be queried;

[0323] The third determining unit is used to determine the target storage location of the real-time indicator to be queried in the real-time indicator database based on the metadata.

[0324] In one optional implementation of this application, the third determining unit is specifically used for: determining a target partition flow table based on the metadata; performing code conversion on the condition expression corresponding to the partition field in the target partition flow table to convert the condition expression into a storage rule code recognizable by the real-time computing engine; and determining the target storage location of the real-time indicator to be queried in the real-time indicator library based on the storage rule code.

[0325] In one optional implementation of this application, the third determining unit is specifically used to: determine the mapping relationship between the real-time indicator to be queried and the storage space based on the metadata; and determine the target storage location of the real-time indicator to be queried in the real-time indicator database according to the mapping relationship between the real-time indicator to be queried and the storage space.

[0326] In one optional implementation of this application, the indicator acquisition module 530 is specifically used to: acquire multiple discrete target fields from the real-time indicator library based on the target storage location; and obtain the real-time indicator to be queried based on the multiple discrete target fields.

[0327] This application provides a query device for indicators. Based on a user's real-time indicator query operation, it obtains the identifier of the real-time indicator to be queried, determines the target storage location of the real-time indicator in a real-time indicator database based on the identifier, retrieves the real-time indicator from the database based on the target storage location, and displays the real-time indicator to the user. This shortens the query time for real-time indicators and improves the query efficiency.

[0328] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 9 The device 500 shown can perform Figure 6 The corresponding method embodiments, and the foregoing and other operations and / or functions of each module in the device 500 are respectively implemented to achieve Figure 6 For the sake of brevity, the corresponding processes in each method are not described in detail here.

[0329] The apparatus 500 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0330] Figure 10 This is a schematic block diagram of an electronic device provided in an embodiment of this application.

[0331] like Figure 10 As shown, the electronic device 600 may include:

[0332] The system includes a memory 610 and a processor 620. The memory 610 stores computer programs and transfers the program code to the processor 620. In other words, the processor 620 can retrieve and run the computer programs from the memory 610 to implement the index storage method or index query method in the embodiments of this application.

[0333] For example, the processor 620 can be used to execute the above-described method embodiments according to instructions in the computer program.

[0334] In some embodiments of this application, the processor 620 may include, but is not limited to:

[0335] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0336] In some embodiments of this application, the memory 610 includes, but is not limited to:

[0337] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0338] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 610 and executed by the processor 620 to complete the index storage method or index query method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the electronic device.

[0339] like Figure 10 As shown, the electronic device may also include:

[0340] Transceiver 630, which can be connected to processor 620 or memory 610.

[0341] The processor 620 can control the transceiver 630 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 630 may include a transmitter and a receiver. The transceiver 630 may further include antennas, and the number of antennas may be one or more.

[0342] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0343] This application also provides a computer-readable storage medium for storing a computer program that causes a computer to execute the index storage method or index query method as described in the above method embodiments, specifically as follows: Figure 11 As shown. Among them, Figure 11 The computer-readable storage medium is 700, and the computer program is 710.

[0344] This application also provides a computer program product containing program instructions, which, when run on an electronic device, cause the electronic device to execute the index storage method or index query method of the above method embodiments.

[0345] When implemented using software, it can be implemented entirely or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0346] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0347] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0348] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0349] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for storing indicators, characterized in that, include: Derived real-time indicators are generated based on atomic real-time indicators. Determine the service line identifier and indicator storage strategy associated with the derived real-time indicator; Based on the business line identifier, the target storage sub-library is determined from the real-time indicator library; Based on the aforementioned indicator storage strategy, the atomic real-time indicators and the derived real-time indicators are stored in the target storage sub-database; The step of storing the atomic real-time indicators and the derived real-time indicators in the target storage sub-library based on the indicator storage strategy includes: obtaining a partition flow table corresponding to the indicator storage strategy; determining the conditional expression corresponding to the partition field in the partition flow table; pushing the conditional expression down to the programming language to code conversion module of the real-time computing engine based on the push-down mechanism of the real-time computing engine, so that the conditional expression is converted into storage rule code recognizable by the real-time computing engine through the programming language to code conversion module; and storing the atomic real-time indicators and the derived real-time indicators in the target storage sub-library based on the storage rule code.

2. The method according to claim 1, characterized in that, The target storage sub-library includes at least one storage space. The step of storing the atomic real-time indicators and the derived real-time indicators into the target storage sub-library based on the storage rule code includes: Based on the storage rule code, determine the first target storage space corresponding to the atomic real-time indicator and the second target storage space corresponding to the derived real-time indicator; The atomic real-time metrics are stored in the first target storage space, and the derived real-time metrics are stored in the corresponding second target storage space.

3. The method according to claim 2, characterized in that, Also includes: The partition flow table and the mapping relationship between the atomic real-time metrics and the first target storage space are stored as the metadata of the atomic real-time metrics. The partitioned flow table, as well as the mapping relationship between the derived real-time metrics and the second target storage space, are stored as metadata for the derived real-time metrics.

4. The method according to claim 1, characterized in that, When the indicator storage strategy is the first storage strategy, obtaining the partition flow table corresponding to the indicator storage strategy includes: Obtain the first partition flow table corresponding to the first storage policy; The first partition flow table includes at least a first partition field and a first conditional expression corresponding to the first partition field. The first conditional expression is determined based on the atomic real-time indicator identifier, the derived real-time indicator identifier, and the preset storage space. The preset storage space is at least two partitions in the target storage sub-database.

5. The method according to claim 4, characterized in that, The first conditional expression is: when the condition satisfies the atomic real-time indicator identifier, the result is returned as the first preset storage space; when the condition satisfies the derived real-time indicator identifier, the result is returned as the second preset storage space. If none of the conditions are met, the result is returned as the third preset storage space and the process ends.

6. The method according to claim 1, characterized in that, When the indicator storage strategy is the second storage strategy, obtaining the partition flow table corresponding to the indicator storage strategy includes: Obtain the second partition flow table corresponding to the second storage strategy; The second partition flow table includes at least a second partition field and a second condition expression corresponding to the second partition field. The second condition expression is determined based on a first modulo value obtained by hashing and modulo operations on the atomic real-time indicator identifier, a second modulo value obtained by hashing and modulo operations on the derived real-time indicator identifier, and a preset storage space. The preset storage space consists of at least two partitions in the target storage sub-database.

7. The method according to claim 6, characterized in that, The second conditional expression is: when the condition is met and the modulus is equal to the first value, the result is returned as the first preset storage space; when the condition is met and the modulus is equal to the second value, the result is returned as the second preset storage space. If none of the conditions are met, the result is returned as the third preset storage space and the process ends.

8. The method according to claim 1, characterized in that, When the indicator storage strategy is the third storage strategy, obtaining the partition flow table corresponding to the indicator storage strategy includes: Obtain the third partition flow table corresponding to the third storage strategy; The third partition flow table includes at least a third partition field and a third condition expression corresponding to the third partition field. The third condition expression is determined based on a first modulo value obtained by hashing and modulo operations on the atomic real-time indicator identifier, a second modulo value obtained by hashing and modulo operations on the derived real-time indicator identifier, a preset storage space, and a preset flow value of the preset storage space. The preset storage space consists of at least two partitions in the target storage sub-database.

9. The method according to claim 8, characterized in that, The third condition expression is: when the condition satisfies that the modulus is equal to the first value, the result returned is the first preset storage space and the preset flow value of the first preset storage space; When the condition is met and the modulus is equal to the second value, the result returned is the second preset storage space and the preset flow rate of the second preset storage space; when all conditions are not met, the result returned is the third preset storage space and the preset flow rate of the third preset storage space, and the process ends.

10. The method according to claim 8, characterized in that, Before storing the atomic real-time metrics and the derived real-time metrics into the target storage sub-library based on the storage rule code, the method further includes: When the storage rule code corresponds to the third storage strategy, the current traffic value of each preset storage space is detected; In response to any current traffic value of the preset storage space being greater than the preset traffic value, the preset traffic value of the preset storage space is expanded.

11. The method according to claim 1, characterized in that, The step of storing the atomic real-time metrics and the derived real-time metrics into the target storage sub-library includes: Obtain the data storage structure corresponding to the target storage sub-library; According to the data storage structure, the atomic real-time indicators and the derived real-time indicators are stored in the target storage sub-library.

12. The method according to claim 1, characterized in that, The generation of derived real-time metrics based on atomic real-time metrics includes: Obtain the processing parameters corresponding to the real-time atomic indicators; The derived real-time index is generated based on the atomic real-time index and the processing parameters.

13. A method for querying an indicator, characterized in that, include: In response to a user's real-time metric query request, obtain the identifier of the real-time metric to be queried; Based on the identifier of the real-time indicator to be queried, determine the target storage location of the real-time indicator to be queried in the real-time indicator database. Based on the target storage location, the real-time indicator to be queried is obtained from the real-time indicator library and displayed to the user; The step of determining the target storage location of the real-time indicator to be queried in the real-time indicator database based on the identifier of the real-time indicator to be queried includes: determining the metadata corresponding to the identifier of the real-time indicator to be queried; and determining the target storage location of the real-time indicator to be queried in the real-time indicator database based on the metadata. The step of determining the target storage location of the real-time indicator to be queried in the real-time indicator library based on the metadata includes: determining the target partition flow table based on the metadata; performing code conversion on the condition expression corresponding to the partition field in the target partition flow table to convert the condition expression into storage rule code recognizable by the real-time computing engine; and determining the target storage location of the real-time indicator to be queried in the real-time indicator library based on the storage rule code.

14. The method according to claim 13, characterized in that, Determining the target storage location of the real-time indicator to be queried in the real-time indicator database based on the metadata includes: Based on the metadata, determine the mapping relationship between the real-time metric to be queried and the storage space; Based on the mapping relationship between the real-time indicator to be queried and the storage space, the target storage location of the real-time indicator to be queried in the real-time indicator library is determined.

15. The method according to claim 13, characterized in that, The step of obtaining the real-time indicator to be queried from the real-time indicator database based on the target storage location includes: Based on the target storage location, multiple target discrete fields are obtained from the real-time indicator library; The real-time metric to be queried is obtained based on the multiple target discrete fields.

16. A storage device for an index, characterized in that, include: The indicator generation module is used to generate derived real-time indicators based on atomic real-time indicators. The first determining module is used to determine the business line identifier and indicator storage strategy associated with the derived real-time indicator; The second determining module is used to determine the target storage sub-library from the real-time indicator library based on the business line identifier; The indicator storage module is used to store the atomic real-time indicator and the derived real-time indicator into the target storage sub-database based on the indicator storage strategy; The indicator storage module includes: an acquisition unit for acquiring a partition flow table corresponding to the indicator storage strategy; a first determination unit for determining a conditional expression corresponding to a partition field in the partition flow table; a conversion unit for pushing the conditional expression down to the programming language to code conversion module of the real-time computing engine based on the push-down mechanism of the real-time computing engine, so as to convert the conditional expression into storage rule code recognizable by the real-time computing engine through the programming language to code conversion module; and a storage unit for storing the atomic real-time indicator and the derived real-time indicator in the target storage sub-library based on the storage rule code.

17. The apparatus according to claim 16, characterized in that, The target storage sub-library includes at least one storage space, and the storage unit is specifically used for: Based on the storage rule code, a first target storage space corresponding to the atomic real-time indicator and a second target storage space corresponding to the derived real-time indicator are determined; the atomic real-time indicator is stored in the first target storage space, and the derived real-time indicator is stored in the corresponding second target storage space.

18. The apparatus according to claim 17, characterized in that, The storage unit is also used for: The partition flow table and the mapping relationship between the atomic real-time metrics and the first target storage space are stored as the metadata of the atomic real-time metrics. The partitioned flow table, as well as the mapping relationship between the derived real-time metrics and the second target storage space, are stored as metadata for the derived real-time metrics.

19. The apparatus according to claim 16, characterized in that, When the indicator storage strategy is the first storage strategy, the acquisition unit is specifically used for: Obtain the first partition flow table corresponding to the first storage policy; The first partition flow table includes at least a first partition field and a first conditional expression corresponding to the first partition field. The first conditional expression is determined based on the atomic real-time indicator identifier, the derived real-time indicator identifier, and the preset storage space. The preset storage space is at least two partitions in the target storage sub-database.

20. The apparatus according to claim 19, characterized in that, The first conditional expression is: when the condition satisfies the atomic real-time indicator identifier, the result is returned as the first preset storage space; when the condition satisfies the derived real-time indicator identifier, the result is returned as the second preset storage space. If none of the conditions are met, the result is returned as the third preset storage space and the process ends.

21. The apparatus according to claim 16, characterized in that, When the indicator storage strategy is the second storage strategy, the acquisition unit is specifically used for: Obtain the second partition flow table corresponding to the second storage strategy; The second partition flow table includes at least a second partition field and a second condition expression corresponding to the second partition field. The second condition expression is determined based on a first modulo value obtained by hashing and modulo operations on the atomic real-time indicator identifier, a second modulo value obtained by hashing and modulo operations on the derived real-time indicator identifier, and a preset storage space. The preset storage space consists of at least two partitions in the target storage sub-database.

22. The apparatus according to claim 21, characterized in that, The second conditional expression is: when the condition is met and the modulus is equal to the first value, the result is returned as the first preset storage space; when the condition is met and the modulus is equal to the second value, the result is returned as the second preset storage space. If none of the conditions are met, the result is returned as the third preset storage space and the process ends.

23. The apparatus according to claim 16, characterized in that, When the indicator storage strategy is the third storage strategy, the acquisition unit is specifically used for: Obtain the third partition flow table corresponding to the third storage strategy; The third partition flow table includes at least a third partition field and a third condition expression corresponding to the third partition field. The third condition expression is determined based on a first modulo value obtained by hashing and modulo operations on the atomic real-time indicator identifier, a second modulo value obtained by hashing and modulo operations on the derived real-time indicator identifier, a preset storage space, and a preset flow value of the preset storage space. The preset storage space consists of at least two partitions in the target storage sub-database.

24. The apparatus according to claim 23, characterized in that, The third condition expression is: when the condition satisfies that the modulus is equal to the first value, the result returned is the first preset storage space and the preset flow value of the first preset storage space; When the condition is met and the modulus is equal to the second value, the result returned is the second preset storage space and the preset flow rate of the second preset storage space; when all conditions are not met, the result returned is the third preset storage space and the preset flow rate of the third preset storage space, and the process ends.

25. The apparatus according to claim 23, characterized in that, The indicator storage module also includes: The detection unit is configured to detect the current traffic value of each preset storage space in response to the storage rule code corresponding to the third storage strategy; An expansion unit is configured to expand the preset storage space to accommodate a preset traffic value in response to any current traffic value of the preset storage space being greater than the preset traffic value.

26. The apparatus according to claim 16, characterized in that, The indicator storage module is specifically used for: obtaining the data storage structure corresponding to the target storage sub-library; and storing the atomic real-time indicator and the derived real-time indicator into the target storage sub-library according to the data storage structure.

27. The apparatus according to claim 16, characterized in that, The indicator generation module is specifically used for: obtaining the processing parameters corresponding to the atomic real-time indicator; and generating the derived real-time indicator based on the atomic real-time indicator and the processing parameters.

28. A device for querying an indicator, characterized in that, include: The identifier acquisition module is used to obtain the identifier of the real-time indicator to be queried in response to the real-time indicator query operation sent by the user. The third determining module is used to determine the target storage location of the real-time indicator to be queried in the real-time indicator database based on the identifier of the real-time indicator to be queried. The indicator acquisition module is used to acquire the real-time indicator to be queried from the real-time indicator library based on the target storage location, and display the real-time indicator to be queried to the user. The third determining module includes: a second determining unit, configured to determine the metadata corresponding to the real-time indicator identifier to be queried; and a third determining unit, configured to determine the target storage location of the real-time indicator to be queried in the real-time indicator database based on the metadata. The third determining unit is specifically used for: determining a target partition flow table based on the metadata; performing code conversion on the condition expression corresponding to the partition field in the target partition flow table to convert the condition expression into a storage rule code recognizable by the real-time computing engine; and determining the target storage location of the real-time indicator to be queried in the real-time indicator library based on the storage rule code.

29. The apparatus according to claim 28, characterized in that, The third determining unit is specifically used to: determine the mapping relationship between the real-time indicator to be queried and the storage space based on the metadata; and determine the target storage location of the real-time indicator to be queried in the real-time indicator database according to the mapping relationship between the real-time indicator to be queried and the storage space.

30. The apparatus according to claim 28, characterized in that, The indicator acquisition module is specifically used for: acquiring multiple target discrete fields from the real-time indicator library based on the target storage location; and obtaining the real-time indicator to be queried based on the multiple target discrete fields.

31. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to perform the index storage method as described in any one of claims 1 to 12, or the index query method as described in claims 13 to 15.

32. A computer-readable storage medium, characterized in that, Used to store computer programs that cause a computer to execute the index storage method as described in any one of claims 1 to 12, or the index query method as described in claims 13 to 15.

Images