Data processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a data processing method and device, electronic equipment and storage medium. The method comprises: receiving a data query request, the data query request carrying a data query condition; obtaining a data query result in an aggregated database based on the data query condition; the aggregated database is used to store aggregated data; determining whether to perform data query based on the data query condition through a detailed database based on the data query result; the detailed database is used to store detailed data, and the aggregated data is obtained by aggregating the detailed data. In this way, the data is aggregated in advance, and the aggregated data is stored in the aggregated database. When data is queried, data is queried in the aggregated database first. Once relevant data is queried in the aggregated database, data query in the detailed database is not needed, and the problem of query efficiency caused by direct query in the detailed database is avoided.

Description

Technical Field

[0001] This disclosure relates to the field of big data technology, and in particular to data processing methods, apparatus, electronic devices and storage media. Background Technology

[0002] With the rapid development of the mobile internet, mobile terminals, represented by smartphones, have become widely used, and applications on these terminals have become increasingly diverse, playing an indispensable role in users' daily work and lives. To enable these applications to better serve users, it is necessary to process large amounts of data to solve user problems.

[0003] Related technologies typically store all data in the ClickHouse database. Due to the high cluster load, data queries using detailed data stored in the ClickHouse database generally have low efficiency. Summary of the Invention

[0004] This disclosure provides a data processing method, apparatus, electronic device, and storage medium.

[0005] According to one aspect of this disclosure, a data processing method is provided, the method comprising:

[0006] Receive a data query request, the data query request carrying data query conditions;

[0007] Based on the data query conditions, the data query results are obtained in the aggregate database; the aggregate database is used to store aggregate data.

[0008] Based on the data query results, it is determined whether to perform a data query through the detailed database based on the data query conditions; the detailed database is used to store detailed data, and the aggregated data is obtained by aggregating the detailed data.

[0009] According to another aspect of this disclosure, a data processing apparatus is provided, characterized in that the apparatus comprises:

[0010] The receiving module is used to receive data query requests, wherein the data query requests carry data query conditions;

[0011] The first data query module is used to obtain data query results in the aggregated database based on the data query conditions; the aggregated database is used to store aggregated data.

[0012] The second data query module is used to determine, based on the data query results, whether to perform a data query through the detailed database based on the data query conditions; the detailed database is used to store detailed data, and the aggregated data is obtained by aggregating the detailed data.

[0013] According to a third aspect of this disclosure, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described above.

[0014] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the methods described above.

[0015] The data processing method, apparatus, electronic device, and storage medium provided in this disclosure, upon receiving a data query request, can first perform a data query in an aggregated database based on the data query conditions, and then determine whether further data querying is needed in a detailed database based on the data query results. This approach allows for pre-aggregation of data and storage of the aggregated data in an aggregated database. During data querying, the query is first performed in the aggregated database. Once the relevant data is found in the aggregated database, a data query in the detailed database is unnecessary, thus avoiding the query efficiency issues associated with directly querying the detailed database. Attached Figure Description

[0016] Further details, features, and advantages of this disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0017] Figure 1 A flowchart of data processing provided for an exemplary embodiment of this disclosure;

[0018] Figure 2 A schematic diagram of a data processing apparatus provided for an exemplary embodiment of this disclosure;

[0019] Figure 3 A structural block diagram of an electronic device provided as an exemplary embodiment of this disclosure;

[0020] Figure 4 A block diagram of a computer system provided for an exemplary embodiment of this disclosure. Detailed Implementation

[0021] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0022] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0023] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0024] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0025] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0026] In ClickHouse data storage scenarios, the data stored using traditional engines is mostly detailed data. Some business operations only need to query fixed aggregated data, resulting in ClickHouse wasting a lot of space storing detailed data and leading to low query efficiency. Conversely, for scenarios where some business operations require a large amount of aggregated data while also needing some detailed data, using only an aggregation engine can result in the loss of detailed data. Therefore, in the embodiments provided in this disclosure, by effectively aggregating detailed data while simultaneously storing it for user queries, the efficiency of data querying can be greatly improved.

[0027] In the embodiments provided in this disclosure, in order to improve the efficiency of data query, there will be corresponding data processing procedures from data collection, data cleaning, data storage and data query.

[0028] During the data collection phase, clients connected to the network periodically report their own application data and network data. Therefore, the sampling rate of the clients can be controlled through sampling configuration, and the clients sample and report according to the issued sampling configuration. The server stores the collected large amounts of data in Kafka for later use.

[0029] It should be noted that the data collected in this embodiment does not involve user privacy information, and data collection will be conducted with the user's consent when it is necessary to obtain relevant user data. The processing of user privacy data in this embodiment will strictly comply with relevant national regulations.

[0030] During the data cleaning phase, after the collected raw data is stored in Kafka, it is necessary to clean the raw data to obtain effective data. In this embodiment, the streaming computing engine Flink can be used. By creating a corresponding Flink Job, the raw data is received from Kafka, and the raw data is sampled, filtered, and aggregated through specific ETL (Extract-Transform-Load) logic to obtain cleaned data, which is then written back to Kafka.

[0031] During the data persistence phase, ClickHouse is selected as the OLAP database. When creating tables, the aggregation engine AggregatingMergeTree is specified. ClickHouse will merge rows with the same sort key within a data fragment, aggregating data according to a certain time granularity. This can reduce the number of data rows, save storage capacity, and reduce cluster load. However, using the aggregation engine can lead to the loss of detailed data. Therefore, the aggregation engine should only be used to create tables to store data when the business only queries aggregated data.

[0032] Therefore, in this embodiment, the aggregated data is stored in the ClickHouse database, while all detailed data is stored in ElasticSearch. This allows users to perform a quick search within the aggregated data in the ClickHouse database first, and if the data is not found there, then they can search through the detailed data in ElasticSearch.

[0033] For queries that only determine whether a certain data exists in the current collection class, the Bitmap engine can be used to create a table. The Bitmap engine is suitable for representing a collection of a large number of entity IDs with specific relationships, converting collection operations into bitwise operations. At the same time, when searching for the existence of an element, Bitmap can directly compare whether its bit is 1, with a time complexity of O(1), which has excellent computational performance. Bitmap only needs one bit to represent an integer value, which can greatly reduce storage space consumption.

[0034] For queries with long spans and small changes in dimensional metrics, such as long-term queries spanning a week or a month, materialized views provided by ClickHouse can be used to speed up the query. Materialized views store the data contained in the SQL query statement, using a space-for-time trade-off to accelerate the query.

[0035] Furthermore, Elasticsearch was chosen as the query engine for detailed data. Elasticsearch is a distributed data analysis and search engine that effectively supports full-text search and analysis. Its inverted index provides high-performance full-text search, overcoming the limitation of ClickHouse, which can only query aggregated data, and enabling efficient querying of detailed data.

[0036] During the data query phase, query requests often contain numerous parameters, which can easily become excessively long and impact Redis key storage and retrieval performance. Therefore, the cached key is the MD5 hash of the query parameters. Determining a cache hit simply requires comparing the MD5 hash of the parameters. The query request returns a complete chart with a large data volume, resulting in a large key-value pair in Redis. Since Redis is a single-threaded architecture, accessing large keys can cause Redis server blocking and timeouts. Therefore, if a key is identified as large, it needs to be split into smaller keys before storing it in Redis. For example, when a string key exceeds 10KB, Redis's single-threaded architecture may cause other Redis operations to time out. Therefore, large keys can be split into multiple smaller keys and reassembled during retrieval.

[0037] In the embodiments provided in this disclosure, based on the above embodiments, a data processing method is also provided, such as... Figure 1 As shown, the method may specifically include the following steps:

[0038] In step S110, a data query request is received.

[0039] The data query request carries the data query conditions.

[0040] For example, in the embodiment, the data query request will request to query what kind of data corresponds to a certain time period, such as the number of views of a certain video within two hours from 20:00 to 22:00 on a certain day.

[0041] In step S120, the data query results in the aggregate database are obtained based on the data query conditions. This aggregate database is used to store aggregated data.

[0042] This embodiment of the disclosure pre-aggregates all detailed data and stores the aggregated data in an aggregated data center. Then, the detailed data is stored in a detailed database. When a data query request is received, the aggregated database is queried first based on the query conditions to obtain the data query result. This data query result is either "data found" or "data not found".

[0043] In step S130, based on the data query results, it is determined whether to perform a data query through the detailed database based on the data query conditions. The detailed database stores detailed data, and aggregated data is obtained by aggregating the detailed data.

[0044] Specifically, if the query result does not yield the requested data, the data query to the detailed database is stopped. Otherwise, if the query result does not yield the requested data, the data query is performed through the detailed database based on the data query conditions.

[0045] Therefore, if the data query result indicates that data was found, the data retrieved from the aggregate database will be sent to the data query request client. Otherwise, if the data query result indicates that no data was found, meaning no data matching the data query conditions was found in the aggregate database, then a further query will be performed through the detailed database, and the data retrieved from the detailed database will be sent to the data query request client.

[0046] Therefore, if the query returns no data, the query to the detailed database is stopped. If the query returns no data, the detailed database is queried based on the query conditions. This leverages the faster query speed of the aggregate database; only when no relevant data is found in the aggregate database is the detailed database queried, avoiding the low query efficiency associated with direct queries to the detailed database.

[0047] Specifically, ClickHouse can be selected as the aggregation database and ElasticSearch as the detail database, as detailed in the above examples, which will not be repeated here.

[0048] The data processing method provided in this disclosure, upon receiving a data query request, can first perform a data query in an aggregated database based on the data query conditions, and then determine whether further data querying is needed in the detailed database based on the data query results. This approach allows for pre-aggregation of data and storage of the aggregated data in an aggregated database. During a data query, the query is first performed in the aggregated database. Once the relevant data is found in the aggregated database, a direct query in the detailed database is unnecessary, thus avoiding the query efficiency issues associated with directly querying the detailed database.

[0049] Based on the above embodiments, in another embodiment provided in this disclosure, the method may further include the following steps:

[0050] In step S140, detailed data is acquired and stored in a detailed database. This detailed data is obtained by pre-processing network data collected from multiple clients.

[0051] In the embodiments, the preprocessed data can be obtained by performing data sampling, filtering and aggregation on the network data collected from multiple clients through the above embodiments.

[0052] In S150, detailed data is aggregated according to preset dimensions, and the resulting aggregated data is stored in the aggregated database.

[0053] In the embodiments provided in this disclosure, data can be aggregated according to a preset time. For example, data within a specific time period can be aggregated based on a specified sorting key and sorting key, which significantly reduces the storage space and the number of rows of the aggregated data.

[0054] In this embodiment, aggregation can be performed in advance according to preset dimensions, including different dimensions such as time and domain name. For example, data with the same country, the same operator, the same CDN provider, and the same interface can be aggregated. Additionally, data can be aggregated based on domain name, meaning data can be aggregated from different dimensions. Therefore, when creating a table, this specific dimension for aggregation needs to be specified. This way, during the data import process from Kafka to the ClickHouse database, the database will automatically aggregate according to the preset aggregation dimensions, making data queries seamless for users and improving query efficiency.

[0055] Based on the above embodiments, in another embodiment provided in this disclosure, the query results obtained based on the data query conditions can also be stored in a cache. The query conditions and query results are stored in the cache as key-value pairs, with the query conditions as the key and the query results as the key-value pairs. This way, when a user requests data again, the cache can be searched first, which not only improves query efficiency but also reduces the query load on the server.

[0056] In the embodiments provided in this disclosure, when the key value is greater than a preset value, the key value can be split into multiple subkey values ​​and stored in a cache. When the key value is retrieved from the cache as a query result, the multiple subkey values ​​are concatenated to obtain the key value. Each subkey value is less than a preset value.

[0057] Specifically, since data query requests often involve numerous parameters, these parameters can easily become too long, potentially impacting Redis key storage and retrieval performance. Therefore, the cached key can be converted to the MD5 hash of the query parameters. Determining a cache hit then only requires comparing the MD5 hash of the parameters. The return value of a data query request is often a complete chart with a large data volume, resulting in large key-value pairs in Redis. Because Redis is a single-threaded architecture, accessing large keys can cause Redis server blocking and easily lead to request timeouts. Therefore, if a key is identified as large, it should be split into smaller keys when storing data in Redis, significantly improving data query efficiency.

[0058] Based on the above embodiments, in another embodiment provided in this disclosure, the data query conditions include data query dimensions and data query granularity. Step S120 may further include the following steps:

[0059] In step S121, it is determined whether the first data corresponding to the query dimension is found in the aggregate database.

[0060] In step S122, if the first data corresponding to the query dimension is found in the aggregate database, it is determined whether the aggregation granularity of the first data is greater than the data query granularity.

[0061] In step S123, if the aggregation granularity of the first data is not greater than the data query granularity, the data query result in the aggregation database is determined based on the data query conditions.

[0062] In the embodiments provided in this disclosure, since the aggregate database is aggregated in advance according to certain aggregation dimensions, such as aggregation according to different dimensions such as time and domain name, when performing data query, it is necessary to first determine whether the data corresponding to the query dimension is found in the aggregate database. For example, if the query dimension is to query data in region A, but the aggregate data does not contain data in region A, then there is no need to query in the aggregate database, and you can directly jump to the detailed database to continue the query.

[0063] If the first data point corresponding to the query dimension is found in the aggregate database, it's necessary to determine if the aggregation granularity of this first data point is larger than the query granularity. For example, if the aggregate database aggregates data hourly, but the query requires data per minute, the aggregation granularity is smaller than the query granularity. In this case, the desired data cannot be obtained directly from the aggregate database. Instead, the query can be redirected to the detailed database to improve query efficiency. Conversely, if the aggregate database aggregates data hourly, but the query requires daily data, the query granularity is larger than the aggregation granularity. The required data can be obtained directly from the aggregate database, eliminating the need to redirect to the detailed database again, significantly improving query efficiency.

[0064] The data processing method provided in this disclosure uses the OLAP database ClickHouse and specifies the AggregatingMergeTree aggregation engine when creating tables. This engine can only store aggregated data and aggregates data according to a specified time granularity. Using this aggregation engine can reduce the number of rows, save storage capacity, reduce cluster load, and improve the efficiency of aggregated data queries. Furthermore, when querying whether a piece of data belongs to the current collection class, the table can be created using the Bitmap engine. The Bitmap engine is suitable for representing collections of a large number of entity IDs with specific relationships, converting collection operations into bitwise operations, thereby improving query performance. Simultaneously, Bitmap only requires 1 bit to represent an integer value, which can greatly reduce storage space consumption.

[0065] For queries with relatively fixed time spans and dimensional metrics, materialized views corresponding to the query SQL can be created. Since materialized views store the data corresponding to the SQL, recalculation and aggregation of the data can be avoided, accelerating queries by trading space for time. Specifically, creating a materialized view using the AggregatingMergeTree engine allows the view to aggregate data based on the sorting key and a specified time granularity. During aggregation, a grouping aggregation is performed based on the sorting key, defined by the order by clause in the table creation statement, thus achieving the effect of speeding up queries.

[0066] By dividing each functional module according to its corresponding function, this disclosure provides a data processing device, which can be a server or a chip applied to a server. Figure 2 This is a schematic block diagram of the functional modules of a data processing apparatus provided for an exemplary embodiment of this disclosure. Figure 2 As shown, the data processing apparatus includes:

[0067] The receiving module 10 is used to receive a data query request, wherein the data query request carries data query conditions;

[0068] The first data query module 20 is used to obtain data query results in the aggregate database based on the data query conditions; the aggregate database is used to store aggregate data.

[0069] The second data query module 30 is used to determine, based on the data query results, whether to perform a data query through the detailed database based on the data query conditions; the detailed database is used to store detailed data, and the aggregated data is obtained by aggregating the detailed data.

[0070] In another embodiment provided in this disclosure, the second data query module includes:

[0071] The first query submodule is used to stop querying the detailed database if the query result is that the query data has been obtained.

[0072] The second query submodule is used to perform data query through the detailed database based on the data query conditions when the query result is that no query data is obtained.

[0073] In yet another embodiment provided in this disclosure, the apparatus further includes:

[0074] The data acquisition module is used to acquire detailed data and store the detailed data in the detailed database; the detailed data is data obtained by pre-processing network data collected from multiple clients.

[0075] The data aggregation module is used to aggregate the detailed data according to preset dimensions and store the resulting aggregated data in the aggregation database.

[0076] In yet another embodiment provided in this disclosure, the apparatus further includes:

[0077] A storage module is used to store the query results obtained based on the data query conditions into a cache; wherein the query conditions and the query results are stored in the cache in the form of key-value pairs, where the query conditions are the key names and the query results are the key values.

[0078] In yet another embodiment provided in this disclosure, the apparatus further includes:

[0079] A first processing module is configured to split the key value into multiple sub-key values ​​and store them in the cache when the key value is greater than a preset value; wherein each sub-key value is less than the preset value;

[0080] The second processing module is used to concatenate the multiple subkey values ​​to obtain the key value when the key value is obtained from the cache as a query result.

[0081] In another embodiment provided in this disclosure, the data query conditions include data query dimensions and data query granularity; the first data query module includes:

[0082] The first judgment submodule is used to determine whether the first data corresponding to the query dimension is found in the aggregated database;

[0083] The second judgment submodule is used to determine whether the aggregation granularity of the first data is greater than the data query granularity when the first data corresponding to the query dimension is found in the aggregate database.

[0084] The determination submodule is used to determine the data query results obtained in the aggregate database based on the data query conditions, provided that the aggregation granularity of the first data is not greater than the data query granularity.

[0085] For details, please refer to the description of the above embodiments, which will not be repeated here.

[0086] The data processing apparatus provided in this embodiment can, upon receiving a data query request, first perform a data query in an aggregated database based on the data query conditions, and then determine whether further data querying is needed in a detailed database based on the query results. This approach allows for pre-aggregation of data and storage of the aggregated data in an aggregated database. During a data query, the data is first queried in the aggregated database. Once the relevant data is found in the aggregated database, a direct query in the detailed database is unnecessary, thus avoiding the query efficiency issues associated with directly querying the detailed database.

[0087] This disclosure also provides an electronic device, including: at least one processor; a memory for storing processor-executable instructions; wherein the at least one processor is configured to execute the instructions to implement the methods disclosed in this disclosure.

[0088] Figure 3 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this disclosure. For example... Figure 3 As shown, the electronic device 1800 includes at least one processor 1801 and a memory 1802 coupled to the processor 1801. The processor 1801 can perform the corresponding steps in the methods disclosed in the embodiments of this disclosure.

[0089] The processor 1801 described above can also be called a central processing unit (CPU), which can be an integrated circuit chip with signal processing capabilities. Each step in the method disclosed in this embodiment can be implemented by the integrated logic circuitry in the processor 1801 or by software instructions. The processor 1801 can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this embodiment can be directly implemented by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can be located in the memory 1802, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor 1801 reads information from the memory 1802 and, in conjunction with its hardware, completes the steps of the method described above.

[0090] Furthermore, various operations / processes according to this disclosure, implemented via software and / or firmware, can be transmitted from a storage medium or network to a computer system with a dedicated hardware architecture, such as... Figure 4 The computer system 1900 shown is equipped with the programs that constitute the software. When various programs are installed, the computer system is able to perform various functions, including those described above. Figure 4 A block diagram of a computer system provided for an exemplary embodiment of this disclosure.

[0091] Computer System 1900 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0092] like Figure 4As shown, the computer system 1900 includes a computing unit 1901, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from a storage unit 1908 into a random access memory (RAM) 1903. The RAM 1903 may also store various programs and data required for the operation of the computer system 1900. The computing unit 1901, ROM 1902, and RAM 1903 are interconnected via a bus 1904. An input / output (I / O) interface 1905 is also connected to the bus 1904.

[0093] Multiple components in computer system 1900 are connected to I / O interface 1905, including: input unit 1906, output unit 1907, storage unit 1908, and communication unit 1909. Input unit 1906 can be any type of device capable of inputting information into computer system 1900. Input unit 1906 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 1907 can be any type of device capable of presenting information and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 1908 may include, but is not limited to, hard disks and optical disks. Communication unit 1909 allows computer system 1900 to exchange information / data with other devices via a network such as the Internet, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0094] The computing unit 1901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1901 performs the various methods and processes described above. For example, in some embodiments, the methods disclosed in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1908. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1900 via ROM 1902 and / or communication unit 1909. In some embodiments, the computing unit 1901 can be configured to perform the methods disclosed in this disclosure by any other suitable means (e.g., by means of firmware).

[0095] This disclosure also provides a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the methods disclosed in this disclosure.

[0096] The computer-readable storage medium in this disclosure can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. The aforementioned computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specifically, the aforementioned computer-readable storage medium may include electrical connections based on one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0097] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0098] This disclosure also provides a computer program product, including a computer program, wherein the computer program, when executed by a processor, implements the methods disclosed in the embodiments of this disclosure.

[0099] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer.

[0100] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0101] The modules, components, or units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules, components, or units do not necessarily constitute a limitation on the module, component, or unit itself.

[0102] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0103] The above description is merely an embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0104] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: Receive a data query request, the data query request carrying data query conditions; The data query results are obtained in the aggregate database based on the data query conditions; the aggregate database is used to store aggregated data, and the aggregate database aggregates data according to a specified time granularity; Based on the data query results, it is determined whether to perform a data query through the detailed database based on the data query conditions; the detailed database is used to store detailed data, and the aggregated data is obtained by aggregating the detailed data; The data query conditions include data query dimensions and data query granularity. The data query based on the data query conditions through a detailed database includes: Determine whether the first data corresponding to the query dimension is found in the aggregated database; If no first data corresponding to the query dimension is found in the aggregate database, data is queried through the detailed database based on the data query conditions. If the aggregated database finds first data corresponding to the query dimension, determine whether the aggregation granularity of the first data is greater than the data query granularity. If the aggregation granularity of the first data is greater than the data query granularity, data query is performed through the detailed database based on the data query conditions.

2. The method according to claim 1, characterized in that, The step of determining whether to perform a data query through the detailed database based on the data query conditions includes: If the query result is that the query data has been obtained, stop querying the detailed database; If the query result is no data obtained, a data query is performed through the detailed database based on the data query conditions.

3. The method according to claim 1, characterized in that, The method further includes: Detailed data is obtained and stored in the detailed database; the detailed data is obtained by performing preset processing on network data collected from multiple clients; The detailed data is aggregated according to preset dimensions, and the resulting aggregated data is stored in the aggregated database.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: The query results obtained based on the data query conditions are stored in a cache; wherein the query conditions and the query results are stored in the cache in the form of key-value pairs, where the query conditions are the key names and the query results are the key values.

5. The method according to claim 4, characterized in that, The method further includes: When the key value is greater than a preset value, the key value is split into multiple subkey values ​​and stored in the cache; wherein each subkey value is less than the preset value; When the key value is obtained from the cache as a query result, the multiple subkey values ​​are concatenated to obtain the key value.

6. The method according to claim 1, characterized in that, The process of obtaining data query results in the aggregated database based on the data query conditions also includes: If the aggregation granularity of the first data is not greater than the data query granularity, the data query result in the aggregated database is determined based on the data query conditions.

7. A data processing apparatus, characterized in that, The device includes: The receiving module is used to receive data query requests, wherein the data query requests carry data query conditions; The first data query module is used to obtain data query results in the aggregate database based on the data query conditions; the aggregate database is used to store aggregated data, and the aggregate database aggregates data according to a specified time granularity; The second data query module is used to determine, based on the data query results, whether to perform a data query through the detailed database based on the data query conditions; the detailed database is used to store detailed data, and the aggregated data is obtained by aggregating the detailed data. The data query conditions include data query dimensions and data query granularity. The data query based on the data query conditions through a detailed database includes: Determine whether the first data corresponding to the query dimension is found in the aggregated database; If no first data corresponding to the query dimension is found in the aggregate database, data is queried through the detailed database based on the data query conditions. If the aggregated database finds first data corresponding to the query dimension, determine whether the aggregation granularity of the first data is greater than the data query granularity. If the aggregation granularity of the first data is greater than the data query granularity, data query is performed through the detailed database based on the data query conditions.

8. The apparatus according to claim 7, characterized in that, The second data query module includes: The first query submodule is used to stop querying the detailed database if the query result is that the query data has been obtained. The second query submodule is used to perform data query through the detailed database based on the data query conditions when the query result is that no query data is obtained.

9. An electronic device, characterized in that, include: At least one processor; Memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1-6.

Images