Data query method, apparatus, device, medium, and product

Abstract

This application relates to a data query method, apparatus, computer device, storage medium, and computer program product. The method includes: first, a server receiving a data query request; then, obtaining pre-aggregated data based on the data query request; the pre-aggregated data being obtained by partially aggregating the data to be consumed; and finally, outputting the pre-aggregated data as the data query result. The method provided in this application, by obtaining pre-aggregated data through partial aggregation of the data to be consumed and outputting the pre-aggregated data as the query result, can improve the timeliness and accuracy of data queries.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, and in particular to a data query method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] With the advent of the big data era, data querying has become increasingly important. However, due to the explosive growth in data volume and the increasing richness of data types, the requirements for data querying methods are becoming more and more demanding.

[0003] Currently, the most common data query method involves batch processing data obtained from user behavior collection systems using MapReduce technology, and then utilizing distributed caching or relational databases to display the existing analysis results. This method uses traditional MapReduce technology for batch data processing, which reduces the timeliness of data processing. In business scenarios requiring rapid iteration, such as marketing and mobile banking operations, it is difficult to guarantee the timeliness and accuracy of the analysis conclusions. Summary of the Invention

[0004] Therefore, it is necessary to provide a data query method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the timeliness and accuracy of data query in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a data query method, the method comprising:

[0006] Receive data query requests;

[0007] Pre-aggregated data is obtained according to the data query request. The pre-aggregated data is obtained by partially aggregating the data to be consumed.

[0008] The pre-aggregated data is output as the data query result.

[0009] In one embodiment, after receiving the data query request, the method further includes:

[0010] The data query request is preprocessed, and the preprocessing includes at least one of the following: mapping the fields to be queried in the data query request to physical fields, verifying the permissions of the user information corresponding to the data query request, masking data layer differences, and optimizing the data query request.

[0011] The step of obtaining pre-aggregated data according to the data query request includes:

[0012] Pre-aggregated data is obtained based on the pre-processed data query request.

[0013] In one embodiment, the data query request carries a business type; before preprocessing the data query request, the method further includes:

[0014] The corresponding basic services are determined based on the business type. The business type includes at least one of event analysis, funnel analysis, and user segmentation. The basic services include metadata access services, different types of task execution, and query engine services.

[0015] When the underlying service is a query engine service, continue to execute the step of preprocessing the data query request.

[0016] In one embodiment, before receiving the data query request, the method further includes:

[0017] Check if the data to be consumed exists in the message queue;

[0018] When there is data to be consumed in the message queue, the data to be consumed is read and stored in the target database.

[0019] In one embodiment, before storing the data to be consumed in the target database, the method further includes:

[0020] The data to be consumed is subjected to data transformation processing to obtain transformed data, wherein the data transformation processing includes at least one of data verification, data processing, data dictionary construction, and data preprocessing;

[0021] The transformed data is partially aggregated to obtain pre-aggregated data;

[0022] The pre-aggregated data is stored in the target database.

[0023] In one embodiment, constructing the data dictionary includes:

[0024] Collect enumerated fields from the user behavior data;

[0025] Construct a data dictionary based on the enumerated fields.

[0026] Secondly, this application provides a data query device, the device comprising:

[0027] The receiving module is used to receive data query requests;

[0028] The acquisition module is used to acquire pre-aggregated data according to the data query request. The pre-aggregated data is obtained by partially aggregating the data to be consumed.

[0029] The output module is used to output the pre-aggregated data as the data query result.

[0030] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the methods in any of the above embodiments.

[0031] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the methods in any of the above embodiments.

[0032] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the methods in any of the above embodiments.

[0033] The aforementioned data query method, apparatus, computer equipment, storage medium, and computer program product first receive a data query request from the server, then obtain pre-aggregated data based on the data query request. This pre-aggregated data is obtained by partially aggregating the data to be consumed. Finally, the pre-aggregated data is output as the data query result. The method provided in this application, by partially aggregating the data to be consumed to obtain pre-aggregated data and outputting the pre-aggregated data as the query result, can improve the timeliness and accuracy of data queries. Attached Figure Description

[0034] Figure 1 This is a diagram illustrating the application environment of a data query method in one embodiment.

[0035] Figure 2 This is a flowchart illustrating a data query method in one embodiment;

[0036] Figure 3 This is a flowchart illustrating the process of constructing a data dictionary in one embodiment;

[0037] Figure 4 This is a structural block diagram of a data query device in one embodiment;

[0038] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0040] The data query method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Server 104 receives data query requests, obtains pre-aggregated data based on the query requests, and then outputs the pre-aggregated data as the data query result to terminal 102 for display. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0041] In one embodiment, such as Figure 2 As shown, a data query method is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:

[0042] S201, Receive data query request.

[0043] The server receives data query requests initiated by users, then controls the application gateway to verify the query content. After successful verification, the application gateway distributes the data query request to various business modules. The application gateway serves as the system's user interface and also as the control center for backend business modules and basic services. It is responsible for health checks of each module, request routing, and maintaining distributed sessions in conjunction with the user authentication module. A distributed session is a special in-memory object created by the server to save user state. The user authentication module can be integrated with existing single sign-on systems to achieve unified login and session management across systems.

[0044] S202. Obtain pre-aggregated data based on the data query request. The pre-aggregated data is obtained by partially aggregating the data to be consumed.

[0045] The data to be consumed includes user behavior data and user tag data. The data to be consumed is stored in the target database. For example, the target database can be HDFS (Hadoop Distributed File System, a distributed storage technology). Local aggregation uses KE (Kyligence Enterprise, an online analytical processing technology with pre-computation as its core) aggregation service to perform local aggregation on the data stored in HDFS. This can reduce query overhead and improve analysis efficiency by trading space for time.

[0046] Specifically, the server first transforms the data to be consumed to obtain transformed data, and then stores the transformed data in HDFS. The server then performs partial aggregation on the transformed data stored in HDFS to obtain pre-aggregated data. For example, using the KE aggregation service to perform partial aggregation on data stored in HDFS can reduce query overhead and improve analysis efficiency by trading space for time.

[0047] Aggregation services refer to pre-processing the data to be consumed based on the data to be queried, in order to obtain pre-aggregated data.

[0048] S203. Output the pre-aggregated data as the data query result.

[0049] Specifically, the server returns the pre-aggregated data to the front end, and the front-end report control outputs the pre-aggregated data to the user.

[0050] In the aforementioned data query method, the server first receives a data query request, then obtains pre-aggregated data based on the request. This pre-aggregated data is obtained by partially aggregating the data to be consumed. Finally, the pre-aggregated data is output as the data query result. The method provided in this application, by partially aggregating the data to be consumed to obtain pre-aggregated data and outputting this pre-aggregated data as the query result, can improve the timeliness and accuracy of data queries.

[0051] In some embodiments, after receiving a data query request, the method further includes: preprocessing the data query request, wherein the preprocessing includes at least one of physical field mapping of the query field in the data query request, permission verification of the user information corresponding to the data query request, masking data layer differences, and query optimization of the data query request; and obtaining pre-aggregated data according to the data query request, including obtaining pre-aggregated data according to the preprocessed data query request.

[0052] In this step, the server transparently completes the preprocessing process by modifying the SQL (Structured Query Language).

[0053] Physical field mapping refers to mapping the field to be queried to the field in the transformed data stored in HDFS, and then dividing the field to be queried into tasks, including scheduled tasks and offline tasks. For example, if the field to be queried is the field corresponding to a customer's purchase of a certain bank product, this field is mapped to the field corresponding to that bank product in the transformed data stored in HDFS, and then tasks are divided into tasks based on the mapping results.

[0054] Permission verification refers to verifying whether the user corresponding to the user information carried in the data query request has the necessary query permissions. Only if the user does so will the scheduled task and offline task mentioned above be generated. For example, if only system administrators have query permissions, the server will determine whether the user is an administrator based on the user information. If the user is an administrator, the scheduled task and offline task mentioned above will be generated.

[0055] Masking data layer differences means that the underlying data layer differences of the query engine can be ignored, thereby achieving the purpose of masking data layer differences. In this context, the query engine serves as a unified query entry point, responding to query requests.

[0056] Query optimization is used to improve query speed and the responsiveness of query results. For example, a combination of rule-based heuristic optimization and cost estimation-based optimization can be used to physically optimize query statements.

[0057] The method provided in this step can effectively improve query efficiency and the security of the query process by preprocessing the data query request.

[0058] In some embodiments, the data query request carries a business type; before preprocessing the data query request, the method further includes: determining the corresponding basic service based on the business type, wherein the business type includes at least one of event analysis, funnel analysis and user segmentation, and the basic service includes metadata access service, different types of task execution and query engine service; when the basic service is query engine service, the step of preprocessing the data query request continues.

[0059] The basic services include metadata access services, execution of different types of tasks, and query engine services. Metadata access services define various basic information such as events, attributes, and field types, providing a RESTful (Representational State Transfer, a design and development style for web application services) interface for on-premises access to the system's metadata database. Execution of different types of tasks refers to executing a large number of offline and scheduled tasks generated by the system. The query engine service is based on real-time queries using Kudu and Spark computing servers. Kudu is an online analytical processing technology that supports random read / write operations and can be used to store user tag data, achieving sub-second response times when user attributes are updated in real time. Spark is a memory-based big data computing framework that can be used to store user behavior data and effectively handle petabyte (PB) level data volumes.

[0060] The business types include at least one of event analysis, funnel analysis, and user segmentation. Event analysis refers to using financial market data to analyze the impact of a specific event on the company's value; funnel analysis refers to analyzing the conversion or churn of users at each step in the user journey from start to finish; user segmentation refers to dividing users into several business-significant groups based on characteristic variables using various methods.

[0061] In this step, the business types depend on and correspond to the basic services, but there are no dependencies or weak relationships between the business types. Each business type can form an independent small application service, which can be developed, tested, deployed and run independently.

[0062] Kudu compute servers and Spark compute servers can be integrated with the HDFS storage system, sharing data while simultaneously seeking a master.

[0063] The method provided in this step, through real-time queries based on Kudu compute servers and Spark compute servers, can significantly improve the system's query performance and response concurrency under the same hardware configuration.

[0064] In some embodiments, before receiving a data query request, the method further includes: detecting whether there is data to be consumed in the message queue; when there is data to be consumed in the message queue, reading the data to be consumed and storing the data to be consumed in the target database.

[0065] In this step, the message queue refers to a container that stores messages during transmission, and the target database refers to a database that stores data to be consumed, transformed data, and pre-aggregated data. The data to be consumed is accessed by the server from a big data platform or data exchange platform via streaming or batch data access methods. User behavior data originates from the upstream system's Kafka. This step uses Spark Streaming Kafka to consume user behavior data, while user tag data is accessed in batches by a separate batch processing module.

[0066] Specifically, when the server detects that there is data to be consumed in the message queue, the server first stores the user behavior data in Spark and the user tag data in Kudu, and then shares the data in Spark and Kudu to the HDFS data storage system.

[0067] The method provided in this step uses a message queue to determine whether to store data to be consumed in the database, which can improve the accuracy of data storage.

[0068] In some embodiments, before storing the data to be consumed in the target database, the method further includes: performing data transformation processing on the data to be consumed to obtain transformed data, wherein the data transformation processing includes at least one of data verification, data processing, constructing a data dictionary, and data preprocessing; performing partial aggregation on the transformed data to obtain pre-aggregated data; and storing the pre-aggregated data in the target database.

[0069] In this step, data validation refers to verifying the data format and legality of user behavior data, removing invalid data and backing it up in the error log. Data processing involves parsing the user behavior data, importing it into the target database according to the system field configuration, building a data dictionary, extracting the corresponding data from the filtered data based on the fields preset by the analyst, and data preprocessing, such as mapping phone numbers to phone models, adding brand codes to phone models, and mapping some attribute values ​​to multiple attributes.

[0070] The method provided in this step transforms and partially aggregates the data to be consumed before storing it in the target database, which can improve the efficiency of data query and analysis.

[0071] In some embodiments, such as Figure 3 As shown, construct the data dictionary, including:

[0072] S301. Collect enumeration fields from user behavior data;

[0073] In this step, enumeration refers to listing out the values ​​of the variables one by one, and the values ​​of the variables are limited to the range of the listed values.

[0074] S302. Construct a data dictionary based on the enumerated fields.

[0075] In this step, the data dictionary refers to the definition and description of data items, data structures, data flows, data storage, and processing logic. Its purpose is to provide a detailed description of each element in the data flow diagram. The data dictionary is used for simple modeling projects.

[0076] The method provided in this step, by constructing a data dictionary, allows for a clearer understanding of the definition and explanation of each data point and related concept.

[0077] In one embodiment, another data query method is provided, including the following steps:

[0078] (1) The data access and transformation module reads data from the data acquisition bus and data exchange platform, processes and transforms it, and then stores it in the computing service cluster (Kudu and Spark). At the same time, when forwarding, it accesses metadata through the metadata access service and retrieves metadata from the metadata database.

[0079] (2) The data calculation and storage module processes and preprocesses the stored raw user behavior data according to the established rules in order to optimize subsequent queries.

[0080] (3) When an analyst initiates a query, the query content will first be uniformly logged in and verified through the gateway, and then the gateway will distribute the request to various business modules.

[0081] (4) Each business module will call the query engine to initiate a query request. Depending on the business needs, it may call and query the metadata database or initiate a scheduled task.

[0082] (5) Scheduled tasks and offline processing will also call the query engine and / or the metadata database.

[0083] (6) The query engine will send a query request to the KE aggregation service.

[0084] (7) The KE aggregation service determines the query method and calculation rules based on the preprocessing results, and uses the preprocessing results for some calculations and pushes the calculation service module down for processing for others.

[0085] (8) The calculation results are returned to the front end and output to the user by the front-end report control.

[0086] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0087] Based on the same inventive concept, this application also provides a data query apparatus for implementing the data query method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data query apparatus embodiments provided below can be found in the limitations of the data query method described above, and will not be repeated here.

[0088] In one embodiment, such as Figure 4 As shown, a data query device 400 is provided, including: a receiving module 401, an acquiring module 402, and an output module 403, wherein:

[0089] The receiving module 401 is used to receive data query requests.

[0090] The acquisition module 402 is used to acquire pre-aggregated data according to the data query request. The pre-aggregated data is obtained by partially aggregating the data to be consumed.

[0091] Output module 403 is used to output the pre-aggregated data as a data query result.

[0092] In some embodiments, the data query device 400 is specifically used for: preprocessing the data query request, the preprocessing including at least one of physical field mapping of the query field in the data query request, permission verification of the user information corresponding to the data query request, masking data layer differences, and query optimization of the data query request; obtaining pre-aggregated data according to the data query request includes: obtaining pre-aggregated data according to the preprocessed data query request.

[0093] In some embodiments, the data query device 400 is further configured to: determine a corresponding basic service based on the business type, wherein the business type includes at least one of event analysis, funnel analysis, and user segmentation, and the basic service includes metadata access service, different types of task execution, and query engine service; when the basic service is a query engine service, continue to execute the step of preprocessing the data query request.

[0094] In some embodiments, the data query device 400 is further configured to: detect whether the data to be consumed exists in the message queue; when the data to be consumed exists in the message queue, read the data to be consumed and store the data to be consumed in the target database.

[0095] In some embodiments, the data query device 400 further includes:

[0096] The data conversion module is used to perform data conversion processing on the data to be consumed to obtain converted data, wherein the data conversion processing includes at least one of data verification, data processing, data dictionary construction, and data preprocessing.

[0097] The local aggregation module is used to perform local aggregation on the transformed data to obtain pre-aggregated data.

[0098] A storage module is used to store the pre-aggregated data into the target database.

[0099] In some embodiments, the data conversion module is further configured to: collect enumeration fields from the user behavior data; and construct a data dictionary based on the enumeration fields.

[0100] Each module in the aforementioned data query device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0101] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data to be consumed. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a data query method.

[0102] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0103] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: receiving a data query request; obtaining pre-aggregated data according to the data query request, the pre-aggregated data being obtained by partially aggregating data to be consumed; and outputting the pre-aggregated data as a data query result.

[0104] In one embodiment, after the processor executes the computer program to receive the data query request, the process further includes: preprocessing the data query request, wherein the preprocessing includes at least one of physical field mapping of the query field in the data query request, permission verification of the user information corresponding to the data query request, masking data layer differences, and query optimization of the data query request; and obtaining pre-aggregated data according to the data query request includes: obtaining pre-aggregated data according to the preprocessed data query request.

[0105] In one embodiment, before the processor executes the computer program to preprocess the data query request, the method further includes: determining a corresponding basic service based on the business type, wherein the business type includes at least one of event analysis, funnel analysis, and user segmentation, and the basic service includes metadata access service, different types of task execution, and query engine service; when the basic service is a query engine service, the step of preprocessing the data query request continues.

[0106] In one embodiment, before the processor executes the computer program to receive a data query request, the method further includes: detecting whether the data to be consumed exists in a message queue; when the data to be consumed exists in the message queue, reading the data to be consumed and storing the data to be consumed in a target database.

[0107] In one embodiment, before the processor executes the computer program to store the data to be consumed into the target database, the method further includes: performing data transformation processing on the data to be consumed to obtain transformed data, wherein the data transformation processing includes at least one of data verification, data processing, constructing a data dictionary, and data preprocessing; performing local aggregation on the transformed data to obtain pre-aggregated data; and storing the pre-aggregated data into the target database.

[0108] In one embodiment, the construction of a data dictionary implemented by a processor when executing a computer program includes: collecting enumeration fields from the user behavior data; and constructing a data dictionary based on the enumeration fields.

[0109] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the following steps: receiving a data query request; obtaining pre-aggregated data according to the data query request, the pre-aggregated data being obtained by partially aggregating data to be consumed; and outputting the pre-aggregated data as a data query result.

[0110] In one embodiment, after the computer program receives the data query request when executed by the processor, it further includes: preprocessing the data query request, the preprocessing including at least one of physical field mapping of the query field in the data query request, permission verification of the user information corresponding to the data query request, masking data layer differences, and query optimization of the data query request; the step of obtaining pre-aggregated data according to the data query request includes: obtaining pre-aggregated data according to the preprocessed data query request.

[0111] In one embodiment, before the computer program is executed by the processor to preprocess the data query request, the method further includes: determining a corresponding basic service based on the business type, wherein the business type includes at least one of event analysis, funnel analysis, and user segmentation, and the basic service includes metadata access service, different types of task execution, and query engine service; when the basic service is a query engine service, the step of preprocessing the data query request continues.

[0112] In one embodiment, before the computer program is executed by the processor to receive a data query request, the method further includes: detecting whether the data to be consumed exists in a message queue; when the data to be consumed exists in the message queue, reading the data to be consumed and storing the data to be consumed in a target database.

[0113] In one embodiment, before the computer program is executed by a processor to store the data to be consumed into a target database, the method further includes: performing data transformation processing on the data to be consumed to obtain transformed data, wherein the data transformation processing includes at least one of data verification, data processing, constructing a data dictionary, and data preprocessing; performing local aggregation on the transformed data to obtain pre-aggregated data; and storing the pre-aggregated data into the target database.

[0114] In one embodiment, the construction of a data dictionary implemented when a computer program is executed by a processor includes: collecting enumeration fields from the user behavior data; and constructing a data dictionary based on the enumeration fields.

[0115] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: receiving a data query request; obtaining pre-aggregated data according to the data query request, the pre-aggregated data being obtained by partially aggregating data to be consumed; and outputting the pre-aggregated data as a data query result.

[0116] In one embodiment, after the computer program receives the data query request when executed by the processor, it further includes: preprocessing the data query request, the preprocessing including at least one of physical field mapping of the query field in the data query request, permission verification of the user information corresponding to the data query request, masking data layer differences, and query optimization of the data query request; the step of obtaining pre-aggregated data according to the data query request includes: obtaining pre-aggregated data according to the preprocessed data query request.

[0117] In one embodiment, before the computer program is executed by the processor to preprocess the data query request, the method further includes: determining a corresponding basic service based on the business type, wherein the business type includes at least one of event analysis, funnel analysis, and user segmentation, and the basic service includes metadata access service, different types of task execution, and query engine service; when the basic service is a query engine service, the step of preprocessing the data query request continues.

[0118] In one embodiment, before the computer program is executed by the processor to receive a data query request, the method further includes: detecting whether the data to be consumed exists in a message queue; when the data to be consumed exists in the message queue, reading the data to be consumed and storing the data to be consumed in a target database.

[0119] In one embodiment, before the computer program is executed by a processor to store the data to be consumed into a target database, the method further includes: performing data transformation processing on the data to be consumed to obtain transformed data, wherein the data transformation processing includes at least one of data verification, data processing, constructing a data dictionary, and data preprocessing; performing local aggregation on the transformed data to obtain pre-aggregated data; and storing the pre-aggregated data into the target database.

[0120] In one embodiment, the construction of a data dictionary implemented when a computer program is executed by a processor includes: collecting enumeration fields from the user behavior data; and constructing a data dictionary based on the enumeration fields.

[0121] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0122] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0123] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data query method, characterized by, The method includes: Receive data query requests; The receiving of data query requests includes: acquiring a data query request initiated by a user and controlling the application gateway to verify the query content; if the verification is successful, controlling the application gateway to distribute the data query request to various business modules. The data query request carries a business type; before receiving the data query request, the method further includes: determining the corresponding basic service based on the business type, wherein the business type includes at least one of event analysis, funnel analysis, and user segmentation, and the basic service includes metadata access service, execution of different types of tasks, and query engine service; when the basic service is a query engine service, a step of preprocessing the data query request is performed. Pre-aggregated data is obtained according to the data query request. The pre-aggregated data is obtained by partially aggregating the data to be consumed. The data to be consumed includes user behavior data and user tag data. The pre-aggregated data is output as the data query result.

2. The method according to claim 1, characterized in that, After receiving the data query request, the method further includes: The data query request is preprocessed, and the preprocessing includes at least one of the following: mapping the fields to be queried in the data query request to physical fields, verifying the permissions of the user information corresponding to the data query request, masking data layer differences, and optimizing the data query request. The step of obtaining pre-aggregated data according to the data query request includes: Pre-aggregated data is obtained based on the pre-processed data query request.

3. The method according to claim 1, characterized in that, Before receiving the data query request, the method further includes: Check if the data to be consumed exists in the message queue; When there is data to be consumed in the message queue, the data to be consumed is read and stored in the target database.

4. The method according to claim 3, characterized in that, Before storing the data to be consumed in the target database, the method further includes: The data to be consumed is subjected to data transformation processing to obtain transformed data, wherein the data transformation processing includes at least one of data verification, data processing, data dictionary construction, and data preprocessing; The transformed data is partially aggregated to obtain pre-aggregated data; The pre-aggregated data is stored in the target database.

5. The method according to claim 4, characterized in that, The construction of the data dictionary includes: Collect enumerated fields from the user behavior data; Construct a data dictionary based on the enumerated fields.

6. A data query device, characterized in that, The device includes: The receiving module is used to receive data query requests; The receiving module is also used to retrieve data query requests initiated by users and control the application gateway to verify the query content; if the verification is successful, the application gateway is controlled to distribute the data query request to various business modules. The data query request carries a business type; the receiving module is further configured to determine the corresponding basic service based on the business type, wherein the business type includes at least one of event analysis, funnel analysis, and user segmentation, and the basic service includes metadata access service, execution of different types of tasks, and query engine service; when the basic service is query engine service, a step of preprocessing the data query request is performed. The acquisition module is used to acquire pre-aggregated data according to the data query request. The pre-aggregated data is obtained by partially aggregating the data to be consumed. The data to be consumed includes user behavior data and user tag data. The output module is used to output the pre-aggregated data as the data query result.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images