Data processing method, device and equipment

Abstract

The embodiment of the invention provides a data processing method, device and equipment, and the method comprises the steps: receiving a business processing request for triggering and executing a target business for a target user; in response to the service processing request, obtaining a first data query condition and a user identifier of the target user; determining a target serial number corresponding to the user identifier of the target user in a preset data lake system, and determining target file data in the data lake system based on metadata index cache according to the target serial number and the first data query condition; sending a data reading task for the target file data to a data reading executor, and receiving first data read from the target file data by the data reading executor; and determining target data corresponding to the first data query condition according to the first data, and executing the target service according to the target data.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to a data processing method, apparatus, and device. Background Art

[0002] With the rapid development of the Internet industry, the amount of data generated during the business service provision process is increasing. Therefore, how to improve the business processing efficiency by improving the data query efficiency to better provide business services for users (such as how to improve the risk detection efficiency by improving the data query efficiency to protect the privacy data of users from being leaked when performing risk detection processing, etc.) has become the focus of attention of network operators.

[0003] When performing a data query, corresponding data can be obtained from a database according to a query statement. However, due to the increasing complexity of the format of the data to be queried and the increasing size of the data stored in the database, the query efficiency of data query by the above method is low, affecting the subsequent business processing efficiency. For this reason, the embodiments of this specification provide a technical solution to improve the business processing efficiency by improving the data query efficiency. Summary of the Invention

[0004] The purpose of the embodiments of this specification is to provide a technical solution to improve the business processing efficiency by improving the data query efficiency.

[0005] To achieve the above technical solution, the embodiments of this specification are implemented as follows: A data processing method provided by the embodiments of this specification, the method includes: receiving a service processing request for triggering the execution of a target service for a target user; in response to the service processing request, obtaining a first data query condition corresponding to the service processing request and the user identifier of the target user; determining a target serial number corresponding to the user identifier of the target user in a preset data lake system, and based on the target serial number and a metadata index cache, determining target file data in the data lake system; sending a data reading task for the target file data to a data reading executor, and receiving first data read by the data reading executor from the target file data according to the first data query condition and the target serial number; determining target data corresponding to the first data query condition according to the first data, and executing the target service according to the target data.

[0006] A data processing device provided by an embodiment of this specification, the device includes: a request receiving module, configured to receive a service processing request for triggering the execution of a target service for a target user; a data acquisition module, configured to, in response to the service processing request, acquire a first data query condition corresponding to the service processing request and the user identifier of the target user; a data query module, configured to determine a target serial number corresponding to the user identifier of the target user in a preset data lake system, and based on the target serial number and the first data query condition, determine target file data in the data lake system based on a metadata index cache; a task sending module, configured to send a data reading task for the target file data to a data reading executor, and receive the first data read from the target file data by the data reading executor according to the first data query condition and the target serial number; a service execution module, configured to determine target data corresponding to the first data query condition according to the first data, and execute the target service according to the target data.

[0007] A data processing device provided by an embodiment of this specification, the data processing device includes: a processor; and a memory arranged to store computer-executable instructions, the executable instructions, when executed, cause the processor to: receive a service processing request for triggering the execution of a target service for a target user; in response to the service processing request, acquire a first data query condition corresponding to the service processing request and the user identifier of the target user; determine a target serial number corresponding to the user identifier of the target user in a preset data lake system, and based on the target serial number and the first data query condition, determine target file data in the data lake system based on a metadata index cache; send a data reading task for the target file data to a data reading executor, and receive the first data read from the target file data by the data reading executor according to the first data query condition and the target serial number; determine target data corresponding to the first data query condition according to the first data, and execute the target service according to the target data.

[0008] An embodiment of this specification also provides a storage medium for storing computer-executable instructions. When the executable instructions are executed by a processor, the following process is implemented: receiving a service processing request for triggering the execution of a target service for a target user; in response to the service processing request, obtaining a first data query condition corresponding to the service processing request and the user identifier of the target user; determining a target serial number corresponding to the user identifier of the target user in a preset data lake system, and based on the target serial number and the first data query condition, determining target file data in the data lake system based on a metadata index cache; sending a data reading task for the target file data to a data reading executor, and receiving the first data read from the target file data by the data reading executor according to the first data query condition and the target serial number; determining target data corresponding to the first data query condition according to the first data, and executing the target service according to the target data.

[0009] An embodiment of this specification also provides a computer program product, including a computer program. When the computer program is executed by a processor, the following process is implemented: receiving a service processing request for triggering the execution of a target service for a target user; in response to the service processing request, obtaining a first data query condition corresponding to the service processing request and the user identifier of the target user; determining a target serial number corresponding to the user identifier of the target user in a preset data lake system, and based on the target serial number and the first data query condition, determining target file data in the data lake system based on a metadata index cache; sending a data reading task for the target file data to a data reading executor, and receiving the first data read from the target file data by the data reading executor according to the first data query condition and the target serial number; determining target data corresponding to the first data query condition according to the first data, and executing the target service according to the target data. Description of the Drawings

[0010] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings described below are only some embodiments recorded in this specification. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings. Figure 1 It is a schematic diagram of a data processing method in this specification. Figure 2 It is a schematic diagram of a point query service processing flow in this specification. Figure 3Schematic diagram of the determination process of a first data query condition in this specification; Figure 4 Schematic diagram of the construction process of a metadata index tree in this specification; Figure 5 Schematic diagram of the construction process of a metadata index tree in this specification; Figure 6 Schematic diagram of the data processing process for determining target file data according to the metadata index tree in this specification; Figure 7 Schematic diagram of the acquisition process of target data in this specification; Figure 8 Schematic diagram of the data processing process for data reading and processing from the local cache in this specification; Figure 9 Schematic diagram of the data processing process for determining the data acquisition result according to the result cache in this specification; Figure 10 Schematic diagram of a local big data query service framework in this specification; Figure 11 Schematic diagram of a data processing device in this specification; Figure 12 Schematic diagram of a data processing device in this specification. Detailed implementation

[0011] The embodiments of this specification provide a data processing method, device and equipment.

[0012] In order to enable those skilled in the art to better understand the technical solutions in this specification, the following will clearly and completely describe the technical solutions in the embodiments of this specification in conjunction with the accompanying drawings in the embodiments of this specification. Obviously, the described embodiments are only a part of the embodiments of this specification, rather than all the embodiments. Based on the embodiments in this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this specification.

[0013] The embodiments of this specification provide a technical solution for improving the business processing efficiency by enhancing the data query efficiency. With the rapid development of the Internet industry, the amount of data generated during the business service provision process is increasing. Therefore, how to improve the business processing efficiency by enhancing the data query efficiency to better provide business services to users (for example, when performing risk detection processing, how to improve the risk detection efficiency by enhancing the data query efficiency to protect the privacy data of users from being leaked, etc.) has become the focus of attention of network operators. When performing data query, corresponding data can be obtained from the database according to the query statement. However, due to the increasing complexity of the formats of the data to be queried and the growing volume of data stored in the database, the query efficiency of data query by the above method is low, which affects the subsequent business processing efficiency. For this reason, the embodiments of this specification provide a technical solution for improving the business processing efficiency by enhancing the data query efficiency. In this solution, by receiving a business processing request for triggering the execution of a target business for a target user, in response to the business processing request, obtaining the first data query condition corresponding to the business processing request and the user identifier of the target user, determining the target serial number corresponding to the user identifier of the target user in the preset data lake system, and based on the target serial number and the first data query condition, determining the target file data in the data lake system based on the metadata index cache, sending the data reading task for the target file data to the data reading executor, and receiving the first data read from the target file data by the data reading executor according to the first data query condition and the target serial number, determining the target data corresponding to the first data query condition according to the first data, and executing the target business according to the target data. In this way, on the one hand, the data source system can store various different formats of data, and the data corresponding to the business processing request can be quickly obtained through the data lake system. On the other hand, when performing data search, the target file data corresponding to the target user can be determined according to the metadata index cache, and then the data reading task corresponding to the target file data can be sent to the data reading executor for data reading processing to improve the data reading efficiency in the point query scenario and the subsequent business processing efficiency. The specific processing can refer to the specific content in the following embodiments.

[0014] As Figure 1 shown, the embodiments of this specification provide a data processing method. The execution subject of this method can be a server. The server can be an independent server or a server cluster composed of multiple servers, etc. The server can be a background server for financial business or online shopping business, etc., or a background server of a certain application program, etc. In this embodiment, the execution subject is taken as an example of a server for detailed description. The method can specifically include the following steps: In step S102, receive a business processing request for triggering the execution of a target business for a target user.

[0015] Among them, the target service can be any data processing service. For example, the target service can be a product recommendation service, a risk detection service, a fraud detection service, etc.

[0016] In implementation, taking the target service as the product recommendation service as an example, the target user can input user requirement information in the product trading application of the terminal device. The terminal device can generate a service processing request for the product recommendation service based on the user requirement information and send the service processing request to the server.

[0017] Or, taking the target service as the risk detection service as an example, the target user can trigger the execution of the resource transfer service in the resource transfer application of the terminal device. The terminal device can generate a service processing request for the risk detection service based on the resource transfer service.

[0018] In addition, the above-mentioned generation method of the service processing request is an optional and implementable generation method. In actual application scenarios, there can also be multiple different generation methods, and different generation methods can be selected according to different actual application scenarios. This embodiment of the specification does not make specific limitations on this.

[0019] In step S104, in response to the service processing request, obtain the first data query condition corresponding to the service processing request and the user identifier of the target user.

[0020] Among them, the first data query condition can include any condition that can be used for data query. For example, the first data query condition can include a time query condition, a numerical range query condition, a location information query condition, etc.

[0021] In implementation, taking the target service as the product recommendation service as an example, the service processing request can carry the user requirement information and the user identifier of the target user. The server can determine the first data query condition based on the user requirement information. For example, the server can use a preset large language model to determine the first data query condition based on the user requirement information. Specifically, the first data query condition related to the product recommendation service determined by the server can include a time query condition (such as querying products launched from January 1st to January 15th) and a data range query condition (such as querying products with a product trading volume greater than 100 in the past thirty days).

[0022] Taking the target service as the risk detection service for a certain resource transfer service as an example, the server can determine the first data query condition corresponding to the service processing request according to the preset detection period, that is, the first query condition determined by the server can include a time query condition. For example, the time query condition (i.e., the first query condition) corresponding to the service processing request can be to query the resource transfer data of the target user in the past thirty days.

[0023] The method and specific content for determining the above first data query condition are an optional and implementable implementation manner. In actual application scenarios, there can be multiple different implementation manners, and different implementation manners can be selected according to different actual application scenarios. The embodiments of this specification do not make specific limitations on this.

[0024] In step S106, determine the target serial number corresponding to the user identifier of the target user in the preset data lake system, and based on the target serial number and the first data query condition, determine the target file data in the data lake system based on the metadata index cache.

[0025] Among them, the preset data lake (paimon) system can be a system that stores data in its natural or original format. The preset data lake system can store object blocks or file data. The data lake system can include different types of data such as structured data (rows and columns) from relational databases, semi-structured data (such as CSV, logs, XML, JSON), unstructured data (such as emails, documents, PDFs, etc.), and binary data (such as images, audio, videos). Metadata can be data used to describe data (data about data), information that can be used to describe data properties (properties), and can be used to support functions such as indicating storage locations, historical data, resource searches, and file records.

[0026] In implementation, the server can determine the target sequence corresponding to the user identifier of the target user in the data lake system according to the preset correspondence between the identifier and the serial number.

[0027] The server can quickly locate the target file data in the data lake system that corresponds to the target serial number and meets the first data query condition based on the target serial number and the first data query condition and the metadata index cache (metaindex cache).

[0028] For example, assume that the target serial number is 208801 and the first query condition is to query data from April 1, 2024 to May 1, 2024. The server can, based on the target serial number and the first data query condition and the metadata index cache, determine the target file data in the data lake system that contains the serial number 208801 and data from April 1, 2024 to May 1, 2024. For example, the target file data can be file data that contains serial numbers 208801 - 208803 and data from March 1, 2024 to June 1, 2024.

[0029] In step S108, send the data reading task for the target file data to the data reading executor, and receive the first data read from the target file data by the data reading executor according to the first data query condition and the target serial number.

[0030] Among them, the data reading executor can be an executor with its own private state and behavior, and there is no shared state information between each data reading executor and other data reading executors. For example, the data reading executor can be an actor in the Actor model. Actors can interact by sending and receiving messages. Message passing is asynchronous, and the sender does not need to wait for the message to be received and can continue other operations, which can enhance the responsiveness and scalability of data processing.

[0031] In implementation, since the target file data may contain data irrelevant to the target serial number and / or the first query condition, in order to improve the data query efficiency, the server can send the data reading task for the target file data to the data reading executor, so that the data reading executor reads the first data from the target file data according to the first data query condition and the target serial number.

[0032] In addition, when there are multiple target file data, the server can send the data reading tasks corresponding to each target file data to the data reading executors in the distributed cluster through a data allocation router (DataSplitRouter) for data reading processing, so as to improve the data reading efficiency through a distributed data processing method.

[0033] For example, as Figure 2 shown, the target file data can include File Data 1, File Data 2, and File Data 3. The server can send the data reading task of File Data 1 to Data Reading Executor 1, send the data reading task of File Data 2 to Data Reading Executor 2, and send the data reading task of File Data 3 to Data Reading Executor 3. Among them, in addition to the data corresponding to the target serial number (such as 208801) and / or the first data query condition in each target file data, it can also include data corresponding to other serial numbers. For example, File Data 1 can include data corresponding to serial numbers 208801 to 208818, File Data 2 can include data corresponding to serial numbers 208801 to 208860, and File Data 3 can include data corresponding to serial numbers 208799 to 208803.

[0034] Each data reading executor can perform data reading processing on the received target file data according to the target serial number and the first data query condition (such as aggregating ACC, sorting limit, grouping by, etc.), and obtain the first data corresponding to the target serial number and the first data query condition. Then, after executing the data reading task, the data reading executor can return the first data read from the target file data to the main query (Query Main) of the server.

[0035] In step S110, according to the first data, determine the target data corresponding to the first data query condition, and execute the target service according to the target data.

[0036] In implementation, in Figure 2 In the processing flow of the point query service shown, for distributed cluster management and remote calls, the pekko model can be used. The actor programming model of pekko can reduce the difficulty of task distribution processing and multi-threaded programming in a distributed environment, allowing developers to spend more time and energy on business logic processing. The point query service framework can include a protocol layer, a task distribution layer, and a task execution layer. Among them, in the protocol layer, the server can determine the first data query condition and the target serial number. In the task execution layer, the data reading executor can read the first data from the target file data according to the first data query condition and the target serial number.

[0037] In the task distribution layer, in addition to the execution plan process and the task distribution process, the server can also execute a result merging process. Among them, the execution plan process is to quickly locate the target file data according to the metadata information. The task distribution process is to distribute the data reading tasks corresponding to the target file data to the data reading executors in the cluster. The result merging process is to perform result merging processing on the first data read by the data reading executors in the cluster, that is, to perform summary processing on multiple first data and determine the summary result as the target data corresponding to the first data query condition.

[0038] For example, taking the target service as a product recommendation service, the first data query condition can include a time query condition (such as querying products launched from January 1 to January 15) and a data range query condition (such as querying products with a product transaction volume greater than 100 in the past thirty days). The first data queried according to the user identifier of the target user and the first data query condition can include product a, product b, and product c, and the product transaction volume of each product in the past thirty days.

[0039] The server can sort these three products according to the product transaction volume of each product in the past thirty days to obtain target data. Then, the server can determine the target products for product recommendation based on the target data and push the product information corresponding to the target products to the terminal device.

[0040] For another example, taking the risk detection service with the target service being the resource transfer service as an example, the first data query condition may include a time query condition (such as querying the resource transfer data of the target user in the past thirty days). The first data obtained by the server may include the resource transfer data of the target user in the past thirty days. The server can determine this first data as the target data and perform risk detection processing based on this target data. For example, the server can perform risk detection processing on the target data according to a preset risk detection model to obtain a risk detection result for this resource transfer service.

[0041] An embodiment of this specification provides a data processing method. By receiving a service processing request for triggering the execution of a target service for a target user, in response to the service processing request, obtaining a first data query condition corresponding to the service processing request and the user identifier of the target user, determining a target serial number corresponding to the user identifier of the target user in a preset data lake system, and based on the target serial number and the first data query condition, determining target file data in the data lake system based on the metadata index cache, sending a data reading task for the target file data to a data reading executor, and receiving the first data read by the data reading executor from the target file data according to the first data query condition and the target serial number, determining target data corresponding to the first data query condition according to the first data, and performing the target service according to the target data. In this way, on the one hand, various different formats of data can be stored in the data source system, and data corresponding to the service processing request can be quickly obtained through the data lake system. On the other hand, when searching for data, the target file data corresponding to the target user can be determined according to the metadata index cache, and then the data reading task corresponding to the target file data can be sent to the data reading executor for data reading processing to improve the data reading efficiency in the point query scenario and improve the subsequent service processing efficiency.

[0042] In practical applications, the specific processing method for obtaining the first data query condition corresponding to the service processing request in step S104 can be various. The following provides an optional processing method, such as Figure 3 shown, which may specifically include the processing steps S1042 to S1046 as follows.

[0043] In step S1042, obtain the data query statement corresponding to the service processing request.

[0044] Among them, the data query statement may include a detailed data query statement and / or an aggregated data query statement. For example, the detailed data query statement may be like "select form database where ID = 208801", that is, query the data with the serial number 208801 from the database. The aggregated data query statement may be a query statement with aggregation functions (such as a function for calculating the number of non-controls (count() function), a function for calculating the sum of a numerical column (sum() function), a function for calculating the average value of a numerical column (avg() function), a function for returning the maximum value ((max() function), a function for returning the minimum value (min() function), etc.). Specifically, the aggregated query statement may be "select count(number) form database", that is, query the total of the resource transfer volume (i.e., number) from the database.

[0045] In implementation, since the sql protocol can only process one query method (i.e., detailed data query or aggregated data query) in one query processing, therefore, in the case of a large number of data query methods, to avoid the problems of slow query efficiency caused by the need for sql parsing processing and multiple query processes in the sql protocol method, at the protocol layer, the data query statement can be determined by parameterization, that is, the data query statement may include a detailed data query statement and / or an aggregated data query statement. In this way, through the data query statement, detailed data query and aggregated data query processing can be performed simultaneously.

[0046] In step S1044, the parameters in the data query statement are subjected to a legality check.

[0047] In implementation, the server also needs to perform a legality check on the parameters in the data query statement at the protocol layer to avoid security problems such as leakage of user privacy data caused by illegal parameters in the data query statement during data query processing.

[0048] Among them, there are various ways to perform a legality check on the parameters in the data query statement. For example, the server can match the parameters in the data query statement according to a preset parameter blacklist and determine the legality check result based on the matching result. Or, the server can also perform a legality check on the data query statement according to a pre-trained parameter detection model to obtain the legality check result. Among them, the parameter detection model can be a model constructed according to a preset machine learning algorithm, etc. Different legality check methods can be selected according to different actual application scenarios, and this specification does not make specific limitations on this.

[0049] In step S1046, when the legality verification of the parameters in the data query statement is successful, the first data query condition is determined according to the parameters in the data query statement.

[0050] In implementation, when the legality verification of the parameters in the data query statement is successful, the server can convert the parameters in the data query statement (such as the attempt parameters) into physical parameters. For example, the server can convert the user representation of the target user in the data query statement into a target serial number. Specifically, the server can convert the cust_name field parameter in the data query statement into a column serial number that can be used by the Java SDK of the preset data lake system (such as Paimon).

[0051] In practical applications, a metadata index tree can be constructed based on the file data in the data lake system. The specific processing methods for constructing the metadata index tree can be diverse. The following provides an optional processing method, as Figure 4 shown, which can specifically include the processing of the following steps S402 to S406.

[0052] In step S402, the metadata of each file data in the data lake system is obtained, and based on the metadata, it is detected whether there is overlapping data between the file data.

[0053] In implementation, the metadata (meta) cache can be used to quickly locate the file block data. Since the file data sorted according to the serial number is not strictly sorted, there may be a situation where there is an interval intersection between a small amount of file data in the same data partition. Therefore, to improve the data query efficiency, before constructing the metadata index, it is necessary to first perform a merging process on the intervals, that is, the server can detect whether there is overlapping data between the file data based on the metadata.

[0054] In step S404, when it is detected that there is overlapping data between multiple file data, the multiple file data with overlapping data are merged to obtain the merged file data.

[0055] In implementation, for example, as Figure 5 shown, since there is overlapping data between file data 1 (including the data corresponding to serial numbers 208801 - 208809) and file data 2 (including the data corresponding to serial numbers 208806 - 208818), the server can merge file data 1 and file data 2 to obtain the merged file data (including the data corresponding to serial numbers 208801 - 208818).

[0056] In step S406, a metadata index tree is constructed based on the index information of the metadata corresponding to the merged file data.

[0057] In implementation, since the server performs data sorting processing at the partition dimension, the server can construct a metadata index tree for each partition.

[0058] As Figure 5 shown, the server can construct a mapping result (RangMap) and a static serialization based on the index information of the metadata corresponding to the merged file data to obtain the metadata index tree Among them, after the server performs file data merging processing, it can construct a metadata index tree based on the index information of the metadata corresponding to the merged file data. In practical applications, the specific processing method of determining the target file data in the data lake system based on the target sequence number and the first data query condition in step S106 above can be various. For example, Figure 6 shown, it can specifically include the processing of the following step S1062.

[0059] In step S1062, based on the target sequence number and the first data query condition, the target file data is determined based on the metadata index tree.

[0060] In implementation, each node of the metadata index tree can store a certain preset interval and the file address corresponding to the interval. In addition, the server can also store the metadata of the file data in an embedded database (such as rocksdb). In this way, when the server receives a target sequence number, it can quickly locate the target file data according to the target sequence number.

[0061] For example, as Figure 5 shown, taking the target sequence number 208801 as an example, it can be determined from the metadata index tree that the merged file data contains 208801. Then, the server can perform fallback filtering on the merged file data to obtain file data 1 and determine file data 1 as the target file data.

[0062] In practical applications, the data reading task includes the address information of the target file data after format conversion processing. The target file data after format conversion processing is the data obtained by converting the data format of the target file data from the external memory format to the in-memory format.

[0063] Taking the data lake system as the paimon system as an example, the server can use the paimon java sdk to read and filter the target file data corresponding to the target sequence number and the first data query condition from the file system (dfs or oss), and then send the data reading task corresponding to the target file data to the data reading executor (such as actor) for processing by different operators.

[0064] Among them, the server can convert the data format of the target file data read from the Paimon system from the InternalRow object (i.e., the external memory format) to the ColumnRow object (i.e., the in-memory format), and then send the address information (i.e., reference) of the target file data after format conversion processing to the mailbox of each downstream actor, thus avoiding waste of space and saving data processing resources.

[0065] At the same time, a bounded queue data structure can be used in the mailbox of the actor operator. In this way, when the downstream processing is not timely and the queue is full, it will block the upstream from reading data from the Paimon system, thus achieving an anti-backpressure effect.

[0066] In practical applications, the first data is the data obtained from the preset data lake system according to the address information of the target file data after format conversion processing when the data reading executor does not contain the text data corresponding to the data query condition and the target serial number in the local cache, and the data filtered from the target file data according to the data query condition and the target serial number.

[0067] In practical applications, the first data is the data read from the local cache when the local cache of the data reading executor meets the preset available conditions and the local cache contains the text data corresponding to the data query condition and the target serial number.

[0068] Among them, after the data reading executor finishes the data reading task, it can store the read data in the local cache. In this way, when the data reading executor executes the next data reading task, it can first judge whether the local cache contains the text data corresponding to the data query condition and the target serial number.

[0069] In addition, as Figure 7 shown, the Consistent Hashing algorithm can be used as the task routing strategy to try to ensure that the writing and reading processing of the same file data are both on the same server (i.e., the server to which the data reading executor belongs).

[0070] In practical applications, the preset available conditions include whether the first key value and the second key value corresponding to the local cache are null values. The first key value is used to represent the start of data writing processing in the local cache, and the second key value is used to represent the data statistical information after data writing.

[0071] In addition, as Figure 8As shown, to avoid the problem of incomplete cached data caused by abnormal situations such as insufficient disk space, when performing data writing processing, a first key value, start key, can be set before writing the data, and a second key value, statistic key, can be written after the writing process is completed. Among them, the start key can be used to represent the start of data writing processing in the local cache, and the statistic key can be used to represent the data statistics information after data writing. The data statistics information can include the number of rows of the written data, the maximum value of each column, the minimum value of each column, and null values, etc. In this way, when the first key value and the second key value are not null values (that is, both the first key value and the second key value exist), it can indicate that the cached data is complete and available. When any one of these two key values exists, it means that the current task has been stored and does not need to be cached again. In this way, as Figure 8 shown, since the consistent hashing algorithm is used to ensure that the writing and reading processing of the same file data are on the same server, when the data reading executor receives a data reading task for the target file data, it can first determine whether the local cache contains the text data corresponding to the data query condition and the target serial number. When the local cache contains the text data corresponding to the data query condition and the target serial number, and the first key value and the second key value corresponding to the local cache are not null values, the data reading executor can quickly read the first data according to the local cache, improving the data reading efficiency.

[0072] For example, assume that the target serial number is 208801, and the first query condition is to query the data from April 1, 2024 to May 1, 2024. If the local cache contains the data with the target serial number 208801 and from March 1, 2024 to June 1, 2024, then the data reading executor can directly read the first data corresponding to 208801 and the above time query condition according to the text data stored in the local cache.

[0073] In addition, if the local cache only contains data related to the target sequence number and the first query condition, then the data reading executor can read the first data from the local cache and the preset data lake system. For example, assuming that the local cache contains data with a target sequence number of 208801 and between March 1, 2024 and April 15, 2024, then the data reading executor can obtain the first data 1 from the local cache, that is, the target sequence number is 208801 and the data is between April 1, 2024 and April 15, 2024, and then obtain the target file data from the preset data lake system according to the address information of the target file data after format conversion in the preset data lake system, and filter out the first data 2 from the target file data according to the data query condition and the target sequence number, that is, the first data 2 is the target sequence number 208801 and the data is between April 16, 2024 and May 1, 2024. In this way, the data reading executor can aggregate the first data based on the first data 1 read from the local cache and the first data 2 read from the preset data lake system.

[0074] When the local cache does not contain text data corresponding to the data query condition and the target sequence number, the first data is read from the preset data lake system. In addition, after the data reading executor reads the first data from the preset data lake system, it can also store the read first data in the local cache. Figure 8 As shown, when the first data is written, the data reading executor also needs to store the corresponding first key value and second key value.

[0075] In actual applications, the server may also store target data, and upon receiving a data acquisition request, determine a data acquisition result based on the stored data. There are many specific processing methods for determining the data acquisition result. The following is an optional processing method, such as Figure 9 As shown, the processing may specifically include the following steps S902 to S910.

[0076] In step S902, the target data is stored in the result cache.

[0077] In implementation, Figure 7 As shown, after the first data is aggregated to obtain the target data, the server may store the target data in a result cache.

[0078] In step S904, a data acquisition request for a target user is received.

[0079] In step S906, in response to the data acquisition request, a second data query condition corresponding to the data acquisition request is determined, and it is determined whether there is a correlation between the first data query condition and the second data query condition.

[0080] Among them, the correlation relationship can be whether there is an overlapping interval. For example, when there are overlapping intervals such as an overlapping time interval, an overlapping location interval, and an overlapping data range interval between the first data query condition and the second data query condition, it can be determined that there is a correlation relationship between the first data query condition and the second data query condition. Specifically, for example, assuming that the first data query condition includes querying products launched from January 1st to January 15th, and the second data query condition includes querying products launched from January 1st to January 18th, since the first data query condition and the second data query condition include an overlapping time interval, that is, January 1st to January 15th, it can be determined that there is a correlation relationship between the first data query condition and the second data query condition.

[0081] In step S908, when there is a correlation relationship between the first data query condition and the second data query condition, obtain the target data from the result cache, and filter the target data according to the second query condition to obtain candidate data.

[0082] In implementation, the server can obtain the target data from the result cache when there is a correlation relationship between the first data query condition and the second data query condition. For example, taking the above example where the first data query condition includes querying products launched from January 1st to January 15th and the second data query condition includes querying products launched from January 1st to January 18th, the server can obtain the target data from the result cache, that is, the products launched from January 1st to January 15th, and filter the target data according to other data query conditions (such as data range query conditions, etc.) in the second data query condition except the time query condition to obtain candidate data.

[0083] In step S910, determine the data acquisition result corresponding to the data acquisition request according to the candidate data, and feedback the data acquisition result.

[0084] In implementation, the server can determine the data acquisition result according to the candidate data. For example, taking the above example where the first data query condition includes querying products launched from January 1st to January 15th and the second data query condition includes querying products launched from January 1st to January 18th, what the server obtains from the result cache is the data corresponding to the products launched from January 1st to January 15th. The server can also obtain the data corresponding to the products launched from January 16th to January 18th from the preset data lake system. In this way, the server can determine the data acquisition result corresponding to the data acquisition request according to the candidate data obtained from the result cache and the data obtained from the preset data lake system.

[0085] Such as Figure 10As described above, under the distributed big data query service framework, a business application can communicate with a remote big data query service cluster (sever cluster) by integrating a client of the big data query service. By adopting the local big data query service framework of the embodiments of this specification, while simplifying the overall system architecture, it can be integrated into the business application in the form of a lightweight middleware, and the resources of the application server itself can be used to complete the data query of the big data source. And through, such as Figure 10 As can be seen from the framework comparison shown, first, the architecture of the local big data query service framework of the embodiments of this specification is relatively simple, and the development and maintenance costs are relatively low. Second, the local big data query service framework of the embodiments of this specification is ready to use and has strong customizability. In addition, the local big data query service framework of the embodiments of this specification does not require additional data storage transfer, and the cost is relatively low.

[0086] The embodiments of this specification provide a data processing method. By receiving a business processing request for triggering the execution of a target business for a target user, in response to the business processing request, obtaining a first data query condition corresponding to the business processing request and the user identifier of the target user, determining a target serial number corresponding to the user identifier of the target user in a preset data lake system, and based on the target serial number and the first data query condition, determining target file data in the data lake system based on the metadata index cache, sending a data reading task for the target file data to a data reading executor, and receiving the first data read from the target file data by the data reading executor according to the first data query condition and the target serial number, determining target data corresponding to the first data query condition according to the first data, and executing the target business according to the target data. In this way, on the one hand, various different formats of data can be stored in the data source system, and the data corresponding to the business processing request can be quickly obtained through the data lake system. On the other hand, when performing data search, the target file data corresponding to the target user can be determined according to the metadata index cache, and then the data reading task corresponding to the target file data can be sent to the data reading executor for data reading processing to improve the data reading efficiency in the point query scenario and improve the subsequent business processing efficiency.

[0087] The above is the data processing method provided by the embodiments of this specification. Based on the same idea, the embodiments of this specification also provide a data processing device, as Figure 11 shown.

[0088] The data processing device includes: a request receiving module 1101, a data obtaining module 1102, a data query module 1103, a task sending module 1104, and a business execution module 1105, where: A request receiving module 1101, configured to receive a service processing request for triggering the execution of a target service for a target user; A data acquisition module 1102, configured to, in response to the service processing request, acquire a first data query condition corresponding to the service processing request and a user identifier of the target user; A data query module 1103, configured to determine a target serial number corresponding to the user identifier of the target user in a preset data lake system, and based on the target serial number and the first data query condition, determine target file data in the data lake system based on a metadata index cache; A task sending module 1104, configured to send a data reading task for the target file data to a data reading executor, and receive first data read by the data reading executor from the target file data according to the first data query condition and the target serial number; A service execution module 1105, configured to determine target data corresponding to the first data query condition according to the first data, and execute the target service according to the target data.

[0089] In an embodiment of the present specification, the data acquisition module 1102 is configured to: acquire a data query statement corresponding to the service processing request, where the data query statement includes a detail data query statement and / or an aggregated data query statement; perform a legality check on parameters in the data query statement; in a case where the legality check on the parameters in the data query statement is successful, determine the first data query condition according to the parameters in the data query statement.

[0090] In an embodiment of the present specification, the apparatus further includes: An overlap detection module, configured to acquire metadata of each file data in the data lake system, and detect whether there is overlapping data between the file data according to the metadata; A file merging module, configured to, in a case where it is detected that there is overlapping data between multiple file data, perform a merging process on the multiple file data with overlapping data to obtain merged file data; An index tree construction module, configured to construct a metadata index tree based on index information of metadata corresponding to the merged file data; The data query module 1103 is configured to: determine target file data based on the target serial number, the first data query condition, and the metadata index tree.

[0091] In the embodiments of the present specification, the data reading task includes the address information of the target file data after format conversion processing, and the target file data after format conversion processing is the data obtained by converting the data format of the target file data from an external memory format to an in-memory format.

[0092] In the embodiments of the present specification, the first data is the data obtained by the data reading executor from the preset data lake system according to the address information of the target file data after format conversion processing and filtering out the data from the target file data according to the data query condition and the target serial number when the local cache does not contain the text data corresponding to the data query condition and the target serial number.

[0093] In the embodiments of the present specification, the first data is the data read from the local cache when the local cache meets the preset available conditions and the local cache contains the text data corresponding to the data query condition and the target serial number.

[0094] In the embodiments of the present specification, the preset available conditions include whether the first key value and the second key value corresponding to the local cache are null values. The first key value is used to represent the start of data writing processing in the local cache, and the second key value is used to represent the data statistical information after data writing.

[0095] In the embodiments of the present specification, the device further includes: A data storage module, configured to store the target data in the result cache; A first receiving module, configured to receive a data acquisition request for the target user; A relationship judgment module, configured to determine a second data query condition corresponding to the data acquisition request in response to the data acquisition request, and judge whether there is a correlation between the first data query condition and the second data query condition; A data filtering module, configured to obtain the target data from the result cache and perform filtering processing on the target data according to the second query condition to obtain candidate data when there is a correlation between the first data query condition and the second data query condition; A result feedback module, configured to determine a data acquisition result corresponding to the data acquisition request according to the candidate data and feedback the data acquisition result.

[0096] An embodiment of this specification provides a data processing device. By receiving a service processing request for triggering the execution of a target service for a target user, in response to the service processing request, obtaining a first data query condition corresponding to the service processing request and the user identifier of the target user, determining a target serial number corresponding to the user identifier of the target user in a preset data lake system, and based on the target serial number and the first data query condition, determining target file data in the data lake system based on a metadata index cache, sending a data reading task for the target file data to a data reading executor, and receiving, from the data reading executor, first data read from the target file data according to the first data query condition and the target serial number, determining target data corresponding to the first data query condition according to the first data, and executing the target service according to the target data. In this way, on the one hand, various different formats of data can be stored in the data source system, and data corresponding to the service processing request can be quickly obtained through the data lake system. On the other hand, when searching for data, the target file data corresponding to the target user can be determined according to the metadata index cache, and then the data reading task corresponding to the target file data can be sent to the data reading executor for data reading processing to improve the data reading efficiency in the point query scenario and improve the subsequent service processing efficiency.

[0097] The above is the data processing device provided by the embodiment of this specification. Based on the same idea, the embodiment of this specification also provides a data processing device, as Figure 12 shown.

[0098] The data processing device may be a terminal device or a server provided in the above embodiment, etc.

[0099] The data processing device may vary greatly due to configuration or performance differences, and may include one or more processors 1201 and a memory 1202. One or more application programs or data may be stored in the memory 1202. Among them, the memory 1202 may be short-term storage or persistent storage. The application programs stored in the memory 1202 may include one or more modules (not shown in the figure), and each module may include a series of computer-executable instructions in the data processing device. Further, the processor 1201 may be set to communicate with the memory 1202 and execute a series of computer-executable instructions in the memory 1202 on the data processing device. The data processing device may also include one or more power supplies 1203, one or more wired or wireless network interfaces 1204, one or more input / output interfaces 1205, and one or more keyboards 1206.

[0100] Specifically, in this embodiment, the data processing device includes a memory and one or more programs. One or more of the programs are stored in the memory, and one or more of the programs may include one or more modules. Each module may include a series of computer-executable instructions in the data processing device and is configured to be executed by one or more processors. The one or more programs include the following computer-executable instructions for: Receiving a service processing request for triggering the execution of a target service for a target user; In response to the service processing request, obtaining a first data query condition corresponding to the service processing request and the user identifier of the target user; Determining a target serial number corresponding to the user identifier of the target user in a preset data lake system, and based on the target serial number and the first data query condition, determining target file data in the data lake system based on a metadata index cache; Sending a data reading task for the target file data to a data reading executor and receiving first data read by the data reading executor from the target file data according to the first data query condition and the target serial number; Determining target data corresponding to the first data query condition according to the first data, and executing the target service according to the target data.

[0101] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. The key point of each embodiment is to illustrate the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and reference can be made to the corresponding parts of the method embodiment for related parts.

[0102] An embodiment of this specification provides a data processing device. By receiving a service processing request for triggering the execution of a target service for a target user, in response to the service processing request, obtaining a first data query condition corresponding to the service processing request and the user identifier of the target user, determining a target serial number corresponding to the user identifier of the target user in a preset data lake system, and based on the target serial number and the first data query condition, determining target file data in the data lake system based on a metadata index cache, sending a data reading task for the target file data to a data reading executor, and receiving the first data read from the target file data by the data reading executor according to the first data query condition and the target serial number, determining target data corresponding to the first data query condition according to the first data, and executing the target service according to the target data. In this way, on the one hand, various different formats of data can be stored in the data source system, and data corresponding to the service processing request can be quickly obtained through the data lake system. On the other hand, when searching for data, the target file data corresponding to the target user can be determined according to the metadata index cache, and then the data reading task corresponding to the target file data can be sent to the data reading executor for data reading processing, so as to improve the data reading efficiency in the point query scenario and improve the subsequent service processing efficiency.

[0103] Further, based on the above Figures 1 to 10 , one or more embodiments of this specification also provide a storage medium for storing computer-executable instruction information. In a specific embodiment, the storage medium can be a USB flash drive, an optical disc, a hard disk, etc. When the computer-executable instruction information stored in the storage medium is executed by a processor, the following process can be implemented: Receiving a service processing request for triggering the execution of a target service for a target user; In response to the service processing request, obtaining a first data query condition corresponding to the service processing request and the user identifier of the target user; Determining a target serial number corresponding to the user identifier of the target user in a preset data lake system, and based on the target serial number and the first data query condition, determining target file data in the data lake system based on a metadata index cache; Sending a data reading task for the target file data to a data reading executor, and receiving the first data read from the target file data by the data reading executor according to the first data query condition and the target serial number; Determining target data corresponding to the first data query condition according to the first data, and executing the target service according to the target data.

[0104] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other, and the key point of each embodiment is to illustrate the differences from other embodiments. In particular, for the above-mentioned embodiment of a storage medium, since it is basically similar to the method embodiment, the description is relatively simple, and for the relevant parts, reference can be made to the partial description of the method embodiment.

[0105] An embodiment of this specification provides a storage medium. By receiving a service processing request for triggering the execution of a target service for a target user, in response to the service processing request, obtaining a first data query condition corresponding to the service processing request and the user identifier of the target user, determining a target serial number corresponding to the user identifier of the target user in a preset data lake system, and based on the target serial number and the first data query condition, determining target file data in the data lake system based on the metadata index cache, sending a data reading task for the target file data to a data reading executor, and receiving the first data read from the target file data by the data reading executor according to the first data query condition and the target serial number, determining target data corresponding to the first data query condition according to the first data, and executing the target service according to the target data. In this way, on the one hand, various different formats of data can be stored in the data source system, and the data corresponding to the service processing request can be quickly obtained through the data lake system. On the other hand, when performing data search, the target file data corresponding to the target user can be determined according to the metadata index cache, and then the data reading task corresponding to the target file data can be sent to the data reading executor for data reading processing to improve the data reading efficiency in the point query scenario and improve the subsequent service processing efficiency.

[0106] Furthermore, based on the above Figures 1 to 10 , one or more embodiments of this specification also provide a computer program product, including a computer program. When the computer program in this computer program product is executed by a processor, the following processes can be implemented: Receiving a service processing request for triggering the execution of a target service for a target user; In response to the service processing request, obtaining a first data query condition corresponding to the service processing request and the user identifier of the target user; Determining a target serial number corresponding to the user identifier of the target user in a preset data lake system, and based on the target serial number and the first data query condition, determining target file data in the data lake system based on the metadata index cache; Sending a data reading task for the target file data to a data reading executor, and receiving the first data read from the target file data by the data reading executor according to the first data query condition and the target serial number; Determine target data corresponding to the first data query condition according to the first data, and execute the target service according to the target data.

[0107] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the above-mentioned embodiment of a computer program product, since it is basically similar to the method embodiment, the description is relatively simple, and reference can be made to the relevant part of the method embodiment for the relevant content.

[0108] An embodiment of this specification provides a computer program product. By receiving a service processing request for triggering the execution of a target service for a target user, in response to the service processing request, obtaining a first data query condition corresponding to the service processing request and the user identifier of the target user, determining a target serial number corresponding to the user identifier of the target user in a preset data lake system, and based on the target serial number and the first data query condition, determining target file data in the data lake system based on a metadata index cache, sending a data reading task for the target file data to a data reading executor, and receiving the first data read from the target file data by the data reading executor according to the first data query condition and the target serial number, determining target data corresponding to the first data query condition according to the first data, and executing the target service according to the target data. In this way, on the one hand, various different formats of data can be stored in the data source system, and data corresponding to the service processing request can be quickly obtained through the data lake system. On the other hand, when performing data search, the target file data corresponding to the target user can be determined according to the metadata index cache, and then the data reading task corresponding to the target file data can be sent to the data reading executor for data reading processing, so as to improve the data reading efficiency in the point query scenario and improve the subsequent service processing efficiency.

[0109] The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the specific order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0110] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to circuit structures such as diodes, transistors, switches, etc.) or software improvements (improvements to method flows). However, with the development of technology, many method flow improvements today can be regarded as direct improvements to hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement to a method flow cannot be implemented using a hardware entity module. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is an integrated circuit whose logic function is determined by the user programming the device. Designers can program themselves to "integrate" a digital system onto a single PLD, without having to ask a chip manufacturer to design and fabricate a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly implemented using "logic compiler" software, which is similar to the software compilers used in program development and writing. The original code before compilation also has to be written in a specific programming language, which is called a Hardware Description Language (HDL), and there is not just one type of HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones currently are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be aware that by simply performing a little logical programming on the method flow using the above-mentioned several hardware description languages and programming it into an integrated circuit, it is easy to obtain the hardware circuit that implements the logical method flow.

[0111] The controller can be implemented in any suitable manner. For example, the controller can take the form of, for example, a microprocessor or a processor and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of the controller include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in the form of pure computer-readable program code, it is entirely possible to logically program the method steps to enable the controller to be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. to achieve the same function. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be regarded as the structures within the hardware component. Or even, the devices for implementing various functions can be regarded as either software modules for implementing the method or structures within the hardware component.

[0112] The systems, devices, modules, or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0113] For the convenience of description, when describing the above devices, they are described separately as various units according to their functions. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0114] Those skilled in the art should understand that the embodiments of this specification can be provided as a method, a system, or a computer program product. Therefore, one or more embodiments of this specification can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, one or more embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0115] Embodiments of the present specification are described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present specification. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable serial-parallel devices for fraud cases to generate a machine, such that the instructions executed by the processor of the computer or other programmable serial-parallel devices for fraud cases generate a means for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or a means for implementing the functions specified in one or more of the blocks

[0116] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable serial-parallel devices for fraud cases to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction means that implements the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or a means for implementing the functions specified in one or more of the blocks

[0117] These computer program instructions can also be loaded onto a computer or other programmable serial-parallel devices for fraud cases, such that a series of operating steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or a means for implementing the functions specified in one or more of the blocks

[0118] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory

[0119] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of computer-readable media

[0120] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0121] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0122] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, one or more embodiments of this specification may be in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Furthermore, one or more embodiments of this specification may be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0123] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0124] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and reference can be made to the corresponding part of the method embodiment for relevant details.

[0125] The above description is only for the embodiments of this specification and is not intended to limit this document. For those skilled in the art, various modifications and changes can be made to this specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included within the scope of the claims of this specification.

Claims

1. A data processing method, comprising: Receiving a service processing request for triggering the execution of a target service for a target user; In response to the service processing request, obtaining a first data query condition corresponding to the service processing request and a user identifier of the target user; Determining a target serial number corresponding to the user identifier of the target user in a preset data lake system, and based on the target serial number and the first data query condition, determining target file data in the data lake system based on a metadata index cache; Sending a data reading task for the target file data to a data reading executor, and receiving first data read by the data reading executor from the target file data according to the first data query condition and the target serial number; Determining target data corresponding to the first data query condition according to the first data, and executing the target service according to the target data.

2. The method according to claim 1, wherein the obtaining of the first data query condition corresponding to the service processing request comprises: Obtaining a data query statement corresponding to the service processing request, where the data query statement includes a detailed data query statement and / or an aggregated data query statement; Performing a legality check on parameters in the data query statement; When the legality check on the parameters in the data query statement is successful, determining the first data query condition according to the parameters in the data query statement.

3. The method according to claim 2, before determining the target file data in the data lake system based on the target serial number and the first data query condition based on a metadata index cache, further comprising: Obtaining metadata of each file data in the data lake system, and detecting whether there is overlapping data between the file data according to the metadata; When it is detected that there is overlapping data between multiple file data, performing a merging process on the multiple file data with overlapping data to obtain merged file data; Constructing a metadata index tree based on index information of the metadata corresponding to the merged file data; The determining the target file data in the data lake system based on the target serial number and the first data query condition based on a metadata index cache includes: Determining the target file data based on the target serial number and the first data query condition based on the metadata index tree.

4. The method according to claim 1, where the data reading task includes address information of the target file data after format conversion processing, and the target file data after format conversion processing is data obtained by converting the data format of the target file data from an external memory format to an in-memory format.

5. The method according to claim 4, where the first data is data obtained by the data reading executor from the preset data lake system according to the address information of the target file data after format conversion processing and filtering out data from the target file data according to the data query condition and the target serial number when the local cache of the data reading executor does not contain text data corresponding to the data query condition and the target serial number.

6. The method according to claim 1, wherein the first data is data read from the local cache by the data reading executor when the local cache meets a preset available condition and the local cache contains text data corresponding to the data query condition and the target serial number.

7. The method according to claim 6, wherein the preset available condition includes whether the first key value and the second key value corresponding to the local cache are null values, the first key value is used to represent the start of data writing processing in the local cache, and the second key value is used to represent the data statistical information after data writing.

8. The method according to claim 1, further comprising: Storing the target data in a result cache; Receiving a data acquisition request for the target user; In response to the data acquisition request, determining a second data query condition corresponding to the data acquisition request, and determining whether there is a correlation between the first data query condition and the second data query condition; When there is a correlation between the first data query condition and the second data query condition, acquiring the target data from the result cache, and filtering the target data according to the second query condition to obtain candidate data; Determining a data acquisition result corresponding to the data acquisition request according to the candidate data, and feeding back the data acquisition result.

9. A data processing apparatus, comprising: A request receiving module, configured to receive a service processing request for triggering and executing a target service for a target user; A data acquisition module, configured to, in response to the service processing request, acquire a first data query condition corresponding to the service processing request and a user identifier of the target user; A data query module, configured to determine a target serial number corresponding to the user identifier of the target user in a preset data lake system, and determine target file data in the data lake system based on the metadata index cache according to the target serial number; A task sending module, configured to send a data reading task for the target file data to a data reading executor, and receive the first data read from the target file data by the data reading executor according to the first data query condition and the target serial number; A service execution module, configured to determine target data corresponding to the first data query condition according to the first data, and execute the target service according to the target data.

10. A data processing device, the data processing device comprising: A processor; And A memory arranged to store computer-executable instructions that, when executed, cause the processor to: Receive a service processing request for triggering and executing a target service for a target user; In response to the service processing request, acquire a first data query condition corresponding to the service processing request and a user identifier of the target user; Determine a target serial number corresponding to the user identifier of the target user in a preset data lake system, and determine target file data in the data lake system based on the metadata index cache according to the target serial number; Send the data reading task for the target file data to the data reading executor, and receive the first data read by the data reading executor from the target file data according to the first data query condition and the target serial number; Determine the target data corresponding to the first data query condition according to the first data, and execute the target service according to the target data.

Images