Data processing method, electronic device, storage medium and program product

Abstract

The embodiment of the invention discloses a data processing method, electronic equipment, a storage medium and a program product. The data processing method comprises the following steps: in response to a data query request for target data, determining data state information of a data file corresponding to the target data; in response to the condition that the data state information indicates that the data file meets a preset change condition, determining an index efficiency index of the data file; the index efficiency index is used for representing the efficiency improvement degree of querying the data file by using an index filtering mode; generating a query strategy of the target data based on the index efficiency index; the query strategy comprises a data query mode for the data file; the data query mode comprises an index filtering mode or a scanning mode; and performing data query on the data file based on the query strategy to obtain a target query result of the target data. According to the method, the data query efficiency in a streaming and batch integrated scene can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, electronic equipment, storage medium and program product. Background Art

[0002] In the stream-batch integrated data processing framework, data is continuously input in the form of a stream, while supporting batch processing tasks. In this mode, data is usually generated in real time, stored after processing, and an index of the data file is generated to quickly respond to user queries in stream mode or batch mode. However, in scenarios where data changes frequently, frequent data updates will cause data files and index structures to change constantly, easily generating a large number of small files, and seriously affecting data query performance. Therefore, it is urgent to provide a data query method that can be applied to stream-batch integrated scenarios. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a data processing method, an electronic device, a storage medium and a program product to solve the technical problem of decreased data query efficiency caused by frequent data updates in a batch-stream integrated scenario.

[0004] To solve the above technical problems, the embodiments of the present application are implemented as follows: On the one hand, an embodiment of the present application provides a data processing method, including: In response to a data query request for target data, determining data status information of a data file corresponding to the target data; In response to the data status information indicating that the data file meets the preset change condition, determining an index efficiency index of the data file; the index efficiency index is used to characterize the degree of efficiency improvement of querying the data file using an index filtering method; the index filtering method is to filter the data file based on the first index information of the data file; Based on the index efficiency index, a query strategy for the target data is generated; the query strategy includes a data query method for the data file; the data query method includes the index filtering method or the scanning method; The data file is queried based on the query strategy to obtain a target query result of the target data.

[0005] On the other hand, an embodiment of the present application provides a data processing device, including: A first determination module, configured to determine data status information of a data file corresponding to the target data in response to a data query request for the target data; A second determination module is configured to determine an index efficiency index of the data file in response to the data status information indicating that the data file meets a preset change condition; the index efficiency index is used to characterize the degree of efficiency improvement of querying the data file using an index filtering method; the index filtering method is to filter the data file based on the first index information of the data file; A generation module, configured to generate a query strategy for the target data based on the index efficiency index; the query strategy includes a data query method for the data file; the data query method includes the index filtering method or the scanning method; The query module is used to perform data query on the data file based on the query strategy to obtain the target query result of the target data.

[0006] On the other hand, an embodiment of the present application provides an electronic device, including a processor and a memory electrically connected to the processor, the memory storing a computer program, and the processor being used to call and execute the computer program from the memory to implement the above-mentioned data processing method.

[0007] On the other hand, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program can be executed by a processor to implement the above-mentioned data processing method.

[0008] On the other hand, an embodiment of the present application provides a computer program product, including a computer program, wherein the computer program is executed by a processor to implement the above-mentioned data processing method.

[0009] According to the technical solution of the embodiment of the present application, after receiving a data query request for target data, by determining the data status information of the data file corresponding to the target data, in response to the data status information indicating that the data file meets the preset change condition, the index efficiency index of the data file is determined, and the index efficiency index is used to characterize the efficiency improvement degree of querying the data file using the index filtering method, and the index filtering method is to filter the data file based on the first index information of the data file. Based on the index efficiency index, a query strategy for the target data is generated, and the query strategy includes a data query method for the data file, and the data query method includes an index filtering method or a scanning method. Based on the query strategy, a data query is performed on the data file to obtain a target query result of the target data. It can be seen that the data query strategy is dynamically adjusted by real-time perception of the data status information, so that a targeted data query method can be adopted to query data in a rapidly changing data environment, ensuring that the index information can be effectively used in a rapidly changing data environment, and improving the data query efficiency and resource utilization. In addition, by flexibly selecting to use the index filtering method or the scanning method, the fragmentation of the index information can be reduced, the maintenance cost of the index information can be reduced, and the data query performance can be ensured to be continuously optimized as the data environment changes, thereby improving the overall stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate one or more embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0011] Figure 1 is a schematic flow chart of a data processing method according to an embodiment of the present application; Figure 2 is a schematic block diagram of a data processing system according to an embodiment of the present application; Figure 3 is a schematic flow chart of a data processing method according to another embodiment of the present application; Figure 4 is a schematic principle diagram of a data change situation of a data file according to an embodiment of the present application; Figure 5 is a schematic principle diagram of a data processing method according to an embodiment of the present application; Figure 6 is a schematic principle diagram of a data processing method according to another embodiment of the present application; Figure 7is a schematic principle diagram of a data processing method according to yet another embodiment of the present application; Figure 8 is a schematic block diagram of a data processing device according to an embodiment of the present application; Fig. 9 It is a schematic block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0012] The embodiments of the present application provide a data processing method, an electronic device, a storage medium, and a program product to solve the technical problem of decreased data query efficiency caused by frequent data updates in a stream-batch integrated scenario.

[0013] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0014] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein.

[0015] The data processing method provided in the embodiments of the present application may be executed by an electronic device or by software installed in the electronic device. Specifically, the electronic device may be a terminal device or a server device. The terminal device may include a smart phone, a laptop computer, a smart wearable device, a vehicle terminal, etc. The server device may include an independent physical server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing.

[0016] Figure 1 is a schematic flow chart of a data processing method according to an embodiment of the present application, such as Figure 1 As shown, the method includes the following steps S102~S108: Step S102: In response to a data query request for target data, determine data status information of a data file corresponding to the target data. The data status information includes at least one of the following: the amount of changed data in the data file, the amount of source data in the data file, and the index validity. The changed data includes additional data and / or deleted record data. The additional data is the newly added data, and the deleted record data is the record of the data deleted in the data file.

[0017] Index validity is used to characterize the validity of the index information of the data file. The higher the index validity, the more valid the index information is and the smaller the amount of changed data is. Conversely, the lower the index validity, the older (invalid) the index information is and the larger the amount of changed data is. The obsolescence of index information is caused by changed data. The more changed data is, the more obsolescent the index information will be, which in turn leads to lower accuracy in querying data based on index information.

[0018] Step S104 , in response to the data status information indicating that the data file meets the preset change condition, determining an index efficiency index of the data file.

[0019] Among them, the index efficiency index is used to characterize the efficiency improvement of using the index filtering method to query data files. It can also be understood that the index efficiency index is used to measure the comprehensive efficiency ratio of the two data query methods, index filtering and scanning, in terms of time and resource consumption under specific query conditions.

[0020] Optionally, before executing step S104, it is first determined whether the data file has first index information. In response to determining that the data file corresponds to the first index information, an index efficiency index of the data file is determined.

[0021] Step S106: Generate a query strategy for target data based on the index efficiency indicator; the query strategy includes a data query method for the data file.

[0022] The data query method includes an index filtering method or a scanning method. The scanning method is a method of scanning the data file in full, and the index filtering method is a method of filtering the data file based on the first index information of the data file. Since the change data usually has no index information, filtering the data file is essentially filtering the source data in the data file.

[0023] Optionally, the index information is represented as a hash table. When the index filtering method is used, the index information is retrieved according to the query conditions, that is, the hash table is queried to determine whether there is data in the hash table that meets the query conditions. If there is data in the hash table that meets the query conditions, the source data cannot be skipped and the source data continues to be scanned in a scanning manner. If there is no data in the hash table that meets the query conditions, it is successfully skipped, indicating that the source data does not contain the target data to be queried, and there is no need to scan the source data again. The hash table includes the hash value corresponding to each piece of data in the source data.

[0024] Since the amount of data in a hash value is much smaller than that in the actual data value, retrieving the hash table through index information can quickly determine whether the source data contains the target data to be queried, thereby quickly deciding whether to skip the source data, that is, whether the source data needs to be scanned, thereby reducing the scanning scope of the data file.

[0025] Step S108: perform a data query on the data file based on the query strategy to obtain a target query result of the target data.

[0026] The data processing method provided in this embodiment can be applied to data storage devices such as databases and data warehouses, or scenarios such as big data processing. As long as it is a scenario involving data query services, the data processing method provided in this application can be used to perform data query.

[0027] According to the technical solution of the embodiment of the present application, after receiving a data query request for target data, by determining the data status information of the data file corresponding to the target data, in response to the data status information indicating that the data file meets the preset change condition, the index efficiency index of the data file is determined, and the index efficiency index is used to characterize the efficiency improvement degree of querying the data file using the index filtering method, and the index filtering method is to filter the data file based on the first index information of the data file. Based on the index efficiency index, a query strategy for the target data is generated, and the query strategy includes a data query method for the data file, and the data query method includes an index filtering method or a scanning method. Based on the query strategy, a data query is performed on the data file to obtain a target query result of the target data. It can be seen that the data query strategy is dynamically adjusted by real-time perception of the data status information, so that a targeted data query method can be adopted to query data in a rapidly changing data environment, ensuring that the index information can be effectively used in a rapidly changing data environment, and improving the data query efficiency and resource utilization. In addition, by flexibly selecting to use the index filtering method or the scanning method, the fragmentation of the index information can be reduced, the maintenance cost of the index information can be reduced, and the data query performance can be ensured to be continuously optimized as the data environment changes, thereby improving the overall stability of the system.

[0028] In one embodiment, there are multiple ways to determine whether a data file meets the preset change condition. Two determination methods are exemplarily described below.

[0029] Mode 1: The data status information includes the amount of changed data in the data file. In response to the amount of changed data being less than or equal to a first preset threshold, it is determined that the data file meets the preset change condition.

[0030] Mode 2: The data status information includes the data volume of the source data, the data volume of the changed data and the index validity in the data file. In response to the index validity being greater than or equal to the second preset threshold, it is determined that the data file meets the preset change condition.

[0031] In the second method, the index validity can be calculated according to the data volume of the source data and the data volume of the changed data in the data file. The index validity is positively correlated with the data volume of the source data and negatively correlated with the total data volume, and the total data volume is the sum of the data volume of the source data and the data volume of the changed data.

[0032] Optionally, the calculation method of index validity can be represented by the following formula:

[0033] The changed data includes additional data and deleted record data. Indicates the amount of source data in the data file. Indicates the amount of deleted record data in the data file. Represents the amount of additional data in the data file.

[0034] In one embodiment, when determining the index efficiency index of a data file, the index parameter information of the data file may be determined first according to the data state information, and then the index efficiency index of the data file may be determined according to the index parameter information.

[0035] Among them, the index parameter information includes at least one of the following: scanning time of source data in the data file, scanning time of changed data in the data file, scanning time of the first index information, expiration time of the first index information, update frequency of the first index information, number of updates of the data file, and index skip probability.

[0036] Optionally, the index efficiency index includes: an efficiency ratio between querying the data file using an index filtering method and querying the data file using a scanning method. That is, the index efficiency index is characterized by using an efficiency ratio.

[0037] The calculation method of efficiency ratio can be expressed as the following formula:

[0038] in, Indicates the scanning time of the source data in the data file, that is, the time required to scan the source data. Indicates the scanning time of the deleted record data in the data file, that is, the time required to scan the deleted record data. Indicates the scanning time of the additional data in the data file, that is, the time required to scan the additional data. Indicates the scanning duration of the first index information, that is, the duration required to scan the first index information. The index skip probability representing the first index information may be updated in real time through performance optimization. Indicates the expiration time of the first index information, that is, the difference between the current time and the generation time of the first index information. , and It is an empirical coefficient used to identify the expiration speed of index information in different scenarios, reflecting the frequency of data changes in data files. The larger the number, the faster the expiration speed and the higher the frequency of data changes in data files. For example, , and They are configured as 1.294, 0.002, and 2 respectively. In scenarios with frequent query requests, in order to keep the index information up to date and increase the probability of successful skipping during query, you can increase and For example, these two values ​​are configured as 3 and 0.01. On the contrary, in a scenario with fewer query requests, since the expiration rate of index information is low and the update of index information has a certain cost, the update frequency of index information can be reduced. and For example, configure these two values ​​to 0.7 and 0.0001.

[0039] Optionally, the calculation method of the efficiency ratio can also be expressed as the following formula:

[0040] in, Indicates the number of times the data file is updated. The meanings of the remaining symbols are the same as those in the previous formula and will not be repeated here.

[0041] When generating a query strategy for target data based on the index efficiency index, in response to the efficiency ratio being higher than a preset efficiency threshold, the data query mode is determined to be an index filtering mode. In response to the efficiency ratio being lower than or equal to the preset efficiency threshold, the data query mode is determined to be a scanning mode.

[0042] If the efficiency ratio is higher than the preset efficiency threshold, it means that the efficiency of using index filtering to query data is high, and the index filtering method can be used to query data. If the efficiency ratio is lower than or equal to the preset efficiency threshold, it means that the efficiency of using index filtering to query data is low, and the scanning method can be used to query data.

[0043] Optionally, when the index parameter information includes the index skip probability, the index skip probability may be updated. The update method may be as follows: First, at least two of the first number of scan skip operations performed on the data file, the second number of scan operations performed on the data file, and the total number of queries on the data file within a preset time period are counted to obtain a statistical result.

[0044] Secondly, based on the statistical results, the index skip probability of the data file is updated.

[0045] In this embodiment, the index efficiency index is characterized by an efficiency ratio, and the data query method is determined to be an index filtering method or a scanning method based on the relationship between the efficiency ratio and a preset efficiency threshold, so that the decision on the data query method is closely related to the efficiency of the data query method, thereby determining the data query method more accurately and efficiently and improving the data query efficiency.

[0046] In one embodiment, a data query is performed on a data file based on the determined query strategy to obtain a target query result of the target data, which can be performed in the following manner: In response to the data query mode being an index filtering mode, the first index information is matched with a query condition of the target data to obtain a matching result. The query condition is carried in the data query request.

[0047] In response to the matching result that the first index information does not match the query condition, a scan skip operation is performed on the data file; or, in response to the matching result that the first index information matches the query condition, a scan operation is performed on the data file to obtain a target query result based on the scan result.

[0048] Among them, the scan skip method for data files means that the source data of the data file does not contain data that matches the query conditions (i.e., target data). In this case, the source data can be skipped and there is no need to scan the source data again. Instead, only the changed data needs to be scanned. It can be seen that the use of index filtering can skip the source data in the data file with a certain probability, greatly reducing the workload of data scanning and improving data query efficiency.

[0049] In one embodiment, data query is performed on the data file based on the determined query strategy to obtain the target query result of the target data, which can be performed in the following manner: First, the source data in the data file is queried based on the determined data query method to obtain a first query result, and the changed data in the data file is scanned based on the scanning method to obtain a second query result of the changed data. Secondly, the target query result is determined based on the first query result and the second query result.

[0050] Optionally, when determining the target query result based on the first query result and the second query result, in response to the change data including the additional data, the first query result and the second query result of the additional data are integrated to obtain the target query result. In response to the change data including the deleted record data, the deleted data corresponding to the deleted record data is deleted from the first query result to obtain the target query result.

[0051] In this embodiment, when querying the source data, the query is performed based on the data query method in the determined query strategy. When querying the changed data, since the changed data usually does not have corresponding index information, there is no need to change the query strategy, and the changed data can be queried in a scanning manner. In addition, the amount of changed data is usually much smaller than the amount of source data. Therefore, even if the scanning method is used to query the changed data, the impact on the overall data query efficiency is relatively small. The two data query methods are flexibly used to query the data files, so that the index information can be effectively utilized in a rapidly changing data environment, thereby improving data query efficiency and resource utilization. In addition, by integrating the first query result and the second query result, the accuracy of the query results can be ensured on the basis of flexible and efficient data query.

[0052] In one embodiment, after a query strategy for target data is generated based on an index efficiency index, if the data query method included in the query strategy is a scanning method, it means that the current index information has a lag, for example, the expiration time (i.e., the difference between the current time and the generation time of the index information) is long, or the index efficiency index (i.e., the efficiency improvement of querying data files using an index filtering method) is low, etc. In this case, the current index information can be updated.

[0053] Optionally, the updating method is: in response to the data query method being a scanning method, generating second index information of the data file, and updating the first index information to the second index information, that is, regenerating new index information of the data file. By timely updating the index information of the data file, the next time a data query request for the data file is received, the new index information can be used to determine the query strategy. In addition, due to the timely update of the index information, the new index information can filter the data file more effectively, that is, the probability of index filtering is higher, so the probability of determining the query strategy as index filtering next time is higher, thereby improving data query efficiency and resource utilization of index information.

[0054] Figure 2 is a schematic block diagram of a data processing system according to an embodiment of the present application, such as Figure 2As shown in FIG, the application scenario includes two main layers: the computing layer and the storage layer. The computing layer and the storage layer communicate with each other through a network connection. Figure 2 The data processing system shown is used to perform Figure 1 The data processing method of the illustrated embodiment. This embodiment is not limited to the operating environment of the data processing system, for example, it can be a physical machine environment, a cloud environment, etc.

[0055] The storage layer is mainly responsible for data storage, including data files, index information, and deletion record data. Deletion record data is the record of data deleted from the data file. The storage layer can store data in a variety of formats, such as Parquet, Avro, or ORC (Optimized Row Columnar). In addition, the storage layer can store data in different storage methods, such as local storage, distributed storage, or cloud storage.

[0056] The computing layer is mainly responsible for data management and access. The computing layer is deployed with a computing engine, which is a tool for processing large-scale data, which can achieve efficient data processing and analysis. This embodiment does not limit the type of computing engine. For example, it can be a computing engine such as Apache Spark, Apache Flink or Ray. Different computing engines play an important role in different scenarios and applications, providing rich data processing and analysis capabilities. Depending on the application scenario, the deployment of the computing engine may include but is not limited to a big data operating environment, a data warehouse operating environment, or a lake warehouse integrated operating environment, etc. The operating environment includes but is not limited to a physical machine environment, a cloud environment, etc.

[0057] based on Figure 2 In the data processing system shown, when executing the data processing method, the data query request is first transmitted to the computing layer, and the computing layer determines the data status information of the data file corresponding to the target data based on the data query request, and determines the index efficiency index of the data file when the data status information indicates that the data file meets the preset change conditions; generates a query strategy for the target data based on the index efficiency index, and then queries the data from the storage layer based on the query strategy.

[0058] The specific implementation of the data processing method is described in detail below.

[0059] Figure 3 is a schematic flow chart of a data processing method according to another embodiment of the present application, such as Figure 3 As shown, the method includes the following steps S301~S311: Step S301, in response to a data query request for target data, determining query conditions for the target data, and acquiring data table information corresponding to the target data.

[0060] The data query request includes or carries the name of the data table and the query condition. The corresponding data table information can be obtained according to the name of the data table. The data table information is stored in the metadata management system of the data processing system. The metadata management system can be a subsystem in the data processing system or an external storage system connected to the data processing system through a network. The data table information includes at least one of the following: the structure of the data table, the field type, the file name of the data file, the storage path of the data file, etc.

[0061] The query condition can be represented as a query predicate, which includes the data dimension and data value of the data to be queried. For example, a data query request is represented as a query SQL (Structured Query Language) statement. ”, where the query predicate “ " is the query condition, " " is the data dimension, and "10" is the data value.

[0062] Step S302: determining data status information of a data file corresponding to the target data according to the data table information, wherein the data status information includes index validity of the index information.

[0063] The data table information includes the file name of the data file. Therefore, all the data files involved in this query can be determined based on the data table information.

[0064] In addition to index validity, data status information also includes at least one of the following information: the amount of changed data in the data file, the amount of source data in the data file, the update frequency of the data file, etc. The changed data includes additional data and / or deleted record data. Additional data refers to newly added data, and deleted record data refers to the record of deleted data in the data file.

[0065] The index validity is used to characterize the validity of the index information of the data file. The higher the index validity, the more valid the index information is and the smaller the amount of changed data is. Conversely, the lower the index validity, the older (invalid) the index information is and the larger the amount of changed data is. The obsolescence of the index information is caused by the changed data. The more changed data is, the more obsolescent the index information is, and the lower the accuracy of querying data based on the index information is. The calculation method of the index validity has been described in detail in the above embodiment and will not be repeated here.

[0066] According to the data table information, it can be determined whether the data file has index information. If the data file has index information, continue to perform subsequent steps. If the data file does not have index information, the target data can be directly queried in a scanning manner.

[0067] Step S303: when the index validity is greater than or equal to a preset threshold, determine the index efficiency ratio of the data file.

[0068] If the index validity is greater than or equal to the preset threshold, it means that the data file meets the preset change condition. If the index validity is less than the preset threshold, it means that the index information is relatively old, that is, the validity is low. At this time, there is no need to determine the index strategy, and the target data can be directly queried by scanning.

[0069] The index efficiency ratio refers to the efficiency ratio between querying a data file using an index filtering method and querying a data file using a scanning method. The calculation method of the index efficiency ratio has been described in detail in the above embodiment and will not be repeated here.

[0070] Step S304: generating a query strategy for target data based on the index efficiency ratio, wherein the query strategy includes a data query method for the data file.

[0071] The data query mode includes an index filtering mode or a scanning mode. If the data query mode is the index filtering mode, then steps S305 and S309 are executed; if the data query mode is the scanning mode, then step S311 is executed.

[0072] Index filtering refers to filtering data files using the index information of data files. Optionally, the index information is represented as a hash table, and the index information is retrieved according to the query conditions, that is, the hash table is queried to determine whether there is data in the hash table that meets the query conditions. If there is data in the hash table that meets the query conditions, the source data cannot be skipped, and the source data continues to be scanned using the scanning method. If there is no data in the hash table that meets the query conditions, it is successfully skipped, indicating that the source data does not contain the target data to be queried, and there is no need to scan the source data again. The hash table includes the hash value corresponding to each piece of data in the source data.

[0073] For example, the query condition is " ". Assume that the hash table corresponding to the source data is retrieved according to the index information. If the hash table does not contain data with a value of 10, the source data is skipped, that is, there is no need to scan the source data again, and only the changed data needs to be scanned. If the hash table contains data with a value of 10, the source data cannot be skipped, and the data file needs to be fully scanned to query the target data. It can be seen that the use of index filtering can skip the source data with a certain probability, greatly reducing the workload of data scanning and improving data query efficiency.

[0074] Step S305: Match the index information of the source data in the data file with the query condition according to the index filtering method. Step S306: determine whether the source data contains data matching the query condition according to the index information. If yes, execute step S307; if no, execute step S308.

[0075] Step S307: Scan the source data based on the scanning method to obtain a first query result.

[0076] The first query result is the data that matches the query condition.

[0077] Step S308: successfully skipping the source data, and no need to scan the source data.

[0078] When the source data is successfully skipped, it means that the source data does not contain the target data to be queried, and the query result of the changed data can be determined as the target query result.

[0079] Step S309: Scan the changed data in the data file based on the scanning method to obtain a second query result of the changed data.

[0080] Step S310: Obtain a target query result according to the first query result and the second query result.

[0081] If the changed data includes additional data, the first query result and the second query result of the additional data are integrated to obtain the target query result. If the changed data includes deleted record data, the deleted data corresponding to the deleted record data is deleted from the first query result to obtain the target query result, thereby ensuring that the deleted data does not appear in the target query result.

[0082] Step S311, scanning the source data and the changed data in the data file based on the scanning method to obtain the target query result.

[0083] In this embodiment, after executing step S304, ie generating a query strategy, the initial query strategy of the data processing system may be optimized according to the query strategy. After optimizing the query strategy, the data query process is executed according to the optimized query strategy.

[0084] It can be seen that by adopting the data processing method provided by the embodiment of the present application, the data query strategy is dynamically adjusted by real-time perception of data status information, so that a targeted data query method can be adopted to query data in a rapidly changing data environment, ensuring that index information can be effectively utilized in a rapidly changing data environment, and improving data query efficiency and resource utilization. In addition, by flexibly choosing to use index filtering or scanning methods, the fragmentation of index information can be reduced, the maintenance cost of index information can be reduced, and it can ensure that data query performance is continuously optimized as the data environment changes, thereby improving the overall stability of the system.

[0085] In the integrated stream and batch processing scenario, taking Flink writing data to the Iceberg table as an example, it supports continuous write jobs and user jobs updating the same data table. During the execution process, it is assumed that there are jobs such as continuous stream writes to generate indexes and perform incremental refreshes, and there are also jobs that use batch mode (such as upsert) to update data. Index information is built for the data files generated by the continuous stream write job to achieve data skipping. Assuming that the amount of data for batch mode upsert is small, index information can be omitted, and additional data files and deleted data files (that is, record data is deleted) can be directly generated to achieve fast refresh, thereby ensuring instant feedback of user operations. Query operations can be performed at any time, using the latest index information and / or change data to ensure the accuracy and response speed of data queries.

[0086] In the data processing system, data can be stored in a mixed form in various ways. For example, in the same data file, source data, additional data, and deleted record data are stored at the same time, and each source data corresponds to an index information. Based on this, the source data can be indexed and filtered through the index information. For the additional data and deleted record data that do not have index information, the scanning method can be used directly to scan them, and then the query results of the source data, additional data, and deleted record data can be integrated.

[0087] Figure 4 is a schematic diagram of a data change situation of a data file according to an embodiment of the present application. Figure 4As shown, for a certain data table, at time T1, data file 1 "{1,2,3,4,5,6}" and its corresponding index 1 "BF({1,2,3,4,5,6})" are written. At time T2, the attachment data file 2 "{10,18}" of data file 1 is written. At time T3, the data "{5,6}" in data file 1 is deleted. At time T4, data file 3 "{7,8,9,11,12}" and its corresponding index 3 "BF({7,8,9,11,12})" are written. At time T5, the system compresses the data file to obtain data file 1 "{1,2,3,4,10,18}" and its corresponding index 1 "BF({1,2,3,4,10,18})".

[0088] based on Figure 4 The following are several data query methods at different times. In the following data query methods, the query statement " " is taken as an example, where the query predicate " " is the query condition.

[0089] Figure 5 The schematic diagram of data query at time T1 is shown. The preset efficiency threshold is 1. Assuming that at time T1, the calculated index efficiency ratio is greater than the preset efficiency threshold 1, the index filtering method is adopted, and the data file 1 is dynamically filtered based on index 1 to determine whether the data file 1 needs to be scanned. Based on the query condition " " and index 1 can determine that data file 1 does not contain data matching the query condition, so data file 1 can be skipped. Figure 5 In the example, the node "Join" means connection, which means integration. Here, it means integrating the query results of different data tables. "T.data" means the data query results of other data tables. The node "Scan" means scanning. The "dynamic filtering" under the node "Scan" means first using the index filtering method to dynamically filter data file 1, and then deciding whether to scan data file 1.

[0090] Assuming that the index efficiency ratio at a certain moment is less than 1, the data file and related change data (including additional data and / or deleted record data) will be directly scanned. In addition, when the index efficiency ratio is less than 1, it means that the index information is relatively old, for example, the amount of changed data is large, or the index information is out of date for a long time, so new index information can be constructed to replace the old index information.

[0091] For the additional data file 2 without index information, its corresponding index efficiency ratio is always a fixed value of 1, and the additional data file 2 is directly scanned using the scanning method. Figure 4The deleted data file 1) shown in the figure will scan the deleted record data at the same time to ensure that the target query data finally returned does not contain the deleted data.

[0092] Figure 6 A schematic diagram of data query at time T3 is shown. At time T3, the data table contains: data file 1 with index information, additional data file 2 without index information, and deleted data file 1. Assuming that the index efficiency ratio calculated at this time is greater than the preset efficiency threshold 1, data file 1 is dynamically filtered based on index 1 to obtain the first query result. At the same time, additional data file 2 and deleted data file 1 are directly scanned to obtain the second query result. Afterwards, the first query result and the second query result are integrated to obtain the target query result.

[0093] exist Figure 6 In the example, the meanings of the nodes "Join" and "Scan" are the same as those in Figure 5 The same, no repetition here. Figure 6 It can be seen that after dynamically filtering data file 1 based on index 1, data file 1 is successfully skipped, and there is no need to scan data file 1 again. The first query result of data file 1 is empty.

[0094] Since the data table may compress data periodically, for example, at time T5, after compressing data file 1, appended data file 2, and deleted data file 1, data file 1' is generated and index 1' is regenerated. That is, at time T5, the data table contains: data file 1', index 1', data file 3, and index 3. Therefore, when querying data at time T5, if the index efficiency ratio calculated at this time is greater than the preset efficiency threshold 1, Figure 7 The query is performed in the manner shown, that is, both data file 1' and data file 3 are dynamically filtered by using an index filtering manner.

[0095] exist Figure 7 In the example, the meanings of the nodes "Join" and "Scan" are the same as those in Figure 5 The same, no repetition here. Figure 7 It can be seen that after dynamically filtering data file 1` based on index 1`, data file 1` is not successfully skipped, indicating that data file 1` contains data that matches the query condition and needs to be scanned. After dynamically filtering data file 3 based on index 3, data file 3 is successfully skipped and does not need to be scanned again.

[0096] It can be seen from the above embodiments that Figure 5 and Figure 6In the example, since it is determined through index 1 that data file 1 does not contain the target data to be queried, data file 1 can be skipped directly. As for additional data file 2, since this is a data file generated by the user's addition operation, there is no corresponding index information and it can be scanned directly. Therefore, according to different data states at different times, it is possible to select a suitable data query method to query data in a targeted manner, thereby avoiding the storage performance impact caused by small index files, saving the time overhead of data file scanning, and improving data query efficiency and resource utilization.

[0097] In summary, specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recorded in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing can be advantageous.

[0098] The above is a data processing method provided in an embodiment of the present application. Based on the same idea, an embodiment of the present application also provides a data processing device.

[0099] Figure 8 is a schematic block diagram of a data processing device according to an embodiment of the present application, such as Figure 8 As shown, the device comprises: A first determination module 81, configured to determine data status information of a data file corresponding to the target data in response to a data query request for the target data; A second determination module 82 is used to determine an index efficiency index of the data file in response to the data status information indicating that the data file meets the preset change condition; the index efficiency index is used to characterize the efficiency improvement degree of querying the data file using the index filtering method; the index filtering method is to filter the data file based on the first index information of the data file; A generating module 83, configured to generate a query strategy for the target data based on the index efficiency index; the query strategy includes a data query method for the data file; the data query method includes the index filtering method or the scanning method; The query module 84 is used to perform data query on the data file based on the query strategy to obtain the target query result of the target data.

[0100] The device of the embodiment of the present application, after receiving a data query request for target data, determines the data status information of the data file corresponding to the target data, and in response to the data status information indicating that the data file meets the preset change condition, determines the index efficiency index of the data file, the index efficiency index is used to characterize the efficiency improvement degree of querying the data file using the index filtering method, the index filtering method is to filter the data file based on the first index information of the data file. Based on the index efficiency index, a query strategy for the target data is generated, the query strategy includes a data query method for the data file, and the data query method includes an index filtering method or a scanning method. Based on the query strategy, the data file is queried to obtain the target query result of the target data. It can be seen that the data query strategy is dynamically adjusted by real-time perception of the data status information, so that the data can be queried in a targeted data query method in a rapidly changing data environment, ensuring that the index information can be effectively used in a rapidly changing data environment, and improving the data query efficiency and resource utilization. In addition, by flexibly selecting to use the index filtering method or the scanning method, the fragmentation of the index information can be reduced, the maintenance cost of the index information can be reduced, and the data query performance can be ensured to be continuously optimized as the data environment changes, and the overall stability of the system can be improved.

[0101] Those skilled in the art should understand that Figure 8 The data processing device in can be used to implement the data processing method described above, and the detailed description thereof should be similar to the description of the method part above. To avoid tediousness, it will not be repeated here.

[0102] Based on the same idea, the embodiment of the present application also provides an electronic device, such as Fig. 9 As shown. The electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 901 and memory 902, and the memory 902 may store one or more storage applications or data. Among them, the memory 902 may be a temporary storage or a permanent storage. The application stored in the memory 902 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the electronic device. Furthermore, the processor 901 can be configured to communicate with the memory 902 to execute a series of computer executable instructions in the memory 902 on the electronic device. The electronic device may also include one or more power supplies 903, one or more wired or wireless network interfaces 904, one or more input and output interfaces 905, and one or more keyboards 906.

[0103] Specifically in this embodiment, the electronic device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions for the electronic device, and the one or more programs are configured to be executed by one or more processors, including the following computer executable instructions: In response to a data query request for target data, determining data status information of a data file corresponding to the target data; In response to the data status information indicating that the data file meets the preset change condition, determining an index efficiency index of the data file; the index efficiency index is used to characterize the degree of efficiency improvement of querying the data file using an index filtering method; the index filtering method is to filter the data file based on the first index information of the data file; Based on the index efficiency index, a query strategy for the target data is generated; the query strategy includes a data query method for the data file; the data query method includes the index filtering method or the scanning method; The data file is queried based on the query strategy to obtain a target query result of the target data.

[0104] According to the technical solution of the embodiment of the present application, after receiving a data query request for target data, by determining the data status information of the data file corresponding to the target data, in response to the data status information indicating that the data file meets the preset change condition, the index efficiency index of the data file is determined, and the index efficiency index is used to characterize the efficiency improvement degree of querying the data file using the index filtering method, and the index filtering method is to filter the data file based on the first index information of the data file. Based on the index efficiency index, a query strategy for the target data is generated, and the query strategy includes a data query method for the data file, and the data query method includes an index filtering method or a scanning method. Based on the query strategy, a data query is performed on the data file to obtain a target query result of the target data. It can be seen that the data query strategy is dynamically adjusted by real-time perception of the data status information, so that a targeted data query method can be adopted to query data in a rapidly changing data environment, ensuring that the index information can be effectively used in a rapidly changing data environment, and improving the data query efficiency and resource utilization. In addition, by flexibly selecting to use the index filtering method or the scanning method, the fragmentation of the index information can be reduced, the maintenance cost of the index information can be reduced, and the data query performance can be ensured to be continuously optimized as the data environment changes, thereby improving the overall stability of the system.

[0105] The embodiment of the present application further proposes a computer-readable storage medium, which stores one or more computer programs, wherein the one or more computer programs include instructions, which, when executed by an electronic device including multiple application programs, enable the electronic device to perform each process of the above data processing method embodiment, and are specifically used to perform: In response to a data query request for target data, determining data status information of a data file corresponding to the target data; In response to the data status information indicating that the data file meets the preset change condition, determining an index efficiency index of the data file; the index efficiency index is used to characterize the degree of efficiency improvement of querying the data file using an index filtering method; the index filtering method is to filter the data file based on the first index information of the data file; Based on the index efficiency index, a query strategy for the target data is generated; the query strategy includes a data query method for the data file; the data query method includes the index filtering method or the scanning method; The data file is queried based on the query strategy to obtain a target query result of the target data.

[0106] According to the technical solution of the embodiment of the present application, after receiving a data query request for target data, by determining the data status information of the data file corresponding to the target data, in response to the data status information indicating that the data file meets the preset change condition, the index efficiency index of the data file is determined, and the index efficiency index is used to characterize the efficiency improvement degree of querying the data file using the index filtering method, and the index filtering method is to filter the data file based on the first index information of the data file. Based on the index efficiency index, a query strategy for the target data is generated, and the query strategy includes a data query method for the data file, and the data query method includes an index filtering method or a scanning method. Based on the query strategy, a data query is performed on the data file to obtain a target query result of the target data. It can be seen that the data query strategy is dynamically adjusted by real-time perception of the data status information, so that a targeted data query method can be adopted to query data in a rapidly changing data environment, ensuring that the index information can be effectively used in a rapidly changing data environment, and improving the data query efficiency and resource utilization. In addition, by flexibly selecting to use the index filtering method or the scanning method, the fragmentation of the index information can be reduced, the maintenance cost of the index information can be reduced, and the data query performance can be ensured to be continuously optimized as the data environment changes, thereby improving the overall stability of the system.

[0107] An embodiment of the present application provides a computer program product, including a computer program, which is executed by a processor to implement the various processes of the above-mentioned data processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0108] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0109] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0110] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0111] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0112] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0114] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0115] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0116] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0117] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0118] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0119] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0120] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A data processing method, comprising: In response to a data query request for target data, determining data status information of a data file corresponding to the target data; In response to the data status information indicating that the data file meets the preset change condition, determining an index efficiency index of the data file; the index efficiency index is used to characterize the degree of efficiency improvement of querying the data file using the index filtering method; The index filtering method is to filter the data file based on the first index information of the data file; Based on the index efficiency index, a query strategy for the target data is generated; the query strategy includes a data query method for the data file; the data query method includes the index filtering method or the scanning method; The data file is queried based on the query strategy to obtain a target query result of the target data.

2. The method according to claim 1, wherein the data status information comprises: The amount of changed data in the data file; After determining the data status information of the data file corresponding to the target data, the method further includes: In response to the data volume of the changed data being less than or equal to a first preset threshold, it is determined that the data file meets the preset change condition.

3. The method according to claim 1, wherein the data status information comprises: The amount of source data, the amount of changed data and index validity in the data file; The determining of data status information of the data file corresponding to the target data includes: The index validity is calculated according to the data volume of the source data and the data volume of the changed data; wherein the index validity is positively correlated with the data volume of the source data and negatively correlated with the total data volume; the total data volume is the sum of the data volume of the source data and the data volume of the changed data; After determining the data status information of the data file corresponding to the target data, the method further includes: In response to the index validity being greater than or equal to a second preset threshold, it is determined that the data file meets the preset change condition.

4. The method according to claim 1, wherein determining the index efficiency index of the data file comprises: Determining index parameter information of the data file according to the data status information; The index parameter information includes at least one of the following: the scanning time of the source data in the data file, the scanning time of the changed data in the data file, the scanning time of the first index information, the expiration time of the first index information, the update frequency of the first index information, the update times of the data file, and the index skip probability; The indexing efficiency index is determined according to the indexing parameter information.

5. The method according to claim 1, wherein the indexing efficiency index comprises: The efficiency ratio between using the index filtering method to query the data file and using the scanning method to query the data file; The step of generating a query strategy for the target data based on the index efficiency indicator includes: In response to the efficiency ratio being higher than a preset efficiency threshold, determining that the data query mode is the index filtering mode; In response to the efficiency ratio being lower than or equal to the preset efficiency threshold, determining the data query mode as the scanning mode.

6. The method according to claim 5, wherein the step of performing a data query on the data file based on the query strategy to obtain a target query result of the target data comprises: In response to the data query mode being the index filtering mode, matching the first index information with the query condition of the target data to obtain a matching result; The query condition is carried in the data query request; In response to the matching result being that the first index information does not match the query condition, performing a scan skip operation on the data file; Alternatively, in response to the matching result that the first index information matches the query condition, a scanning operation is performed on the data file, and the target query result is obtained according to the scanning result.

7. The method according to claim 1, wherein the step of performing a data query on the data file based on the query strategy to obtain a target query result of the target data comprises: Performing a data query on the source data in the data file based on the data query method to obtain a first query result; Scan the changed data in the data file based on the scanning method to obtain a second query result of the changed data; The target query result is determined according to the first query result and the second query result.

8. The method according to claim 7, wherein the change data comprises additional data and / or deleted record data; The determining the target query result according to the first query result and the second query result includes: In response to the change data including the additional data, integrating the first query result and the second query result of the additional data to obtain the target query result; In response to the change data including the deletion record data, the deleted data corresponding to the deletion record data is deleted from the first query result to obtain the target query result.

9. The method according to claim 1, wherein determining the index efficiency index of the data file comprises: In response to determining that the data file corresponds to the first index information, an index efficiency index of the data file is determined.

10. The method according to claim 1, after generating the query strategy for the target data based on the index efficiency index, further comprising: In response to the data query mode being the scanning mode, generating second index information of the data file; The second index information is updated with the first index information.

11. The method according to claim 4, wherein the index parameter information includes the index skip probability; The method further comprises: Counting at least two of a first number of times the scan skip operation is performed on the data file, a second number of times the scan operation is performed on the data file, and a total number of times the data file is queried within a preset time period to obtain a statistical result; The index skip probability of the data file is updated according to the statistical result.

12. An electronic device, comprising a processor and a memory electrically connected to the processor, the memory storing a computer program, and the processor being configured to call and execute the computer program from the memory to implement the data processing method according to any one of claims 1 to 11.

13. A computer-readable storage medium, wherein the storage medium is used to store a computer program, wherein the computer program can be executed by a processor to implement the data processing method according to any one of claims 1 to 11.

14. A computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the data processing method according to any one of claims 1 to 11.

Images