Data processing method and data processing equipment
By directly scanning the file directory of the Hive metadata database through the metadata service to obtain file metadata information and generate a scan completion report, the problem of high resource consumption in Hive data volume statistics is solved, and efficient utilization of computing resources is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JUHAOKAN TECH CO LTD
- Filing Date
- 2025-12-12
- Publication Date
- 2026-05-08
AI Technical Summary
Existing technologies consume a large amount of computing resources when executing SQL queries in Hive data volume statistics scenarios, leading to resource contention and blocking other computing tasks.
The file directory of the Hive metadata database is scanned directly by the metadata service. The file metadata information is obtained by full scan or incremental scan and a scan completion report is generated, avoiding the use of Hive or Spark SQL select count(*) query.
It reduces the consumption of computing resources, solves the resource consumption problem of executing SQL in Hive data volume statistics scenarios, and improves the utilization efficiency of computing resources.
Smart Images
Figure CN121996712A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of big data technology, and in particular to a data processing method and a data processing device. Background Technology
[0002] Hive, as a data warehouse tool within the Hadoop ecosystem, stores its metadata (table structure, etc.) in a relational database (such as MySQL), while the actual data files are stored in a distributed file system (such as the Hadoop Distributed File System (HDFS) or Amazon Simple Storage Service (S3)). Data volume statistics are a core task of data management and governance, requiring monitoring of table size, partition changes, and data growth trends.
[0003] The current industry mainstream adopts the method of executing Structured Query Language (SQL) (such as select count(*)) to count data volume, but this method has significant drawbacks: executing SQL requires starting a Hive or Spark job, which consumes a lot of computing resources (Central Processing Unit (CPU) and memory), especially in batch processing scenarios, it is easy to cause resource contention, thereby blocking other computing tasks.
[0004] Therefore, how to reduce the resource consumption of executing SQL in Hive data volume statistics scenarios has become an urgent problem to be solved. Summary of the Invention
[0005] To address the aforementioned technical problems, this disclosure provides a data processing method and a data processing device.
[0006] In a first aspect, this disclosure provides a data processing device, comprising: a communicator configured to communicate with a scheduler; and a controller configured to: when preset conditions for performing a scanning operation are met, query at least one table contained in a target metadata database and the partition file directory of each table based on a metadata service; traverse the actual files in each partition file directory according to a pre-configured scanning method, and obtain the file metadata information of each traversed actual file through the metadata service; wherein the file metadata information includes one or more of the actual modification time and the actual data volume, and the scanning method includes either a full scan or an incremental scan; and generate a scan completion report corresponding to the scanning operation based on the file metadata information of each actual file.
[0007] Secondly, this disclosure provides a data processing method, comprising: when preset conditions for performing a scan operation are met, obtaining at least one table contained in the target metadata database and the partition file directory of each table through a metadata service; traversing the actual files in each partition file directory according to a pre-configured scan method, and obtaining the file metadata information of each actual file after traversal through the metadata service; wherein the file metadata information includes one or more of the actual modification time and the actual data volume, and the scan method includes any one of full scan and incremental scan; and generating a scan completion report corresponding to the scan operation based on the file metadata information of each actual file.
[0008] Thirdly, this disclosure provides a computer-readable storage medium, comprising: storing a computer program on the computer-readable storage medium, the computer program being executed by a controller using a data processing method as provided in any of the second aspects.
[0009] Fourthly, this disclosure provides a computer program product that, when run on a computer, causes the computer to perform any of the data processing methods provided in the second aspect.
[0010] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on the first computer-readable storage medium. The first computer-readable storage medium may be encapsulated together with the controller of the data processing device, or it may be encapsulated separately from the controller of the data processing device; this disclosure does not impose any limitations on this.
[0011] The descriptions of the second, third, and fourth aspects in this disclosure can be referenced to the detailed description of the first aspect; and the beneficial effects of the descriptions of the second, third, and fourth aspects can be referenced to the analysis of the beneficial effects of the first aspect, which will not be repeated here.
[0012] In this disclosure, the names of the aforementioned data processing devices do not limit the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear under other names. As long as the functions of each device or functional module are similar to those of this disclosure, they fall within the scope of this disclosure and its equivalents.
[0013] These or other aspects of this disclosure will become more readily apparent in the following description.
[0014] The technical solution provided in this disclosure has the following advantages compared with the prior art: This disclosure provides a data processing device, including: a communicator configured to communicate with a scheduler; and a controller configured to: when preset conditions for performing a scan operation are met, query at least one table contained in the target metadata database and the partition file directory of each table based on a metadata service; traverse the actual files in each partition file directory according to a pre-configured scanning method, and obtain the file metadata information of each traversed actual file through the metadata service; and generate a scan completion report corresponding to the scan operation based on the file metadata information of each actual file. When the target metadata database is Hive metadata, the file directories of each table in the Hive metadata database are directly scanned through the metadata service, the actual files in each partition file directory are traversed according to a pre-configured scanning method, and the file metadata information of each traversed actual file is obtained through the metadata service; and a scan completion report corresponding to the scan operation is generated based on the file metadata information of each actual file. By avoiding the use of Hive or Spark SQL to execute select count(*) queries, the computational resource consumption is reduced, thus solving the problem of how to reduce the resource consumption of executing SQL in Hive data volume statistics scenarios. Attached Figure Description
[0015] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0016] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1 One of the flowcharts illustrating the data processing method provided in the embodiments of this application; Figure 2 An architecture diagram of the server for the data processing method provided in the embodiments of this application; Figure 3 A second schematic flowchart illustrating the data processing method provided in this application embodiment; Figure 4 The third schematic flowchart of the data processing method provided in the embodiments of this application; Figure 5 The fourth flowchart illustrating the data processing method provided in the embodiments of this application; Figure 6 Fifth flowchart illustrating the data processing method provided in the embodiments of this application; Figure 7 A flowchart illustrating the data processing method provided in this application embodiment is shown in Figure 6. Figure 8 Seventh schematic flowchart of the data processing method provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of the data processing device provided in the embodiments of this application; Figure 10 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation
[0018] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.
[0019] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.
[0020] The data processing device provided in this application can have various implementation forms, such as a television, a smart television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figure 1 and Figure 2 This is one specific implementation of the data processing device of this application.
[0021] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0022] Azure Blob Storage in this disclosure is an object storage service provided by the Microsoft Azure cloud platform. Its core purpose is to store massive amounts of unstructured data (such as images, videos, documents, logs, etc.), and it supports high availability, scalability, and data protection.
[0023] In this disclosure, Google Cloud Storage is an object storage service of the Google Cloud Platform, which also focuses on the storage of massive amounts of unstructured data, combining flexibility and cost-effectiveness.
[0024] In this disclosure, file size refers to the storage space occupied by a computer file on a storage device, typically expressed in units such as bytes (Byte), kilobytes (KB), or megabytes (MB). It directly depends on the file's content and format, such as the raw data size of a text document, image, or video. The file size is fixed; as long as the file content remains unchanged, its size will not change.
[0025] In this disclosure, "data volume" refers more broadly to the scale or quantity of data. It can be synonymous with file size, but is sometimes used to describe the overall volume of data, including additional overhead involved in transmission, processing, or storage. For example, in data transmission, data volume may include additional data such as protocol headers and checksums; in database or big data scenarios, data volume may refer to abstract metrics such as the number of records or the total throughput of the data stream. Therefore, data volume does not necessarily directly correspond to the physical space occupied by a file on the disk.
[0026] In the following embodiments, the execution subject of the data processing method provided in the embodiments of this disclosure is the aforementioned server, which will be used as an example to illustrate the method of the embodiments of this application.
[0027] This application provides a data processing method, such as... Figure 1 As shown, the data processing method may include S11-S13.
[0028] S11. When the preset conditions for performing the scan operation are met, obtain at least one table contained in the target metadata database and the partition file directory of each table through the metadata service. In some examples, the data processing method provided in this disclosure is applied to a data processing system that includes a client and a server; the server architecture is as follows: Figure 2As shown, the system includes a metadata service, a Hive metadata database (such as MySQL), a distributed file system (such as HDFS, S3), and a results database (such as MySQL). The metadata service, triggered via API or scheduled tasks, accesses the Hive metadata database to obtain table structure and partition information. It then traverses the file directory, collecting file metadata (such as filename, size, and modification time) and calculating data volume (e.g., obtaining the number of rows by parsing the file format). Afterward, a scan completion report is stored in the results database, containing fields such as table name, partition name, filename, data volume, and scan time. The metadata service directly scans the Hive table file directory to achieve efficient data volume statistics. The metadata service is based on the following principle: automatically collecting data volume information from Hive tables within the big data platform, supporting data monitoring, governance, and cost analysis.
[0029] In some examples, the scan completion report is shown in Table 1.
[0030] Table 1
[0031] In some examples, users can pre-configure the target application programming interface (API) for external access to the application on the server and set it to execute the data processing method provided in this disclosure embodiment when the user calls the target API through a client. Thus, when the user calls the target API through a client, if the server determines that the preset conditions for performing the scan operation are met, it obtains at least one table contained in the target metadata database, and the partition file directory of each table, through the metadata service; it traverses the actual files in each partition file directory according to the pre-configured scan method, and obtains the file metadata information of each traversed actual file through the metadata service; based on the file metadata information of each actual file, it generates a scan completion report corresponding to the scan operation.
[0032] In some examples, a user can set up a timed trigger task on the server to execute the data processing method provided in the embodiments of this disclosure. Thus, when the timed scan task is triggered at the current moment, the server determines that preset conditions for performing the scan operation are met, and queries the target metadata database for at least one table and the partition file directory of each table based on the metadata service.
[0033] For example, the configuration of a scanning task is shown in Table 2.
[0034] Table 2
[0035] In some examples, the target metadata database can be a Hive metadata database, and the result database can be a MySQL database.
[0036] In some examples, the metadata service can be deployed on a standalone server or in a container, interacting with the Hive metadata database and distributed file system over the network.
[0037] In some examples, the distributed file system can also be Azure Blob Storage or Google Cloud Storage, and interaction with data processing devices can be achieved simply by reconfiguring the file access interface in the distributed file system.
[0038] S12. Traverse the actual files in each partition file directory according to the pre-configured scanning method, and obtain the file metadata information of each actual file after traversal through the metadata service; wherein, the file metadata information includes one or more of the actual modification time and the actual data volume, and the scanning method includes any one of full scan and incremental scan.
[0039] In some examples, after the server starts, it performs service initialization, and then reads a pre-configured scan method, such as either a full scan or an incremental scan. When the scan method is a full scan, the historical scan time is 0; when the scan method is an incremental scan, the historical scan time is the actual scan time when the previous scan method was incremental.
[0040] In some examples, the decision to scan a file is made by comparing its last_modified_time (the last actual modification time) with the last scan time of the database record. For example, if the actual modification time is earlier than the historical scan time of the last scan operation in the target metadata database, it means that the actual file has not changed, so the scan of the actual file is skipped; or, if the actual modification time is later than the historical scan time of the last scan operation in the target metadata database, it means that the actual file has changed, so the actual file is scanned.
[0041] In some examples, the actual modification time of the actual file can also be obtained through file verification, Message Digest Algorithm 5 (MD5), etc.
[0042] In some examples, lightweight parsing is used for different file formats, such as Optimized RowColumnar (ORC) and Parquet.
[0043] In some examples, Hive metadata databases commonly use ORC and PARQUET formats to directly obtain the row count by reading the footer metadata, without requiring a full file scan.
[0044] In some examples, the scan completion report records a global timestamp for each scan operation, and the next scan only processes actual files whose actual modification time is later than that global timestamp.
[0045] In some examples, when actual files are updated, the metadata service scans the changed files, updates the resulting database, and records a new global timestamp. For instance, a daily scheduled scan processes only files modified within the last 24 hours.
[0046] In some examples, if the actual file is inaccessible (e.g., due to insufficient permissions or network failure), an error log is recorded in the scan completion report, and that actual file is skipped without affecting the scanning of other actual files.
[0047] In some examples, if there are no actual file changes, the metadata service does not perform a scan on the actual file and returns the result directly, reducing resource waste.
[0048] In some examples, full mode (initial scan) and incremental mode (subsequent scans) are supported, which users can switch between via configuration. This avoids redundant scans and reduces resource consumption through file-level incremental updates.
[0049] S13. Based on the file metadata of each actual file, generate a scan completion report corresponding to the scan operation.
[0050] In some feasible examples, the scan completion report includes at least one or more of the following: the table name of at least one table, the filenames of the actual files in the partition file directory of each table, the actual data volume of each actual file, and the current scan time when the scan operation was performed.
[0051] As can be seen from the above, this disclosure provides a data processing method. When the target metadata database is Hive metadata, the file directories of each table in the Hive metadata database are directly scanned through the metadata service. The actual files in each partition file directory are traversed according to the pre-configured scanning method, and the file metadata information of each actual file after traversal is obtained through the metadata service. Based on the file metadata information of each actual file, a scan completion report corresponding to the scan operation is generated. Since the use of Hive or Spark SQL to execute select count(*) queries is avoided, the consumption of computing resources is reduced.
[0052] In some feasible examples, combining Figure 1 ,like Figure 3As shown, the above S11 can be implemented by the following S110.
[0053] S110. Upon receiving a call request for invoking the target application programming interface, determine that the preset conditions for performing the scan operation are met, and query at least one table contained in the target metadata database, as well as the partition file directory of each table, based on the metadata service.
[0054] As can be seen from the above, the data processing method provided in this disclosure includes: when the target metadata database is Hive metadata, directly scanning the file directories of each table in the Hive metadata database through the metadata service, traversing the actual files in each partition file directory according to the pre-configured scanning method, and obtaining the file metadata information of each actual file after traversal through the metadata service; generating a scan completion report corresponding to the scan operation based on the file metadata information of each actual file, thereby reducing the consumption of computing resources by avoiding the use of Hive or Spark SQL to execute select count(*) queries.
[0055] In some feasible examples, combining Figure 1 ,like Figure 4 As shown, the above S11 can be implemented in the following S111.
[0056] S111. When a scheduled scan task is triggered at the current time, determine whether the preset conditions for performing the scan operation are met, and query at least one table contained in the target metadata database and the partition file directory of each table based on the metadata service.
[0057] As can be seen from the above, this disclosure provides a data processing method. When the target metadata database is Hive metadata, the file directories of each table in the Hive metadata database are directly scanned through the metadata service. The actual files in each partition file directory are traversed according to the pre-configured scanning method, and the file metadata information of each actual file after traversal is obtained through the metadata service. Based on the file metadata information of each actual file, a scan completion report corresponding to the scan operation is generated. Since the use of Hive or Spark SQL to execute select count(*) queries is avoided, the consumption of computing resources is reduced.
[0058] In some feasible examples, file metadata includes the actual modification time; combined with Figure 1 ,like Figure 5 As shown, the above S12 can be implemented by the following S120.
[0059] S120. Traverse the actual files in each partition file directory according to the pre-configured scanning method, and obtain the actual modification time of each actual file after traversal through the metadata service.
[0060] As can be seen from the above, this disclosure provides a data processing method. When the target metadata database is Hive metadata, the file directories of each table in the Hive metadata database are directly scanned through the metadata service. The actual files in each partition file directory are traversed according to the pre-configured scanning method, and the file metadata information of each actual file after traversal is obtained through the metadata service. Based on the file metadata information of each actual file, a scan completion report corresponding to the scan operation is generated. Since the use of Hive or Spark SQL to execute select count(*) queries is avoided, the consumption of computing resources is reduced.
[0061] In some feasible examples, file metadata also includes the actual amount of data; combined with Figure 5 ,like Figure 6 As shown, the above S12 can be implemented in the following S121.
[0062] S121. Based on the historical scan time of the last scan operation performed in the target metadata database and the actual modification time of each actual file, determine the actual data volume of each actual file.
[0063] As can be seen from the above, the data processing method provided in this disclosure, when the target metadata database is Hive metadata, determines the actual data volume of each actual file based on the historical scan time of the last scan operation performed in the target metadata database and the actual modification time of each actual file. This can provide accurate data volume change information and support historical trend analysis. Then, based on the file metadata of each actual file, a scan completion report corresponding to the scan operation is generated. Since the use of Hive or Spark SQL to execute select count(*) queries is avoided, the consumption of computing resources is reduced.
[0064] In some feasible examples, file metadata also includes the actual amount of data; combined with Figure 6 ,like Figure 7 As shown, the above S121 can be specifically implemented through the following S1210.
[0065] S1210. For each actual file, when the actual modification time is later than the historical scan time of the last scan operation in the target metadata database, obtain the file format of the actual file; calculate the theoretical data volume according to the data calculation method corresponding to the file format; and determine the actual data volume of the actual file as the theoretical data volume based on the theoretical data volume.
[0066] In some examples, file formats include OCR and Parquet. When the file format is OCR, the data calculation method involves parsing the actual file's footer to obtain the number of lines in the actual file and the average length of each line. Based on the number of lines and the average length, the theoretical data size is determined. For example, each ORC file has a FileFooter, which stores the number of lines in each stripe and the data type information for each column. When reading the file, the process seeks to the end of the file and reads the PostScript, parsing the FileFooter length. Then, it reads the FileFooter again and parses the information for each stripe, including the number of lines in each stripe. By adding up the line counts of all stripes, the total number of lines in the file can be obtained. Combining this with the data types stored in the file and the approximate average length of each line (which can be obtained through statistical estimation of partial data), the file's data size can be estimated. For example, if an ORC file has 3 stripes with 1000, 1200 and 1100 rows respectively, and the average length of each row is estimated to be 50 bytes, then the number of rows is 1000 + 1200 + 1100 = 3300 rows, and the data volume is approximately 3300 × 50 = 165000 bytes.
[0067] When the file format is Parquet, the data calculation method involves reading the file metadata of the actual file to determine the number of lines contained in the actual file and the average length of the data stored in each line. Based on the number of lines and the average length, the theoretical data volume is then determined.
[0068] As described above, this disclosure provides a data processing method. When the target metadata database is Hive metadata, for each actual file, if the actual modification time is later than the historical scan time of the last scan operation performed on the target metadata database, the file format of the actual file is obtained; the theoretical data volume is calculated according to the data calculation method corresponding to the file format; based on the theoretical data volume, the actual data volume of the actual file is determined to be the theoretical data volume, thus providing accurate data volume change information and supporting historical trend analysis; subsequently, based on the file metadata of each actual file, a scan completion report corresponding to the scan operation is generated. This reduces the consumption of computing resources by avoiding the use of Hive or Spark SQL to execute select count(*) queries.
[0069] In some feasible examples, file metadata also includes the actual amount of data; combined with Figure 7 ,like Figure 8 As shown, the data processing method provided in this embodiment of the disclosure further includes S14.
[0070] S14. When the actual modification time is earlier than the historical scan time of the last scan operation in the target metadata database, the actual data volume of the actual file is determined to be the historical data volume corresponding to the last scan operation in the target metadata database.
[0071] As can be seen from the above, this disclosure provides a data processing method that, when the target metadata database is Hive metadata, By determining the relationship between historical scan times and actual modification times, it is possible to accurately determine whether actual files have changed. An incremental update mechanism avoids repeatedly scanning unchanged files. For example, for each actual file, if the actual modification time is later than the historical scan time of the last scan operation in the target metadata database, the file format of the actual file is obtained. The theoretical data volume is calculated according to the data calculation method corresponding to the file format. Based on the theoretical data volume, the actual data volume of the actual file is determined as the theoretical data volume, thus providing accurate information on data volume changes and supporting historical trend analysis. Subsequently, based on the file metadata of each actual file, a scan completion report corresponding to the scan operation is generated. This avoids using Hive or Spark SQL to execute select count(*) queries, thereby reducing computational resource consumption.
[0072] In some feasible examples, the historical scan time is 0 when the last scan was a full scan; When the previous scan method was incremental scan, the historical scan time is the actual scan time when the previous scan operation was performed.
[0073] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0074] This application embodiment can divide the data processing device into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing unit. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.
[0075] like Figure 9 As shown in the diagram, an embodiment of this application provides a schematic diagram of a data processing device. It includes a communication unit 101 and a controller 102.
[0076] Communicator 101 is configured to communicate with the scheduler; Controller 102 is configured as follows: When the preset conditions for performing the scan operation are met, query at least one table contained in the target metadata database and the partition file directory of each table based on the metadata service. The system traverses the actual files in each partition's file directory according to a pre-configured scanning method, and obtains the file metadata information of each actual file after traversal through the metadata service. The file metadata information includes one or more of the actual modification time and the actual data volume, and the scanning method includes either full scan or incremental scan. Based on the file metadata of each actual file, a scan completion report is generated corresponding to the scan operation.
[0077] In some implementable examples, when the controller 102 obtains at least one table contained in the target metadata database and the partition file directory of each table through the metadata service when the preset conditions for performing the scan operation are met, it is further configured to: Upon receiving a call request to invoke the target application programming interface, it determines that the preset conditions for performing the scan operation are met, and queries the target metadata database for at least one table and the partition file directory of each table based on the metadata service.
[0078] In some implementable examples, when the controller 102 obtains at least one table contained in the target metadata database and the partition file directory of each table through the metadata service when the preset conditions for performing the scan operation are met, it is further configured to: When a scheduled scan task is triggered at the current time, the preset conditions for performing the scan operation are determined to be met. Based on the metadata service, at least one table contained in the target metadata database, as well as the partition file directory of each table, are queried.
[0079] In some feasible examples, file metadata includes the actual modification time; When traversing the actual files in each partition's file directory according to the pre-configured scanning method, and obtaining the file metadata information of each actual file after traversal through the metadata service, it is further configured as follows: The system traverses the actual files in each partition's file directory according to a pre-configured scanning method, and obtains the actual modification time of each actual file after traversal through the metadata service.
[0080] In some feasible examples, file metadata also includes the actual amount of data; Controller 102 is also configured as follows: Based on the historical scan time of the last scan operation in the target metadata database and the actual modification time of each actual file, the actual data volume of each actual file is determined.
[0081] In some implementable examples, controller 102, when determining the actual data volume of each actual file based on the historical scan time of the last scan operation performed on the target metadata database and the actual modification time of each actual file, is further configured to: For each actual file, if the actual modification time is later than the historical scan time of the last scan operation in the target metadata database, the file format of the actual file is obtained; according to the data calculation method corresponding to the file format, the theoretical data volume is calculated; based on the theoretical data volume, the actual data volume of the actual file is determined as the theoretical data volume.
[0082] In some implementable examples, controller 102 is also configured as follows: When the actual modification time is earlier than the historical scan time of the last scan operation in the target metadata database, the actual data volume of the actual file is determined to be the historical data volume corresponding to the last scan operation in the target metadata database.
[0083] In some feasible examples, the historical scan time is 0 when the last scan was a full scan; When the previous scan method was incremental scan, the historical scan time is the actual scan time when the previous scan operation was performed.
[0084] In some feasible examples, the scan completion report includes at least one or more of the following: the table name of at least one table, the filenames of the actual files in the partition file directory of each table, the actual data volume of each actual file, and the current scan time when the scan operation was performed.
[0085] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and their functions will not be repeated here.
[0086] Of course, the data processing device provided in this application embodiment includes, but is not limited to, the modules described above. For example, the data processing device may also include a memory 103. The memory 103 may be used to store the program code of the data processing device, and may also be used to store data generated by the data processing device during operation, such as data in write requests.
[0087] like Figure 10As shown, this application embodiment also provides a chip system that can be applied to the data processing device in the foregoing embodiments. The chip system includes at least one processor 1501 and at least one interface circuit 1502. The processor 1501 may be the processor in the aforementioned data processing device. The processor 1501 and the interface circuit 1502 are interconnected via a line. The processor 1501 can receive and execute computer instructions from the memory of the aforementioned data processing device through the interface circuit 1502. When the computer instructions are executed by the processor 1501, the data processing device can perform the various steps executed by the data processing device in the foregoing embodiments. Of course, the chip system may also include other discrete devices, and this application embodiment does not specifically limit this.
[0088] This application also provides a computer-readable storage medium for storing computer instructions for operating the aforementioned data processing device.
[0089] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data processing device, characterized in that, include: The communicator is configured to communicate with the scheduler. The controller is configured as follows: When the preset conditions for performing the scan operation are met, query at least one table contained in the target metadata database and the partition file directory of each table based on the metadata service. The system traverses the actual files in each of the partition file directories according to a pre-configured scanning method, and obtains the file metadata information of each actual file after traversal through the metadata service; wherein, the file metadata information includes one or more of the actual modification time and the actual data volume, and the scanning method includes any one of full scan and incremental scan; Based on the file metadata of each actual file, a scan completion report corresponding to the scan operation is generated.
2. The data processing device according to claim 1, characterized in that, When the controller meets the preset conditions for performing a scan operation, and obtains at least one table contained in the target metadata database and the partition file directory of each table through the metadata service, it is further configured to: Upon receiving a call request to invoke the target application programming interface, it determines that the preset conditions for performing the scan operation are met, and queries the target metadata database for at least one table and the partition file directory of each table based on the metadata service.
3. The data processing device according to claim 1, characterized in that, When the controller meets the preset conditions for performing a scan operation, and obtains at least one table contained in the target metadata database and the partition file directory of each table through the metadata service, it is further configured to: When a scheduled scan task is triggered at the current time, the preset conditions for performing the scan operation are determined to be met. Based on the metadata service, at least one table contained in the target metadata database, as well as the partition file directory of each table, are queried.
4. The data processing device according to claim 1, characterized in that, The file metadata includes the actual modification time; When traversing the actual files in each of the partition file directories according to a pre-configured scanning method and obtaining the file metadata information of each actual file after traversal through the metadata service, it is further configured as follows: The system traverses the actual files in each of the partition file directories according to a pre-configured scanning method, and obtains the actual modification time of each actual file after traversal through the metadata service.
5. The data processing device according to claim 4, characterized in that, The file metadata also includes the actual data volume; The controller is also configured to: Based on the historical scan time of the last scan operation performed on the target metadata database and the actual modification time of each actual file, the actual data volume of each actual file is determined.
6. The data processing device according to claim 5, characterized in that, The controller, when determining the actual data volume of each actual file based on the historical scan time of the last scan operation performed on the target metadata database and the actual modification time of each actual file, is further configured as follows: For each actual file, when the actual modification time is later than the historical scan time of the last scan operation performed on the target metadata database, the file format of the actual file is obtained; the theoretical data volume is calculated according to the data calculation method corresponding to the file format; based on the theoretical data volume, the actual data volume of the actual file is determined as the theoretical data volume.
7. The data processing device according to claim 6, characterized in that, The controller is also configured to: When the actual modification time is earlier than the historical scan time of the last scan operation performed on the target metadata database, the actual data volume of the actual file is determined to be the historical data volume corresponding to the last scan operation performed on the target metadata database.
8. The data processing device according to claim 5, characterized in that, When the previous scan method was a full scan, the historical scan time was 0; When the previous scan method was incremental scan, the historical scan time is the actual scan time when the previous scan operation was performed.
9. The data processing apparatus according to any one of claims 1-8, characterized in that, The scan completion report includes at least one or more of the following: the table name of the at least one table, the file name of each actual file in the partition file directory of each table, the actual data volume of each actual file, and the current scan time when the scan operation was performed.
10. A data processing method, characterized in that, include: When the preset conditions for performing the scan operation are met, at least one table contained in the target metadata database and the partition file directory of each table are obtained through the metadata service. The system traverses the actual files in each of the partition file directories according to a pre-configured scanning method, and obtains the file metadata information of each actual file after traversal through the metadata service; wherein, the file metadata information includes one or more of the actual modification time and the actual data volume, and the scanning method includes any one of full scan and incremental scan; Based on the file metadata of each actual file, a scan completion report corresponding to the scan operation is generated.