Partition table data processing method and device, computer equipment and readable storage medium

Abstract

The invention discloses a partition table data processing method and device, computer equipment and a readable storage medium, and belongs to the technical field of big data processing. According to the partition table data processing method, a to-be-exported target partition table name can be obtained according to a data export request; generating a unique key value associated with the data export request and a corresponding target folder used for storing exported data according to the data export request; by adopting a parallel processing mode of the first thread pool, the target partition tables corresponding to all the target partition table names associated with the data export request are respectively written into the corresponding partition files, so that the data export efficiency is improved, the data export time is shortened, memory management can be effectively realized, the risk of memory overflow is reduced, and the data export efficiency is improved. Meanwhile, the data processing performance is improved; according to the method, all the partition files in the target folder are combined into the target file, and then the target file is split into the target sub-files with the fixed length, so that the consistency of exporting all the data files is ensured.

Description

Technical Field

[0001] The present invention relates to the field of big data processing technology, and in particular to a partition table data processing method, device, computer equipment and readable storage medium. Background Art

[0002] At present, the amount of data in the database every hour can reach millions or even tens of millions. Therefore, in order to facilitate the management of data in the database, the data table is usually partitioned by hour. This data management mode takes a long time to query data; it is conceivable that if the data in the data table is stored in a file, it will take a lot of time. The existing data export method takes a lot of time to export the data table in the database. At the same time, memory overflow may occur during the data export process due to the large amount of data, and the exported file may also be too large to be opened. Summary of the invention

[0003] In view of the problem that the existing data export method for a large amount of data in a data table takes a lot of time and the memory is prone to overflow, a partition table data processing method, device, computer equipment and readable storage medium are provided to quickly export data and avoid memory overflow.

[0004] The present invention provides a partition table data processing method, comprising the following steps:

[0005] In response to a data export request, obtaining a target partition table name to be exported;

[0006] Generating a key value and a corresponding target folder associated with the data export request according to the data export request;

[0007] Starting a first thread pool, writing target partition tables corresponding to all target partition table names associated with the data export request into corresponding partition files respectively, and storing the partition files in the target folder;

[0008] All the partition files in the target folder are merged into a target file, and then the target file is split into target sub-files of fixed length.

[0009] Preferably, in response to the data export request, obtaining the name of the target partition table to be exported includes:

[0010] The data export request includes an export time period and a target partition table name prefix;

[0011] The database is queried to obtain a target partition table name that matches the export time period and the prefix of the target partition table name.

[0012] Preferably, generating a key value and a corresponding target folder associated with the data export request according to the data export request includes:

[0013] The data export request also includes the current time;

[0014] Generate a unique key value according to the data export request;

[0015] Creating the target folder associated with the target partition table name and a storage path corresponding to the target folder according to the key value and the current time;

[0016] The format of the target folder is current time_key value;

[0017] The storage path is associated with the target partition table name prefix.

[0018] Preferably, the starting of the first thread pool to write the target partition tables corresponding to all target partition table names associated with the data export request into corresponding partition files respectively includes:

[0019] Each first thread in the first thread pool corresponds to at least one target partition table;

[0020] The corresponding target partition table is written into the corresponding partition file through the first thread.

[0021] Preferably, writing the corresponding target partition table into the corresponding partition file through the first thread includes:

[0022] Query the total number of records in the target partition table according to the data export request;

[0023] Determine whether the total number of records is greater than or equal to a threshold number;

[0024] If the total number of records is less than the number threshold, writing the record data in the target partition table into the corresponding partition file through the first thread;

[0025] If the total number of records is greater than or equal to the record number threshold, the number of record pages is calculated according to the total number of records and the record number threshold, and the data in the target partition table is paging according to the record page number;

[0026] Start the second thread pool, write the record data of all pages in the target partition table into corresponding slice files respectively, the slice files are stored in the target folder, and each page is associated with one slice file;

[0027] All the slice files are merged to generate the partition file.

[0028] Preferably, the starting of the second thread pool to write the record data of all pages in the target partition table into corresponding shard files respectively includes:

[0029] Each second thread in the second thread pool corresponds to at least one page;

[0030] The record data of the corresponding pages are written into the associated fragment files respectively through the second thread.

[0031] Preferably, the step of merging all the fragment files to generate the partition file includes:

[0032] The FileChannel class is used to merge all the fragment files to generate the partition file.

[0033] Preferably, merging all the partition files in the target folder into a target file, and then splitting the target file into target sub-files of fixed lengths, comprises:

[0034] Using the FileChannel class to merge all the partition files in the target folder into a target file;

[0035] The target file is split into target sub-files of fixed length using the FileChannel class.

[0036] Preferably, it also includes:

[0037] The target sub-file is compressed and saved in the target folder.

[0038] The present invention also provides a partition table data processing device, comprising:

[0039] An acquisition unit, used for acquiring the name of the target partition table to be exported according to the data export request;

[0040] A generating unit, configured to generate a key value and a corresponding target folder associated with the data export request according to the data export request;

[0041] A writing unit, used to start a first thread pool, and write the target partition tables corresponding to all target partition table names associated with the data export request into corresponding partition files respectively, and the partition files are stored in the target folder;

[0042] A merging unit, used for merging all the partition files in the target folder into a target file;

[0043] The splitting unit is used to split the target file into target sub-files of fixed length.

[0044] The present invention also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0045] The present invention also provides a computer-readable storage medium on which a computer program is stored, and the computer program implements the steps of the above method when executed by a processor.

[0046] Beneficial effects of the above technical solution:

[0047] In the technical scheme, the partition table data processing method of the present invention can obtain the target partition table name to be exported according to the data export request; generate a unique key value associated with the data export request and a corresponding target folder for storing the exported data according to the data export request; adopt the parallel processing method of the first thread pool to write the target partition tables corresponding to all the target partition table names associated with the data export request into the corresponding partition files respectively, thereby improving the data export efficiency, shortening the data export time, and being able to effectively realize memory management, reduce the risk of memory overflow, and improve data processing performance at the same time; by first merging all the partition files in the target folder into a target file, and then splitting the target file into target sub-files of fixed length, the consistency of all exported data files is ensured, and the risk of data confusion is avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of an embodiment of the partition table data processing method of the present invention;

[0049] Figure 2 It is a flowchart of an embodiment of writing a corresponding target partition table into a corresponding partition file through a first thread according to the present invention;

[0050] Figure 3 A module of an embodiment of the partition table data processing device of the present invention;

[0051] Figure 4 A schematic diagram of the hardware structure of a computer device for executing a partition table data processing method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The advantages of the present invention are further described below in conjunction with the accompanying drawings and specific embodiments.

[0053] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0054] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0055] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0056] In the description of the present invention, it should be understood that the numerical labels before the steps do not identify the order of executing the steps, but are only used to facilitate the description of the present invention and distinguish each step, and therefore cannot be understood as a limitation of the present invention.

[0057] The present invention proposes a partition table data processing method to solve the defects that the existing data export method takes a lot of time and the memory is prone to overflow when a large amount of data in the data table is used. Figure 1 , which is a flow chart of a partition table data processing method according to a preferred embodiment of the present invention. As can be seen from the figure, the partition table data processing method provided in this embodiment includes the following steps:

[0058] S1. In response to the data export request, obtain the name of the target partition table to be exported;

[0059] It should be noted that: the data export request includes the export time period and the target partition table name prefix.

[0060] Furthermore, step S1 may include: querying a database to obtain a target partition table name that matches the export time period and the prefix of the target partition table name.

[0061] In this embodiment, the data export request can be generated based on the export conditions, export time period, and partition table name to be exported selected / entered by the user on the export page. Considering that the partition table name is divided into hours, the target partition table names that meet the export time period and export conditions are screened from the target partition table names, that is, the screened target partition table names are partition tables that meet the export time range. The matching target partition table names are saved in the tableList component container.

[0062] S2. Generate a key value and a corresponding target folder associated with the data export request according to the data export request;

[0063] It should be noted that: the data export request may also include the current time.

[0064] Further, step S2 may include:

[0065] S21. Generate a unique key value according to the data export request;

[0066] S22. Create the target folder associated with the target partition table name and the storage path corresponding to the target folder according to the key value and the current time;

[0067] Wherein, the format of the target folder is current time_key value;

[0068] In this embodiment, the storage path is associated with the target partition table name prefix.

[0069] In this embodiment, a unique key value is generated for each data export request as the temporary directory name of the downloaded file to distinguish each export request; at the same time, the current time of this export is recorded to prevent newly inserted data from affecting subsequent export results.

[0070] Storage path filePath: / a path of the server / target partition table name prefix / yyyyMMddHHmmssSSS_the unique key value exported this time. In this embodiment, subsequent file operations are related to this storage path.

[0071] In this embodiment, the above steps can be implemented by the main thread, and the main thread ends before executing step S3. The end of the main thread can quickly feedback the export status to the user, so as to proceed with the download step and start a new thread to handle subsequent downloads. If the main thread is used later, the entire download process will block other operations of the user. Quickly ending the main thread ensures that other operations of the user are not affected by the main thread.

[0072] S3. Start the first thread pool, write the target partition tables corresponding to all target partition table names associated with the data export request into corresponding partition files, and store the partition files in the target folder;

[0073] In this embodiment, the first thread pool pool1 is started to cyclically process the target partition tables corresponding to all target partition table names in the tableList component container, and each first thread processes one target partition table, so that the threads do not interfere with each other and the queried data will not be confused.

[0074] In this embodiment, each first thread in the first thread pool corresponds to at least one target partition table;

[0075] See also Figure 2 As shown, step S3 may include: writing the corresponding target partition table into the corresponding partition file through the first thread, specifically including the following steps:

[0076] S31. Query the total number of records in the target partition table according to the data export request;

[0077] In this step, the total number of records that meet the conditions in a target partition table is queried based on the export conditions, target partition table name, export time period and other information on the user page.

[0078] S32. Determine whether the total number of records is greater than or equal to the number threshold. If not, execute step S33; if yes, execute step S34;

[0079] In this step, the number threshold is the preset single-page record number pageSize (for example: 1000). Determine whether the total number of records exceeds the single-page record number pageSize. If not, directly query all the data in the target partition table.

[0080] S33. Write the record data in the target partition table into the corresponding partition file through the first thread;

[0081] In this step, the FileChannel class is used to write the recorded data into the corresponding partition file (the file is named "target partition table name prefix yyyyMMddHH_current table record number.csv", such as staweblog2024052011_300.csv) and saved to filePath. The partition file stores the data of the current target partition table.

[0082] S34. Calculate the number of record pages according to the total number of records and the number threshold, and paginate the data in the target partition table according to the number of record pages;

[0083] In this step, if the total number of records exceeds the number of records per page, pageSize, the number of record pages, pageCount, is calculated based on the total number of records, totalCount, and the number of records per page, pageSize. The data in the target partition table is paginated based on the number of record pages and the number of entries threshold.

[0084] S35. Start the second thread pool, write the record data of all pages in the target partition table into the corresponding fragment files, the fragment files are stored in the target folder, and each page is associated with a fragment file;

[0085] It should be noted that each second thread in the second thread pool corresponds to at least one page.

[0086] Further, step S35 may include: writing the record data of the corresponding pages into the associated fragment files respectively through the second thread.

[0087] In this step, the second thread pool pool2 is started to process the record data of each page in a loop. First, the record data is queried according to the export conditions, current page number, number threshold, export time period, and target partition table name selected / entered by the user on the export page. Each record data queried is a string separated by "," so that it can be written into the csv file. Then the found record data is written into the corresponding fragment file. The fragment file name is "target partition table name prefix yyyyMMddHH_fixed length page number_number of records on the current page.csv", where yyyy represents year, MM represents month, dd represents day, and HH represents hour (such as staweblog2024052006_10000001_20000.csv. The file is named in this way to ensure the order of data when merging fragment files). The fragment file is a fragment file of the record data in the current target partition table name.

[0088] In this embodiment, due to the large amount of data, the exported files are all in csv format. The csv format is simple, each line represents a piece of data, and the fields are separated by commas. In this way, when querying, the data can be directly assembled into such a format without secondary processing. csv is a plain text format, which is very convenient for writing files or merging and splitting files. And csv can also be presented to users in the form of a table for easy reference.

[0089] S36. Merge all the fragment files to generate the partition file.

[0090] Specifically, the FileChannel class is used to merge all the fragment files to generate the partition file.

[0091] In this step, after the second threads in the second thread pool pool2 are processed, the FileChannel class is used to merge all the fragment files of the current target partition table name into one partition file, that is, the data of the current target partition table name, and each target partition table name corresponds to a partition file (such as staweblog2024052008_1099065.csv). And delete the corresponding fragment files to save disk space.

[0092] S4. Merge all the partition files in the target folder into a target file, and then split the target file into target sub-files of fixed length.

[0093] Furthermore, the FileChannel class is used to merge all the partition files in the target folder into a target file; and the FileChannel class is used to split the target file into target sub-files of fixed length.

[0094] In this step, after the first threads in the first thread pool pool1 are processed, the partition files of all target partition tables are merged into a csv target file using the FileChannel class, and the merged partition files are deleted; the target file is then split according to a fixed size using the FileChannel class, and the target subfiles are named: target partition table name prefix_fixed length count.csv (such as staWebLog_10000001.csv, staWebLog_10000002.csv, ...). After splitting, the original target file is deleted.

[0095] In this embodiment, because there are many partition tables and the amount of data is also large, multi-threaded table paging query data is adopted to improve the query efficiency. Each page of data is written into a file and named with "table name prefix yyyyMMddHH_fixed length page number_number of records on the current page.csv" (yyyy represents year, MM represents month, dd represents day, and HH represents hour), so that the order of query data can be guaranteed. After all paging data in a table are queried, the data files of each page are merged into one file and named with "table name prefix yyyyMMddHH_current table record number.csv". After the data query of all tables is completed, at this point, the data of each table has a file. Since the file sizes at this time are very different, in order to make each file size basically the same, these files need to be merged and split. Here, the FileChannel class is used to merge and split the files (split by fixed size when splitting), which reduces the number of data copies between the operating system kernel and the user, thereby improving the efficiency of file operations.

[0096] In this embodiment, the partition table data processing method can obtain the target partition table name to be exported according to the data export request; generate a unique key value associated with the data export request and a corresponding target folder for storing the exported data according to the data export request; adopt the parallel processing method of the first thread pool to write the target partition tables corresponding to all the target partition table names associated with the data export request into the corresponding partition files, thereby improving the data export efficiency, shortening the data export time, and being able to effectively implement memory management, reduce the risk of memory overflow, and improve data processing performance; by first merging all the partition files in the target folder into a target file, and then splitting the target file into target sub-files of fixed length, the consistency of all exported data files is ensured, and the risk of data confusion is avoided.

[0097] In a preferred embodiment, the partition table data processing method may further include:

[0098] S5. compress the target sub-file and save it in the target folder.

[0099] In this step, the target sub-files after splitting are compressed and saved for downloading, which also saves disk space. The compressed target sub-file name is: target partition table name prefix_yyyyMMddHHmmssSSS_unique key value exported this time.zip (such as staWebLog_20240520110431529_fd5784a0195242958c0c5256f70f9d13.zip). Delete other temporary files. Record the download path of the compressed file and feedback to the user.

[0100] The partition table data processing method of this embodiment can be used for exporting data of tens of millions of large partition tables, with high export efficiency; it can save a large number of data files and slice the files into fixed sizes to ensure that the files will not be too large to be opened. The advantages of the partition table data processing method are: using parallel processing technology to query and process data, using the FileChannel class to process data files, significantly improving the export efficiency; having an effective memory management mechanism to reduce the risk of memory overflow and improve performance; when exporting a large amount of data, it can ensure the consistency and integrity of the data and avoid the risk of data confusion; using data compression technology to reduce data storage space and data transmission time.

[0101] like Figure 3 As shown, a partition table data processing device 1 may include: an acquisition unit 11, a generation unit 12, a writing unit 13, a merging unit 14 and a splitting unit 15, wherein:

[0102] The acquisition unit 11 is used to acquire the name of the target partition table to be exported according to the data export request;

[0103] It should be noted that: the data export request includes the export time period and the target partition table name prefix.

[0104] Furthermore, the acquisition unit 11 can query a database to acquire a target partition table name that matches the export time period and the prefix of the target partition table name.

[0105] In this embodiment, the data export request can be generated based on the export conditions, export time period, and partition table name to be exported selected / entered by the user on the export page. Considering that the partition table name is divided into hours, the target partition table names that meet the export time period and export conditions are screened from the target partition table names, that is, the screened target partition table names are partition tables that meet the export time range. The matching target partition table names are saved in the tableList component container.

[0106] A generating unit 12, configured to generate a key value and a corresponding target folder associated with the data export request according to the data export request;

[0107] It should be noted that: the data export request may also include the current time.

[0108] Further, the generating unit 12 may be used to generate the unique key value according to the data export request; create the target folder associated with the target partition table name and the storage path corresponding to the target folder according to the key value and the current time;

[0109] Wherein, the format of the target folder is current time_key value;

[0110] In this embodiment, the storage path is associated with the target partition table name prefix.

[0111] In this embodiment, a unique key value is generated for each data export request as the temporary directory name of the downloaded file to distinguish each export request; at the same time, the current time of this export is recorded to prevent newly inserted data from affecting subsequent export results.

[0112] Storage path filePath: / a path of the server / prefix of the target partition table name / yyyyMMddHHmmssSSS_unique key value exported this time. All subsequent file operations in this embodiment are related to this storage path. The writing unit 13 is used to start the first thread pool, and write the target partition tables corresponding to all the target partition table names associated with the data export request into corresponding partition files respectively. The partition files are stored in the target folder;

[0113] In this embodiment, the first thread pool pool1 is started to cyclically process the target partition tables corresponding to all target partition table names in the tableList component container, and each first thread processes one target partition table, so that the threads do not interfere with each other and the queried data will not be confused.

[0114] In this embodiment, each first thread in the first thread pool corresponds to at least one target partition table; the writing unit 13 can be used to write the corresponding target partition table into the corresponding partition file through the first thread. The writing unit 13 can query the total number of records in the target partition table according to the data export request; determine whether the total number of records is greater than or equal to the number threshold, if the total number of records is less than the number threshold, write the record data in the target partition table into the corresponding partition file through the first thread; if the total number of records is greater than or equal to the number threshold, calculate the number of record pages according to the total number of records and the number threshold, and paginate the data in the target partition table according to the number of record pages; start the second thread pool, write all the paging record data in the target partition table into the corresponding shard files, the shard files are stored in the target folder, each paging is associated with a shard file, and all the shard files are merged to generate the partition file.

[0115] In this embodiment, the FileChannel class is used to write the record data into the corresponding partition file (the file is named "target partition table name prefix yyyyMMddHH_current table record number.csv", such as staweblog2024052011_300.csv") and saved to filePath. The partition file stores the data of the current target partition table. When the total number of records exceeds the number of single-page records pageSize, the number of record pages pageCount is calculated based on the total number of records totalCount and the number of single-page records pageSize. The data in the target partition table is paginated according to the number of record pages and the number of records threshold.

[0116] The second thread pool pool2 is started to process the record data of each page in a loop. First, the record data is queried according to the export conditions, current page number, number threshold, export time period, and target partition table name selected / entered by the user on the export page. Each record data queried is a string separated by "," so that it can be written into the csv file. Then the found record data is written into the corresponding fragment file. The fragment file name is "target partition table name prefix yyyyMMddHH_fixed length page number_number of records on the current page.csv", where yyyy represents the year, MM represents the month, dd represents the day, and HH represents the hour (such as staweblog2024052006_10000001_20000.csv. The file is named in this way to ensure the order of data when merging fragment files). This fragment file is a fragment file of the record data in the current target partition table name.

[0117] In this embodiment, due to the large amount of data, the exported files are all in csv format. The csv format is simple, each line represents a piece of data, and the fields are separated by commas. In this way, when querying, the data can be directly assembled into such a format without secondary processing. csv is a plain text format, which is very convenient for writing files or merging and splitting files. And csv can also be presented to users in the form of a table for easy reference.

[0118] The FileChannel class is used to merge all the fragment files to generate the partition file.

[0119] In the embodiment, after the second threads in the second thread pool pool2 are processed, the FileChannel class is used to merge all the fragment files of the current target partition table name into one partition file, that is, the data of the current target partition table name, and each target partition table name corresponds to a partition file (such as staweblog2024052008_1099065.csv). And delete the corresponding fragment files to save disk space.

[0120] A merging unit 14, configured to merge all the partition files in the target folder into a target file;

[0121] The splitting unit 15 is used to split the target file into target sub-files of fixed length.

[0122] Furthermore, the FileChannel class is used to merge all the partition files in the target folder into a target file; and the FileChannel class is used to split the target file into target sub-files of fixed length.

[0123] In this embodiment, after the first threads in the first thread pool pool1 are processed, the partition files of all target partition tables are merged into a csv target file using the FileChannel class, and the merged partition files are deleted; the target file is then split according to a fixed size using the FileChannel class, and the target subfiles are named: target partition table name prefix_fixed length count.csv (such as staWebLog_10000001.csv, staWebLog_10000002.csv, ...). After splitting, the original target file is deleted.

[0124] In this embodiment, because there are many partition tables and the amount of data is also large, multi-threaded table paging query data is adopted to improve the query efficiency. Each page of data is written into a file and named with "table name prefix yyyyMMddHH_fixed length page number_number of records on the current page.csv" (yyyy represents year, MM represents month, dd represents day, and HH represents hour), so that the order of query data can be guaranteed. After all paging data in a table are queried, the data files of each page are merged into one file and named with "table name prefix yyyyMMddHH_current table record number.csv". After the data query of all tables is completed, at this point, the data of each table has a file. Since the file sizes at this time are very different, in order to make each file size basically the same, these files need to be merged and split. Here, the FileChannel class is used to merge and split the files (split by fixed size when splitting), which reduces the number of data copies between the operating system kernel and the user, thereby improving the efficiency of file operations.

[0125] In this embodiment, the acquisition unit 11 in the partition table data processing device 1 can obtain the target partition table name to be exported according to the data export request; the generation unit 12 generates a unique key value associated with the data export request and a corresponding target folder for storing the exported data according to the data export request; the writing unit 13 adopts the parallel processing method of the first thread pool to write the target partition tables corresponding to all target partition table names associated with the data export request into the corresponding partition files, thereby improving the data export efficiency, shortening the data export time, and being able to effectively implement memory management, reduce the risk of memory overflow, and improve data processing performance; first, all the partition files in the target folder are merged into a target file through the merging unit 14, and then the target file is split into target sub-files of fixed length through the splitting unit 15, thereby ensuring the consistency of all exported data files and avoiding the risk of data confusion.

[0126] like Figure 4 As shown, a computer device 2, the computer device 2 includes:

[0127] A memory 21, used for storing executable program codes; and

[0128] The processor 22 is used to call the executable program code in the memory 21, and the execution steps include the above-mentioned partition table data processing method.

[0129] Figure 4 A processor 22 is taken as an example.

[0130] The memory 21 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the partition table data processing method in the embodiment of the present application (for example, Figure 3 The acquisition unit 11, the generation unit 12, the writing unit 13, the merging unit 14 and the splitting unit 15 are shown. The processor 22 executes various functional applications and data processing of the computer device 2 by running the non-volatile software programs, instructions and modules stored in the memory 21, that is, the method for generating the voice barrage in the above method embodiment is realized.

[0131] The memory 21 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and at least one application required for the function; the data storage area may store the user's voice input request on the computer device 2, convert the voice information into text information, generate barrage information, send the barrage information, the client identification or the voice inputter's identity identification, etc. In addition, the memory 21 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 21 may optionally include a memory 21 remotely arranged relative to the processor 22, and these remote memories 21 may be connected to the partition table data processing device 1 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0132] The one or more modules are stored in the memory 21, and when executed by the one or more processors 22, execute the partition table data processing method in any of the above method embodiments, for example, execute the partition table data processing method described above. Figure 1 Steps S1 to S4 of the method, Figure 2 Steps S31 to S36 of the method are implemented Figure 3 The functions of the acquisition unit 11, the generation unit 12, the writing unit 13, the merging unit 14 and the splitting unit 15 are shown.

[0133] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0134] The computer device 2 of the embodiment of the present application exists in various forms, including but not limited to:

[0135] (1) Mobile communication devices: These devices are characterized by their mobile communication functions and their main purpose is to provide voice and data communications. These terminals include: smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones.

[0136] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, such as iPad.

[0137] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players (such as iPods), handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0138] (4) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, system bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0139] (5) Other electronic devices with data interaction functions.

[0140] The embodiment of the present application provides a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, such as Figure 4 A processor 22 in the embodiment may enable the one or more processors 22 to execute the partition table data processing method in any of the above method embodiments, for example, to execute the above described Figure 1 Steps S1 to S4 of the method, Figure 2 Steps S31 to S36 of the method are implemented Figure 3 The functions of the acquisition unit 11, the generation unit 12, the writing unit 13, the merging unit 14 and the splitting unit 15 are shown.

[0141] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on at least two network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present application. Those of ordinary skill in the art may understand and implement them without creative effort.

[0142] Through the description of the above implementation methods, a person of ordinary skill in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A partition table data processing method, characterized in that: include: In response to a data export request, obtaining a target partition table name to be exported; Generating a key value and a corresponding target folder associated with the data export request according to the data export request; Starting a first thread pool, writing target partition tables corresponding to all target partition table names associated with the data export request into corresponding partition files respectively, and storing the partition files in the target folder; All the partition files in the target folder are merged into a target file, and then the target file is split into target sub-files of fixed length.

2. The partition table data processing method according to claim 1, characterized in that: The step of obtaining the name of the target partition table to be exported in response to the data export request includes: The data export request includes an export time period and a target partition table name prefix; The database is queried to obtain a target partition table name that matches the export time period and the prefix of the target partition table name.

3. The partition table data processing method according to claim 2, characterized in that: Generating a key value associated with the data export request and a corresponding target folder according to the data export request includes: The data export request also includes the current time; Generate a unique key value according to the data export request; Creating the target folder associated with the target partition table name and a storage path corresponding to the target folder according to the key value and the current time; The format of the target folder is current time_key value; The storage path is associated with the target partition table name prefix.

4. The partition table data processing method according to claim 1, characterized in that: The starting of the first thread pool, writing the target partition tables corresponding to all target partition table names associated with the data export request into corresponding partition files respectively, comprises: Each first thread in the first thread pool corresponds to at least one target partition table; The corresponding target partition table is written into the corresponding partition file through the first thread.

5. The partition table data processing method according to claim 4, characterized in that: Writing the corresponding target partition table into the corresponding partition file through the first thread includes: Query the total number of records in the target partition table according to the data export request; Determine whether the total number of records is greater than or equal to a threshold number; If the total number of records is less than the number threshold, writing the record data in the target partition table into the corresponding partition file through the first thread; If the total number of records is greater than or equal to the record number threshold, the number of record pages is calculated according to the total number of records and the record number threshold, and the data in the target partition table is paging according to the record page number; Start the second thread pool, write the record data of all pages in the target partition table into corresponding slice files respectively, the slice files are stored in the target folder, and each page is associated with one slice file; All the slice files are merged to generate the partition file.

6. The partition table data processing method according to claim 5, characterized in that: The starting of the second thread pool to write all the paging record data in the target partition table into the corresponding shard files respectively includes: Each second thread in the second thread pool corresponds to at least one page; The record data of the corresponding pages are written into the associated fragment files respectively through the second thread.

7. The partition table data processing method according to claim 5, characterized in that: The step of merging all the slice files to generate the partition file includes: The FileChannel class is used to merge all the fragment files to generate the partition file.

8. The partition table data processing method according to claim 1, characterized in that: The step of merging all the partition files in the target folder into a target file, and then splitting the target file into target sub-files of fixed lengths, comprises: Using the FileChannel class to merge all the partition files in the target folder into a target file; The target file is split into target sub-files of fixed length using the FileChannel class.

9. The partition table data processing method according to claim 1, characterized in that: Also includes: The target sub-file is compressed and saved in the target folder.

10. A partition table data processing device, characterized in that: include: An acquisition unit, used for acquiring the name of the target partition table to be exported according to the data export request; A generating unit, configured to generate a key value and a corresponding target folder associated with the data export request according to the data export request; A writing unit, used to start a first thread pool, and write the target partition tables corresponding to all target partition table names associated with the data export request into corresponding partition files respectively, and the partition files are stored in the target folder; A merging unit, used for merging all the partition files in the target folder into a target file; The splitting unit is used to split the target file into target sub-files of fixed length.

11. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 9 when executing the computer program.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images