File data processing method and device

By analyzing configuration files and splitting large text files, and using thread pools to process them in parallel, the problem of poor processing performance of large text data in the existing technology is solved, efficient and flexible file data processing is achieved, and user experience is improved.

CN115114247BActive Publication Date: 2024-11-22BANK OF CHINA FINANCIAL TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210605685.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-30
Publication Date
2024-11-22
Estimated Expiration
2042-05-30

AI Technical Summary

Technical Problem

In the prior art, large text data processing methods have problems such as poor flexibility, poor performance and low efficiency, especially when processing large text files, resulting in poor data processing performance and poor user experience.

Method used

By obtaining the configuration file of batch files, parsing the file size and processing logic, and splitting the target file when the file size is greater than the preset threshold, using the thread pool to perform write and read operations in parallel, realizing file writing and reading Isolate from processing to improve the accuracy and efficiency of data processing.

Benefits of technology

It improves the flexibility, performance and efficiency of file data processing, is suitable for various batch files, enhances the universality and reuse rate of processing, and improves the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115114247B_ABST
    Figure CN115114247B_ABST
Patent Text Reader

Abstract

The present invention provides a file data processing method and device, the method comprising: obtaining a configuration file of a batch file, parsing the configuration information of each file from the configuration file; in the case where there is a target file whose file size is greater than a preset threshold in the batch file, splitting the target file to obtain multiple sub-files of the target file; calling multiple first threads from a thread pool, and based on each of the multiple first threads, writing the sub-file corresponding to each first thread into a cache queue; calling multiple second threads from the thread pool, and based on each of the multiple second threads, reading the sub-file corresponding to each second thread from the cache queue, and executing corresponding processing logic on the sub-file corresponding to each second thread; and obtaining the processing result of the target file according to the processing results of the multiple sub-files. The present invention effectively improves the flexibility, performance and efficiency of file data processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a file data processing method and device. Background Art

[0002] At present, many Internet companies are developing programs. Due to the complexity of business scenarios, there are more and more batch text downloads and interactions between systems, and more and more large text data are involved. In addition, the amount of text data is increasing, and the data content contained in a single text file is also increasing. For example, a txt text file contains more than 300G of data content, and the frequency of large text data being sent is also high, which places high demands on business timeliness. With the increase in business scenarios and the increasing frequency of business changes, more and more systems are connected, and the corresponding large texts in the system are also increasing. Therefore, in order to enable business personnel to view and use data quickly and efficiently, the demand for efficient and fast processing of large text data is becoming more and more urgent.

[0003] In the prior art, the processing method of large text data in program development is generally to write code manually, then read the text data one by one, and then process the currently read text data, and then save the result, and then use it for the next step of data processing. After the processing is completed, the next data is read, processed, and then the result is saved, and then used for the next step of data processing. Such serial processing one by one, the data processing performance is very slow, the processing is also very slow, resulting in poor data processing performance flexibility, poor performance, and low efficiency. Summary of the invention

[0004] The present invention provides a file data processing method and device to solve the defects of poor flexibility, poor performance and relatively low efficiency in the prior art of processing large text data by manually writing codes, so as to improve the flexibility, performance and efficiency of file data processing.

[0005] The present invention provides a file data processing method, comprising:

[0006] Obtaining a configuration file of a batch of files, and parsing configuration information of each file from the configuration file; the configuration information includes file size and processing logic;

[0007] If there is a target file whose file size is greater than a preset threshold in the batch file, split the target file to obtain multiple sub-files of the target file;

[0008] Calling a plurality of first threads from the thread pool, and based on each of the plurality of first threads, writing a sub-file corresponding to each of the plurality of first threads into a cache queue; the plurality of first threads are used to perform write operations in parallel;

[0009] Calling a plurality of second threads from the thread pool, based on each of the plurality of second threads, reading a sub-file corresponding to each second thread from the cache queue, and executing corresponding processing logic on the sub-file corresponding to each second thread; the plurality of second threads are used to execute the reading operation and the processing logic in parallel;

[0010] The processing result of the target file is obtained according to the processing results of the multiple sub-files.

[0011] According to a file data processing method provided by the present invention, the configuration file also includes thread pool parameters;

[0012] The calling of a plurality of first threads from the thread pool and, based on each of the plurality of first threads, writing a sub-file corresponding to each of the first threads into a cache queue comprises:

[0013] Determine the total number of the first threads in the thread pool according to the thread pool parameters;

[0014] Determining a target number of calling the first threads according to the total number of the first threads and the running status of each first thread;

[0015] The target number of first threads is called, and based on each first thread in the target number of first threads, a sub-file corresponding to each first thread is read, and the read sub-file is written into the cache queue.

[0016] According to a file data processing method provided by the present invention, calling the target number of first threads, based on each first thread in the target number of first threads, reading a sub-file corresponding to each first thread, and writing the read sub-file into the cache queue, comprises:

[0017] In the case that the target number is less than the number of sub-files to be written in the multiple sub-files, calling the target number of first threads, based on each first thread in the target number of first threads, reading the sub-file to be written corresponding to each first thread, writing the read sub-file to be written into the cache queue, and continuously monitoring the running status of each first thread in the thread pool and obtaining the remaining sub-files to be written in the multiple sub-files;

[0018] When it is determined that there is a target first thread in the thread pool whose running state is idle, the target first thread is called, the remaining sub-files to be written corresponding to the target first thread are read, and the remaining sub-files to be written are written into the cache queue until the multiple sub-files are written into the cache queue.

[0019] According to a file data processing method provided by the present invention, the method further includes:

[0020] Writing the processing result of the sub-file corresponding to each second thread into the storage file corresponding to each second thread;

[0021] According to the processing logic of each file in the batch files, obtaining files in the batch files having the same processing logic;

[0022] According to the storage files, the processing results corresponding to the files with the same processing logic are merged and written into the database;

[0023] Alternatively, according to the storage files, the processing results corresponding to the files with the same processing logic are written into the database in batches.

[0024] According to a file data processing method provided by the present invention, the step of obtaining a configuration file of a batch of files and parsing the configuration information of each file from the configuration file includes:

[0025] Obtaining a configuration file of the batch file, and verifying the correctness of the configuration file;

[0026] When the configuration file passes the verification, the configuration information of each file is parsed from the configuration file;

[0027] If the configuration file fails to pass the verification, the configuration file is updated according to the attribute information of the batch file, and the correctness of the updated configuration file is verified until the updated configuration file passes the verification;

[0028] When the updated configuration file passes the verification, the configuration information of each file is parsed from the updated configuration file.

[0029] According to a file data processing method provided by the present invention, splitting the target file to obtain multiple sub-files of the target file includes:

[0030] Determine the number of splits of the target file according to the preset threshold and the attribute information of the target file;

[0031] According to the split number, the target file is split to obtain a plurality of sub-files whose number is the split number;

[0032] Alternatively, the split size of the target file is determined according to the preset threshold and the attribute information of the target file;

[0033] The target file is split according to the split size to obtain the multiple sub-files whose file sizes are smaller than or equal to the split size; wherein the split size is smaller than the preset threshold.

[0034] According to a file data processing method provided by the present invention, the configuration information also includes a decompression state;

[0035] Before splitting the target file to obtain a plurality of sub-files of the target file, the method further includes:

[0036] Determine whether the decompression state of each file in the batch file is a to-be-decompressed state;

[0037] Decompression processing is performed on the files in the batch files whose decompression state is a to-be-decompressed state.

[0038] The present invention also provides a file data processing device, comprising:

[0039] A parsing module, used to obtain a configuration file of a batch of files, and parse the configuration information of each file from the configuration file; the configuration information includes the file size and processing logic;

[0040] A splitting module, configured to split the target file into multiple sub-files of the target file when there is a target file whose file size is larger than a preset threshold in the batch file;

[0041] A cache module, configured to call a plurality of first threads from a thread pool, and based on each of the plurality of first threads, write a sub-file corresponding to each of the plurality of first threads into a cache queue; the plurality of first threads are configured to perform write operations in parallel;

[0042] A processing module, configured to call a plurality of second threads from the thread pool, read a sub-file corresponding to each second thread from the cache queue based on each second thread in the plurality of second threads, and execute corresponding processing logic on the sub-file corresponding to each second thread; the plurality of second threads are configured to execute the reading operation and the processing logic in parallel;

[0043] The acquisition module is used to acquire the processing result of the target file according to the processing results of the multiple sub-files.

[0044] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the file data processing methods described above is implemented.

[0045] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the file data processing method described in any one of the above is implemented.

[0046] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the file data processing method described above is implemented.

[0047] The file data processing method and device provided by the present invention, on the one hand, perform different processing operations on each file in the batch file according to the configuration file of the batch file, so as to realize unified processing of multiple files in the batch file, and can be applicable to various batch files, with good versatility, high reuse rate and good flexibility; on the other hand, when there is a target file with a file size greater than a preset threshold in the batch file, multiple first threads are used to slice and concurrently write the target file into a cache queue; and a second thread is used to slice and concurrently read sub-files from the cache queue and perform logical processing on the sub-files, so as to effectively realize the isolation of file writing, file reading and file processing operations, and can improve the accuracy and reliability of file data processing, as well as the efficiency of file processing, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 It is a flowchart of the file data processing method provided by the present invention;

[0050] Figure 2 It is a structural schematic diagram of the file data processing device provided by the present invention;

[0051] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Currently, there is no configurable processing method for large files. Most of them implement file logic processing by writing code, which has very poor versatility, flexibility and efficiency, cannot process files quickly, and human factors have a great impact on the efficiency of large file processing.

[0054] Therefore, the existing technology implements file processing by writing code, which not only has a slow processing speed, but also requires writing different codes for different processing rules of large text data, which increases unnecessary workload and leads to inconsistent processing methods; and due to different business change scenarios, it is difficult to reuse processing methods, resulting in a low reuse rate of development and processing methods, low development efficiency, and also introduces quality risks to efficient data processing; in addition, when the amount of data text is too large, the processing efficiency is low, which increases the risk of parsing errors and delays, and also increases the workload of developers, resulting in a poor user experience.

[0055] In order to solve the problems existing in the prior art, the present embodiment provides a file data processing method, which obtains the configuration information of each file in the batch file by obtaining the configuration file of the batch file, and obtains the file size and processing logic of each file according to the configuration information. When the file is larger than a preset threshold, the file is split for parallel writing, split for parallel reading and logical processing, so as to realize the decoupling and asynchronous execution of the writing operation, the reading and logical processing operation, and effectively improve the flexibility, performance and efficiency of text data processing, and can be applied to various batch file processing, with good versatility and high reuse rate, which effectively improves the user experience.

[0056] It should be noted that the execution subject of the method may be an electronic device, a component or a chip in an electronic device. The electronic device may be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device may be a mobile phone, a tablet computer, a laptop computer, etc., and the non-mobile electronic device may be a server, a network attached storage, a personal computer, etc., which is not specifically limited in the present invention.

[0057] The file transmission method of this embodiment can be implemented based on JAVA language or C language.

[0058] Combine the following Figure 1 The file data processing method of the present invention is described, and the method comprises the following steps:

[0059] Step 101, obtaining a configuration file of a batch of files, and parsing the configuration information of each file from the configuration file; the configuration information includes the file size and processing logic;

[0060] The batch files are multiple files with the same processing logic or the same type, and may be batch files to be translated, batch files to be format updated, etc. This embodiment does not specifically limit this.

[0061] Each batch file corresponds to a configuration file. The format of the configuration file may be XML (Extensible Markup Language) format, or JOSN (JavaScript Object Notation) format or other formats, which is not specifically limited in this embodiment.

[0062] The configuration file contains configuration information of each file in the batch file. The configuration information includes the file size and processing logic, and may also include the file name and processing steps, etc. This embodiment does not specifically limit this.

[0063] The processing logic is the business logic that needs to be executed on the file, and the processing steps are the specific implementation steps for executing the business logic on the file.

[0064] The configuration file can be expanded according to actual needs, such as modifying, deleting, and adding configuration information.

[0065] Optionally, obtain a batch file to be processed, and obtain a configuration file of the batch file;

[0066] After obtaining the configuration file, the configuration file can be directly parsed to obtain the configuration information of each file; or the configuration file can be processed and then parsed to obtain the configuration information of each file, which is not specifically limited in this embodiment. The processing method of the configuration file includes but is not limited to verification or format conversion.

[0067] Step 102, when there is a target file whose file size is greater than a preset threshold in the batch file, split the target file to obtain multiple sub-files of the target file;

[0068] The preset threshold may be preset according to thread performance parameters in the thread pool or according to actual needs, which is not specifically limited in this embodiment, for example, it may be determined according to the transmission capacity of each thread in the thread pool.

[0069] The number of target files can be one or more, and the processing steps for each target file are the same. In order to simplify the description, the following takes one target file as an example to describe the file data processing method in this embodiment.

[0070] Optionally, after obtaining the configuration information of each file, the file size of each file is obtained from the configuration information.

[0071] The file size of each file is compared with a preset threshold to determine whether each file is a large file; if there is no target file with a file size larger than the preset threshold in the batch file, it indicates that the files in the batch file are all small and can be directly processed by multi-threading; if there is a target file with a file size larger than the preset threshold in the batch file, it indicates that there is a target file with large data content in the batch file, and the target file needs to be split into multiple sub-files.

[0072] Among them, the splitting method includes but is not limited to determining the number of splits according to a preset threshold and the file size of the target file, and splitting the file to be transferred into multiple sub-files according to the number of splits; or determining the split size according to a preset threshold and the file size, and splitting the file to be transferred into multiple sub-files.

[0073] Step 103, calling a plurality of first threads from the thread pool, and based on each of the plurality of first threads, writing a sub-file corresponding to each of the plurality of first threads into a cache queue; the plurality of first threads are used to perform write operations in parallel;

[0074] Among them, multithreading refers to the technology of implementing the concurrent execution of multiple threads from software or hardware. Computers with multithreading capabilities can execute more than one thread at the same time due to hardware support, thereby improving overall processing performance. Systems with this capability include symmetric multiprocessors, multi-core processors, and chip-level multiprocessing or simultaneous multithreading processors. In a program, these independently running program fragments are threads, and the concept of programming using them is multithreading. Computers with multithreading capabilities can execute more than one thread at the same time due to hardware support, thereby improving overall processing performance.

[0075] The thread pool is a form of multithreading that adds tasks to a queue and then automatically starts them after the thread is created. Thread pool threads are background threads. Each thread uses the default stack size, runs at the default priority, and is in a multithreaded apartment. If a thread is idle in managed code (such as waiting for an event), the thread pool can insert another auxiliary thread to keep all processors busy. If all thread pool threads are always busy, but the queue contains pending work, the thread pool will create another auxiliary thread after a period of time, but the number of threads will never exceed the maximum. Threads exceeding the maximum can be queued, but they will not start until other threads are completed.

[0076] The first thread is used to perform a write operation to write the file into the cache queue;

[0077] Optionally, after obtaining multiple target sub-files, multiple first threads may be called from the thread pool to write the corresponding sub-files to the cache queue in parallel;

[0078] The number of calls to the first thread may be determined according to the number of first threads in the thread pool and the running status of the first thread, as well as the number of sub-files that need to be written into the cache queue.

[0079] Step 104, calling a plurality of second threads from the thread pool, based on each of the plurality of second threads, reading a sub-file corresponding to each second thread from the cache queue, and executing corresponding processing logic on the sub-file corresponding to each second thread; the plurality of second threads are used to execute the reading operation and the processing logic in parallel;

[0080] The second thread is used to execute the reading operation and processing logic to read the file from the cache queue and perform corresponding processing logic on the file.

[0081] It should be noted that the first thread and the second thread asynchronously execute their respective corresponding operations.

[0082] Optionally, while calling the first thread, multiple second threads can be asynchronously called from the thread pool to perform parallel actions, so as to read the sub-files corresponding to each second thread from the cache queue, and execute corresponding processing logic on the sub-files corresponding to each second thread. After the processing logic is executed, the execution results of the sub-files corresponding to each second thread are saved.

[0083] The number of calls to the second thread may be determined according to the number of second threads in the thread pool and the running status of the second threads, as well as the number of sub-files in the cache queue that need to be read and logically processed.

[0084] In this embodiment, asynchronous calls are made to multiple first threads and multiple second threads to achieve isolation of file writing, file reading, and file processing operations, effectively avoiding the impact of abnormal operations on data processing and improving the accuracy and reliability of data processing; and, when multiple first threads process sub-file writing tasks in parallel, multiple second threads can also process sub-file reading tasks and processing tasks in parallel, effectively improving the efficiency of file data processing.

[0085] Step 105: Obtain the processing result of the target file according to the processing results of the multiple sub-files.

[0086] Optionally, when it is determined that multiple sub-files of the target file have been processed, processing results of the multiple sub-files are obtained, and the processing results of the multiple files are summarized to obtain the processing result of the target file.

[0087] The file data processing method in this embodiment, on the one hand, performs different processing operations on each file in the batch file according to the configuration file of the batch file, so as to realize unified processing of multiple files in the batch file, and can be applicable to various batch files, with good versatility, high reuse rate and good flexibility; on the other hand, when there is a target file with a file size greater than a preset threshold in the batch file, multiple first threads are used to slice and concurrently write the target file into a cache queue; and a second thread is used to slice and concurrently read sub-files from the cache queue and perform logical processing on the sub-files, so as to effectively realize the isolation of file writing, file reading and file processing operations, and can improve the accuracy and reliability of file data processing, as well as the efficiency of file processing, thereby improving the user experience.

[0088] In some embodiments, the configuration file also includes thread pool parameters; calling multiple first threads from the thread pool, and based on each first thread in the multiple first threads, writing the sub-file corresponding to each first thread into the cache queue, includes: determining the total number of the first threads in the thread pool according to the thread pool parameters; determining the target number of calls to the first threads according to the total number of the first threads and the running status of each first thread; calling the target number of first threads, and based on each first thread in the target number of first threads, reading the sub-file corresponding to each first thread, and writing the read sub-file into the cache queue.

[0089] The thread pool parameters include at least the total number of first threads and the total number of second threads;

[0090] Optionally, in step 103, the step of calling multiple first threads to write multiple sub-files into the cache queue includes:

[0091] First, the total number of first threads is obtained from the thread pool parameters, and the running status of each first thread in the thread pool is monitored;

[0092] Then, according to the total number of first threads in the thread pool and the running status of each first thread, the total number of first threads in the thread pool in an idle state is determined, and according to the total number of first threads in an idle state, the target number of calling first threads is determined;

[0093] Then, the target number of first threads are called to execute the write operation of the sub-files corresponding to each of the target number of first threads in parallel, so as to realize parallel reading of the sub-files corresponding to each of the first threads, and write the read sub-files into the cache queue in parallel.

[0094] Similarly, the steps of calling multiple second threads to read multiple sub-files and performing logical processing on the multiple sub-files in step 104 include:

[0095] First, the total number of second threads is obtained from the thread pool parameters, and the running status of each second thread in the thread pool is monitored;

[0096] Then, based on the total number of second threads in the thread pool and the running status of each second thread, the total number of second threads in the thread pool that are in an idle state is determined, and the number of second threads to be called is determined based on the total number of second threads in an idle state, so as to call a corresponding number of second threads, and execute the read and write operations and processing logic of sub-files corresponding to each second thread in the corresponding number of second threads in parallel, so as to read the sub-files corresponding to each second thread from the cache queue in parallel, and execute the corresponding processing logic on the read sub-files.

[0097] In this embodiment, different file processing strategies are adaptively determined according to the running status of each thread in the thread pool, and sub-files in the target file are segmented and write operations, read operations and processing logic are performed in parallel according to the corresponding file processing strategies, thereby effectively ensuring the effectiveness and accuracy of file processing and avoiding file data loss and file data processing abnormalities due to congestion.

[0098] In some embodiments, the calling of the target number of first threads, based on each first thread in the target number of first threads, reading the sub-file corresponding to each first thread, and writing the read sub-file into the cache queue, includes: when the target number is less than the number of sub-files to be written among the multiple sub-files, calling the target number of first threads, based on each first thread in the target number of first threads, reading the sub-file to be written corresponding to each first thread, and writing the read sub-file to be written into the cache queue, and continuously monitoring the running status of each first thread in the thread pool and obtaining the remaining sub-files to be written among the multiple sub-files; when it is determined that there is a target first thread in the thread pool whose running status is idle, calling the target first thread, reading the remaining sub-files to be written corresponding to the target first thread, and writing the read remaining sub-files to be written into the cache queue until the multiple sub-files are all written into the cache queue.

[0099] The sub-file to be written is a sub-file in the target file waiting to be written.

[0100] The remaining sub-files to be written are the sub-files to be written that remain after the current calling thread performs a file writing operation on the sub-files to be written in the target file.

[0101] Optionally, in step 103, the specific step of calling multiple first threads to write multiple sub-files into the cache queue further includes:

[0102] After determining the target number for calling the first thread, the target number can also be compared with the number of sub-files to be written in the target file. When the target number is greater than or equal to the number of sub-files to be written in the target file, multiple sub-files to be written can be written in parallel to the cache queue at one time. At this time, the first thread of the target number is directly called, and the corresponding sub-files to be written are written in parallel to the cache queue at one time based on the first thread of the target number.

[0103] When the target number is smaller than the number of sub-files to be written in the target file, multiple sub-files to be written cannot be written into the cache queue in parallel at one time; at this time, the target number of first threads is called first, and the corresponding sub-files to be written are written into the cache queue in parallel based on the target number of first threads; and the running status of each first thread in the thread pool is continuously monitored. When it is determined that there is a target first thread in the thread pool with an idle running status, the target first thread is called again to write the corresponding remaining sub-files to be written into the cache queue until all sub-files in the target file are written.

[0104] Similarly, in step 104, the specific steps of calling multiple second threads to read multiple sub-files and perform logical processing on the multiple sub-files also include:

[0105] After determining the target number of calls to the second thread, the target number of the second thread can also be compared with the number of sub-files to be processed in the cache queue. When the target number of the second thread is greater than or equal to the number of sub-files to be processed in the cache queue, multiple sub-files to be processed can be read out from the cache queue in parallel at one time, and corresponding processing logic can be executed on the read sub-files to be processed. At this time, the target number of second threads is directly called, and based on the target number of second threads, the corresponding sub-files to be processed are read out from the cache queue in parallel at one time and the corresponding processing logic is executed.

[0106] When the target number of second threads is smaller than the number of sub-files to be processed in the target file, it is impossible to read out multiple sub-files to be processed in parallel and execute corresponding processing logics in parallel at one time; at this time, the target number of second threads is first called, and based on the target number of second threads, the corresponding sub-files to be processed are read out from the cache queue in parallel and the corresponding processing logics are executed in parallel; and the running status of each second thread in the thread pool is continuously monitored. When it is determined that there is a target second thread in the thread pool with an idle running status, the target second thread is called again to read out the remaining corresponding sub-files to be processed from the cache queue and execute corresponding processing logics until all sub-files in the target file are processed.

[0107] The sub-file to be processed is a sub-file in the target file waiting to be processed.

[0108] The remaining sub-files to be processed are the sub-files to be processed that remain after the second thread is currently called to perform a file processing operation on the test processing file in the target file.

[0109] In this embodiment, different file processing strategies are adaptively determined based on the running status of each thread in the thread pool and the number of sub-files to be written in the target file and the number of sub-files to be processed in the cache queue. The sub-files in the target file are segmented and write operations, read operations and processing logic are performed in parallel according to the corresponding file processing strategies, thereby effectively ensuring the effectiveness and accuracy of file processing.

[0110] In some embodiments, the method further includes: writing the processing results of the sub-files corresponding to each second thread into the storage file corresponding to each second thread; obtaining the files with the same processing logic in the batch file according to the processing logic of each file in the batch file; merging the processing results corresponding to the files with the same processing logic according to the storage file and writing them into the database; or, writing the processing results corresponding to the files with the same processing logic into the database in batches according to the storage file.

[0111] Each second thread corresponds to a storage file.

[0112] Optionally, after the processing of the sub-file corresponding to each second thread is completed, the execution result of the sub-file corresponding to each second thread may be written into the storage file corresponding to each second thread based on each second thread;

[0113] Then, the processing logic of each file in the batch file is compared with each other to determine the files in the batch file with the same processing logic;

[0114] Then, the processing results corresponding to the files with the same processing logic are merged and written into the database or written into the database in batches. The specific writing step includes: determining the execution result corresponding to each sub-file in the files with the same processing logic from the storage file, merging the execution results corresponding to all sub-files in the files with the same processing logic and writing them into the database or writing them into the database in batches.

[0115] In this embodiment, files with the same processing logic can be saved in batches, effectively improving the file saving efficiency.

[0116] In some embodiments, obtaining the configuration file of the batch files and parsing the configuration information of each file from the configuration file includes: obtaining the configuration file of the batch files and verifying the correctness of the configuration file; if the configuration file passes the verification, parsing the configuration information of each file from the configuration file; if the configuration file fails the verification, updating the configuration file according to the attribute information of the batch files, and verifying the correctness of the updated configuration file until the updated configuration file passes the verification; if the updated configuration file passes the verification, parsing the configuration information of each file from the updated configuration file.

[0117] Optionally, the specific steps of parsing the configuration information of each file in executing step 101 include:

[0118] First, obtain the configuration file of the batch file. In order to ensure the correctness of the configuration file and thus the correctness of the parsing result, before parsing the configuration file, it is necessary to verify the correctness of each configuration information in the configuration file;

[0119] When all configuration information in the configuration file passes the verification, it is determined that the configuration information passes the verification; at this time, the configuration file can be directly parsed to obtain the configuration information of each file in the configuration file.

[0120] In the case that any configuration information in the configuration file fails to pass the verification, the configuration file is updated according to the attribute information of the batch file, and the correctness verification of the updated configuration file continues until all the configuration information in the updated configuration file passes the correctness verification.

[0121] Then, the configuration information of each file is parsed from the configuration file that has passed the correctness verification.

[0122] In this embodiment, when parsing the configuration file, the configuration file is correctly verified and corrected and updated, which effectively ensures the correctness of the configuration file and further ensures the correctness of the file processing.

[0123] In some embodiments, splitting the target file to obtain multiple sub-files of the target file includes: determining the number of splits of the target file according to the preset threshold and the attribute information of the target file; splitting the target file according to the number of splits to obtain multiple sub-files whose number is the number of splits; or determining the split size of the target file according to the preset threshold and the attribute information of the target file; splitting the target file according to the split size to obtain the multiple sub-files whose file size is less than or equal to the split size; wherein the split size is less than the preset threshold.

[0124] The attribute information of the target file includes but is not limited to the file size of the target file.

[0125] Optionally, the step of splitting the target file in step 102 specifically includes:

[0126] The number of splits of the target file is determined according to a preset threshold and the file size of the target file, and the target file is split according to the number of splits to split the target file into multiple sub-files whose data meets the number of splits.

[0127] Alternatively, the split size of the target file is determined according to a preset threshold and the file size of the target file, and according to the split size, the target file is split into a plurality of sub-files whose file sizes are less than or equal to the split size.

[0128] In this embodiment, according to the attribute information of the target file and the preset threshold value, the target file can be quickly and accurately split into multiple sub-files, and then the target file can be quickly and effectively processed in parallel.

[0129] In some embodiments, the configuration information also includes a decompression status; before splitting the target file to obtain multiple sub-files of the target file, the method also includes: determining whether the decompression status of each file in the batch file is a state to be decompressed; and decompressing the files in the batch file whose decompression status is a state to be decompressed.

[0130] The configuration information also includes the decompression status of each file, which is used to indicate whether each file needs to be decompressed.

[0131] Optionally, before executing step 102, it is also necessary to determine whether the decompression state of each file is a to-be-compressed state. If the decompression state of any file is a to-be-decompressed state, the file is decompressed to obtain a decompressed file.

[0132] Steps 102 to 104 are executed according to the decompressed files to obtain the processing result of each file in the batch file.

[0133] The complete process of the file data processing method in this embodiment is specifically described below. The specific steps are as follows:

[0134] Step (1), obtaining a configuration file of a batch file, verifying the correctness of the configuration file and checking the configuration file;

[0135] Step (2), reading and parsing the configuration file that has passed the correctness verification to obtain the configuration information of each file; wherein the configuration file includes but is not limited to the file name, file size, decompression status and processing steps of each file, and thread pool parameters;

[0136] Step (3), judging the decompression status of each file in the batch file to determine whether a decompression operation needs to be performed on the file, and if a decompression operation is required, first decompressing the file;

[0137] Step (4), determining whether the file size of each file in the batch file is greater than a preset threshold, and splitting the target file that is greater than the preset threshold to obtain multiple sub-files of the target file;

[0138] Step (5), calling multiple first threads from the thread pool to perform the write operation in parallel, wherein each first thread reads a sub-file and writes it into the cache queue;

[0139] Step (6), calling multiple second threads from the thread pool to execute the reading operation and the processing operation in parallel, wherein each second thread reads a sub-file from the cache queue and then executes the related processing logic;

[0140] Step (7), after the processing is completed, save the processing results; specifically, as in steps (8) and (9);

[0141] Step (8), after each second thread completes processing of the corresponding sub-file, the processing result of the sub-file is written into the corresponding storage file; wherein each second thread corresponds to one storage file;

[0142] Step (9), based on the storage files, the processing results corresponding to the files with the same processing logic are written into the database in batches or merged and written into the database.

[0143] The existing technology relies on manual coding of each processing logic file one by one to achieve file processing. Not only is the processing performance relatively low and the data usage latency relatively high, but also manual coding is required for each file. The processing is neither flexible nor universal, and the development efficiency is low.

[0144] The file data processing method proposed in this embodiment can not only flexibly configure the file processing method according to the file size, but also realize asynchronous processing of files in steps, and for the writing task and file processing task of the file, it can realize concurrent processing of fragments, effectively reduce the delay time of file processing, and improve the efficiency and accuracy of file processing; in addition, the user only needs to call the configuration file according to the operation steps in the file data processing method to efficiently process large files, and can directly complete the development and use according to the operation steps, and can also apply the method to file processing of various types and various file sizes, and can be directly used without development, reducing a lot of unnecessary workload for developers; at the same time, it can also solve the problem of difficulty in unified processing of batch large text data, and can save the data in real time to realize fast and efficient processing of batch files, and the use of subsequent data is more and more efficient, and the delay is relatively low. And various processing logics can be configured in the configuration file, the processing logic is flexible, the versatility is relatively good, and the function reuse rate of development is relatively high.

[0145] The file data processing device provided by the present invention is described below. The file data processing device described below and the file data processing method described above can be referenced to each other.

[0146] like Figure 2 As shown, this embodiment provides a file data processing device, which includes a parsing module 201, a splitting module 202, a cache module 203, a processing module 204 and an acquisition module 205, wherein:

[0147] The parsing module 201 is used to obtain the configuration file of the batch file, and parse the configuration information of each file from the configuration file; the configuration information includes the file size and processing logic;

[0148] Optionally, obtain a batch file to be processed, and obtain a configuration file of the batch file;

[0149] After obtaining the configuration file, the configuration file can be directly parsed to obtain the configuration information of each file; or the configuration file can be processed and then parsed to obtain the configuration information of each file, which is not specifically limited in this embodiment. The processing method of the configuration file includes but is not limited to verification or format conversion.

[0150] The splitting module 202 is used for splitting the target file into multiple sub-files of the target file when there is a target file whose file size is larger than a preset threshold in the batch file;

[0151] Optionally, after obtaining the configuration information of each file, the file size of each file is obtained from the configuration information.

[0152] The file size of each file is compared with a preset threshold to determine whether each file is a large file; if there is no target file with a file size larger than the preset threshold in the batch file, it indicates that the files in the batch file are all small and can be directly processed by multi-threading; if there is a target file with a file size larger than the preset threshold in the batch file, it indicates that there is a target file with large data content in the batch file, and the target file needs to be split into multiple sub-files.

[0153] The cache module 203 is used to call a plurality of first threads from the thread pool, and based on each of the plurality of first threads, write the sub-file corresponding to each of the first threads into the cache queue; the plurality of first threads are used to perform the write operation in parallel;

[0154] Optionally, after obtaining multiple target sub-files, multiple first threads may be called from the thread pool to write the corresponding sub-files to the cache queue in parallel;

[0155] The number of calls to the first thread may be determined according to the number of first threads in the thread pool and the running status of the first thread, as well as the number of sub-files that need to be written into the cache queue.

[0156] The processing module 204 is used to call a plurality of second threads from the thread pool, read a sub-file corresponding to each second thread from the cache queue based on each second thread in the plurality of second threads, and execute corresponding processing logic on the sub-file corresponding to each second thread; the plurality of second threads are used to execute the reading operation and the processing logic in parallel;

[0157] Optionally, while calling the first thread, multiple second threads can be asynchronously called from the thread pool to perform parallel actions, so as to read the sub-files corresponding to each second thread from the cache queue, and execute corresponding processing logic on the sub-files corresponding to each second thread. After the processing logic is executed, the execution results of the sub-files corresponding to each second thread are saved.

[0158] The number of calls to the second thread may be determined according to the number of second threads in the thread pool and the running status of the second threads, as well as the number of sub-files in the cache queue that need to be read and logically processed.

[0159] The acquisition module 205 is used to acquire the processing result of the target file according to the processing results of the multiple sub-files.

[0160] Optionally, when it is determined that multiple sub-files of the target file have been processed, processing results of the multiple sub-files are obtained, and the processing results of the multiple files are summarized to obtain the processing result of the target file.

[0161] The file data processing method in this embodiment, on the one hand, performs different processing operations on each file in the batch file according to the configuration file of the batch file, so as to realize unified processing of multiple files in the batch file, and can be applicable to various batch files, with good versatility, high reuse rate and good flexibility; on the other hand, when there is a target file with a file size greater than a preset threshold in the batch file, multiple first threads are used to slice and concurrently write the target file into a cache queue; and a second thread is used to slice and concurrently read sub-files from the cache queue and perform logical processing on the sub-files, so as to effectively realize the isolation of file writing, file reading and file processing operations, and can improve the accuracy and reliability of file data processing, as well as the efficiency of file processing, thereby improving the user experience.

[0162] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3As shown, the electronic device may include: a processor (processor) 301, a communication interface (Communications Interface) 302, a memory (memory) 303 and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304. The processor 301 can call the logic instructions in the memory 303 to execute the file data processing method, which includes: obtaining a configuration file of a batch of files, and parsing the configuration information of each file from the configuration file; the configuration information includes the file size and processing logic; when there is a target file whose file size is greater than a preset threshold in the batch file, split the target file to obtain multiple sub-files of the target file; calling multiple first threads from the thread pool, and based on each first thread of the multiple first threads, writing the sub-file corresponding to each first thread into a cache queue; the multiple first threads are used to perform write operations in parallel; calling multiple second threads from the thread pool, and based on each second thread of the multiple second threads, reading the sub-file corresponding to each second thread from the cache queue, and executing corresponding processing logic for the sub-file corresponding to each second thread; the multiple second threads are used to perform read operations and processing logic in parallel; and obtaining the processing result of the target file according to the processing results of the multiple sub-files.

[0163] In addition, the logic instructions in the above-mentioned memory 303 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0164] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the file data processing method provided by the above methods, which includes: obtaining a configuration file of a batch of files, and parsing the configuration information of each file from the configuration file; the configuration information includes a file size and a processing logic; when there is a target file whose file size is greater than a preset threshold in the batch file, splitting the target file to obtain multiple sub-files of the target file; calling multiple first threads from a thread pool, and based on each first thread of the multiple first threads, writing the sub-file corresponding to each first thread into a cache queue; the multiple first threads are used to perform write operations in parallel; calling multiple second threads from the thread pool, and based on each second thread of the multiple second threads, reading the sub-file corresponding to each second thread from the cache queue, and executing corresponding processing logic on the sub-file corresponding to each second thread; the multiple second threads are used to perform read operations and processing logic in parallel; and obtaining the processing result of the target file according to the processing results of the multiple sub-files.

[0165] On the other hand, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it is implemented to execute the file data processing method provided by the above methods, the method comprising: obtaining a configuration file of a batch of files, and parsing the configuration information of each file from the configuration file; the configuration information includes the file size and processing logic; when there is a target file whose file size is greater than a preset threshold in the batch file, the target file is split to obtain multiple sub-files of the target file; calling multiple first threads from a thread pool, and based on each first thread of the multiple first threads, writing the sub-file corresponding to each first thread into a cache queue; the multiple first threads are used to perform write operations in parallel; calling multiple second threads from the thread pool, and based on each second thread of the multiple second threads, reading the sub-file corresponding to each second thread from the cache queue, and executing corresponding processing logic for the sub-file corresponding to each second thread; the multiple second threads are used to perform read operations and processing logic in parallel; and obtaining the processing result of the target file according to the processing results of the multiple sub-files.

[0166] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0167] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A file data processing method, characterized in that: include: Obtaining a configuration file of a batch of files, and parsing configuration information of each file from the configuration file; the configuration information includes file size and processing logic; If there is a target file whose file size is greater than a preset threshold in the batch file, split the target file to obtain multiple sub-files of the target file; Calling a plurality of first threads from the thread pool, and based on each of the plurality of first threads, writing a sub-file corresponding to each of the first threads into a cache queue; The multiple first threads are used to perform write operations in parallel; Calling a plurality of second threads from the thread pool, based on each of the plurality of second threads, reading a sub-file corresponding to each second thread from the cache queue, and executing corresponding processing logic on the sub-file corresponding to each second thread; The plurality of second threads are used to execute the reading operation and the processing logic in parallel; According to the processing results of the multiple sub-files, obtaining the processing result of the target file; Among them, the preset threshold is pre-set according to the thread performance parameters in the thread pool; the first thread and the second thread asynchronously execute their respective corresponding operations; the number of calls of the first thread is determined according to the number of threads of the first thread in the thread pool and the running status of the first thread, and the number of sub-files that need to be written into the cache queue; the number of calls of the second thread is determined according to the number of threads of the second thread in the thread pool and the running status of the second thread, and the number of sub-files that need to be read and logically processed in the cache queue.

2. The file data processing method according to claim 1, characterized in that: The configuration file also includes thread pool parameters; The calling of a plurality of first threads from the thread pool and, based on each of the plurality of first threads, writing a sub-file corresponding to each of the first threads into a cache queue comprises: Determine the total number of the first threads in the thread pool according to the thread pool parameters; Determining a target number of calling the first threads according to the total number of the first threads and the running status of each first thread; The target number of first threads is called, and based on each first thread in the target number of first threads, a sub-file corresponding to each first thread is read, and the read sub-file is written into the cache queue.

3. The file data processing method according to claim 2, characterized in that: The calling of the target number of first threads, based on each first thread in the target number of first threads, reading a sub-file corresponding to each first thread, and writing the read sub-file into the cache queue, comprises: In the case that the target number is less than the number of sub-files to be written in the multiple sub-files, calling the target number of first threads, based on each first thread in the target number of first threads, reading the sub-file to be written corresponding to each first thread, writing the read sub-file to be written into the cache queue, and continuously monitoring the running status of each first thread in the thread pool and obtaining the remaining sub-files to be written in the multiple sub-files; When it is determined that there is a target first thread in the thread pool whose running state is idle, the target first thread is called, the remaining sub-files to be written corresponding to the target first thread are read, and the remaining sub-files to be written are written into the cache queue until the multiple sub-files are written into the cache queue.

4. The file data processing method according to any one of claims 1 to 3, characterized in that: The method further comprises: Writing the processing result of the sub-file corresponding to each second thread into the storage file corresponding to each second thread; According to the processing logic of each file in the batch files, obtaining files in the batch files having the same processing logic; According to the storage files, the processing results corresponding to the files with the same processing logic are merged and written into the database; Alternatively, according to the storage files, the processing results corresponding to the files with the same processing logic are written into the database in batches.

5. The file data processing method according to any one of claims 1 to 3, characterized in that: The step of obtaining a configuration file of a batch of files and parsing configuration information of each file from the configuration file includes: Obtaining a configuration file of the batch file, and verifying the correctness of the configuration file; When the configuration file passes the verification, the configuration information of each file is parsed from the configuration file; If the configuration file fails to pass the verification, the configuration file is updated according to the attribute information of the batch file, and the correctness of the updated configuration file is verified until the updated configuration file passes the verification; When the updated configuration file passes the verification, the configuration information of each file is parsed from the updated configuration file.

6. The file data processing method according to any one of claims 1 to 3, characterized in that: The step of splitting the target file to obtain multiple sub-files of the target file includes: Determine the number of splits of the target file according to the preset threshold and the attribute information of the target file; According to the split number, the target file is split to obtain a plurality of sub-files whose number is the split number; Alternatively, the split size of the target file is determined according to the preset threshold and the attribute information of the target file; The target file is split according to the split size to obtain the multiple sub-files whose file sizes are smaller than or equal to the split size; wherein the split size is smaller than the preset threshold.

7. The file data processing method according to any one of claims 1 to 3, characterized in that: The configuration information also includes a decompression status; Before splitting the target file to obtain a plurality of sub-files of the target file, the method further includes: Determine whether the decompression state of each file in the batch file is a to-be-decompressed state; Decompression processing is performed on the files in the batch files whose decompression state is a to-be-decompressed state.

8. A file data processing device, characterized in that: include: A parsing module, used to obtain a configuration file of a batch of files, and parse the configuration information of each file from the configuration file; the configuration information includes the file size and processing logic; A splitting module, configured to split the target file into multiple sub-files of the target file when there is a target file whose file size is larger than a preset threshold in the batch file; A cache module, configured to call a plurality of first threads from a thread pool, and based on each of the plurality of first threads, write a sub-file corresponding to each of the first threads into a cache queue; The multiple first threads are used to perform write operations in parallel; A processing module, configured to call a plurality of second threads from the thread pool, read a sub-file corresponding to each second thread from the cache queue based on each second thread in the plurality of second threads, and execute corresponding processing logic on the sub-file corresponding to each second thread; The plurality of second threads are used to execute the reading operation and the processing logic in parallel; An acquisition module, used for acquiring a processing result of the target file according to the processing results of the multiple sub-files; Among them, the preset threshold is pre-set according to the thread performance parameters in the thread pool; the first thread and the second thread asynchronously execute their respective corresponding operations; the number of calls of the first thread is determined according to the number of threads of the first thread in the thread pool and the running status of the first thread, and the number of sub-files that need to be written into the cache queue; the number of calls of the second thread is determined according to the number of threads of the second thread in the thread pool and the running status of the second thread, and the number of sub-files that need to be read and logically processed in the cache queue.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the file data processing method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the file data processing method according to any one of claims 1 to 7 is implemented.