File processing method and device, equipment and storage medium

Abstract

Embodiments of the invention provide a file processing method and apparatus, a device and a storage medium. The method comprises the steps of obtaining a to-be-processed file; analyzing the to-be-processed file to obtain file data; performing row verification and column verification on the file data to obtain verified file data; the checked file data has a preset batch processing data volume; according to a preset batch processing data volume, generating a data set according to the checked file data; counting the file data in the data set to obtain target counting data; screening file data in the data set to obtain target file data; and storing the target counting data and the target file data. According to the embodiment of the invention, asynchronous thread submission support is provided for links of file acquisition, processing, storage and the like, and the speed of a data processing link is effectively improved by realizing decoupling design of file discovery, processing, writing and index counting, so that a more standard, stable, efficient and controllable file processing method and scheme are provided.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a file processing method and device, an electronic device and a storage medium. Background Art

[0002] In the related FTP (File Transfer Protocol) file processing technology, the operation is usually carried out in the form of full-process single-threaded offline periodic scheduling. This processing method can ensure the stability and reliability of the system because it avoids the resource competition and data consistency problems that may be caused by multi-threaded concurrent operations, but it also brings obvious disadvantages of insufficient timeliness.

[0003] Specifically, due to the single-threaded processing mechanism, the system must schedule according to a predetermined cycle when processing massive files, which means that the file processing time is limited by the length of the cycle and cannot achieve instant response. For those application scenarios that require real-time processing of a large number of FTP files, this traditional processing method obviously cannot meet the needs. For example, in the fields of financial transactions, real-time monitoring, data analysis, etc., the real-time processing capability of files directly affects the efficiency and accuracy of the business.

[0004] Therefore, how to improve the timeliness of FTP file processing to meet real-time requirements has become an urgent problem to be solved. Summary of the invention

[0005] The embodiment of the present application provides a file processing method to solve the problem of how to improve the timeliness of FTP file processing to meet the real-time requirements.

[0006] Correspondingly, an embodiment of the present application also provides a file processing device, an electronic device and a storage medium to ensure the implementation and application of the above method.

[0007] In order to solve the above problems, an embodiment of the present application discloses a file processing method, which is applied to a server, and the method includes:

[0008] Get the files to be processed;

[0009] Parsing the file to be processed to obtain file data;

[0010] Performing row verification and column verification on the file data to obtain verified file data; the verified file data has a preset batch processing data volume;

[0011] According to the preset batch processing data volume, generating a data set based on the verified file data;

[0012] Counting the file data in the data set to obtain target counting data;

[0013] Filtering the file data in the data set to obtain target file data;

[0014] The target count data and the target file data are stored.

[0015] In an optional embodiment of the present application, the server includes an original directory, a file processing directory and a file queue, the original directory includes the to-be-processed file, and obtaining the to-be-processed file includes:

[0016] Acquire the to-be-processed file from the original directory, where the to-be-processed file has a corresponding file name;

[0017] Move the to-be-processed file to the file processing directory;

[0018] Write the file name to the file queue.

[0019] In an optional embodiment of the present application, the server includes a file processor, the file processor includes a plurality of processing sub-threads, and the parsing of the to-be-processed file includes:

[0020] Controlling the processing subthread to obtain a target file name from the file queue;

[0021] According to the target file name, obtain the target file to be processed from the file processing directory;

[0022] The target file to be processed is parsed in the processing sub-thread.

[0023] In an optional embodiment of the present application, the to-be-processed file includes a structured file, the file data includes a key field and a primary key identifier, and the performing of row verification and column verification on the file data to obtain the verified file data includes:

[0024] Parsing the file to be processed to obtain the structure file;

[0025] Performing row verification on the file data using the structured file to obtain row-verified file data;

[0026] Filter the file data whose key fields are empty from the file data after the row verification to obtain abnormal file data;

[0027] The primary key identifier of the abnormal file data is set to a null value to obtain the verified file data.

[0028] In an optional embodiment of the present application, the server includes a data queue, the data queue includes the verified file data, and after the data set is generated according to the verified file data in accordance with the preset batch processing data volume, the method further includes:

[0029] Detecting the amount of remaining file data in the data queue;

[0030] If the amount of the remaining file data does not change, and the amount of the remaining file data is not zero, then updating the number of detection pass times;

[0031] When the number of detection passes reaches a preset number threshold, the data queue is completed according to the preset batch processing data volume to obtain a completed data queue;

[0032] Acquire the remaining file data in the completed data queue;

[0033] The data set is generated according to the remaining file data.

[0034] In an optional embodiment of the present application, the file data has a corresponding data type, and counting the file data in the data set to obtain target count data includes:

[0035] Counting the file data according to the data type to obtain count data of the data type;

[0036] Determine whether the count data of the data type reaches a preset count threshold;

[0037] When the count data of the data type reaches the preset count threshold, the filtering of the file data of the data type in the data set is stopped.

[0038] In an optional embodiment of the present application, the method further includes:

[0039] Determine whether there are any remaining files to be processed in the file processing directory;

[0040] If there are the retained files to be processed in the file processing directory, the retained files to be processed are transferred to the original directory.

[0041] The present application also discloses a file processing device, which is applied to a server. The device includes:

[0042] A module for obtaining files to be processed, used for obtaining files to be processed;

[0043] A file parsing module, used for parsing the file to be processed to obtain file data;

[0044] A file data verification module, used for performing row verification and column verification on the file data to obtain verified file data; the verified file data has a preset batch processing data volume;

[0045] A data set generation module, used to generate a data set according to the preset batch processing data volume and the verified file data;

[0046] A data counting module, used for counting the file data in the data set to obtain target counting data;

[0047] A data screening module, used for screening the file data in the data set to obtain target file data;

[0048] A data storage module is used to store the target counting data and the target file data.

[0049] An embodiment of the present application further discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes the file processing method as described in one or more of the embodiments of the present application.

[0050] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes the file processing method as described in one or more of the embodiments of the present application.

[0051] Compared with the prior art, the embodiments of the present application have the following advantages:

[0052] In an embodiment of the present application, a file to be processed is obtained; the file to be processed is parsed to obtain file data; row and column checks are performed on the file data to obtain the checked file data; the checked file data has a preset batch processing data volume; a data set is generated according to the checked file data according to the preset batch processing data volume; the file data in the data set is counted to obtain target count data; the file data in the data set is filtered to obtain target file data; and the target count data and the target file data are stored. The embodiments of the present application provide asynchronous thread submission support for file acquisition, processing, storage and other links respectively, and by realizing the decoupling design of file discovery, processing, writing and indicator counting, the speed of the data processing link is effectively improved, and the critical line configuration between the storage medium performance and the thread in the storage link is balanced, thereby providing a more standard, stable, efficient and controllable file processing method and solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a flowchart of the steps of an embodiment of a file processing method of the present application;

[0054] Figure 2 is a system structure diagram of an embodiment of a file processing method of the present application;

[0055] Figure 3 is a flow chart of an embodiment of a file processing method of the present application;

[0056] Figure 4 is a structural block diagram of an embodiment of a file processing device of the present application;

[0057] Figure 5 It is a schematic diagram of the structure of a device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0059] Reference Figure 1 , is a flowchart of a file processing method embodiment of the present application, comprising the following steps:

[0060] Step 101: Obtain the file to be processed.

[0061] A file processing method shown in an embodiment of the present application can be applied to a server, which may include an FTP service. The FTP service is a file storage service that can store file type data.

[0062] In one embodiment, before step 101, a thread pool (pool) created using an unbounded blocking queue LinkedBlockingDeque (a class in the Java concurrent package) can be constructed using ThreadPoolExecutor (a class in the Python standard library for managing a thread pool), and the number of thread pools can be dynamically configured using a configuration file. In an embodiment of the present application, the thread pool can implement file discovery, processing, writing, and indicator counting by providing child thread support.

[0063] In the embodiment of the present application, the file monitor, processor, data monitor, tail data monitor and index monitor can be constructed by the thread pool to realize the file discovery, processing, writing and index counting in the embodiment of the present application. Among them, the file monitor, data monitor, tail data monitor and index monitor can adopt the asynchronous single-thread mode to ensure the stability of the processing process, thereby avoiding abnormal situations caused by the overload of the file server and data writing carrier thread processing; the processor can adopt the asynchronous multi-thread mechanism to meet the data timeliness requirements of real-time applications.

[0064] In one embodiment, before step 101, Apache common-pool2 (a library for object pooling) can also be used to build a connection pool (fileConnectionPool) for reading files, and the number of connections in the connection pool can be dynamically configured using a configuration file. In an embodiment of the present application, the thread pool can support asynchronous sub-threads for file discovery, processing, writing, and indicator counting by providing a connection with an FTP server.

[0065] In step 101, the to-be-processed file may be acquired by using an interval polling method through a file monitor using an asynchronous sub-thread method. In one embodiment, the to-be-processed file may be a ZIP file (compressed file).

[0066] Step 102: parse the file to be processed to obtain file data.

[0067] In step 102, the file to be processed may be parsed by a processor using an asynchronous multi-threading method, thereby obtaining file data.

[0068] In one embodiment, the number of threads required by the processor can be calculated using the following formula (1). According to formula (1), the number of threads of the processor can be adjusted in real time according to the number of users and QPS (Queries Per Second) in actual scenarios.

[0069]

[0070] Among them, P s Set a value for the number of processor threads; T n is the number of tasks; QPS is the number of concurrent tasks at the same time; P n is the number of users; F n is the number of file data; Δt is the time interval for file generation.

[0071] In one embodiment, the number of file data items can be set to a default value of 5000 or less, and the number of tasks can be set to a default value of no more than 100. For the file generation time interval, assuming that the processing time of a single piece of data is 10 ms (milliseconds), the default value can be set to 30×60×(1000 / 10), that is, the file generation time interval is set according to the amount of file data that can be processed within 30 minutes.

[0072] Step 103: Perform row verification and column verification on the file data to obtain verified file data; the verified file data has a preset batch processing data volume.

[0073] In step 103, the processor may perform row verification and column verification on the parsed file data. The row verification needs to detect the data accuracy, and the column verification needs to screen the data availability and matching degree. The verified file data has a preset batch processing data volume. When the number of verified file data meets the preset batch processing data volume, it will trigger the generation of a data set based on the verified file data, and further process the verified file data in the data set.

[0074] In one embodiment, in step 102 and step 103, if a non-logical abnormal situation occurs, the file to be processed needs to be re-parsed for a second time.

[0075] In one embodiment, the server may include a storage directory save, and before or after parsing the files to be processed, the files to be processed may be moved to save for storage, so as to facilitate subsequent historical storage and file processing auditing.

[0076] Step 104: Generate a data set based on the verified file data according to the preset batch processing data volume.

[0077] In step 104, a data monitor using an asynchronous sub-thread method can be used to detect whether the number of file data after verification meets the preset batch processing data volume. If the number of file data after verification meets the preset batch processing data volume, a data set is generated according to the verified file data according to the preset batch processing data volume. Subsequently, the file data will be counted, screened, and written in batches according to the data set, that is, the data monitor processes the file data of one data set each time.

[0078] For example, if the preset batch processing data volume is 3000, and the number of file data after verification is greater than or equal to 3000, the preset batch processing data volume is met, and a data set is generated based on the verified file data, each of which has 3000 data. Step 105: Count the file data in the data set to obtain target count data.

[0079] Step 105: Count the file data in the data set to obtain target count data.

[0080] In step 105, before writing the file data in the data set to storage, the data monitor also needs to count the file data in the data set to obtain target count data. Specifically, in one embodiment, a key-value index cache count can be performed to count the number of file data of different data types and the number of abnormal file data.

[0081] Step 106: Filter the file data in the data set to obtain target file data.

[0082] In step 106, before writing the file data in the data set into storage, the data monitor needs to screen the file data to screen out abnormal file data, that is, file data with unreasonable logic, and remove the abnormal file data to obtain the target file data.

[0083] Step 107: Store the target counting data and the target file data.

[0084] In step 107, the target count data is stored by using an indicator monitor in an asynchronous sub-thread manner, that is, the target count data is superimposed on the historical count data, and the historical count data may include the count data for the day and the total historical count data.

[0085] For the target file data, a data monitor is used to write the target file data into the storage medium doris (Apache Doris, an analytical database) in the form of streamLoad. In one embodiment, if there is a non-logical abnormal error in step 106 or step 107, the data set needs to be re-screened and re-written.

[0086] In an embodiment of the present application, a file to be processed is obtained; the file to be processed is parsed to obtain file data; row and column checks are performed on the file data to obtain the checked file data; the checked file data has a preset batch processing data volume; a data set is generated according to the checked file data according to the preset batch processing data volume; the file data in the data set is counted to obtain target count data; the file data in the data set is filtered to obtain target file data; and the target count data and the target file data are stored. The embodiments of the present application provide asynchronous thread submission support for file acquisition, processing, storage and other links respectively, and by realizing the decoupling design of file discovery, processing, writing and indicator counting, the speed of the data processing link is effectively improved, and the critical line configuration between the storage medium performance and the thread in the storage link is balanced, thereby providing a more standard, stable, efficient and controllable file processing method and solution.

[0087] In an optional embodiment of the present application, the server includes an original directory, a file processing directory and a file queue, the original directory includes the to-be-processed file, and step 101 includes:

[0088] Acquire the to-be-processed file from the original directory, where the to-be-processed file has a corresponding file name;

[0089] Move the to-be-processed file to the file processing directory;

[0090] Write the file name to the file queue.

[0091] In this embodiment, an unbounded blocking queue LinkedBlockingDeque can be used to construct a file queue (fileQueue); a file processing directory consumer can also be created to isolate it from the original directory, so that file monitoring scanning and file processing can be performed simultaneously.

[0092] Specifically, in this embodiment, when using a file monitor to obtain a file to be processed, the following steps may be included.

[0093] The thread pool is used to asynchronously submit the child thread, and the ftpClient (a class used to interact with the FTP server) is obtained through the connection pool. The original directory is then scanned by interval polling. The ftpClient obtains the to-be-processed file with the file name from the original directory, moves the to-be-processed file to the file processing directory consumer, and writes the file name corresponding to the to-be-processed file into the file queue fileQueue.

[0094] In one embodiment, when moving the to-be-processed files to the file processing directory, a micro-delay process may be added to reserve sufficient time for the to-be-processed files to flow. The larger the to-be-processed files are, the more time may be reserved for the micro-delay process.

[0095] In one embodiment, if the movement of the pending file to the file processing directory fails due to a non-logical error, a secondary movement of the pending file can be triggered, and the file name is written into the file queue. After the file name is written, it is returned to the ftpClient.

[0096] The embodiment of the present application ensures the isolation of processes such as file data generation, processing, and historical retention by formulating a hierarchical decoupling design for directories such as original directories and file processing directories, thereby realizing a dynamic link for effectively monitoring file data movement.

[0097] In an optional embodiment of the present application, the server includes a file processor, and the file processor includes a plurality of processing sub-threads. Step 102 includes:

[0098] Controlling the processing subthread to obtain a target file name from the file queue;

[0099] According to the target file name, obtain the target file to be processed from the file processing directory;

[0100] The target file to be processed is parsed in the processing sub-thread.

[0101] In this embodiment, parsing the to-be-processed file by a processor using an asynchronous multi-threading method may specifically include the following steps.

[0102] Since the processor adopts the form of asynchronous multithreading, the processor includes a number of processing sub-threads, and the number of threads can be calculated according to the above formula (1). According to the calculated number of threads, the thread pool is used to asynchronously submit a corresponding number of processing sub-threads, each processing sub-thread obtains the target file name from the file queue, and the connection pool obtains the target to-be-processed file corresponding to the target file name in the file processing directory according to the ftpClient.

[0103] Parse the target file to be processed in the processing subthread. In one embodiment, if the target file to be processed is a ZIP file, the file data needs to be read and parsed by an IO (Input / Output) stream. Since the ZIP file includes a structure file and a data file, the IO stream needs to extract the internal file name of the ZIP file, and parse the structure file and the data file separately according to the file type.

[0104] The processor in the embodiment of the present application adopts an asynchronous multi-threading mechanism, which can improve the efficiency of file processing and ensure that the requirements of real-time applications for data timeliness are met.

[0105] In an optional embodiment of the present application, the to-be-processed file includes a structured file, the file data includes a key field and a primary key identifier, and step 103 includes:

[0106] Parsing the file to be processed to obtain the structure file;

[0107] Performing row verification on the file data using the structured file to obtain row-verified file data;

[0108] Filter the file data whose key fields are empty from the file data after the row verification to obtain abnormal file data;

[0109] The primary key identifier of the abnormal file data is set to a null value to obtain the verified file data.

[0110] In this embodiment, if the file to be processed is a ZIP file, since the ZIP file includes a structure file and a data file, the row check and column check of the file to be processed may specifically include the following steps.

[0111] The file to be processed is parsed to obtain a structure file and a data file. After obtaining the file data from the data file, the file data in the data file is checked according to the structure file to verify the accuracy of the data.

[0112] After that, the file data after row verification is verified by column verification. The file data with empty key fields are filtered out from the file data. These file data are abnormal file data. The primary key identifier of the abnormal file data needs to be set to a null value, so as to mark the abnormal file data to obtain the verified file data. Subsequently, the abnormal file data can be filtered out according to the characteristics of the primary key identifier being a null value for counting, elimination, and other operations.

[0113] In one embodiment, after verification is completed, the verified file data may be converted into a usable object and written into a data queue to wait for subsequent processing.

[0114] The embodiment of the present application can perform row verification and column verification on the parsed file data, thereby screening abnormal file data in the file data.

[0115] In an optional embodiment of the present application, the server includes a data queue, and the data queue includes the verified file data. After step 104, the method further includes:

[0116] Detecting the amount of remaining file data in the data queue;

[0117] If the amount of the remaining file data does not change, and the amount of the remaining file data is not zero, then updating the number of detection pass times;

[0118] When the number of detection passes reaches a preset number threshold, the data queue is completed according to the preset batch processing data volume to obtain a completed data queue;

[0119] Acquire the remaining file data in the completed data queue;

[0120] The data set is generated according to the remaining file data.

[0121] In the embodiment of the present application, since the number of file data after inspection in the data queue meets the preset batch processing data volume, it will trigger the generation of a data set based on the inspected file data and perform subsequent processing on the data set. If there is tail data in the data queue that does not meet the preset batch processing data volume, the generation of the data set and subsequent steps cannot be triggered. For example, if there are 10,000 file data in the data queue, and the preset batch processing data volume is 3,000, then after generating 3 data sets according to the preset batch processing data volume, there will be 1,000 file data left in the data queue. At this time, since 1,000 do not meet the preset batch processing data volume, the generation of the data set and subsequent steps cannot be triggered. These 1,000 data are called tail data.

[0122] This embodiment sets a tail data monitor by adopting an asynchronous sub-thread mode to trigger subsequent processing of the tail data, which may specifically include the following steps.

[0123] The thread pool is used to asynchronously submit a child thread to detect the number of remaining file data in the data queue. If the number of remaining file data does not change and the number of remaining file data is not zero, the number of detection passes is updated. In one embodiment, the data queue can be detected several times, with each detection interval being a preset time.

[0124] When the number of detection passes reaches a preset threshold, the data queue is completed according to the preset batch processing data volume to obtain a completed data queue. In one example, if each detection satisfies that the number of remaining file data has not changed and the number of remaining file data is not zero, the data queue is completed according to the preset batch processing data volume, wherein the data queue can be completed with empty elements or preset elements. For example, if the preset batch processing data volume is 3000 and the remaining file data is 1000, the elements in the data queue can be completed to 3000, thereby triggering the generation of a data set and subsequent steps.

[0125] After the data queue is completed, the remaining file data in the completed data queue is obtained, and when the data monitor is polled and monitored, a data set is generated based on the remaining file data and merged for subsequent processing.

[0126] The embodiment of the present application can complete the elements of the remaining file data in the data queue, thereby triggering the generation of a data set and subsequent processing steps on the data set.

[0127] In an optional embodiment of the present application, the file data has a corresponding data type, and step 105 includes:

[0128] Counting the file data according to the data type to obtain count data of the data type;

[0129] Determine whether the count data of the data type reaches a preset count threshold;

[0130] When the count data of the data type reaches the preset count threshold, the filtering of the file data of the data type in the data set is stopped.

[0131] In this embodiment, an indicator monitor is set by adopting an asynchronous sub-thread method to count file data, which may specifically include the following steps.

[0132] The thread pool is used to asynchronously submit child threads. When the indicator monitor is first started, the historical count data is loaded into the cache, and the daily data volume of file data of a single data type exceeds the preset count threshold, and the access to file data of this data type is stopped.

[0133] For different data types, the file data is counted to obtain the count data of different data types, and it is determined whether the count data of the data type reaches a preset count threshold. When the count data of the data type reaches the preset count threshold, the filtering of the file data of the data type in the data set is stopped, that is, the access to the file data of the data type is stopped.

[0134] For file data of data types that do not reach the preset count threshold, the database statistics can be updated uniformly based on the key-value indicator cache.

[0135] In one embodiment, abnormal file data may be screened out for counting based on the characteristic that the primary key identifier is a null value.

[0136] In an optional embodiment, the indicator cache of the indicator monitor can be implemented using static area memory. However, this method has some problems, especially in terms of statistics on the amount of data for the day. Specifically, the count will be lost after the program is restarted, and the counting time may be misplaced. In order to solve these problems, Redis (RemoteDictionary Server, a data structure storage system) can also be used instead of static area memory to implement the indicator cache. As an independent storage component, Redis can effectively isolate changes in internally stored data when the program is restarted, thereby avoiding data loss. In addition, Redis's key-value storage structure is very suitable for such scenarios. By using the time of the day as the key and setting the TTL (Time To Live) to one day, it can ensure that the correct value is obtained when querying the amount of data for the day, thereby avoiding the problem of misaligned query time.

[0137] The embodiment of the present application can count file data for different data types and determine whether it is necessary to stop accessing file data of the data type. It can flexibly control the amount of file data access for each data type to avoid data overload or exceeding processing capacity.

[0138] In an optional embodiment of the present application, the method further includes:

[0139] Determine whether there are any remaining files to be processed in the file processing directory;

[0140] If there are the retained files to be processed in the file processing directory, the retained files to be processed are transferred to the original directory.

[0141] In this embodiment, an FTP service monitoring mechanism task can also be added. Specifically, a file processing directory is scanned once a day through crontab (a tool for periodically executing tasks on Unix and Linux systems) to determine whether there are any remaining files to be processed in the file processing directory. If there are any remaining files to be processed in the file processing directory, the remaining files to be processed are transferred to the original directory and wait for the file monitor to obtain them again.

[0142] The embodiment of the present application determines whether there are any remaining files to be processed in the file processing directory. If there are any remaining files to be processed in the file processing directory, the remaining files to be processed are transferred to the original directory, thereby solving the problem of FTP file parsing thread blocking.

[0143] In an embodiment of the present application, a file to be processed is obtained; the file to be processed is parsed to obtain file data; row and column checks are performed on the file data to obtain the checked file data; the checked file data has a preset batch processing data volume; a data set is generated according to the checked file data according to the preset batch processing data volume; the file data in the data set is counted to obtain target count data; the file data in the data set is filtered to obtain target file data; and the target count data and the target file data are stored. The embodiments of the present application provide asynchronous thread submission support for file acquisition, processing, storage and other links respectively, and by realizing the decoupling design of file discovery, processing, writing and indicator counting, the speed of the data processing link is effectively improved, and the critical line configuration between the storage medium performance and the thread in the storage link is balanced, thereby providing a more standard, stable, efficient and controllable file processing method and solution.

[0144] On the basis of the above embodiments, in order to enable those skilled in the art to have a deeper understanding of an embodiment of a file processing method of the present application, the following Figure 2 , Figure 3 A file processing method shown in an embodiment of the present application is explained.

[0145] Reference Figure 2 , shows a system structure diagram of an embodiment of a file processing method of the present application.

[0146] like Figure 2 As shown, the embodiment of the present application may include a file monitor, a processor, a data monitor, a tail data monitor, and an index monitor. The file monitor monitors the file changes of the FTP in real time and obtains the files to be processed; the processor processes the files to be processed in the file queue in real time and writes the results into the data queue; the data monitor and the tail data monitor write the data in the data queue into a database or other data carrier; the index monitor performs index statistics on the file data in the data queue.

[0147] Reference Figure 3 , is a flow chart of an embodiment of a file processing method of the present application.

[0148] like Figure 3 As shown, an embodiment of the present application may include a file monitor, a processor, a data monitor, a tail data monitor, and an index monitor. At the start (Start), the file monitor polls and listens to the FTP source data folder, and adds the pending files to the file queue. The processor polls and listens to the file queue, parses the pending files in the file queue, obtains the file data, and encapsulates the file data verification into the data queue. The data monitor polls and listens to the data queue, obtains the file data, and writes the file data to the storage medium. The tail monitor can fill the data queue. The index monitor counts the file data in the data monitor, and performs index caching, polls and listens to the index cache, writes the index to the storage medium, and ends (End) the processing of the file.

[0149] In an embodiment of the present application, a file to be processed is obtained; the file to be processed is parsed to obtain file data; row and column checks are performed on the file data to obtain the checked file data; the checked file data has a preset batch processing data volume; a data set is generated according to the checked file data according to the preset batch processing data volume; the file data in the data set is counted to obtain target count data; the file data in the data set is filtered to obtain target file data; and the target count data and the target file data are stored. The embodiments of the present application provide asynchronous thread submission support for file acquisition, processing, storage and other links respectively, and by realizing the decoupling design of file discovery, processing, writing and indicator counting, the speed of the data processing link is effectively improved, and the critical line configuration between the storage medium performance and the thread in the storage link is balanced, thereby providing a more standard, stable, efficient and controllable file processing method and solution.

[0150] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0151] On the basis of the above embodiments, this embodiment further provides a file processing device, which is applied to electronic devices such as terminal devices and servers.

[0152] Reference Figure 4, shows a structural block diagram of an embodiment of a file processing device of the present application, which may specifically include the following modules:

[0153] The to-be-processed file acquisition module 401 is used to acquire the to-be-processed file;

[0154] A file parsing module 402, used to parse the file to be processed to obtain file data;

[0155] The file data verification module 403 is used to perform row verification and column verification on the file data to obtain verified file data; the verified file data has a preset batch processing data volume;

[0156] A data set generation module 404, configured to generate a data set according to the preset batch processing data volume and the verified file data;

[0157] The data counting module 405 is used to count the file data in the data set to obtain target counting data;

[0158] A data screening module 406 is used to screen the file data in the data set to obtain target file data;

[0159] The data storage module 407 is used to store the target counting data and the target file data.

[0160] In an optional embodiment of the present application, the server includes an original directory, a file processing directory and a file queue, the original directory includes the to-be-processed files, and the to-be-processed file acquisition module 401 includes:

[0161] The original directory scanning submodule is used to obtain the to-be-processed file from the original directory, where the to-be-processed file has a corresponding file name;

[0162] A to-be-processed file moving submodule, used for moving the to-be-processed file to the file processing directory;

[0163] The file name writing submodule is used to write the file name into the file queue.

[0164] In an optional embodiment of the present application, the server includes a file processor, the file processor includes a plurality of processing sub-threads, and the file parsing module 402 includes:

[0165] A target file name acquisition submodule, used for controlling the processing subthread to acquire a target file name from the file queue;

[0166] A target to-be-processed file acquisition submodule is used to acquire the target to-be-processed file from the file processing directory according to the target file name;

[0167] The target to-be-processed file parsing submodule is used to parse the target to-be-processed file in the processing subthread.

[0168] In an optional embodiment of the present application, the to-be-processed file includes a structured file, the file data includes a key field and a primary key identifier, and the file data verification module 403 includes:

[0169] A structure file acquisition submodule is used to parse the file to be processed to obtain the structure file;

[0170] A row check submodule, used to perform row check on the file data using the structure file to obtain the file data after row check;

[0171] A column check submodule, used for screening the file data whose key fields are empty from the file data after the row check, to obtain abnormal file data;

[0172] The primary key identifier setting submodule is used to set the primary key identifier of the abnormal file data to a null value to obtain the verified file data.

[0173] In an optional embodiment of the present application, the server includes a data queue, the data queue includes the verified file data, and the device further includes:

[0174] A file quantity detection module, used to detect the quantity of remaining file data in the data queue;

[0175] A detection number updating module, configured to update the detection pass number if the amount of the remaining file data does not change and the amount of the remaining file data is not zero;

[0176] A queue completion module, used for completing the data queue according to the preset batch processing data volume to obtain a completed data queue when the number of detection passes reaches a preset number threshold;

[0177] A remaining file acquisition module, used to acquire the remaining file data in the completed data queue;

[0178] The remaining file processing module is used to generate the data set according to the remaining file data.

[0179] In an optional embodiment of the present application, the file data has a corresponding data type, and the data counting module 405 includes:

[0180] A file data counting submodule, for counting the file data according to the data type, to obtain count data of the data type;

[0181] A counting threshold judgment submodule, used to judge whether the counting data of the data type reaches a preset counting threshold;

[0182] The stop acquisition submodule is used to stop screening the file data of the data type in the data set when the count data of the data type reaches the preset count threshold.

[0183] In an optional embodiment of the present application, the device further includes:

[0184] A retained file scanning submodule is used to determine whether there are retained files to be processed in the file processing directory;

[0185] The retained file transfer submodule is used to transfer the retained files to be processed to the original directory if there are the retained files to be processed in the file processing directory.

[0186] In an embodiment of the present application, a file to be processed is obtained; the file to be processed is parsed to obtain file data; row and column checks are performed on the file data to obtain the checked file data; the checked file data has a preset batch processing data volume; a data set is generated according to the checked file data according to the preset batch processing data volume; the file data in the data set is counted to obtain target count data; the file data in the data set is filtered to obtain target file data; and the target count data and the target file data are stored. The embodiments of the present application provide asynchronous thread submission support for file acquisition, processing, storage and other links respectively, and by realizing the decoupling design of file discovery, processing, writing and indicator counting, the speed of the data processing link is effectively improved, and the critical line configuration between the storage medium performance and the thread in the storage link is balanced, thereby providing a more standard, stable, efficient and controllable file processing method and solution.

[0187] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0188] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0189] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as terminal devices and servers (clusters). Figure 5 An exemplary apparatus 500 that can be used to implement various embodiments described in this application is schematically shown.

[0190] For one embodiment, Figure 5 An exemplary apparatus 500 is shown having one or more processors 502, a control module (chip set) 504 coupled to at least one of the processor(s) 502, a memory 506 coupled to the control module 504, a non-volatile memory (NVM) / storage device 508 coupled to the control module 504, one or more input / output devices 510 coupled to the control module 504, and a network interface 512 coupled to the control module 504.

[0191] The processor 502 may include one or more single-core or multi-core processors, and the processor 502 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 500 can be used as a terminal device, server (cluster), etc. described in the embodiments of the present application.

[0192] In some embodiments, the apparatus 500 may include one or more computer-readable media (e.g., memory 506 or NVM / storage device 508) having instructions 514 and one or more processors 502 configured in conjunction with the one or more computer-readable media to execute the instructions 514 to implement a module to perform the actions described in the present disclosure.

[0193] For one embodiment, control module 504 may include any suitable interface controller to provide any suitable interface to at least one of processor(s) 502 and / or any suitable device or component in communication with control module 504 .

[0194] The control module 504 may include a memory controller module to provide an interface to the memory 506. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0195] The memory 506 may be used, for example, to load and store data and / or instructions 514 for the device 500. For one embodiment, the memory 506 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 506 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0196] For one embodiment, control module 504 may include one or more input / output controllers to provide an interface to NVM / storage device 508 and input / output device(s) 510 .

[0197] For example, NVM / storage 508 may be used to store data and / or instructions 514. NVM / storage 508 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0198] NVM / storage device 508 may include storage resources that are physically part of the device on which apparatus 500 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 508 may be accessed via input / output device(s) 510 over a network.

[0199] (One or more) input / output devices 510 may provide an interface for the apparatus 500 to communicate with any other appropriate device, and the input / output device 510 may include a communication component, an audio component, a sensor component, etc. The network interface 512 may provide an interface for the apparatus 500 to communicate through one or more networks, and the apparatus 500 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0200] For one embodiment, at least one of the processor(s) 502 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 504. For one embodiment, at least one of the processor(s) 502 may be packaged together with the logic of one or more controllers of the control module 504 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 502 may be integrated on the same die with the logic of one or more controllers of the control module 504. For one embodiment, at least one of the processor(s) 502 may be integrated on the same die with the logic of one or more controllers of the control module 504 to form a system-on-chip (SoC).

[0201] In various embodiments, the apparatus 500 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the apparatus 500 may have more or fewer components and / or a different architecture. For example, in some embodiments, the apparatus 500 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0202] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0203] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0204] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0205] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable file processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable file processing terminal device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0206] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable file processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0207] These computer program instructions may also be loaded onto a computer or other programmable file processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0208] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0209] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0210] The above is a detailed introduction to a file processing method and device, an electronic device and a storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A file processing method, characterized in that: Applied to a server, the method comprises: Get the files to be processed; Parsing the file to be processed to obtain file data; Performing row verification and column verification on the file data to obtain verified file data; the verified file data has a preset batch processing data volume; According to the preset batch processing data volume, generating a data set based on the verified file data; Counting the file data in the data set to obtain target counting data; Filtering the file data in the data set to obtain target file data; The target count data and the target file data are stored.

2. The method according to claim 1, characterized in that The server includes an original directory, a file processing directory and a file queue, the original directory includes the to-be-processed files, and obtaining the to-be-processed files includes: Acquire the to-be-processed file from the original directory, where the to-be-processed file has a corresponding file name; Move the to-be-processed file to the file processing directory; Write the file name to the file queue.

3. The method according to claim 2, characterized in that The server includes a file processor, and the file processor includes a plurality of processing sub-threads. The parsing of the to-be-processed file includes: Controlling the processing subthread to obtain a target file name from the file queue; According to the target file name, obtain the target file to be processed from the file processing directory; The target file to be processed is parsed in the processing sub-thread.

4. The method according to claim 1, characterized in that: The to-be-processed file includes a structured file, the file data includes a key field and a primary key identifier, and the row check and column check are performed on the file data to obtain the checked file data, including: Parsing the file to be processed to obtain the structure file; Performing row verification on the file data using the structured file to obtain row-verified file data; Filter the file data whose key fields are empty from the file data after the row verification to obtain abnormal file data; The primary key identifier of the abnormal file data is set to a null value to obtain the verified file data.

5. The method according to claim 1, characterized in that The server includes a data queue, the data queue includes the verified file data, and after the preset batch processing data volume is followed and a data set is generated according to the verified file data, the method further includes: Detecting the amount of remaining file data in the data queue; If the amount of the remaining file data does not change, and the amount of the remaining file data is not zero, then updating the number of detection pass times; When the number of detection passes reaches a preset number threshold, the data queue is completed according to the preset batch processing data volume to obtain a completed data queue; Acquire the remaining file data in the completed data queue; The data set is generated according to the remaining file data.

6. The method according to claim 1, characterized in that The file data has a corresponding data type, and counting the file data in the data set to obtain target count data includes: Counting the file data according to the data type to obtain count data of the data type; Determine whether the count data of the data type reaches a preset count threshold; When the count data of the data type reaches the preset count threshold, the filtering of the file data of the data type in the data set is stopped.

7. The method according to claim 2, characterized in that The method further comprises: Determine whether there are any remaining files to be processed in the file processing directory; If there are the retained files to be processed in the file processing directory, the retained files to be processed are transferred to the original directory.

8. A file processing device, characterized in that: Applied to a server, the device comprises: A module for obtaining files to be processed, used for obtaining files to be processed; A file parsing module, used for parsing the file to be processed to obtain file data; A file data verification module, used for performing row verification and column verification on the file data to obtain verified file data; the verified file data has a preset batch processing data volume; A data set generation module, used to generate a data set according to the preset batch processing data volume and the verified file data; A data counting module, used for counting the file data in the data set to obtain target counting data; A data screening module, used for screening the file data in the data set to obtain target file data; A data storage module is used to store the target counting data and the target file data.

9. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, enables the processor to execute the file processing method as described in one or more of claims 1-7.

10. One or more machine-readable media having executable codes stored thereon, which, when executed, cause a processor to execute the file processing method according to one or more of claims 1-7.

Images