File processing method and related equipment

Abstract

The invention provides a file processing method and related equipment, and the method comprises the steps: periodically employing a plurality of threads in a preset timing scheduling thread pool and a preset SFTP connection channel, carrying out the state updating detection of each file in an SFTP file system in batches, collecting each target file every time when one or more target files with updated states are detected, and carrying out the collection of the target files. Determining a state updating type of each target file; when one or more target files are collected, establishing an asynchronous reading task corresponding to each target file; and executing each asynchronous reading task in parallel by utilizing a plurality of threads in a preset data processing thread pool to obtain file data of each target file, and calling corresponding processing logic to perform data processing on the file data of each target file on the basis of the state updating type of each target file, the purpose of improving the resource utilization efficiency and the file processing real-time performance is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data transmission, and in particular to a file processing method and related equipment. BACKGROUND

[0002] In the modern enterprise information environment, SFTP (Secure File Transfer Protocol, secure file transfer protocol) is widely used in the data exchange scene of the SFTP file system as a safe and reliable file transfer protocol. Enterprises usually need to monitor the file changes on the remote SFTP server in real time, and collect the newly added or modified data files in time for subsequent business processing.

[0003] The traditional remote file collection scheme is mainly implemented based on a timing polling mechanism: the client connects the SFTP server (i.e. I / O intensive SFTP file system) at a fixed time interval, scans the target directory, compares the file state, and downloads and processes the data after finding the changes. The traditional scheme has two main problems: 1. Low resource utilization efficiency: the fixed interval (such as every 10 minutes) polling mechanism will establish an SFTP connection and scan whether there is a file update on the remote server. Since the SFTP connection establishment requires TCP handshake, SSH key exchange, identity authentication and other multi-step operations, when the file update frequency is much lower than the polling frequency, a large number of invalid polling will cause significant network bandwidth waste and server resource consumption. 2. Insufficient real-time performance: mainly manifested in two aspects. On the one hand, the traditional scheme adopts a single-threaded sequential processing mode, and even if multiple files change at the same time, they can only be downloaded and processed one by one. When a large file is encountered, the entire monitoring process is blocked, causing the processing of other files to be severely delayed. On the other hand, the fixed interval polling strategy introduces an unavoidable detection delay. For example, with a 10-minute polling interval, the delay from the generation of a new file to its detection can be up to 10 minutes, and the average delay is 5 minutes. Shortening the polling interval can improve real-time performance, but will further exacerbate the resource waste problem, creating a contradiction between real-time performance and resource efficiency.

[0004] In summary, the existing technology has the problems of low resource utilization efficiency and insufficient real-time performance, which need to be solved urgently. SUMMARY

[0005] Therefore, the embodiments of the present application provide a file processing method and related equipment to improve resource utilization efficiency and file processing real-time performance.

[0006] To achieve the above object, the embodiments of the present application provide the following technical solutions:

[0007] The first aspect of the embodiments of the present application discloses a file processing method, which comprises:

[0008] Periodically, a plurality of threads in a preset timing scheduling thread pool and a pre-created SFTP connection channel are used to detect state update of each file in the SFTP file system in batches, and each target file that has a state update is collected and a state update type of each target file is determined;

[0009] An asynchronous reading task corresponding to each target file is established when one or more target files are collected;

[0010] A plurality of threads in a preset data processing thread pool are used to execute each asynchronous reading task in parallel to obtain file data of each target file, and a corresponding processing logic is called based on the state update type of each target file to perform data processing on the file data of each target file.

[0011] Optionally, periodically, a plurality of threads in a preset timing scheduling thread pool and a pre-created SFTP connection channel are used to detect state update of each file in the SFTP file system in batches, and each target file that has a state update is collected and a state update type of each target file is determined, including:

[0012] Periodically, a plurality of threads in a preset timing scheduling thread pool and a pre-created SFTP connection channel are used to obtain current state information of each file in the SFTP file system in batches, and for each file for which the current state information is obtained, the current state information is compared with historical state information stored locally to obtain a comparison result;

[0013] Based on the comparison result corresponding to each file in the SFTP file system, one or more target files that have a state update and a state update type corresponding to each target file are determined, and each target file is collected.

[0014] Optionally, the method further includes:

[0015] When the state update detection of each file in the SFTP file system is completed and the data processing on the file data of each collected target file is completed, the historical state information of each target file is updated.

[0016] Optionally, the plurality of threads in the preset data processing thread pool are used to execute each asynchronous reading task in parallel to obtain file data of each target file, including:

[0017] The plurality of threads in the preset data processing thread pool are used to perform the asynchronous reading tasks in parallel based on a streaming data processing technology to read each of the target files and obtain file data in the form of a character stream corresponding to each of the target files.

[0018] Optionally, the use of the plurality of threads in the preset data processing thread pool to perform the asynchronous reading tasks in parallel based on the streaming data processing technology to read each of the target files and obtain file data in the form of a character stream corresponding to each of the target files includes:

[0019] For each of the target files of the addition type, the plurality of threads in the preset data processing thread pool are used to perform the corresponding asynchronous reading tasks in parallel based on the streaming data processing technology to read all data in each of the target files of the addition type and create a corresponding file offset record, and obtain file data in the form of a character stream corresponding to each of the target files of the addition type; the file offset record is used to record content of the target file that has been read.

[0020] For each of the target files of the modification type, the plurality of threads in the preset data processing thread pool are used to perform the corresponding asynchronous reading tasks in parallel based on the streaming data processing technology and the file offset record corresponding to each of the target files of the modification type to read data that has been modified in each of the target files of the modification type and update the corresponding file offset record, and obtain file data in the form of a character stream corresponding to each of the target files of the modification type.

[0021] Optionally, the calling of the corresponding processing logic for data processing of the file data of each of the target files based on the state update type of each of the target files includes:

[0022] For each of the target files, the state update type of the target file and the file data of the target file are input to an event dispatcher for recording a plurality of pre-registered file event listeners, so that each of the state update types triggers a corresponding file event listener to use processing logic associated with the triggered file event listener to perform data processing on the file data of the target file.

[0023] Optionally, before the file data of the target file is input to the event dispatcher for recording a plurality of pre-registered file event listeners, the method further includes:

[0024] Convert the file data of the target file to obtain file data in a format meeting preset data processing requirements.

[0025] The second aspect of the embodiment of the application discloses a file processing system, and the system comprises:

[0026] The core control layer is configured to periodically use multiple threads in a preset timing scheduling thread pool and a pre-created SFTP connection channel to detect the state of each file in the SFTP file system in batches, collect one or more target files whenever a state update occurs, and determine the state update type of each target file; and establish an asynchronous reading task corresponding to each target file whenever one or more target files are collected.

[0027] The data processing layer is configured to use multiple threads in a preset data processing thread pool to execute each asynchronous reading task in parallel, obtain the file data of each target file, and call a corresponding processing logic based on the state update type of each target file to perform data processing on the file data of each target file.

[0028] The third aspect of the embodiment of the application discloses an electronic device, which comprises:

[0029] The memory is configured to save a computer program.

[0030] The processor is configured to execute the computer program to implement the method according to any one of the first aspect of the embodiment of the application.

[0031] The fourth aspect of the embodiment of the application discloses a computer readable storage medium configured to save a computer program; wherein the computer program is executed by a processor to implement the method according to any one of the first aspect of the embodiment of the application.

[0032] Based on the above-mentioned file processing method and related equipment provided by the embodiment of the application, the state update detection is periodically performed on each file in the SFTP file system by using multiple threads in the preset timing scheduling thread pool and the pre-created SFTP connection channel, whenever one or more target files with state update are detected, each target file is collected, and the state update type of each target file is determined; whenever one or more target files are collected, an asynchronous reading task corresponding to each target file is established; multiple threads in the preset data processing thread pool are used to perform each asynchronous reading task in parallel, file data of each target file is obtained, and based on the state update type of each target file, a corresponding processing logic is called to perform data processing on the file data of each target file. In the present scheme, the state update detection is periodically performed on each file in the SFTP file system by using the pre-created SFTP connection channel, and multiple SFTP connections do not need to be created, thereby improving the resource utilization efficiency; the double-thread pool asynchronous parallel architecture is used to replace the existing single-thread serial processing, the asynchronous parallel of file state detection and file data processing is realized, and the speed and real-time performance of file processing are improved. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0034] Figure 1 The flow chart of the file processing method disclosed by the embodiment of the present application;

[0035] Figure 2 The architecture diagram of the file processing system disclosed by the embodiment of the present application;

[0036] Figure 3 The running flow chart of the file processing system disclosed by the embodiment of the present application;

[0037] Figure 4 The structural diagram of the electronic device disclosed by the embodiment of the present application. DETAILED DESCRIPTION

[0038] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below, obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0039] In the present application, the term "comprising" or "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes the elements inherent to such process, method, article or equipment. Without more limitation, the element defined by the sentence "including a" does not exclude the presence of other identical elements in the process, method, article or equipment including the element.

[0040] As known from the background, the existing technology has the problems of low resource utilization efficiency and insufficient real-time performance, which are urgent problems to be solved at present.

[0041] Therefore, the embodiments of the present application disclose a file processing method and related equipment, in the present scheme, the state update detection of each file in the SFTP file system is periodically performed by using the pre-created SFTP connection channel, without the need to create SFTP connection multiple times, the resource utilization efficiency is improved; the existing single-thread serial processing is replaced by the double-thread pool asynchronous parallel architecture, the asynchronous parallel of file state detection and file data processing is realized, and the speed and real-time performance of file processing are improved.

[0042] As shown in FIG. 1, it is a flowchart of a file processing method disclosed by the embodiments of the present application, mainly including the following steps: Figure 1

[0043] Step S101: periodically using multiple threads in a preset timing scheduling thread pool and a pre-created SFTP connection channel, batch processing state update detection of each file in the SFTP file system, whenever one or more target files are detected to have state update, collecting each target file, and determining the state update type of each target file.

[0044] ​In step S101, a timing scheduling thread pool scheduler is created in advance, which is responsible for periodic SFTP file system scanning tasks (i.e. state update detection of each file in the SFTP file system in batches), and the number of threads is set to a small fixed value to ensure stable execution of the timing task. The thread pool is implemented by using a scheduled execution service (ScheduledExecutorService), which can accurately control the scanning interval and avoid task accumulation and resource competition.

[0045] The number of threads determines the number of files for which state update detection is performed in each batch.

[0046] The connection management efficiency of the prior art is low: the strategy of re-establishing connection each time cannot reuse the SFTP connection. The connection establishment process involves complex cryptographic calculations (RSA key exchange, AES encryption and decryption initialization, etc.), and frequent connection establishment and disconnection not only waste local computing resources, but also generate a large amount of connection logs and session management overhead on the remote server (SFTP file system).

[0047] Therefore, in the embodiment of the application, an SFTP connection channel is created in advance to realize pool management of the SFTP connection, and the connection establishment overhead is greatly reduced by using connection reuse technology.

[0048] In the specific implementation process of step S101, the current state information of each file in the SFTP file system is obtained in batches by using the plurality of threads in the preset timing scheduling thread pool and the pre-created SFTP connection channel, and for each file for which the current state information is obtained, the corresponding current state information is compared with the locally stored historical state information to obtain a comparison result.

[0049] Based on the comparison result corresponding to each file in the SFTP file system, one or more target files for which state update occurs and the state update type corresponding to each target file are determined, and each target file is collected.

[0050] It should be noted that the application uses an incremental detection algorithm based on a timestamp to determine whether a file has changed by comparing the last modification time of the file.

[0051] In the specific implementation, a file state mapping table (Map<String,Long>) is maintained to store the last modification timestamp of each file (i.e. the locally stored historical state information).

[0052] The file state mapping table is implemented by using a thread-safe concurrent hash mapping (ConcurrentHashMap) to support high-concurrency read-write operations. The key of the mapping table is the full path name of a file, and the value is the last modification timestamp of the file (in milliseconds).

[0053] The state update detection process is divided into three stages:

[0054] Current state information acquisition: the current state information of all files in the remote directory is acquired through an SFTP connection. Historical state information comparison: the current state information is compared with the historical state information in the file state mapping table. Change identification: whether the current detected file is a target file with a state update is identified and determined according to the comparison result, and if so, the state update type is determined, including addition, modification, deletion, and the like.

[0055] It can be understood that if the target file is a deleted file, since the file data has been deleted, the content of the target file collected only contains the information (such as the name) of the target file, and the information of the target file is taken as the file data during subsequent data processing, and corresponding data processing (such as cleaning related state information and read file data) is performed.

[0056] Step S102: whenever one or more target files are collected, an asynchronous reading task corresponding to each target file is established.

[0057] In step S102, to solve the task coordination problem in multi-file parallel processing, the asynchronous task arrangement technology is used. Each file reading task is encapsulated as a complete asynchronous reading task (CompletableFuture) to support non-blocking parallel execution.

[0058] In each round of scanning, all target files that need to be processed are collected, and an asynchronous reading task corresponding to each target file is established, and these asynchronous reading tasks are uniformly submitted to the data processing thread pool.

[0059] Optionally, the asynchronous task arrangement technology also supports task-level exception isolation: the failure of a single target file corresponding asynchronous reading task will not affect the normal processing of other target file corresponding asynchronous reading tasks. When the asynchronous reading task fails, the exception information is recorded and the remaining asynchronous reading tasks are processed, improving the robustness of the system.

[0060] Step S103: using multiple threads in the preset data processing thread pool, each asynchronous reading task is executed in parallel to obtain the file data of each target file, and based on the state update type of each target file, the corresponding processing logic is called to perform data processing on the file data of each target file.

[0061] The data processing thread pool (dataProcessor) is specially used for processing reading, parsing and data processing of file content, and the number of threads thereof is dynamically adjusted according to system load. The data processing thread pool is implemented by using an execution service (Executors.newFixedThreadPool) with a fixed size, and through a task queue management mechanism, asynchronous parallel processing of asynchronous reading tasks corresponding to multiple target files is realized, and after file data of the multiple target files are read out, parallel data processing of the multiple file data is realized.

[0062] The extensibility of the prior art is limited: the prior art adopts a single-thread sequential processing mode, and when the number of monitored directories or files increases, the execution time of a single polling increases linearly. If the time consumption of a single polling exceeds the polling interval, task accumulation and conflict will occur, which fundamentally limits the concurrent processing capability and horizontal expansion capability.

[0063] Therefore, in the embodiment of the application, the two thread pools of the timing scheduling thread pool and the data processing thread pool interact through a task submission interface: after the timing scheduling thread pool detects a target file, the timing scheduling thread pool immediately generates an asynchronous reading task corresponding to the target file and submits the asynchronous reading task to the data processing thread pool, thereby realizing asynchronous decoupling of detection (monitoring) and processing, and significantly improving the concurrent processing capability, reducing resource consumption, enhancing the scalability and stability.

[0064] In step S103, multiple threads in the preset data processing thread pool are utilized to execute each asynchronous reading task in parallel based on a stream processing technology (STP), read each target file, and obtain file data in the form of a character stream corresponding to each target file.

[0065] Each asynchronous reading task is executed asynchronously and in parallel in the data processing thread pool, that is, each thread in the data processing thread pool processes one asynchronous reading task, thereby avoiding blocking threads in the timing scheduling thread pool.

[0066] It should be noted that, in order to solve the problem of memory overflow in processing of large files, the stream processing technology is adopted. File content is read through an input stream (InputStream), thereby avoiding loading the entire file into memory. The stream processing technology wraps the file input stream into a character stream by establishing a buffer reader (BufferedReader) and setting an appropriate buffer size. The file content is processed by reading line by line, and only a small amount of data lines are maintained in memory each time, thereby greatly reducing the memory usage.

[0067] The whole flow data processing technology performs an asynchronous reading task, which includes obtaining the input stream of the file, creating a buffered reader, reading the file content line by line, and constructing a data row list (i.e., file data).

[0068] In a specific implementation, incremental processing of data rows is supported, and only the changed part of the content is processed for the modified file, as follows:

[0069] For each target file of the addition type, multiple threads in the preset data processing thread pool are used to perform corresponding asynchronous reading tasks in parallel based on the flow data processing technology, read all data in each target file of the addition type, and create corresponding file offset records to obtain file data in the form of character streams corresponding to each target file of the addition type.

[0070] For each target file of the modification type, multiple threads in the preset data processing thread pool are used to perform corresponding asynchronous reading tasks in parallel based on the flow data processing technology and the file offset records corresponding to each target file of the modification type, read the modified data in each target file of the modification type, and update the corresponding file offset records to obtain file data in the form of character streams corresponding to each target file of the modification type.

[0071] The file offset record is used to record the content of the target file that has been read, i.e., to locate the changed position of the file.

[0072] For a target file of the modification type, if the efficiency of reading from the beginning to the end each time is too low, a file offset record (recorded in the cache or database) is created. That is, the file offset record is used to remember where it was read last time, and the reading is continued from that position next time to achieve efficient and accurate incremental processing. In Java, the seek() scheme of RandomAccessFile can be used to jump to the position of the recorded file content using the file offset record.

[0073] In step S103, based on the state update type of each target file, the corresponding processing logic is called to perform data processing on the file data of each target file, including:

[0074] For each target file, the state update type of the target file and the file data of the target file are input to an event dispatcher for recording a plurality of pre-registered file event listeners, so that each state update type triggers a corresponding file event listener to perform data processing on the file data of the target file using the processing logic associated with the triggered file event listener.

[0075] It should be noted that the present application adopts the observer mode to construct an event-driven architecture, and completely decouples file state detection and file data processing logic. In an embodiment of the present application, a standardized file event interface, i.e., a registered file event listener of an event dispatcher record, is defined, which is responsible for listening to basic event types including a file creation event, a file modification event, and a file deletion event, and a data event listener is defined, which is responsible for listening to a data processing completion event, and a business processor is defined, which is responsible for subsequent business processing based on complete data content obtained after data processing.

[0076] The event dispatcher, as a core component of the observer mode, maintains a thread-safe event listener list. When a file change is detected, the event dispatcher iterates through all registered listeners and calls corresponding event processing methods. To ensure thread safety, the listener list is implemented using a CopyOnWriteArrayList array list, which supports concurrent reading and safe updating.

[0077] The event processing process adopts a synchronous calling mode to ensure the orderliness and consistency of event processing. At the same time, an exception isolation mechanism is implemented: an exception in a single listener will not affect the normal work of other listeners, and the system will record exception information and continue to process subsequent listeners.

[0078] Each event type corresponds to specific processing logic:

[0079] File creation event: triggered when the target file is of the new type, carrying basic information of the file, used for initializing the file processing flow;

[0080] File modification event: triggered when the target file is of the modified type, carrying modification time information, used for incremental update processing;

[0081] File deletion event: triggered when the target file is of the deleted type, used for cleaning related state information and file data;

[0082] Data processing completion event: triggered when file data processing is completed, carrying complete data content, used for subsequent business processing.

[0083] Among them, the event dispatcher automatically routes to the corresponding processing method according to the event type, realizing automatic management of event processing.

[0084] In an embodiment, before inputting the file data of the target file into the event dispatcher for recording a plurality of pre-registered file event listeners, the file data of the target file is converted to obtain file data conforming to the preset data processing requirements.

[0085] In the embodiment of the present application, the original file line data is converted into a data format required by preset data processing. The conversion process includes steps of data cleaning, format standardization, type conversion, business rule application, etc.

[0086] In an embodiment, when the status update detection of each file in the SFTP file system is completed, and the data processing of the file data of each collected target file is completed, the historical state information of each target file is updated.

[0087] Through the task aggregation waiting mechanism (CompletableFuture.allOf), it is ensured that each batch of detected target files is processed before the detection of the next batch of files is started, ensuring the integrity and consistency of data processing.

[0088] Based on the above-mentioned file processing method disclosed in the embodiment of the present application, in the present scheme, the state update detection of each file in the SFTP file system is periodically performed by using the pre-created SFTP connection channel, without the need to create SFTP connection multiple times, improving the resource utilization efficiency; the existing single-thread serial processing is replaced by a double-thread pool asynchronous parallel architecture, realizing asynchronous parallel of file state detection and file data processing, improving the speed and real-time performance of file processing; the incremental detection algorithm based on timestamp is adopted to avoid repeated processing, combined with the stream data processing technology to solve the large file memory occupation problem, and through the observer mode to establish a standardized event distribution mechanism, realizing complete decoupling of file state detection and file data processing, thereby significantly improving the concurrent processing capability, reducing resource consumption, enhancing the scalability and stability of the system, effectively solving the problems of resource waste, insufficient real-time performance, performance bottleneck and expansion limitation in the traditional scheme.

[0089] As shown in Figure 2 , it is an architecture diagram of a file processing system disclosed in the embodiment of the present application, which includes a core control layer 201, a data processing layer 202, a connection management layer 203, a thread pool management layer 204 and an event processing layer 205, each layer interacts through a standardized interface, forming a loosely coupled modular design, ensuring the scalability and maintainability of the system.

[0090] The core control layer 201 includes an asynchronous SFTP data collector, a file state manager and an event distributor;

[0091] The asynchronous SFTP data collector is configured to periodically detect state updates of each file in the SFTP file system by using multiple threads in a preset timing scheduling thread pool and a pre-created SFTP connection channel, collect each target file and determine a state update type of each target file whenever one or more target files with state updates are detected, and establish an asynchronous reading task corresponding to each target file whenever one or more target files are collected.

[0092] The data processing layer 202 is configured to execute each asynchronous reading task in parallel by using multiple threads in a preset data processing thread pool and an asynchronous file reader, obtain file data of each target file, and perform data processing on the file data of each target file by calling a corresponding processing logic based on the state update type of each target file.

[0093] The connection management layer 203 is configured to create an SFTP connection channel by using a connection factory and implement pool management of the SFTP connection by using an SFTP connector, and greatly reduce connection establishment overhead by using connection multiplexing technology.

[0094] The thread pool management layer 204 is configured to create and manage the timing scheduling thread pool and the data processing thread pool.

[0095] The event processing layer 205 is configured to create and manage a file event listener, a data event listener and a business processor.

[0096] Optionally, the asynchronous SFTP data collector configured to periodically detect state updates of each file in the SFTP file system by using multiple threads in a preset timing scheduling thread pool and a pre-created SFTP connection channel, collect each target file and determine a state update type of each target file whenever one or more target files with state updates are detected, is specifically configured to:

[0097] The asynchronous SFTP data collector is configured to periodically detect state updates of each file in the SFTP file system by using multiple threads in a preset timing scheduling thread pool and a pre-created SFTP connection channel, collect each target file and determine a state update type of each target file whenever one or more target files with state updates are detected, and establish an asynchronous reading task corresponding to each target file whenever one or more target files are collected.

[0098] The asynchronous SFTP data collector is configured to periodically detect state updates of each file in the SFTP file system by using multiple threads in a preset timing scheduling thread pool and a pre-created SFTP connection channel, collect each target file and determine a state update type of each target file whenever one or more target files with state updates are detected, and establish an asynchronous reading task corresponding to each target file whenever one or more target files are collected.

[0099] Correspondingly, the asynchronous SFTP data collector is further configured to: when the state update detection on each file in the SFTP file system is completed, and the data processing on the file data of each collected target file is completed, update the historical state information of each target file.

[0100] Optionally, the data processing layer 202 configured to utilize a plurality of threads in the preset data processing thread pool to perform each asynchronous reading task in parallel to obtain the file data of each target file, is specifically configured to:

[0101] The data processing layer 202 configured to utilize a plurality of threads in the preset data processing thread pool and the stream data parser to perform each asynchronous reading task in parallel based on the stream data processing technology to read each target file and obtain the file data of each target file in the form of a character stream.

[0102] Specifically, for each target file of the state update type of the addition type, the data processing layer 202 is configured to utilize a plurality of threads in the preset data processing thread pool to perform each corresponding asynchronous reading task in parallel based on the stream data processing technology to read all data in each target file of the state update type of the addition type and create a corresponding file offset record, and obtain the file data of each target file of the state update type of the addition type in the form of a character stream; the file offset record is configured to record the content of the target file that has been read.

[0103] For each target file of the state update type of the modification type, the data processing layer 202 is configured to utilize a plurality of threads in the preset data processing thread pool to perform each corresponding asynchronous reading task in parallel based on the stream data processing technology and the file offset record corresponding to each target file of the state update type of the modification type to read the modified data in each target file of the state update type of the modification type and update the corresponding file offset record, and obtain the file data of each target file of the state update type of the modification type in the form of a character stream.

[0104] Optionally, the data processing layer 202 configured to call corresponding processing logic based on the state update type of each target file to perform data processing on the file data of each target file, is specifically configured to:

[0105] For each target file, the data processing layer 202 is configured to input the state update type of the target file and the file data of the target file to an event dispatcher configured to record a plurality of pre-registered file event listeners, so that each state update type triggers a corresponding file event listener to utilize the processing logic associated with the triggered file event listener to perform data processing on the file data of the target file.

[0106] Optionally, the data processing layer 202 is further configured to:

[0107] Before inputting the file data of the target file into an event dispatcher for recording a plurality of pre-registered file event listeners, the data converter is used to perform conversion processing on the file data of the target file to obtain file data conforming to preset data processing requirements in format.

[0108] As Figure 3 shown, a running flow chart of a file processing system disclosed in the embodiment of the application, wherein the running flow corresponds to the file processing method disclosed in the embodiment of the application, and mutual reference can be made, and thus no further description is given here.

[0109] Based on the file processing system disclosed in the embodiment of the application, in the present scheme, the state update detection of each file in the SFTP file system is periodically performed by using the pre-created SFTP connection channel, and thus the SFTP connection does not need to be created multiple times, and the resource utilization efficiency is improved; the double-thread pool asynchronous parallel architecture is used to replace the existing single-thread serial processing, the asynchronous parallel of the file state detection and the file data processing is realized, and thus the speed and the real-time performance of the file processing are improved; the incremental detection algorithm based on the time stamp is used to avoid repeated processing, the stream data processing technology is combined to solve the large file memory occupation problem, and the standardized event distribution mechanism is established by using the observer mode, the complete decoupling of the file state detection and the file data processing is realized, and thus the concurrent processing capability is significantly improved, the resource consumption is reduced, the scalability and the stability of the system are enhanced, and the problems of the resource waste, the insufficient real-time performance, the performance bottleneck and the scalability limitation in the traditional scheme are effectively solved.

[0110] The embodiment of the application further provides an electronic device, please refer to Figure 4 The electronic device includes a memory 401 and a processor 402.

[0111] The memory 401 is configured to store a computer program.

[0112] The processor 402 is configured to execute the computer program, and specifically configured to implement the file processing method provided in any of the embodiments of the application.

[0113] The application further provides a computer storage medium for storing a computer program, and the computer program is executed to specifically implement the file processing method provided in any of the embodiments of the application.

[0114] The various embodiments described in this specification are described in progressive order of complexity, from the simplest embodiment to more complex embodiments. Identify commonalities in the various embodiments so that the disclosure is not redundant. Each embodiment is directed to the differences between that embodiment and the other embodiments. In particular, the system or system embodiments are described more simply because they are substantially similar to the method embodiments. The system or system embodiments are described with reference to the method embodiments. The systems and system embodiments described above are merely illustrative of the principles of this application. Any feature described in relation to one example can be used in relation to another example. The system and system embodiments described above are merely illustrative of the principles of this application. The units described as separate units can or can not be physically separate and the units shown as separate units can or can not be physical units, i.e. can be located at one place or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. Those skilled in the art can understand and implement without creative labor.

[0115] Those skilled in the art will further appreciate that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or any combination thereof. To clearly illustrate the interchangeability of hardware and software, the elements of the examples have been described generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0116] The above description of disclosed embodiments enables one of ordinary skill in the art to make and use the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles described herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of processing a file, characterized by, The method comprises: Periodically using multiple threads in a preset timing scheduling thread pool and a pre-created SFTP connection channel, batch detecting state update of each file in an SFTP file system, collecting each target file whenever one or more target files with state update are detected, and determining state update type of each target file; Whenever one or more target files are collected, establishing an asynchronous reading task corresponding to each target file; Using multiple threads in a preset data processing thread pool, executing each asynchronous reading task in parallel to obtain file data of each target file, and based on the state update type of each target file, calling corresponding processing logic to process file data of each target file.

2. The method of claim 1, wherein, The periodically using multiple threads in a preset timing scheduling thread pool and a pre-created SFTP connection channel, batch detecting state update of each file in an SFTP file system, collecting each target file whenever one or more target files with state update are detected, and determining state update type of each target file comprises: Periodically using multiple threads in a preset timing scheduling thread pool and a pre-created SFTP connection channel, batch obtaining current state information of each file in an SFTP file system, and comparing corresponding current state information with locally stored historical state information for each file with obtained current state information to obtain a comparison result; Based on the comparison result corresponding to each file in an SFTP file system, determining one or more target files with state update and state update type corresponding to each target file, and collecting each target file.

3. The method of claim 2, wherein, The method further comprises: When state update detection of each file in the SFTP file system is completed, and data processing of file data of each collected target file is completed, updating the historical state information of each target file.

4. The method of claim 1, wherein, The using multiple threads in a preset data processing thread pool to execute each asynchronous reading task in parallel to obtain file data of each target file comprises: Using multiple threads in a preset data processing thread pool to execute each asynchronous reading task in parallel based on streaming data processing technology, reading each target file to obtain file data in the form of character stream corresponding to each target file.

5. The method of claim 4, wherein, The using multiple threads in a preset data processing thread pool to execute each asynchronous reading task in parallel based on streaming data processing technology, reading each target file to obtain file data in the form of character stream corresponding to each target file comprises: The using multiple threads in a preset data processing thread pool to execute each asynchronous reading task in parallel based on streaming data processing technology, reading each target file to obtain file data in the form of character stream corresponding to each target file comprises: For each of the target files of the state update type of the new type, multiple threads in a preset data processing thread pool are used to perform corresponding each of the asynchronous reading tasks based on the streaming data processing technology, read all data in each of the target files of the state update type of the new type, and create corresponding file offset records to obtain file data in the form of a character stream corresponding to each of the target files of the state update type of the new type; the file offset record is used to record the content of the target file that has been read currently; For each of the target files of the state update type of the modification type, multiple threads in a preset data processing thread pool are used to perform corresponding each of the asynchronous reading tasks based on the streaming data processing technology and the file offset record corresponding to each of the target files of the state update type of the modification type, read the data that has been modified in each of the target files of the state update type of the modification type, and update the corresponding file offset record to obtain file data in the form of a character stream corresponding to each of the target files of the state update type of the modification type.

6. The method of claim 1, wherein, The processing logic corresponding to each of the target files is called based on the state update type of each of the target files to perform data processing on the file data of each of the target files, including: For each of the target files, the state update type of the target file and the file data of the target file are input into an event dispatcher for recording a plurality of pre-registered file event listeners, so that each of the state update types triggers a corresponding file event listener to perform data processing on the file data of the target file by using the processing logic associated with the triggered file event listener.

7. The method of claim 6, wherein, Before the file data of the target file is input into the event dispatcher for recording a plurality of pre-registered file event listeners, the method further includes: The file data of the target file is converted to obtain file data in a format that meets preset data processing requirements.

8. A system for processing files, characterized by The system includes: A core control layer is configured to periodically use multiple threads in a preset timing scheduling thread pool and a pre-created SFTP connection channel to detect the state update of each file in an SFTP file system, collect one or more target files whenever a state update occurs, and determine the state update type of each of the target files; an asynchronous reading task corresponding to each of the target files is established whenever one or more of the target files are collected; A data processing layer is configured to use multiple threads in a preset data processing thread pool to perform each of the asynchronous reading tasks in parallel to obtain file data of each of the target files, and call corresponding processing logic based on the state update type of each of the target files to perform data processing on the file data of each of the target files.

9. An electronic device, comprising: including: A memory is configured to save a computer program; A processor is configured to execute the computer program to implement the file processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program product for saving a computer program; wherein the computer program, when executed by a processor, implements the method of processing a file according to any one of claims 1 to 7.

Images