A method of processing a file and related apparatus

Abstract

The application provides a file processing method and related equipment, periodically using multiple threads in a preset timing scheduling thread pool and a preset SFTP connection channel to detect the state update of each file in an SFTP file system in batches, collecting one or more target files whenever a state update is detected, and determining the state update type of each target file; establishing an asynchronous reading task corresponding to each target file whenever one or more target files are collected; using multiple threads in a preset data processing thread pool to execute each asynchronous reading task in parallel, obtaining file data of each target file, and calling corresponding processing logic based on the state update type of each target file to process the file data of each target file, thereby improving resource utilization efficiency and file processing real-time.

Description

Technical Field

[0001] This invention relates to the field of data transmission technology, specifically to a file processing method and related equipment. Background Technology

[0002] In modern enterprise information environments, SFTP (Secure File Transfer Protocol) is widely used as a secure and reliable file transfer protocol for data exchange scenarios within the SFTP file system. Enterprises typically need to monitor file changes on remote SFTP servers in real time and promptly collect newly added or modified data files for subsequent business processing.

[0003] Traditional remote file acquisition solutions primarily rely on a timed polling mechanism: the client connects to an SFTP server (i.e., an I / O-intensive SFTP file system) at fixed time intervals, scans the target directory, compares file statuses, and downloads and processes data upon detecting changes. This traditional approach suffers from two main problems: 1. Low resource utilization efficiency: Using a fixed-interval (e.g., every 10 minutes) polling mechanism, an SFTP connection is established and scanned regardless of whether the remote server has file updates. Since establishing an SFTP connection requires multiple steps such as TCP handshake, SSH key exchange, and authentication, a large number of invalid polls when the file update frequency is much lower than the polling frequency leads to significant waste of network bandwidth and server resources. 2. Insufficient real-time performance: This manifests in two ways. First, traditional solutions use a single-threaded sequential processing mode. Even if multiple files change simultaneously, they can only be downloaded and processed one by one. When encountering large files, the entire monitoring process is blocked, severely delaying the processing of other files. Second, the fixed-interval polling strategy introduces unavoidable detection latency. For example, with a 10-minute polling interval, the delay from the creation of a new file to its detection can be as long as 10 minutes, with an average delay of 5 minutes. While shortening the polling interval can improve real-time performance, it will further exacerbate the problem of resource waste, creating a contradiction between real-time performance and resource efficiency.

[0004] In summary, existing technologies suffer from low resource utilization efficiency and insufficient real-time performance, which are problems that urgently need to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a file processing method and related equipment to improve resource utilization efficiency and real-time file processing.

[0006] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0007] The first aspect of this invention discloses a method for processing files, the method comprising:

[0008] The system periodically utilizes multiple threads in a pre-defined timed scheduling thread pool and a pre-created SFTP connection channel to perform status update detection on each file in the SFTP file system in batches. Whenever one or more target files that have undergone status updates are detected, each target file is collected, and the status update type of each target file is determined.

[0009] Whenever one or more of the target files are collected, an asynchronous reading task is created for each target file;

[0010] Multiple threads in a pre-defined data processing thread pool are used to execute each asynchronous reading task in parallel to obtain the file data of each target file. Based on the status update type of each target file, the corresponding processing logic is called to process the file data of each target file.

[0011] Optionally, the periodic use of multiple threads in a preset timed scheduling thread pool and a pre-created SFTP connection channel to perform status update detection on each file in the SFTP file system in batches. Whenever one or more target files with status updates are detected, each target file is collected, and the status update type of each target file is determined, including:

[0012] The system periodically utilizes multiple threads in a pre-defined timed thread pool and a pre-created SFTP connection channel to obtain the current status information of each file in the SFTP file system in batches. For each file whose current status information has been obtained, the corresponding current status information is compared with the historical status information stored locally to obtain the comparison result.

[0013] Based on the comparison results corresponding to each file in the SFTP file system, one or more target files that have undergone status updates and the status update type corresponding to each target file are determined, and each target file is collected.

[0014] Optionally, the method further includes:

[0015] When the status update detection of each file in the SFTP file system is completed, and the data processing of the file data of each collected target file is completed, the historical status information of each target file is updated.

[0016] Optionally, the step of using multiple threads from a preset data processing thread pool to execute each of the asynchronous reading tasks in parallel to obtain the file data of each target file includes:

[0017] Using multiple threads in a pre-defined data processing thread pool, and based on streaming data processing technology, each asynchronous reading task is executed in parallel to read each target file and obtain file data in the form of a character stream corresponding to each target file.

[0018] Optionally, the step of utilizing multiple threads in a preset data processing thread pool to execute each asynchronous reading task in parallel based on streaming data processing technology, reading each target file, and obtaining file data in character stream form corresponding to each target file includes:

[0019] For each target file whose status update type is "new", multiple threads in a preset data processing thread pool are used to execute the corresponding asynchronous reading tasks in parallel based on streaming data processing technology. All data in each target file whose status update type is "new" is read, and a corresponding file offset record is created to obtain the file data in character stream form corresponding to each target file whose status update type is "new". The file offset record is used to record the content that has been read from the target file.

[0020] For each target file whose status update type is "modified", multiple threads in a preset data processing thread pool are used to execute the corresponding asynchronous reading tasks in parallel based on streaming data processing technology and the file offset records corresponding to each target file whose status update type is "modified". The modified data in each target file whose status update type is "modified" is read, and the corresponding file offset records are updated to obtain file data in the form of a character stream corresponding to each target file whose status update type is "modified".

[0021] Optionally, the step of invoking corresponding processing logic based on the state update type of each target file to perform data processing on the file data of each target file includes:

[0022] For each target file, the status update type and file data of the target file are input to an event dispatcher that records multiple pre-registered file event listeners, so that each status update type triggers the corresponding file event listener, and the file data of the target file is processed using the processing logic associated with the triggered file event listener.

[0023] Optionally, before inputting the file data of the target file into an event dispatcher for recording multiple pre-registered file event listeners, the method further includes:

[0024] The file data of the target file is converted to obtain file data in a format that meets the preset data processing requirements.

[0025] A second aspect of this invention discloses a file processing system, the system comprising:

[0026] The core control layer is used to periodically utilize multiple threads in a preset timed scheduling thread pool and a pre-created SFTP connection channel to perform status update detection on each file in the SFTP file system in batches. Whenever one or more target files with status updates are detected, each target file is collected and the status update type of each target file is determined. Whenever one or more target files are collected, an asynchronous read task corresponding to each target file is established.

[0027] The data processing layer is used to utilize multiple threads in a preset data processing thread pool to execute each asynchronous reading task in parallel, obtain the file data of each target file, and, based on the status update type of each target file, call the corresponding processing logic to process the file data of each target file.

[0028] A third aspect of this invention discloses an electronic device, comprising:

[0029] Memory, used to store computer programs;

[0030] A processor is configured to execute the computer program to implement the method as described in any of the first aspects of the embodiments of the present invention.

[0031] A fourth aspect of the present invention discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the method described in any of the first aspects of the present invention.

[0032] Based on the file processing method and related equipment provided in the above embodiments of the present invention, the method periodically utilizes multiple threads in a preset timed scheduling thread pool and a pre-created SFTP connection channel to perform status update detection on each file in the SFTP file system in batches. Whenever one or more target files with status updates are detected, each target file is collected, and the status update type of each target file is determined. Whenever one or more target files are collected, an asynchronous read task corresponding to each target file is established. Multiple threads in a preset data processing thread pool are used to execute each asynchronous read task in parallel to obtain the file data of each target file. Based on the status update type of each target file, the corresponding processing logic is called to process the file data of each target file. In this solution, the periodic use of a pre-created SFTP connection channel to perform status update detection on each file in the SFTP file system eliminates the need to create multiple SFTP connections, improving resource utilization efficiency. The use of a dual-thread pool asynchronous parallel architecture to replace the existing single-thread serial processing achieves asynchronous parallelism of file status detection and file data processing, improving the speed and real-time performance of file processing. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0034] Figure 1 This is a flowchart of a file processing method disclosed in an embodiment of the present invention;

[0035] Figure 2 This is an architecture diagram of a file processing system disclosed in an embodiment of the present invention;

[0036] Figure 3 This is a flowchart illustrating the operation of a file processing system disclosed in an embodiment of the present invention;

[0037] Figure 4 This is a structural diagram of an electronic device disclosed in an embodiment of the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0040] As can be seen from the background technology, existing technologies suffer from low resource utilization efficiency and insufficient real-time performance, which are problems that urgently need to be solved.

[0041] Therefore, this invention discloses a file processing method and related equipment. In this solution, a pre-created SFTP connection channel is used periodically to perform status update detection on each file in the SFTP file system, eliminating the need to create multiple SFTP connections and improving resource utilization efficiency. A dual-thread pool asynchronous parallel architecture is adopted to replace the existing single-thread serial processing, realizing asynchronous parallelism of file status detection and file data processing, thereby improving the speed and real-time performance of file processing.

[0042] like Figure 1 The diagram shown is a flowchart of a file processing method disclosed in an embodiment of the present invention, which mainly includes the following steps:

[0043] Step S101: Periodically use multiple threads in the preset timed thread pool and the pre-created SFTP connection channel to perform status update detection on each file in the SFTP file system in batches. Whenever one or more target files that have undergone status update are detected, collect each target file and determine the status update type of each target file.

[0044] In step S101, a pre-created scheduler thread pool is used to handle periodic SFTP file system scan tasks (i.e., performing status update checks on each file in the SFTP file system in batches). The number of its threads is set to a small, fixed value to ensure stable execution of the scheduled tasks. This thread pool is implemented using a ScheduledExecutorService, which can precisely control the scan interval and avoid task backlog and resource contention.

[0045] The number of threads determines the number of files for which state update checks are performed in each batch.

[0046] The existing connection management technology suffers from inefficiency: the strategy of re-establishing a connection with each poll cannot reuse SFTP connections. The connection establishment process involves complex cryptographic calculations (RSA key exchange, AES encryption / decryption initialization, etc.), and frequent connection establishment and disconnection not only waste local computing resources, but also generate a large amount of connection logs and session management overhead on the remote server (SFTP file system).

[0047] Therefore, in this embodiment of the invention, an SFTP connection channel is pre-created to achieve pooled management of SFTP connections, and the connection establishment overhead is greatly reduced through connection reuse technology.

[0048] In the specific implementation of step S101, multiple threads in the preset timed scheduling thread pool and the pre-created SFTP connection channel are periodically used to obtain the current status information of each file in the SFTP file system in batches. For each file whose current status information has been obtained, the corresponding current status information is compared with the historical status information stored locally to obtain the comparison result.

[0049] Based on the comparison results for each file in the SFTP file system, one or more target files that have undergone status updates and the status update type for each target file are determined, and each target file is collected.

[0050] It should be noted that this invention uses a timestamp-based incremental detection algorithm to determine whether a file has changed by comparing its last modification time.

[0051] In the specific implementation, a file state mapping table (Map) is maintained.<String,Long> It stores the last modification timestamp of each file (i.e., the historical state information stored locally).

[0052] The file status mapping table is implemented using a thread-safe concurrent hash map (ConcurrentHashMap), supporting high-concurrency read and write operations. The key of the mapping table is the full path name of the file, and the value is the last modification timestamp of the file (in milliseconds).

[0053] The state update detection process consists of three phases:

[0054] Current Status Information Acquisition: Retrieves the current status information of all files in the remote directory via SFTP connection. Historical Status Information Comparison: Compares the current status information with the historical status information in the file status mapping table. Change Identification: Based on the comparison results, identifies and determines whether the currently detected file is a target file that has undergone a status update. If so, it determines the status update type, which includes: addition, modification, deletion, etc.

[0055] It is understandable that if the target file is a deleted file, since the file data has been deleted, the content of the target file collected only contains the information of the target file (such as the name). When performing data processing later, the information of the target file is used as the file data, and the corresponding data processing is performed (such as cleaning up related status information and read file data).

[0056] Step S102: Whenever one or more target files are collected, an asynchronous reading task is created for each target file.

[0057] In step S102, to solve the task coordination problem in multi-file parallel processing, this invention employs asynchronous task orchestration technology. Each file reading task is encapsulated as a complete asynchronous reading task (CompletableFuture), supporting non-blocking parallel execution.

[0058] In each round of scanning, all target files that need to be processed are collected, and an asynchronous reading task is created for each target file. These asynchronous reading tasks are then submitted to the data processing thread pool.

[0059] Optionally, asynchronous task orchestration technology also supports task-level exception isolation: the failure of an asynchronous read task for a single target file will not affect the normal processing of asynchronous read tasks for other target files. When an asynchronous read task fails, exception information is recorded and the remaining asynchronous read tasks are processed, improving the robustness of the system.

[0060] Step S103: Utilize multiple threads in the preset data processing thread pool to execute each asynchronous reading task in parallel, obtain the file data of each target file, and call the corresponding processing logic based on the status update type of each target file to process the file data of each target file.

[0061] The data processing thread pool (dataProcessor) is specifically designed for reading, parsing, and processing file content. Its thread count is dynamically adjusted based on system load. The data processing thread pool is implemented using a fixed-size execution service (Executors.newFixedThreadPool). Through a task queue management mechanism, it enables asynchronous parallel processing of asynchronous read tasks for multiple target files, as well as parallel data processing of multiple files after their data has been read.

[0062] The scalability limitations of existing technologies: Existing technologies employ a single-threaded sequential processing approach. As the number of monitored directories or files increases, the execution time of a single poll increases linearly. If the time taken for a single poll exceeds the polling interval, task backlog and conflicts will occur, fundamentally limiting concurrent processing capabilities and horizontal scalability.

[0063] Therefore, in this embodiment of the invention, the timed scheduling thread pool and the data processing thread pool interact through a task submission interface: after the timed scheduling thread pool detects the target file, it immediately generates an asynchronous reading task corresponding to the target file and submits it to the data processing thread pool, thereby realizing asynchronous decoupling between detection (monitoring) and processing, which significantly improves concurrent processing capabilities, reduces resource consumption, and enhances scalability and stability.

[0064] In step S103, multiple threads in a preset data processing thread pool are used to execute asynchronous reading tasks in parallel based on Stream Processing (STP) technology, read each target file, and obtain file data in the form of character streams corresponding to each target file.

[0065] Each asynchronous read task is executed asynchronously and in parallel within the data processing thread pool. That is, each thread in the data processing thread pool processes one asynchronous read task, thus avoiding blocking the threads in the timed scheduling thread pool.

[0066] It should be noted that, to address the memory overflow issue in large file processing, this invention employs streaming data processing technology. File content is read through an input stream, avoiding loading the entire file into memory. Streaming data processing utilizes a buffered reader to wrap the file input stream as a character stream and sets an appropriate buffer size. By processing file content line by line, only a small number of data lines are kept in memory at a time, significantly reducing memory usage.

[0067] The entire process of streaming data processing technology to perform asynchronous reading tasks is as follows: obtain the input stream of the file, create a buffered reader, read the file content line by line, and build a list of data lines (i.e., file data).

[0068] In the specific implementation, incremental processing of data rows is supported. For modified files, only the changed parts are processed, as follows:

[0069] For each target file whose status update type is "new", multiple threads in the preset data processing thread pool are used to execute the corresponding asynchronous reading tasks in parallel based on streaming data processing technology. All data in each target file whose status update type is "new" are read, and corresponding file offset records are created to obtain the file data in the form of character stream for each target file whose status update type is "new".

[0070] For each target file whose status update type is "modified", multiple threads in a preset data processing thread pool are used to execute asynchronous read tasks in parallel based on streaming data processing technology and the file offset records corresponding to each target file whose status update type is "modified". The modified data in each target file whose status update type is "modified" is read and the corresponding file offset records are updated to obtain file data in the form of character streams for each target file whose status update type is "modified".

[0071] The file offset record is used to record the content that has been read from the target file, that is, to locate the changed position of the file.

[0072] For target files whose status update type is "modification," reading from beginning to end every time is too inefficient. Therefore, file offset records are created (recorded in a cache or database). In other words, the file offset record remembers where the file was last read and resumes from that position next time, achieving efficient and accurate incremental processing. In Java, the `seek()` method of `RandomAccessFile` can be used to jump to the location of the recorded file content using the file offset record.

[0073] In step S103, based on the status update type of each target file, the corresponding processing logic is invoked to process the file data of each target file, including:

[0074] For each target file, the target file's status update type and file data are input into an event dispatcher that records multiple pre-registered file event listeners. This causes each status update type to trigger the corresponding file event listener, and the processing logic associated with the triggered file event listener is used to process the file data of the target file.

[0075] It should be noted that this invention employs the observer pattern to construct an event-driven architecture, completely decoupling file status detection and file data processing logic. In this embodiment, a standardized file event interface is defined, namely, the registered file event listeners recorded by the event dispatcher, responsible for listening to basic event types including file creation events, file modification events, and file deletion events. A data event listener is also defined, responsible for listening to data processing completion events. Finally, a business processor is defined, responsible for performing subsequent business processing based on the complete data content obtained after data processing.

[0076] The event dispatcher, as the core component of the observer pattern, maintains a thread-safe list of event listeners. When a file change is detected, the event dispatcher iterates through all registered listeners and calls the corresponding event handling methods. To ensure thread safety, the listener list is implemented using a copy-on-write array list (CopyOnWriteArrayList), supporting concurrent reads and safe updates.

[0077] The event handling process employs a synchronous call approach to ensure the sequential and consistent handling of events. It also implements an exception isolation mechanism: an exception in a single listener will not affect the normal operation of other listeners; the system will record the exception information and continue processing subsequent listeners.

[0078] Each event type corresponds to specific processing logic:

[0079] File creation event: Triggered when the target file is a new file type, carrying basic file information and used to initialize the file processing flow;

[0080] File modification event: Triggered when the target file is modified, carrying modification time information for incremental update processing;

[0081] File deletion event: Triggered when the target file is deleted, used to clean up related status information and file data;

[0082] Data processing complete event: Triggered when file data processing is complete, carrying the complete data content for subsequent business processing.

[0083] The event dispatcher automatically routes events to the appropriate handling methods based on the event type, thus achieving automated management of event handling.

[0084] In one embodiment, before inputting the file data of the target file into the event dispatcher for recording multiple pre-registered file event listeners, the file data of the target file is converted to obtain file data in a format that conforms to preset data processing requirements.

[0085] In this embodiment of the invention, the original file line data is converted into a data format that meets preset data processing requirements. The conversion process includes steps such as data cleaning, format standardization, type conversion, and application of business rules.

[0086] In one embodiment, when the status update detection of each file in the SFTP file system is completed and the data processing of the file data of each collected target file is completed, the historical status information of each target file is updated.

[0087] By using a task aggregation waiting mechanism (CompletableFuture.allOf), the detection of the next batch of files is only started after each batch of detected target files has been processed, thus ensuring the integrity and consistency of data processing.

[0088] Based on the file processing method disclosed in the above embodiments of the present invention, this solution periodically uses a pre-created SFTP connection channel to perform status update detection on each file in the SFTP file system, eliminating the need to create multiple SFTP connections and improving resource utilization efficiency. A dual-thread pool asynchronous parallel architecture replaces the existing single-threaded serial processing, achieving asynchronous parallelism between file status detection and file data processing, thus improving the speed and real-time performance of file processing. A timestamp-based incremental detection algorithm avoids redundant processing, and streaming data processing technology addresses the memory consumption problem of large files. Furthermore, a standardized event distribution mechanism is established through the observer pattern, achieving complete decoupling between file status detection and file data processing. This significantly improves concurrent processing capabilities, reduces resource consumption, enhances system scalability and stability, and effectively solves the problems of resource waste, insufficient real-time performance, performance bottlenecks, and scalability limitations inherent in traditional solutions.

[0089] like Figure 2 The diagram shown is an architecture diagram of a file processing system disclosed in an embodiment of the present invention. The system includes: a core control layer 201, a data processing layer 202, a connection management layer 203, a thread pool management layer 204, and an event processing layer 205. Each layer interacts through standardized interfaces to form a loosely coupled modular design, ensuring the scalability and maintainability of the system.

[0090] The core control layer 201 includes: an asynchronous SFTP data collector, a file status manager, and an event dispatcher;

[0091] An asynchronous SFTP data collector is used to periodically utilize multiple threads in a pre-defined timed thread pool and a pre-created SFTP connection channel to perform status update detection on each file in the SFTP file system in batches. Whenever one or more target files with status updates are detected, each target file is collected, and the status update type of each target file is determined. Whenever one or more target files are collected, an asynchronous read task corresponding to each target file is established.

[0092] The data processing layer 202 is used to utilize multiple threads in the preset data processing thread pool and asynchronous file readers to execute various asynchronous reading tasks in parallel, obtain the file data of each target file, and call the corresponding processing logic based on the status update type of each target file to process the file data of each target file.

[0093] The connection management layer 203 is used to create SFTP connection channels using the connection factory, implement pooled management of SFTP connections using the SFTP connector, and significantly reduce connection establishment overhead through connection reuse technology.

[0094] The thread pool management layer 204 is used to create and manage the timed scheduling thread pool and the data processing thread pool.

[0095] Event handling layer 205 is used to create and manage file event listeners, data event listeners, and business processors.

[0096] Optionally, an asynchronous SFTP data collector is used to periodically utilize multiple threads in a preset timed thread pool and pre-created SFTP connection channels to perform status update detection on each file in the SFTP file system in batches. Whenever one or more target files with status updates are detected, each target file is collected, and the status update type of each target file is determined. Specifically, it is used for:

[0097] The system periodically utilizes multiple threads in a pre-defined timed thread pool and a pre-created SFTP connection channel to obtain the current status information of each file in the SFTP file system in batches. For each file whose current status information has been obtained, the system compares the corresponding current status information with the historical status information stored in the file status manager to obtain the comparison result.

[0098] Based on the comparison results for each file in the SFTP file system, one or more target files that have undergone status updates and the status update type for each target file are determined, and each target file is collected.

[0099] Correspondingly, the asynchronous SFTP data collector is also used to update the historical status information of each target file when the status update detection of each file in the SFTP file system is completed and the data processing of the file data of each collected target file is completed.

[0100] Optionally, a data processing layer 202 is used to utilize multiple threads in a preset data processing thread pool to execute various asynchronous read tasks in parallel to obtain the file data of each target file. Specifically, it is used for:

[0101] By utilizing multiple threads in a pre-defined data processing thread pool and a streaming data parser, asynchronous reading tasks are executed in parallel based on streaming data processing technology to read various target files and obtain file data in character stream format corresponding to each target file.

[0102] Specifically, for each target file whose status update type is "new", multiple threads in the preset data processing thread pool are used to execute the corresponding asynchronous read tasks in parallel based on streaming data processing technology. All data in each target file whose status update type is "new" is read, and a corresponding file offset record is created to obtain the file data in the form of a character stream for each target file whose status update type is "new". The file offset record is used to record the content that has been read from the target file.

[0103] For each target file whose status update type is "modified", multiple threads in a preset data processing thread pool are used to execute asynchronous read tasks in parallel based on streaming data processing technology and the file offset records corresponding to each target file whose status update type is "modified". The modified data in each target file whose status update type is "modified" is read and the corresponding file offset records are updated to obtain file data in the form of character streams for each target file whose status update type is "modified".

[0104] Optionally, a data processing layer 202 is used to call the corresponding processing logic based on the state update type of each target file to process the file data of each target file. Specifically, it is used for:

[0105] For each target file, the target file's status update type and file data are input into an event dispatcher that records multiple pre-registered file event listeners. This causes each status update type to trigger the corresponding file event listener, and the processing logic associated with the triggered file event listener is used to process the file data of the target file.

[0106] Optionally, the data processing layer 202 is also used for:

[0107] Before the target file's data is input into the event dispatcher, which records multiple pre-registered file event listeners, the target file's data is converted using a data converter to obtain file data in a format that meets preset data processing requirements.

[0108] like Figure 3 The diagram shown is a flowchart of a file processing system disclosed in an embodiment of the present invention. The operation flow therein corresponds to the file processing method disclosed in the above embodiment of the present invention, and can be referred to each other. It will not be described again here.

[0109] Based on the file processing system disclosed in the above embodiments of the present invention, this solution periodically uses a pre-created SFTP connection channel to perform status update detection on each file in the SFTP file system, eliminating the need to create multiple SFTP connections and improving resource utilization efficiency. A dual-thread pool asynchronous parallel architecture replaces the existing single-threaded serial processing, achieving asynchronous parallelism between file status detection and file data processing, thus improving the speed and real-time performance of file processing. A timestamp-based incremental detection algorithm avoids redundant processing, and streaming data processing technology addresses the memory consumption problem of large files. Furthermore, a standardized event distribution mechanism is established through the observer pattern, achieving complete decoupling between file status detection and file data processing. This significantly improves concurrent processing capabilities, reduces resource consumption, and enhances the system's scalability and stability, effectively solving the problems of resource waste, insufficient real-time performance, performance bottlenecks, and scalability limitations inherent in traditional solutions.

[0110] This invention also provides an electronic device, please refer to... Figure 4 The electronic device includes a memory 401 and a processor 402.

[0111] Among them, memory 401 is used to store computer programs;

[0112] The processor 402 is used to execute computer programs, specifically to implement the file processing method provided in any embodiment of this application.

[0113] This application also provides a computer storage medium for storing a computer program, which, when executed, is specifically used to implement the file processing method provided in any embodiment of this application.

[0114] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0115] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0116] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing files, characterized in that, The method includes: The system periodically utilizes multiple threads in a pre-defined timed thread pool and a pre-created SFTP connection channel to acquire the current status information of each file in the SFTP file system in batches. For each file whose current status information has been acquired, the corresponding current status information is compared with the historical status information stored locally to obtain the comparison result. The number of threads in the thread pool determines the number of files for status update detection in each batch. Based on the comparison results corresponding to each file in the SFTP file system, one or more target files that have undergone status updates and the status update type corresponding to each target file are determined, and each target file is collected. Whenever one or more of the target files are collected, an asynchronous reading task is created for each target file; Multiple threads in a pre-defined data processing thread pool are used to execute each asynchronous reading task in parallel to obtain the file data of each target file. Based on the status update type of each target file, the corresponding processing logic is called to process the file data of each target file.

2. The method according to claim 1, characterized in that, The method further includes: When the status update detection of each file in the SFTP file system is completed, and the data processing of the file data of each collected target file is completed, the historical status information of each target file is updated.

3. The method according to claim 1, characterized in that, The method of utilizing multiple threads in a preset data processing thread pool to execute each asynchronous reading task in parallel to obtain the file data of each target file includes: Using multiple threads in a pre-defined data processing thread pool, and based on streaming data processing technology, each asynchronous reading task is executed in parallel to read each target file and obtain file data in the form of a character stream corresponding to each target file.

4. The method according to claim 3, characterized in that, The method utilizes multiple threads from a pre-defined data processing thread pool to execute asynchronous reading tasks in parallel based on streaming data processing technology, reading each target file to obtain file data in character stream format corresponding to each target file, including: For each target file whose status update type is "new", multiple threads in a preset data processing thread pool are used to execute the corresponding asynchronous reading tasks in parallel based on streaming data processing technology. All data in each target file whose status update type is "new" is read, and a corresponding file offset record is created to obtain the file data in character stream form corresponding to each target file whose status update type is "new". The file offset record is used to record the content that has been read from the target file. For each target file whose status update type is "modified", multiple threads in a preset data processing thread pool are used to execute the corresponding asynchronous reading tasks in parallel based on streaming data processing technology and the file offset records corresponding to each target file whose status update type is "modified". The modified data in each target file whose status update type is "modified" is read, and the corresponding file offset records are updated to obtain file data in the form of a character stream corresponding to each target file whose status update type is "modified".

5. The method according to claim 1, characterized in that, The step of invoking corresponding processing logic based on the state update type of each target file to process the file data of each target file includes: For each target file, the status update type and file data of the target file are input to an event dispatcher that records multiple pre-registered file event listeners, so that each status update type triggers the corresponding file event listener, and the file data of the target file is processed using the processing logic associated with the triggered file event listener.

6. The method according to claim 5, characterized in that, Before inputting the file data of the target file into the event dispatcher used to record multiple pre-registered file event listeners, the method further includes: The file data of the target file is converted to obtain file data in a format that meets the preset data processing requirements.

7. A file processing system, characterized in that, The system includes: The core control layer periodically utilizes multiple threads in a pre-defined timed thread pool and a pre-created SFTP connection channel to acquire the current status information of each file in the SFTP file system in batches. For each file with acquired current status information, the corresponding current status information is compared with historical status information stored locally to obtain a comparison result. The number of threads in the thread pool determines the number of files for status update detection in each batch. Based on the comparison result for each file in the SFTP file system, one or more target files that have undergone status updates and the corresponding status update type for each target file are determined, and each target file is collected. Whenever one or more target files are collected, an asynchronous read task is established for each target file. The number of threads in the thread pool determines the number of files for status update detection in each batch. The data processing layer is used to utilize multiple threads in a preset data processing thread pool to execute each asynchronous reading task in parallel, obtain the file data of each target file, and, based on the status update type of each target file, call the corresponding processing logic to process the file data of each target file.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the file processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer program is executed by a processor, it implements the file processing method as described in any one of claims 1 to 6.

Images