File concurrent processing method and system and electronic equipment

Abstract

The invention provides a file concurrent processing method and system and electronic equipment, and belongs to the technical field of file transmission, and the method comprises the following steps: determining a target concurrent level according to the number of available processing cores of the equipment; initializing a semaphore for controlling the concurrent task according to the target concurrent level; respectively operating a virtual thread for each to-be-processed file to be concurrent to obtain permission of the semaphore, and transmitting the to-be-processed file from the source storage position to the target storage position during the period of holding the permission to obtain a target file; after it is detected that all the virtual threads in the structured task domain are operated, processing results of all the target files are output. According to the file concurrent processing method based on the virtual thread and semaphore control, the overall efficiency and system stability of large-scale file concurrent processing are improved.

Description

Technical Field

[0001] The present invention relates to the field of file transmission technology, and in particular to a file concurrent processing method, system and electronic equipment. Background Art

[0002] With the widespread adoption of distributed systems and data-intensive businesses, concurrent processing tasks involving large numbers of files are becoming increasingly common. Typical applications include log archiving, data synchronization, and batch transfers. In these scenarios, systems often need to process hundreds or even thousands of files to be transferred simultaneously, placing higher demands on concurrency and resource scheduling.

[0003] Existing technical solutions include using a statically configured thread pool to control the number of concurrent requests. When the number of threads is far below the upper limit of the device's capacity, the hardware resources are idle. When the number of threads exceeds the device's processing capacity, CPU utilization drops sharply, accompanied by frequent memory copies and context switches, which trigger system performance bottlenecks. In addition, the file copying method based on the user-mode buffer requires reading data from the disk into the user buffer and then writing it back to the kernel, resulting in additional CPU overhead and memory usage caused by bidirectional copying, further exacerbating performance degradation. It can be seen that the existing solution can neither achieve hardware-aware dynamic concurrency control nor provide efficient I / O transmission methods, thereby restricting the overall efficiency and system stability of large-scale concurrent file processing. Summary of the Invention

[0004] The present invention provides a file concurrent processing method, system, electronic device, storage medium and computer program product to address the defects in the existing technology and improve the overall efficiency and system stability of large-scale file concurrent processing.

[0005] The present invention provides a method for concurrently processing files, comprising the following steps: Determine the target concurrency level based on the number of available processing cores on the device; Initialize a semaphore that controls concurrent tasks according to the target concurrency level; Running a virtual thread for each pending file to be processed concurrently to obtain a license of the semaphore, and transferring the pending file from a source storage location to a target storage location during the period of holding the license to obtain a target file; After detecting that all the virtual threads in the structured task domain have completed execution, the processing results of all the target files are output.

[0006] According to a file concurrent processing method provided by the present invention, a virtual thread is respectively run for each file to be processed concurrently to obtain a license of the semaphore, and the file to be processed is transferred from a source storage location to a target storage location during the period of holding the license to obtain the target file, comprising: For each of the to-be-processed files, a virtual thread is created, and an uncaught exception handler is registered for the virtual thread, where the uncaught exception handler is used to capture and report exception information during the execution of the virtual thread; When the current virtual thread starts executing, it applies for a license by calling the license acquisition interface of the semaphore; When the current number of permissions for the semaphore is greater than zero, obtaining a permission for the current virtual thread; When the current number of permissions of the semaphore is equal to zero, the current virtual thread is put into a blocked waiting state until another virtual thread releases its permission and obtains the permission of the current virtual thread; During the period of holding the license, the to-be-processed file is transferred from the source storage location to the target storage location in a kernel zero-copy manner to obtain the target file.

[0007] According to a concurrent file processing method provided by the present invention, the method of transferring the to-be-processed file from a source storage location to a target storage location in a kernel zero-copy manner to obtain the target file includes: Opening a source file channel corresponding to the source storage location and a target file channel corresponding to the target storage location in the virtual thread respectively; Call the zero-copy interface, starting from the starting position of the source file channel, using the current size of the source file channel as the number of transferred bytes, and directly transfer the file to be processed to the target file channel to obtain the target file; wherein the transfer process is completed in the kernel space without passing through the user-mode buffer.

[0008] According to a method for concurrently processing files provided by the present invention, the method further comprises: transferring the to-be-processed file from the source storage location to the target storage location during the period of holding the license, and obtaining the target file; After the virtual thread completes the transmission of the to-be-processed file, or when the exception information is captured during the execution process, the license release interface of the semaphore is called to release the license corresponding to the virtual thread.

[0009] A method for concurrently processing files according to the present invention further includes: When it is detected that any of the virtual threads in the structured task domain fails to execute, the remaining virtual threads in the structured task domain are canceled, and the execution failure result is reported.

[0010] According to a file concurrent processing method provided by the present invention, determining a target concurrency level based on the number of available processing cores of a device includes: Obtaining the number of available processing cores; The target concurrency level is obtained by multiplying the number of available processing cores by a preset concurrency coefficient.

[0011] According to a file concurrency processing method provided by the present invention, the initializing a semaphore for controlling concurrent tasks according to the target concurrency level includes: An initial number of grants for the semaphore is set to the target concurrency level.

[0012] A method for concurrently processing files according to the present invention further includes: When instantiating the semaphore, the semaphore is set to a fair scheduling mode, so that when multiple virtual threads apply for permission for the semaphore, the permission is granted in sequence according to the permission application order of each virtual thread.

[0013] The present invention also provides a file concurrent processing system, comprising the following modules: A first processing module, configured to determine a target concurrency level based on the number of available processing cores of the device; A second processing module, configured to initialize a semaphore for controlling concurrent tasks according to the target concurrency level; a third processing module, configured to respectively run a virtual thread for each pending file to be processed concurrently to obtain a license of the semaphore, and to transfer the pending file from the source storage location to the target storage location during the period of holding the license to obtain the target file; The fourth processing module is configured to output processing results of all the target files after detecting that all the virtual threads in the structured task domain have completed execution.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-described concurrent file processing methods is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for concurrently processing files.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned concurrent file processing methods.

[0017] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: The target concurrency level is determined based on the number of available processing cores on the device, enabling concurrent scheduling to adaptively match the device's actual processing capabilities and avoiding resource congestion caused by overconfigured threads. Furthermore, by initializing a semaphore to control concurrent tasks based on the target concurrency level, the number of simultaneously executing virtual threads is limited, ensuring that the execution of each task is subject to permission management, ensuring that the system load remains stable and within a controllable range, and enhancing the predictability and security of concurrent scheduling. A virtual thread is then run for each concurrently processed file to obtain a semaphore permission. During this permission period, the file is transferred from the source storage location to the target storage location to obtain the target file. The lightweight nature of virtual threads improves thread creation and switching efficiency, and combined with the kernel's zero-copy transfer mechanism, significantly reduces CPU overhead and memory pressure during data transfer, enabling efficient execution of large-scale file transfer tasks. When all virtual threads within the structured task domain are detected to have successfully executed, the processing results of all target files are output, ensuring the semantic integrity and consistency of concurrent processing tasks and effectively preventing the issue of triggering result output while some tasks are not yet completed, thereby improving the overall efficiency and system stability of large-scale concurrent file processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 This is one of the flow charts of the file concurrent processing method provided by the present invention.

[0020] Figure 2 This is the second flowchart of the file concurrent processing method provided by the present invention.

[0021] Figure 3 This is the third flow chart of the file concurrent processing method provided by the present invention.

[0022] Figure 4 This is the fourth flow chart of the file concurrent processing method provided by the present invention.

[0023] Figure 5 This is the fifth flow chart of the file concurrent processing method provided by the present invention.

[0024] Figure 6 It is a structural diagram of the file concurrent processing system provided by the present invention.

[0025] Figure 7It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field without making creative efforts based on the embodiments of the present invention are within the scope of protection of the present invention.

[0027] It should be noted that, in the description of the present invention, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. The orientation or positional relationship indicated by the terms "upper" and "lower" is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the system or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0028] The terms "first," "second," and so forth, used herein are used to distinguish similar objects, not to describe a specific order or precedence. It should be understood that such terms are interchangeable where appropriate, allowing embodiments of the present invention to be implemented in an order other than that illustrated or described herein. Furthermore, the terms "first," "second," and so forth generally distinguish objects of a single type, and do not limit the number of objects. For example, the first object may be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the connected objects.

[0029] The following combination Figures 1-7 The present invention describes the method, system, electronic device and storage medium for concurrent file processing.

[0030] Figure 1 This is one of the flowcharts of the file concurrent processing method provided by the present invention, such as Figure 1 As shown, it includes steps 101-104: Step 101: Determine a target concurrency level based on the number of available processing cores of the device.

[0031] Due to differences in processing capabilities between devices, using a fixed concurrency level for task scheduling can lead to idle resources on high-performance devices or resource contention on low-performance devices, resulting in reduced processing efficiency, system instability, and other issues. Therefore, in this embodiment, step 101 aims to determine the target concurrency level based on the number of available processing cores on the device, providing a dynamic, adaptable concurrency upper limit for subsequent virtual thread concurrency control, thereby achieving a balance between resource utilization and processing efficiency.

[0032] In one possible implementation, Figure 2 This is the second flowchart of the file concurrent processing method provided by the present invention. Figure 2 As shown, step 101 specifically includes steps 201-202: Step 201: Obtain the number of available processing cores.

[0033] Step 202: Multiply the number of available processing cores by the preset concurrency coefficient to obtain a target concurrency level.

[0034] In this embodiment, steps 201 and 202 together constitute a specific implementation of step 101, aiming to dynamically obtain the number of available processing cores on the device and, in combination with a preset concurrency coefficient, calculate a target concurrency level that is appropriate for the current system resources. This strategy effectively addresses the resource waste and performance bottlenecks associated with traditional concurrency configurations that rely on fixed concurrency numbers or manual configuration, thereby enabling automated adjustment of concurrency capabilities without increasing system complexity.

[0035] In step 201, the computer obtains the number of available processing cores on the current device in real time by calling the core resource management interface provided by the operating system. This number of available processing cores can be expressed as the number of physical cores, the number of logical cores, or the effective number derived from the operating system's scheduling policy. In some multitasking or virtualized environments, this number may change dynamically due to system resource allocation policies. Therefore, using a runtime acquisition method ensures that concurrency control is highly consistent with the current system state.

[0036] In step 202, the computer multiplies the number of available processing cores obtained in step 201 by the preset concurrency coefficient to calculate the target concurrency level. The preset concurrency coefficient is usually set empirically based on the average blocking behavior of virtual threads in the target application scenario. For example, in an I / O-intensive scenario, the value can be 10 to ensure that the system still has sufficient execution concurrency when most virtual threads are in a waiting state. The target concurrency level generated by this multiplication operation will subsequently serve as a key parameter for controlling the number of semaphore permissions to dynamically limit the total number of concurrent virtual threads in the system.

[0037] Step 102: Initialize the semaphore for controlling concurrent tasks according to the target concurrency level.

[0038] In concurrent file processing scenarios, a large number of virtual threads may be created simultaneously, and system resources (such as CPU cores, memory, and I / O bandwidth) have a certain upper limit on their availability. If left uncontrolled, this can lead to increased task competition, increased system load, and even system resource exhaustion. Therefore, in this embodiment, the purpose of step 102 is to initialize a semaphore for controlling concurrent tasks based on the target concurrency level, thereby limiting the number of simultaneously executing virtual threads and achieving precise control over task concurrency.

[0039] In a possible implementation, step 102 specifically includes the following steps: Sets the semaphore's initial number of permits to the target concurrency level.

[0040] In practice, when a computer initializes a semaphore object, it uses the target concurrency level as the initial number of permitted executions, ensuring that the system only allows a maximum of this number of virtual threads to execute at any given time. Semaphores can be instantiated using the Semaphore class in a concurrent library (such as the JDK concurrency package), passing the target concurrency level in the constructor and configuring it for either fair or unfair scheduling. The initialized semaphore maintains a permit counter. Each time a virtual thread attempts to execute a task, it must first request a permit. Execution can only begin after a permit is granted, and upon completion, the permit must be explicitly released to allow other waiting threads to continue using it.

[0041] Step 103: Run a virtual thread for each concurrently processed file to obtain a semaphore permit, and transfer the processed file from the source storage location to the target storage location while holding the permit to obtain the target file.

[0042] In this embodiment, the purpose of step 103 is to create a corresponding virtual thread for each pending file. After the virtual thread is started, it obtains a semaphore license. While holding the license, the pending file is transferred from the source storage location to the target storage location, thereby completing the concurrent file transfer process. This step is the core processing stage of the entire method. It is designed to leverage the lightweight concurrency of virtual threads to improve processing throughput, while also combining semaphores to control concurrency scale and ensure that system resources are not over-consumed.

[0043] In the traditional concurrency model, when using a thread pool or a fixed number of threads to execute batch file tasks, it is easy to face problems such as high thread context switching overhead, uneven resource utilization, and thread starvation. To this end, this embodiment adopts a virtual thread model to construct the task execution logic. Specifically, the system creates a virtual thread for each file to be processed. After the virtual thread is started, it first attempts to apply for a license by calling the semaphore's license acquisition interface. If there is an available license in the semaphore, the current virtual thread immediately obtains the license and enters the execution state; if the number of licenses is zero, the thread will enter a blocked wait until other threads release the license. This mechanism ensures that the number of virtual threads in the execution state at the same time does not exceed the target concurrency level set when the semaphore is initialized, thereby avoiding system resources being occupied by excessive threads.

[0044] While holding the license, the virtual thread performs file transfer operations. To improve transfer efficiency and reduce memory consumption, the transfer process uses a kernel zero-copy approach. This involves invoking the zero-copy interface of the file channel to transfer data directly from the file channel corresponding to the source storage location to the writable channel corresponding to the target storage location. This avoids multiple data copies between user mode and kernel mode, significantly reducing CPU load and context switches.

[0045] It should be noted that, in a possible implementation, after creating the virtual thread, the method further includes the following steps: When instantiating a semaphore, the semaphore is set to a fair scheduling mode, so that when multiple virtual threads apply for permission for the semaphore, the permission is granted in sequence according to the permission application order of each virtual thread.

[0046] During large-scale concurrent processing of virtual threads, if multiple threads apply for permission at the same time and the semaphore adopts an unfair scheduling strategy, some threads may be unable to obtain permission for a long time due to being frequently "jumped in line" by other newly created threads, which may lead to accumulated execution delays and even task starvation or priority inversion problems at the system level. To this end, in this embodiment, when creating a semaphore instance, the system sets the semaphore to fair mode by configuring fair scheduling parameters. In the Java implementation, this configuration can be achieved by passing the true parameter into the semaphore constructor, that is, newSemaphore(permits,true), where permits is the target concurrency level.

[0047] Once fair scheduling mode is enabled, the semaphore maintains a first-in-first-out queue when processing virtual thread permit requests, granting permits in the order in which threads enter the queue. This eliminates competition for permits between threads, ensuring stable and predictable response times for all threads after requesting permits, preventing some threads from being stuck in the queue for extended periods.

[0048] In the following embodiments, the implementation method of step 103 will be further described. In one possible implementation, Figure 3 This is the third flow chart of the concurrent file processing method provided by the present invention. Figure 3 As shown, step 103 specifically includes steps 301-305: Step 301: For each file to be processed, a virtual thread is created, and an uncaught exception handler is registered for the virtual thread. The uncaught exception handler is used to capture and report exception information during the execution of the virtual thread.

[0049] In the large-scale concurrent file processing scenario, the traditional thread pool management mode is difficult to meet the performance and stability requirements of million-level file processing due to the expensive thread resources, limited concurrency, and poor exception observability. Therefore, the virtual thread mechanism introduced by JDK 21 is used in the embodiment to allocate an independent virtual thread for each file to be processed without increasing the thread management burden. Specifically, when initializing the virtual thread execution environment, the system first constructs a thread builder for creating a virtual thread based on the Thread.Builder API, and sets the thread model to a virtual thread through the Thread.ofVirtual() method. In order to realize unified identification and classified management of thread instances, the builder further sets the thread naming rule through chain calling, for example, adding a "vt-worker-" prefix to each thread and automatically incrementing the number, so as to ensure that each thread has a unique and readable identifier in the log tracking and monitoring interface. Further, the virtual thread executor (vtExecutor) is used for task scheduling, which can significantly improve the concurrent processing capability and reduce the system resource occupation.

[0050] At the same time of creating a virtual thread, the system registers an uncaught exception handler logException for each virtual thread. The exception handler is used to trigger automatically when an uncaught exception occurs during the execution of the virtual thread, to record the exception stack information in real time and push the exception details to the operation and maintenance channel through the monitoring alarm system, to realize the automatic monitoring and rapid response of the exception event. By calling the factory() method, a unified virtual thread factory can be generated based on the above configuration, to ensure that all virtual threads created by the factory are consistent in execution parameters, naming rules and exception handling strategies.

[0051] Further, in order to improve the thread observability and fault response capability of the method in the high-concurrency file processing scenario, the system introduces a thread management and monitoring alarm mechanism based on the virtual thread creation in step 301. In the thread running phase, the monitoring system continuously collects the running indicators of each virtual thread through the context information associated with the thread factory, including the creation time, current state, CPU occupancy and memory usage of the thread, and other key resource indicators. The above indicators will be reported in a periodic manner, and when detecting risk behaviors such as abnormal use of thread pool resources, high load or abnormal termination of virtual threads, the system will actively trigger the notification mechanism based on the preset alarm rules, and send warning information to the operation and maintenance personnel to assist them in timely intervention before or at the initial stage of the problem.

[0052] Step 302: When the current virtual thread starts to execute, the permission acquisition interface of the semaphore is called to apply for permission.

[0053] During actual operation, the system creates a virtual thread for each file to be processed. After it is started, it does not immediately perform the file transfer operation. Instead, it first calls the semaphore's license acquisition interface to request execution permission from the system. The semaphore here is a concurrency control object initialized according to the target concurrency level in step 102. It maintains a set of license numbers to limit the total number of virtual threads that are in execution at the same time. When a virtual thread initiates a license application, if the semaphore currently still has an available license, the virtual thread successfully obtains the license and enters the transfer phase; otherwise, the virtual thread will enter a waiting state until another thread releases the license.

[0054] This mechanism ensures controlled execution of each file transfer task by proactively requesting permission after the virtual thread is started, enabling refined scheduling of concurrent system resources. Compared to the traditional thread pool model, which implicitly regulates execution order through thread queues, this implementation enhances the predictability of task scheduling and controllability of system load through explicit permission management.

[0055] Step 303: When the current number of permissions of the semaphore is greater than zero, obtain the permission of the current virtual thread.

[0056] In a specific implementation, after the current virtual thread calls the semaphore's permission acquisition interface, the system determines whether the current number of permissions in the semaphore is greater than zero. If the number of permissions is greater than zero, indicating that the system still has concurrent execution capacity, the computer allocates a permission to the current virtual thread and allows it to enter the execution phase. The virtual thread that obtains the permission will then proceed with its subsequent tasks, completing the transfer of the pending file in step 305. This permission acquisition process can be implemented using a standard semaphore interface provided by the operating system or runtime environment, and strictly corresponds to the target concurrency level set in step 102, ensuring that concurrent behavior is always controlled by the system's preset threshold.

[0057] Step 304: When the current number of permissions of the semaphore is equal to zero, the current virtual thread is put into a blocked waiting state until another virtual thread releases its permission and obtains the permission of the current virtual thread.

[0058] During concurrent file processing, if a large number of virtual threads simultaneously request execution permissions, the semaphore's permissions may be fully occupied at a certain point. In this case, allowing newly created virtual threads to continue executing would exceed the system's concurrency limit, leading to core overload or increased memory pressure, seriously impacting system stability. To address this, this implementation introduces a permission blocking mechanism. When the current virtual thread detects that the number of permissions is zero when calling the semaphore's permission acquisition interface, the system will place the virtual thread in a waiting queue, pausing its execution.

[0059] Specifically, virtual threads in a waiting state will no longer consume execution resources until other virtual threads complete their corresponding tasks and release permits. Once the number of semaphore permits increases, the system will then wake up the leading waiting thread according to the fair scheduling policy, allowing it to re-enter the permit acquisition process. This blocking waiting mechanism allows virtual thread scheduling to be completely constrained by the real-time state of system resources, achieving a dynamic match between task concurrency and resource carrying capacity.

[0060] Step 305: During the license holding period, the file to be processed is transferred from the source storage location to the target storage location through the kernel zero copy method to obtain the target file.

[0061] In this embodiment, the purpose of step 305 is to transfer the file to be processed from the source storage location to the target storage location using kernel zero-copy technology, after the virtual thread obtains the semaphore grant and while holding the grant, to obtain the target file. This step, as the key processing action of step 103, is responsible for the actual data transfer process. It is designed to effectively improve file transfer efficiency and reduce system overhead in high-concurrency scenarios. It also works in conjunction with the grant control mechanism to ensure controllable and high-performance resource utilization.

[0062] In one possible implementation, Figure 4 This is the fourth flowchart of the file concurrent processing method provided by the present invention, such as Figure 4 As shown, step 305 specifically includes steps 401-402: Step 401: Open a source file channel corresponding to a source storage location and a target file channel corresponding to a target storage location in a virtual thread.

[0063] In implementation, the system opens a source file channel in each virtual thread based on the source storage path of the file to be processed and establishes a read-only channel connection. Simultaneously, it opens or creates the target file based on the target file path and establishes a writable target file channel. To efficiently complete file transfers, the FileChannel.transferTo method is used. First, the source and destination file channels are opened using the FileChannel.open method. The source file channel src is used to read the source file, and the destination file channel dest is opened in create and write mode to receive the transferred data.

[0064] Step 402: Call the zero-copy interface, starting from the starting position of the source file channel, using the current size of the source file channel as the number of transferred bytes, and directly transfer the file to be processed to the target file channel to obtain the target file; wherein, the transfer process is completed in the kernel space without passing through the user-mode buffer.

[0065] In this embodiment, the purpose of step 402 is to call the zero-copy interface to complete the transfer of the pending file after establishing the source and target file channels. This transfer process occurs entirely within kernel space, thus avoiding redundant data copy operations between user and kernel space, significantly improving transfer efficiency and reducing system resource overhead. This step, as the key technical implementation path for step 305, relies on the zero-copy mechanism supported by modern operating systems and is particularly suitable for high-concurrency, large-volume file transfer scenarios.

[0066] Specifically, after opening the source file channel corresponding to the source storage location and the target file channel corresponding to the target storage location, the virtual thread will call the zero-copy interface to perform the data transfer operation. In this embodiment, the interface call format is src.transferTo(0, src.size(), dest), where src represents the source file channel, dest represents the target file channel, the offset is set to 0 to indicate that the transfer starts from the beginning of the file, and the transfer length is set to the current file size of the source file channel. This method call allows data to be mapped directly from the source file to the target file in the kernel buffer without passing through the application layer memory buffer, avoiding the two-way copy and related context switching between user mode and kernel mode in traditional I / O.

[0067] To ensure the safe release of file resources, this zero-copy transfer operation is encapsulated in a try-with-resources structure. This structure automatically closes the source and destination file channels upon successful completion or if an exception occurs during the transfer, preventing resource leaks. This not only improves code robustness but also ensures the sustainability and controllability of system resources during highly concurrent tasks.

[0068] Through the implementation of this step, the method uses zero-copy technology to achieve efficient transmission of file content in kernel space, significantly reducing CPU load, reducing memory usage, improving the system's processing performance during large-scale concurrent transmission, and further improving the performance ceiling and operating efficiency of the entire file concurrent processing system.

[0069] In a possible implementation, after step 103, the method further includes: After the virtual thread completes the transfer of the pending file, or when an exception message is captured during the execution process, the semaphore's license release interface is called to release the license corresponding to the virtual thread.

[0070] In this embodiment, the purpose of this step is to call the release interface of the semaphore after the virtual thread completes the transmission operation of the corresponding to-be-processed file or when an exception information is captured during execution, so as to release the permission held by the current virtual thread, so that other virtual threads in a waiting state can obtain the permission in time and continue to execute the task.

[0071] Specifically, when the virtual thread successfully completes the file transmission process of step 305 or is interrupted due to a running exception during execution, the system performs a permission release operation. To ensure that the permission release logic can be reliably called regardless of the change of the execution path, the present embodiment encapsulates the call of the permission release interface in a structured resource management statement or an equivalent try-finally block, so that it can be triggered in all cases of normal end of the task or throwing of an exception. By calling the semaphore.release() interface to release one permission unit, the available permission number of the semaphore is increased, so as to ensure that other virtual threads that are blocked are woken up in time and obtain an execution opportunity when the resource allows.

[0072] Step 104: After detecting that all virtual threads in the structured task domain run completely, output the processing results of all target files.

[0073] In specific implementation, the system creates a virtual thread for each to-be-processed file and includes it in a structured task domain. The structured task domain is a concurrent execution model, and its core feature is that the system can automatically track the life cycle of all concurrent sub-tasks before the main task collects all sub-tasks, and uniformly listens to the execution results when the main task is in a waiting state. Each virtual thread in the current structured task domain corresponds to a file processing task, and when all virtual threads are executed and no exception is thrown, the structured task domain considers that the overall task execution is successful, at which time the system outputs a processing result set composed of all target files or writes it to a specified storage path or task callback interface.

[0074] The use of the structured task domain ensures the closure of task management and the consistency of exception handling. Unlike the traditional concurrent model, the structured task domain provides thread-level life cycle isolation and aggregation operation capability, can uniformly cancel the tasks of other threads in the domain when any thread fails (see step 501), and can uniformly converge the results when all threads are successful, avoiding the risk of partial task isolated completion or disconnection. In addition, the structured model cooperates with the scheduling flexibility provided by the virtual thread, so that the method has a consistent guarantee mechanism for results while maintaining high concurrent performance.

[0075] Through the implementation of this step, the method not only achieves efficient concurrent execution control, but also provides a stable and unified task completion judgment and output mechanism, improving the practicality of the method.

[0076] In one possible implementation, Figure 5 This is the fifth flowchart of the file concurrent processing method provided by the present invention, such as Figure 5 As shown, after step 104, the following steps are also included: Step 501: When it is detected that any virtual thread in the structured task domain fails to execute, the remaining virtual threads in the structured task domain are canceled, and the execution failure result is reported.

[0077] In this embodiment, the purpose of step 501 is to promptly cancel the execution of the remaining virtual threads within the structured task domain and report the corresponding execution failure results when any virtual thread within the structured task domain fails. This step, as a supplementary path to step 104, ensures that if a local anomaly occurs during concurrent file processing, the system can respond immediately and terminate all unfinished tasks, avoiding resource waste and state inconsistencies. This is a core control mechanism for ensuring task consistency and system robustness.

[0078] During implementation, after constructing a structured task domain and incorporating multiple virtual threads into it, the system continuously monitors the execution status of each virtual thread. When any virtual thread fails due to an exception during execution (e.g., I / O error, unavailable file channel, permission exception, etc.), the structured task domain immediately identifies the exception and proactively cancels all virtual threads in the current task domain that are still running or waiting based on a "failure-first" scheduling strategy. This cancellation operation is initiated by the structured task domain's unified scheduling, avoiding the chaotic behavior of multiple threads attempting to interrupt separately and ensuring that each task can complete resource release operations before being terminated.

[0079] At the same time, the system structures and encapsulates the exception information caused by the failed virtual thread, including the thread identifier, failure cause, and stack trace information. This exception result is then reported to the monitoring system or external caller as evidence of the overall task failure. Because structured task domains inherently possess task aggregation and error propagation mechanisms, the reported failure results clearly identify thread ownership and provide full domain coverage, helping external systems quickly locate the source of the problem and improving troubleshooting efficiency.

[0080] In an optional embodiment, to improve the adaptability of the method in large-scale distributed environments, the system further introduces a node-aware task partitioning strategy, and independently executes the concurrent file processing process defined in steps 101 to 501 above within each computing node. This strategy is particularly suitable for practical scenarios based on container or cluster deployment architectures, such as Kubernetes, Docker Swarm, or distributed scheduling platforms.

[0081] Specifically, during the task initialization phase, the system first performs task partitioning based on the number and characteristics of the set of files to be processed, combined with the number and load of available nodes in the current cluster. Each subtask contains a set of files to be processed and is scheduled and assigned to a specific node. Subsequently, each node independently runs the concurrent processing flow within its local resource constraints, executing steps 101 to 501 within its local node, including determining the target concurrency level, initializing semaphores, creating virtual threads, executing file transfers, releasing permissions, and outputting results.

[0082] To prevent resource contention and inter-node interference, each node independently determines its target concurrency level based on its CPU core count and manages its license quantity and scheduling policy independently. For example, if Node A has a 16-core CPU, its target concurrency level can be set to 160; while Node B, with 8 cores, can have its concurrency level set to 80. This prevents performance degradation on low-resource nodes due to excessive concurrency.

[0083] Reference Figure 6 , Figure 6 Schematic diagram of the structure of the concurrent file processing system provided by the present invention, the system includes: A first processing module, configured to determine a target concurrency level based on the number of available processing cores of the device; A second processing module is used to initialize a semaphore for controlling concurrent tasks according to a target concurrency level; The third processing module is configured to run a virtual thread for each file to be processed concurrently to obtain a semaphore license, and transfer the file to be processed from the source storage location to the target storage location during the period of holding the license to obtain the target file; The fourth processing module is configured to output processing results of all target files after detecting that all virtual threads in the structured task domain have completed execution.

[0084] In a possible implementation, the fourth processing module is further configured to cancel the remaining virtual threads in the structured task domain and report the execution failure result when detecting that any virtual thread in the structured task domain fails to execute.

[0085] In a possible implementation, the third processing module is further configured to: For each file to be processed, a virtual thread is created, and an uncaught exception handler is registered for the virtual thread, the uncaught exception handler being configured to capture and report exception information in the execution process of the virtual thread; When the current virtual thread starts execution, a permission obtaining interface of the semaphore is called to apply for permission; When the current number of permissions of the semaphore is greater than zero, the current virtual thread obtains permission; When the current number of permissions of the semaphore is equal to zero, the current virtual thread enters a blocked waiting state until the current virtual thread obtains permission after other virtual threads release permission; During the holding of the permission, the file to be processed is transmitted from the source storage location to the target storage location by a kernel zero-copy manner to obtain a target file.

[0086] In a possible implementation, the third processing module is further configured to: The source file channel corresponding to the source storage location and the target file channel corresponding to the target storage location are respectively opened in the virtual thread; The zero-copy interface is called to directly transmit the file to be processed to the target file channel from the starting position of the source file channel with the current size of the source file channel as the transmission byte number, so as to obtain a target file; wherein the transmission process is completed in the kernel space without passing through a user state buffer.

[0087] In a possible implementation, the third processing module is further configured to: When the virtual thread completes transmission of the file to be processed or captures exception information in the execution process, the permission release interface of the semaphore is called to release the permission corresponding to the virtual thread.

[0088] In a possible implementation, the first processing module is further configured to: The number of available processing cores is obtained; The number of available processing cores is multiplied by a preset concurrency coefficient to obtain a target concurrency level.

[0089] In a possible implementation, the second processing module is further configured to set the initial number of permissions of the semaphore as the target concurrency level.

[0090] In a possible implementation, the third processing module is further configured to, when instantiating the semaphore, set the semaphore to a fair scheduling mode, so that when multiple virtual threads apply for permission of the semaphore, the permission is granted in turn according to the permission application order of each virtual thread.

[0091] It should be noted that the file concurrency processing system provided by the present application can execute the file concurrency processing method of any of the above embodiments when actually running, and thus the present embodiment will not be described in detail.

[0092] Figure 7 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 7 As shown, the electronic device may include: a processor 710 (processor), a communication interface 720 (CommunicationsInterface), a memory 730 (memory) and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute a file concurrent processing method, which includes: determining a target concurrency level based on the number of available processing cores of the device; initializing a semaphore for controlling concurrent tasks based on the target concurrency level; running a virtual thread for each file to be processed concurrently to obtain a semaphore license, and transferring the file to be processed from a source storage location to a target storage location during the period of holding the license to obtain a target file; and outputting the processing results of all target files after detecting that all virtual threads in the structured task domain have completed execution.

[0093] Furthermore, the logic instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0094] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the file concurrent processing method provided in the above embodiments.

[0095] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented by a processor to execute the file concurrent processing method provided in the above embodiments.

[0096] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. That is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0097] Through the description of the above embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods of each embodiment or certain portions of the embodiments.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for concurrent file processing, characterized in that: include: Determine the target concurrency level based on the number of available processing cores on the device; Initialize a semaphore that controls concurrent tasks according to the target concurrency level; Running a virtual thread for each pending file to be processed concurrently to obtain a license of the semaphore, and transferring the pending file from a source storage location to a target storage location during the period of holding the license to obtain a target file; After detecting that all the virtual threads in the structured task domain have completed execution, the processing results of all the target files are output.

2. The file concurrent processing method according to claim 1, characterized in that: The method includes running a virtual thread for each file to be processed concurrently to obtain a license of the semaphore, and transferring the file to be processed from a source storage location to a target storage location during the period of holding the license to obtain the target file, including: For each of the to-be-processed files, a virtual thread is created, and an uncaught exception handler is registered for the virtual thread, where the uncaught exception handler is used to capture and report exception information during the execution of the virtual thread; When the current virtual thread starts executing, it applies for a license by calling the license acquisition interface of the semaphore; When the current number of permissions for the semaphore is greater than zero, obtaining a permission for the current virtual thread; When the current number of permissions of the semaphore is equal to zero, the current virtual thread is put into a blocked waiting state until another virtual thread releases its permission and obtains the permission of the current virtual thread; During the period of holding the license, the to-be-processed file is transferred from the source storage location to the target storage location in a kernel zero-copy manner to obtain the target file.

3. The file concurrent processing method according to claim 2, characterized in that: The transferring the to-be-processed file from the source storage location to the target storage location in a kernel zero-copy manner to obtain the target file includes: Opening a source file channel corresponding to the source storage location and a target file channel corresponding to the target storage location in the virtual thread respectively; Call the zero-copy interface, starting from the starting position of the source file channel, using the current size of the source file channel as the number of transferred bytes, and directly transfer the file to be processed to the target file channel to obtain the target file; wherein the transfer process is completed in the kernel space without passing through the user-mode buffer.

4. The file concurrent processing method according to claim 2, characterized in that: The method further comprises: transferring the to-be-processed file from the source storage location to the target storage location during the period of holding the license, and obtaining the target file; After the virtual thread completes the transmission of the to-be-processed file, or when the exception information is captured during the execution process, the license release interface of the semaphore is called to release the license corresponding to the virtual thread.

5. The file concurrent processing method according to claim 1, characterized in that: Also includes: When it is detected that any of the virtual threads in the structured task domain fails to execute, the remaining virtual threads in the structured task domain are canceled, and the execution failure result is reported.

6. The method for concurrently processing files according to claim 1, wherein: Determining the target concurrency level based on the number of available processing cores of the device includes: Obtaining the number of available processing cores; The target concurrency level is obtained by multiplying the number of available processing cores by a preset concurrency coefficient.

7. The file concurrent processing method according to claim 1, characterized in that: Initializing a semaphore for controlling concurrent tasks according to the target concurrency level includes: An initial number of grants for the semaphore is set to the target concurrency level.

8. The file concurrent processing method according to claim 1, characterized in that: Also includes: When instantiating the semaphore, the semaphore is set to a fair scheduling mode, so that when multiple virtual threads apply for permission for the semaphore, the permission is granted in sequence according to the permission application order of each virtual thread.

9. A concurrent file processing system, characterized in that: include: A first processing module, configured to determine a target concurrency level based on the number of available processing cores of the device; A second processing module, configured to initialize a semaphore for controlling concurrent tasks according to the target concurrency level; a third processing module, configured to respectively run a virtual thread for each pending file to be processed concurrently to obtain a license of the semaphore, and to transfer the pending file from the source storage location to the target storage location during the period of holding the license to obtain the target file; The fourth processing module is configured to output processing results of all the target files after detecting that all the virtual threads in the structured task domain have completed execution.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method for concurrently processing files according to any one of claims 1 to 8 is implemented.

Images