Processing method, computer equipment and storage medium

Abstract

The invention discloses a processing method, computer equipment and a storage medium. The processing method comprises the following steps: receiving a preset instruction; generating an asynchronous processing request according to the preset instruction; based on the asynchronous processing request, performing asynchronous processing on the Excel file to be processed by utilizing a dynamic thread pool to obtain a result file; and executing a preset process on the result file. Therefore, by utilizing an asynchronous processing mechanism and a dynamic thread pool, uploading and downloading tasks about Excel files are decomposed into a plurality of subtasks through a multi-thread concurrent processing mechanism, and thread resources are dynamically allocated, so that the file transmission speed and efficiency are improved, and system resources are reasonably allocated.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a processing method, computer equipment and storage medium. Background Art

[0002] Common Excel file operations on the Internet include uploading and downloading, such as Figure 1 As shown, there is an interactive diagram for uploading and downloading Excel files. Figure 1 In the process, the server receives the Excel file uploaded by the client, fills in the data according to the data template based on the Excel file, and then parses the data according to the defined template, saves it to the database, and returns a success message to the client, otherwise it returns an error message to the client; after receiving the client's download request, the server queries the database for data according to the filtering conditions, writes the query results to the Excel file, and returns it to the client.

[0003] During the process of conceiving and implementing this application, the inventors discovered that there are at least the following problems: Synchronous operations are blocked, preventing the program from responding to other tasks and causing the user interface to become unresponsive. Single-threaded processing results in limited performance and slow processing speeds, making it impossible to perform multiple tasks simultaneously. Resource utilization is inadequate, and CPU and other resources remain idle during waiting periods. Multi-threaded processing suffers from irrational resource allocation, with thread data unable to adjust in real time based on load and demand. This can lead to either too many threads, consuming excessive system resources, or too few threads, underutilizing system performance. Batch transfers or transfers of very large files can result in low transfer efficiency, high resource usage, and slow processing speeds, especially when network bandwidth is limited and server loads are high. Improving file upload performance and reducing resource usage are pressing issues. Summary of the Invention

[0004] The purpose of this application is to provide a processing method, computer device and storage medium that can utilize asynchronous processing mechanism and dynamic thread pool to decompose the upload and download tasks of Excel files into multiple subtasks with a multi-threaded concurrent processing mechanism, dynamically allocate thread resources, so as to improve the speed and efficiency of file transfer, and reasonably allocate resources.

[0005] To achieve the above objectives: In a first aspect, an embodiment of the present application provides a method, the method comprising: Receive preset instructions; Generate an asynchronous processing request according to the preset instruction; Based on the asynchronous processing request, the Excel file to be processed is asynchronously processed using a dynamic thread pool to obtain a result file; Execute a preset process on the result file.

[0006] In one embodiment, the step of generating an asynchronous processing request includes: Based on the preset instructions sent by the client, an asynchronous processing request is generated using the Java asynchronous mechanism to respond to the client. In one embodiment, before the step of generating the asynchronous processing request, the method further includes: Determine a transmission task for an Excel file, and then execute a multi-threaded processing flow based on the task type of the transmission task; wherein, when the preset instruction is an upload instruction, the transmission task is determined to be an upload task; when the preset instruction is a download instruction, the transmission task is determined to be a download task.

[0007] In one embodiment, the step of asynchronously processing the Excel file to be processed using a dynamic thread pool includes: A dynamic thread pool is used to perform multi-threaded processing on the data content of the Excel file to obtain a result file. In one embodiment, before the step of performing multi-threaded processing on the data content of the Excel file using a dynamic thread pool, the following steps are included: Counting the number of requests for the asynchronous processing; Extracting thread information of the dynamic thread pool; Thread data of multi-threaded concurrent execution is dynamically adjusted based on the request amount and the thread information. In one embodiment, the thread information includes a maximum number of threads and a core number of threads; and the step of dynamically adjusting the thread data of the multi-threaded concurrent execution based on the request volume and the thread information includes: Compare the request amount with the maximum number of threads and the number of core threads; When the request amount is between the maximum number of threads and the number of core threads, the thread data remains unchanged; When the number of requests exceeds the maximum number of threads, the excess ratio between the number of requests and the maximum number of threads is used to calculate the additional thread data, thereby adjusting the maximum number of threads and the number of core threads. When the request amount is less than the number of core threads, the ratio of underutilized threads is calculated and the reduction ratio is calculated to adjust the number of core threads.

[0008] In one embodiment, the step of performing multi-threaded processing on the data content of the Excel file using a dynamic thread pool includes: When executing the upload task, the data content of the Excel file is divided into multiple data blocks, and each data block is uploaded in parallel using multiple threads. The uploaded data blocks are parsed, and the data blocks are merged based on the parsing results to obtain the data results; When executing the download task, multi-threading is used to perform paging query on the preset database to obtain a data set, and the data content in the data set is written into the preset worksheet to obtain the Excel file to be downloaded.

[0009] In one embodiment, the step of performing multi-threaded processing on the data content of the Excel file using a dynamic thread pool further includes: Get the file stream when processing data in multiple threads; Use the preset reading and writing tools to parse or write the file stream to obtain the result file.

[0010] In one embodiment, the step of executing a preset process on the result file includes one of the following: When executing the upload task, the data result is verified. If the data result passes the verification, the data result is saved. If the data result fails the verification, the error cause is obtained, the error cause is mapped to the corresponding row information of the Excel file, and the error cause is written into the imported Excel file. When executing the download task, the obtained Excel file is sent to the client.

[0011] In a second aspect, an embodiment of the present application provides a computer device, comprising: a processor and a memory storing a computer program, wherein when the processor runs the computer program, the steps of the above method are implemented.

[0012] In a third aspect, an embodiment of the present application provides a storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0013] The processing method, computer device, and storage medium provided in the embodiments of the present application employ the following methods: receiving a preset instruction; generating an asynchronous processing request based on the preset instruction; asynchronously processing the Excel file to be processed using a dynamic thread pool based on the asynchronous processing request to obtain a result file; and executing a preset process on the result file. In this manner, the asynchronous processing mechanism is utilized to dynamically adjust the thread data in the dynamic thread pool to decompose the upload or download task corresponding to the preset instruction into multiple subtasks corresponding to the thread data, thereby dynamically allocating thread resources, improving file transfer speed and efficiency, and rationally allocating system resources.

[0014] Since an asynchronous processing mechanism is used to execute Excel file upload and download, a timely response can be made after receiving the upload or download instruction, avoiding the other party waiting for the response result while the server is processing.

[0015] Due to the use of a dynamic thread pool, thread data can be dynamically adjusted according to the generated asynchronous processing requests, thereby improving the processing performance of multiple threads, rationally utilizing system resources, avoiding insufficient processing performance caused by a sudden increase in the number of business requests, and avoiding waste of resources when the number of business requests is small. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a schematic diagram of the existing interaction for uploading and downloading Excel files.

[0017] Figure 2 This is an application environment diagram of the processing method provided in the first embodiment of the present application.

[0018] Figure 3 A flowchart of the processing method provided in the second embodiment of the present application.

[0019] Figure 4 This is a logical block diagram of the asynchronous processing mechanism and dynamic thread pool provided in the second embodiment of this application.

[0020] Figure 5 A flowchart of the processing method provided in the third embodiment of the present application.

[0021] Figure 6 This is a structural block diagram of the processing system provided in the fifth embodiment of the present application.

[0022] Figure 7 FIG. 4 is a structural block diagram of a processing device in an embodiment.

[0023] Figure 8 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment.

[0024] Figure 9 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0026] It should be noted that, in this document, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.

[0027] It should be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, without departing from the scope of this document, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the term "if" as used herein may be interpreted as "at the time of," "when," or "in response to a determination." Furthermore, as used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context indicates otherwise. It should be further understood that the terms "comprising" and "including" indicate the presence of the described features, steps, operations, elements, components, items, types, and / or groups, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, types, and / or groups. The terms "or" and "and / or" as used herein are to be interpreted as inclusive, meaning any one or any combination. Thus, “A, B, or C” or “A, B, and / or C” means “any of: A; B; C; A and B; A and C; B and C; A, B, and C.” An exception to this definition occurs only when a combination of elements, functions, steps, or operations are inherently mutually exclusive in some manner.

[0028] It should be understood that, although the various steps in the flowchart in the embodiment of the present application are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless clearly stated herein, the execution of these steps is not strictly limited in order, and they can be performed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and their execution order is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0029] It should be noted that in this article, step codes such as S101 and S102 are used for the purpose of expressing the corresponding content more clearly and concisely, and do not constitute a substantial limitation on the order. When implementing the step, those skilled in the art may execute S102 first and then S101, etc., but these should all be within the scope of protection of this application.

[0030] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0031] In the subsequent description, the use of suffixes such as "module", "component" or "unit" to represent elements is only for the purpose of facilitating the description of the present application and has no specific meaning. Therefore, "module", "component" or "unit" can be used interchangeably.

[0032] First embodiment In one embodiment, the processing method provided by this application can be applied to Figure 2 In the application environment shown, the server 200 communicates with the client 100 via a network. The server 200 receives a preset instruction; generates an asynchronous processing request based on the preset instruction; based on the asynchronous processing request, asynchronously processes the Excel file to be processed using a dynamic thread pool to obtain a result file; and executes a preset process on the result file. After the client 200 issues the preset instruction to the server 200, it receives a response from the server 200.

[0033] The client 100 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, etc., and the server 200 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0034] Second embodiment In one embodiment, if Figure 3 As shown, a processing method is provided, which is applied to Figure 2 The server in the example is used as an example to illustrate the following steps: S101: receiving a preset instruction; S102: Generate an asynchronous processing request according to a preset instruction; S103: Based on the asynchronous processing request, the Excel file to be processed is asynchronously processed using a dynamic thread pool to obtain a result file; S104: Execute a preset process on the result file.

[0035] In one embodiment, step S101: receiving a preset instruction.

[0036] Among them, the preset instruction is an upload instruction or a download instruction. When receiving an upload instruction, the server receives the Excel file transmitted by the client based on the upload instruction; when receiving a download instruction, the server transmits the Excel file to the client based on the download instruction. Traditional file transfer has technical problems in terms of transmission efficiency, resource occupation, processing speed, etc. when large-scale file transfer or ultra-large file transfer occurs. Based on this, when this embodiment receives a preset instruction, such as an upload instruction or a download instruction, it will perform corresponding preset processing on the Excel file to be transmitted to solve technical problems such as transmission efficiency, resource occupation, and processing speed.

[0037] Step S102: Generate an asynchronous processing request according to a preset instruction.

[0038] Asynchronous request processing is a programming pattern in which the server or application does not immediately complete processing and return a response after receiving a request. Instead, it queues the request or assigns it to another thread or process for processing. It then immediately returns a "received" confirmation, allowing the client to continue performing other operations without having to wait for a final result. In contrast, in asynchronous processing, after receiving a request, synchronous processing requires waiting until the request is processed and a response is returned. In asynchronous processing, a response is returned immediately after receiving the request and processed in the background, eliminating the need to wait for a final result and providing notification of the result later.

[0039] In one embodiment, the step of generating an asynchronous processing request in step S102 includes: Based on the preset instruction sent by the receiving client, an asynchronous processing request is generated using the Java asynchronous mechanism to respond to the client.

[0040] Java's asynchronous mechanism allows programs to continue executing other tasks while waiting for time-consuming operations to complete, improving application responsiveness and throughput. Asynchronous request processing typically uses tools in the java.util.concurrent package, such as ExecutorService, or uses CompletableFuture and the java.util.concurrent.CompletionStage interface introduced in Java 8 and above. This allows for asynchronous execution of time-consuming tasks and processing of the results after completion.

[0041] In one embodiment, before step S102, the step of generating an asynchronous processing request, the method further includes: Determine a transmission task for an Excel file, and then execute a multi-threaded processing flow based on the task type of the transmission task; wherein, when the preset instruction is an upload instruction, the transmission task is determined to be an upload task; when the preset instruction is a download instruction, the transmission task is determined to be a download task.

[0042] In one embodiment, step S103: based on the asynchronous processing request, asynchronously process the Excel file to be processed using a dynamic thread pool to obtain a result file.

[0043] Among them, the dynamic thread pool (Thread Pool) is a form of multi-threaded processing based on pooling technology, which is used to manage the creation and life cycle of threads, and provide a thread queue for parallel execution of tasks. The main purpose of the dynamic thread pool is to reduce the overhead and resources spent on creating and destroying threads, improve program performance, and also provide better management of concurrent execution tasks, such as controlling thread data. Threads in the dynamic thread pool can be reused to perform multiple tasks, avoiding the performance overhead of frequent creation and destruction of threads, thereby improving response speed; the dynamic thread pool can limit the maximum number of threads in the system to prevent excessive memory consumption due to too many threads, or excessive context switching overhead; the dynamic thread pool provides configurable parameters, such as the number of core threads, the maximum number of threads, the idle thread survival time, the size of the task queue, etc., allowing customization to suit different application needs. Multithreading refers to multiple computational scheduling units within a dynamic thread pool. A thread is the smallest unit of computational scheduling within an operating system and represents an execution flow within a process. A process can contain multiple threads. All threads share process resources (such as memory space and file descriptors), but each thread has its own execution stack and program counter. In this embodiment, the upload or download process is divided into multiple threads for execution.

[0044] In one embodiment, the step of asynchronously processing the Excel file to be processed by using a dynamic thread pool includes: performing multi-threaded processing on the data content of the Excel file by using the dynamic thread pool to obtain a result file.

[0045] Optionally, refer to the attached Figure 4 The figure shows a logical block diagram of the asynchronous processing mechanism and dynamic thread pool provided by an embodiment of the present application. Before executing multi-threaded processing, the dynamic thread pool dynamically adjusts the thread data of the multi-threads based on the request volume of the generated asynchronous processing requests, and executes the upload task or download task after updating the thread data of the multi-threads.

[0046] Before the step of using the dynamic thread pool to perform multi-threaded processing on the data content of the Excel file, the method includes: counting the request volume of the asynchronous processing requests; extracting thread information of the dynamic thread pool; and dynamically adjusting the thread data of the multi-threaded concurrent execution based on the request volume and the thread information.

[0047] Exemplarily, the thread information includes a maximum number of threads and a core number of threads; and the step of dynamically adjusting thread data for concurrent multi-threaded execution based on the request volume and the thread information includes: Compare the request amount with the maximum number of threads and the number of core threads; When the request amount is between the maximum number of threads and the number of core threads, the thread data remains unchanged; When the number of requests exceeds the maximum number of threads, the excess ratio between the number of requests and the maximum number of threads is used to calculate the additional thread data, thereby adjusting the maximum number of threads and the number of core threads. When the request amount is less than the number of core threads, the ratio of underutilized threads is calculated and the reduction ratio is calculated to adjust the number of core threads.

[0048] Among them, the dynamic thread pool pre-sets the maximum number of threads (maximumPoolSize) and the number of core threads (corePoolSize), which jointly control the behavior and performance of the dynamic thread pool. The maximum number of threads indicates the maximum number of threads that the dynamic thread pool is allowed to create, which can cope with a large number of sudden tasks and provide additional processing capabilities. When the number of requests exceeds the processing capacity of the core threads and the work queue is full, the dynamic thread pool will create new threads until the maximum number of threads is reached. Threads that exceed the number of core threads will be recycled after being idle for a period of time (controlled by the keepAliveTime parameter). The number of core threads indicates the minimum number of threads that remain active in the dynamic thread pool, ensuring that the dynamic thread pool always has a certain number of threads available, avoiding the overhead of frequent thread creation and destruction. In other words, even if these threads are idle, they will not be recycled and destroyed. When a task arrives, the core threads will be used first. The core threads are usually created when the dynamic thread pool is initialized.

[0049] Optionally, based on the maximum number of threads and the number of core threads, the concurrently executed thread data is dynamically adjusted.

[0050] For example, when the request volume is between the maximum number of threads and the number of core threads, the thread data remains unchanged. When the request volume exceeds the maximum number of threads, the excess ratio between the request volume and the maximum number of threads is calculated: excess ratio = (request volume - maximum number of threads) / maximum number of threads. The maximum number of threads is adjusted based on the excess ratio. The increased thread data = maximum number of threads * min(floor(excess ratio / 10), 1), and the adjusted maximum number of threads does not exceed the system-set maximum thread threshold. In the formula, "min" represents the minimum value, and "floor" represents rounding down. This ensures that when the excess ratio is between 0 and 10, the increased thread data is at most the maximum number of threads. For larger excess ratios, the increased thread data does not increase significantly and remains at the maximum number of threads. Further, after adjusting the maximum number of threads, the core thread number is dynamically adjusted by half the increase in the maximum number of threads. That is, the increased thread data for the core thread number = 1 / 2 * maximum number of threads * min(floor(excess ratio / 10), 1). When the number of requests is less than the number of core threads, the underutilized thread ratio is calculated using the reduction ratio = min((number of core threads - number of requests) / number of core threads, 0.5). In this formula, "min" represents the minimum value, and the numerator, "(number of core threads - number of requests) / number of core threads," represents the underutilized thread ratio. To avoid excessive thread reduction, a maximum value of 0.5 (i.e., 50%) is set. In this embodiment, the number of core threads is reduced based on the underutilized thread ratio, and the number of core threads must not be less than the minimum core thread count threshold configured for the system.

[0051] For example, the request volume is 1000, the current maximum number of threads is 20, the number of core threads is 8, and the system's preset maximum thread threshold is 60. The calculated excess ratio is (1000-20) / 20=49, and the added thread data is 20*min(floor(49 / 10), 1)=20*1=20, so the added thread volume is 20, and the final adjusted maximum number of threads is 40, and the number of core threads is 18.

[0052] For example, if the request volume is 10 and the number of core threads is 25, the minimum core thread threshold configured by the system is 12. The reduction ratio is min((25-10) / 25, 0.5) = min(0.6, 0.5) = 0.5, which means that the number of core threads is reduced by 50%, and the final number of core threads is 12.

[0053] In one embodiment, after dynamically adjusting the thread data of the concurrently executed threads based on the request volume and the thread information, the method further includes: performing a cooldown process based on a preset cooldown period, so that the thread data of the dynamic thread pool remains unchanged during the cooldown period. For example, a cooldown period of, for example, 5 minutes, is preset, and the dynamic thread pool is not adjusted during the cooldown period to avoid system instability caused by overly frequent adjustments.

[0054] In one embodiment, step S103: asynchronously processing the Excel file to be processed using a dynamic thread pool based on the asynchronous processing request, further includes: Synchronize parsing status and update result files in real time; visually display the running status of asynchronous processing processes and thread pools.

[0055] The parsing status includes: downloading, download successful, and download failed.

[0056] Among them, in the step of visually displaying the running status of the asynchronous processing process and the thread pool, it is allowed to view uploaded files and downloaded files.

[0057] Step S104: executing a preset process on the result file.

[0058] When executing an upload task, the result file is the data result obtained by merging and processing multiple threads of upload. When executing a download task, the result file is an Excel file with the data content written in it.

[0059] In one embodiment, the step of executing a preset process on the result file includes one of the following: When executing the upload task, the data result is verified. If the data result passes the verification, the data result is saved. If the data result fails the verification, the error cause is obtained, the error cause is mapped to the corresponding row information of the Excel file, and the error cause is written into the imported Excel file. When executing the download task, the obtained Excel file is sent to the client.

[0060] Further explanation: when the data result passes the verification, the data result is saved in the preset database and imported into the original Excel file. When the data result fails the verification, the error reason is written into the imported Excel file.

[0061] Third embodiment Based on the first embodiment and the second embodiment, in one embodiment, as Figure 5 As shown, a processing method is provided, which is applied to Figure 2 The server in the example is used as an example to illustrate the following steps: S201: In response to receiving a preset instruction, determining a transmission task for an Excel file, generating an asynchronous processing request, and performing asynchronous processing on the Excel file to be processed; S202: During asynchronous processing, based on the task type of the transmission task, a dynamic thread pool is used to perform multi-threaded processing on the data content of the Excel file to obtain a result file.

[0062] In step S201, in the step of determining a transmission task regarding an Excel file, when the preset instruction is an upload instruction, the transmission task is determined to be an upload task.

[0063] Among them, the processing logic of the dynamic thread pool is selected based on the task type of the transmission task. It is further explained that the asynchronous processing in this embodiment is implemented by the dynamic thread pool. When the transmission task of asynchronous processing is initiated, the transmission task is submitted to the dynamic thread pool for execution, and the main thread can continue to perform other operations. For example, the CompletableFuture in Java or the Task in C# use a dynamic thread pool at the bottom layer to perform asynchronous operations. The dynamic thread pool provides controllable concurrent resources for asynchronous processing, avoiding the exhaustion of system resources caused by unlimited thread creation. The concurrency of the transmission task is controlled by configuring the size of the dynamic thread pool. The dynamic thread pool provides a task queue mechanism. When all threads are busy, new transmission tasks can queue up. In addition, the dynamic thread pool can also provide a task rejection policy and take corresponding measures when the system is overloaded. When executing the transmission task, this embodiment combines asynchronous processing with the dynamic thread pool to build a reliable and efficient concurrent application.

[0064] In step S202, the step of using a dynamic thread pool to perform multi-threaded processing on the data content of the Excel file includes: When executing the upload task, the data content of the Excel file is divided into multiple data blocks, and each data block is uploaded in parallel using multi-threading. The uploaded data blocks are parsed, and the data blocks are merged based on the parsing results to obtain the data results.

[0065] In one embodiment, in the step of dividing the data content of the Excel file into multiple data blocks, the data content of the Excel file is obtained, row information of the data content is obtained, and the data content is divided into multiple data blocks based on the row information.

[0066] In one embodiment, the step of parsing each uploaded data block includes: Multiple data blocks uploaded in parallel by multiple threads form multiple file streams; Use the preset reading and writing tools to parse the file stream and obtain the parsing results.

[0067] Exemplarily, the reading and writing tool can use the EasyExcel open source tool. Optionally, the reading and writing tool uses JavaBean to perform data parsing based on the list fields and data relationships of the data content. Among them, the EasyExcel open source tool is an Excel processing framework that uses the EasyExcel open source tool to read data line by line from the disk and parse it one by one, without loading all the file data into the memory at once, thereby reducing memory usage. JavaBean is a reusable component in the Java language. JavaBean can be used in JSP development to reduce duplicate code, making the development of the entire JSP code more concise.

[0068] Fourth embodiment Based on the third embodiment, in step S201 , in the step of determining the transmission task regarding the Excel file, when the preset instruction is a download instruction, the transmission task is determined to be a download task.

[0069] In step S202, the step of using a dynamic thread pool to perform multi-threaded processing on the data content of the Excel file includes: When executing the download task, multi-threading is used to perform paging query on the preset database to obtain a data set, and the data content in the data set is written into the preset worksheet to obtain the Excel file to be downloaded.

[0070] The steps of writing the data content in the dataset into the worksheet of the Excel file include: Query data based on multi-threaded paging to obtain file stream; Use the preset reading and writing tools to write the file stream so that the data content in the dataset is written into the worksheet.

[0071] Exemplarily, the reading and writing tool can adopt the EasyExcel open source tool, or can adopt multi-threaded paging writing. Among them, when the data content exceeds the maximum data size of the preset worksheet, multi-threaded paging writing is adopted. Specifically, when the EasyExcel open source tool is used to write data content to the worksheet, the writing operation is performed according to the mapping of Excel list fields and data relationships in JavaBean; when the data content exceeds the maximum data size of the threshold form, the multi-threaded paging writing operation is adopted.

[0072] In summary, the method provided in the above embodiment adopts the method of receiving preset instructions; generating an asynchronous processing request according to the preset instructions; asynchronously processing the Excel file to be processed using a dynamic thread pool based on the asynchronous processing request to obtain a result file; and executing a preset process on the result file. Thus, it is determined that this method combines the asynchronous processing mechanism with the dynamic thread pool to implement the upload or download of Excel files. The asynchronous processing mechanism is used to dynamically adjust the thread data in the dynamic thread pool to decompose the upload task or download task corresponding to the preset instruction into multiple subtasks corresponding to the thread data, thereby dynamically allocating thread resources, improving the file transfer rate and efficiency, and reasonably allocating system resources.

[0073] Because an asynchronous processing mechanism is used to execute Excel file upload and download, a timely response can be made after receiving the upload or download instruction, avoiding the other party waiting for the response result while the server is processing; the speed of file upload and download is significantly improved, and the transmission time is greatly shortened, especially for the processing of large files and a large number of small files; the system resources are fully utilized, the overall performance and concurrent processing capabilities of the system are improved, and resource waste is reduced; the stability and reliability of file transmission are enhanced, and it can adapt to different network environments and system loads.

[0074] Due to the use of a dynamic thread pool, thread data can be dynamically adjusted according to the generated asynchronous processing requests, thereby improving multi-threaded processing performance, rationally utilizing system resources, avoiding insufficient processing performance caused by a sudden increase in business requests, and avoiding resource waste when the number of business requests is small; improving user experience and making file transfer operations smoother and more efficient.

[0075] Fifth embodiment In one embodiment, if Figure 6 As shown, a processing system is provided for implementing any one of the processing methods described in the second to fourth embodiments.

[0076] The processing system provided in this embodiment is divided into two areas, including: a file processing area and a thread management area.

[0077] The file processing area includes: an asynchronous file manager 211 , a WEB_UI file manager 212 , an Excel data parser 213 , an Excel data loader 214 , and a file counter 215 .

[0078] The asynchronous file manager 211, connected to the WEB_UI file manager 212, the Excel data parser 213, and the Excel data loader 214, maintains the upload or download process. Upon receiving an upload instruction, it generates an asynchronous processing request and dispatches it to the Excel data parser. Upon receiving a download instruction, it generates an asynchronous processing request and dispatches it to the Excel data loader. It also records the updated upload or download progress status (uploading, upload success, upload failure, and downloading, download success, download failure) in real time and records the path of the uploaded or downloaded file.

[0079] The WEB_UI file manager 212 uses a web browser to manage asynchronously uploaded or downloaded Excel files based on a visual user interface. The interface displays file information in the asynchronous file manager's processing flow, allowing real-time query and tracking of upload or download progress. Optionally, the uploaded file can be downloaded from the displayed upload list. If an upload fails, the error cause can be viewed in the downloaded uploaded file. The resulting file can also be downloaded from the displayed download list.

[0080] The Excel data parser 213 is used to parse the data content of Excel files in the processing flow. Upon receiving an asynchronous processing request generated based on an upload instruction issued by the asynchronous file manager, it calls a dynamic thread pool to execute the multi-threaded upload task and generate a file stream. It then calls a preset read / write tool to parse the generated file stream. The read / write tool can be the open-source EasyExcel tool. The parsing process can be performed based on the relationship between the Excel list fields and the data content mapped in the JavaBean. This utilizes dynamic thread pool technology to improve parsing performance and rationally allocate system resources. The parsing status (uploading, upload successful, upload failed) and the resulting file are synchronized to the asynchronous file manager in real time.

[0081] The Excel data loader 214 loads the required data content into the Excel file. Upon receiving an asynchronous processing request generated based on a download instruction distributed by the asynchronous file manager, the multithreaded process of the dynamic thread pool is invoked to perform a paged query on the data content and generate a file stream. A preset read / write tool is then called to write to the generated file stream. The read / write tool can be the open-source EasyExcel tool. The writing process can be performed according to the relationship between the Excel list fields and the data content mapped in the JavaBean. Thus, the use of dynamic thread pool technology improves loading performance and rationally allocates system resources. The loading status (downloading, download successful, download failed) and the resulting file are synchronized to the asynchronous file manager in real time.

[0082] The file counter 215 is connected to the Excel data parser and the Excel data loader, and counts the number of upload or download requests based on the running status of the Excel data parser and the Excel data loader.

[0083] The thread management area includes: a policy adjustment center 221 , a WEB_UI thread manager 222 , a data transceiver center 223 , a timing collector 224 , and a cache middleware 225 .

[0084] The WEB_UI thread manager 222 is connected to the data transceiver center 223 and uses a web browser to query or modify the thread data of the dynamic thread pool based on a visual user interface. The dynamic thread pool thread data is displayed through the visual user interface, allowing manual adjustment of the maximum number of threads and the number of core threads in the dynamic thread pool. The modified thread data is fed back to the data transceiver center for timely implementation.

[0085] The data transceiver center 223 is connected to the WEB_UI thread manager 222 and the policy adjustment center 221, receives the thread data after the dynamic thread pool is changed from the WEB_UI thread manager 222 or the policy adjustment center 221, and uploads it to the cache middleware; provides a unified entry for query and modification for the dynamic thread pool, so as to obtain the thread data about the dynamic thread pool in the cache, the thread data after the dynamic thread pool is changed, and synchronize the latest thread data of the dynamic thread pool to the cache middleware.

[0086] The policy adjustment center 221 is connected to the file counter 215, the data transceiver center 223, and the timing collector 224. It monitors the request volume counted by the file counter and the thread data of the dynamic thread pool, dynamically adjusts the thread data of the dynamic thread pool to optimize the use of system resources, and feeds back the adjusted thread data to the data transceiver center.

[0087] The timing collector 224 regularly collects thread data of the dynamic thread pool and feeds it back to the policy adjustment center.

[0088] The cache middleware 225 stores the dynamic thread pool information. Redis is used as the cache middleware to store the thread data of the dynamic thread pool, ensuring that the data transceiver center can efficiently query and modify the dynamic thread pool information.

[0089] Exemplarily, due to the use of the processing system provided by this embodiment, a dynamic thread pool can be implemented. Among them, when the server is initialized and started, the configuration information of the dynamic thread pool is obtained, it is assembled as a service, and the Bean is registered in the Spring container so that other services can call it to obtain the file processing-specific thread pool information in the system. The service of the thread pool is called on time by the timer collector, and the current thread data of the queried thread pool is published; then, based on the thread data of the current thread pool monitored, the maximum number of threads and the number of core threads in the dynamic thread pool are dynamically adjusted in combination with the number of upload and download requests counted by the file counter, and then the scheduled thread data is changed to run in the thread pool through the data transceiver center, and the results are stored in the Redis of the cache middleware for query. Optionally, the thread pool thread data can be viewed and modified in the WEB_UI thread manager.

[0090] In summary, the processing system provided in this application is used to implement any of the processing methods described in the second to fourth embodiments, and to realize the uploading and downloading of Excel files through an asynchronous processing mechanism; after the server receives the upload instruction or download instruction sent by the client, it can respond in a timely manner, which can avoid the client waiting for the response result while the server is processing, and can display the upload or download process results.

[0091] Multi-threading based on dynamic thread pool can parse Excel data, load data, improve processing performance, and monitor the usage of thread pool and the number of requests in the process in real time, balance and adjust the maximum number of threads and core threads in the thread pool, and rationally utilize system resources to avoid insufficient processing performance caused by a sudden increase in business requests and waste of resources when the number of business requests is small.

[0092] In one embodiment, Figure 7 As shown, a processing device is provided, including: a communication module 301, an asynchronous module 302, a processing module 303 and an execution module 304, wherein: The communication module 301 is used to receive preset instructions.

[0093] The asynchronous module 302 is used to generate an asynchronous processing request according to the preset instruction.

[0094] The processing module 303 is used to perform asynchronous processing on the Excel file to be processed by using a dynamic thread pool based on the asynchronous processing request to obtain a result file.

[0095] The execution module 304 is used to execute a preset process on the result file.

[0096] For the specific definition of the processing device, please refer to the definition of the processing method above and will not be repeated here. Each module in the above-mentioned processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software so that the processor can call and execute the operations corresponding to each of the above modules.

[0097] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store processing data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a processing method is implemented.

[0098] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0099] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented: receiving a preset instruction; generating an asynchronous processing request according to the preset instruction; based on the asynchronous processing request, asynchronously processing an Excel file to be processed using a dynamic thread pool to obtain a result file; and executing a preset process on the result file.

[0100] In one embodiment, a computer-readable storage medium is provided, which stores a computer program. When executed by a processor, the computer program implements the following steps: receiving a preset instruction; generating an asynchronous processing request according to the preset instruction; asynchronously processing the Excel file to be processed using a dynamic thread pool based on the asynchronous processing request to obtain a result file; executing a preset process on the result file Based on the same inventive concept as the above embodiments, the embodiment of the present invention further provides a computer device, such as Figure 9 As shown, the device includes: a processor 310 and a memory 311 storing a computer program; wherein, Figure 9 The processor 310 shown in the figure is not used to indicate that the number of processors 310 is one, but is only used to indicate the positional relationship of the processor 310 relative to other devices. In actual applications, the number of processors 310 may be one or more; similarly, Figure 9 The memory 311 shown in the figure has the same meaning, that is, it is only used to refer to the position relationship of the memory 311 relative to other devices. In actual application, the number of memories 311 can be one or more. When the processor 310 runs the computer program, the method applied to the above device is implemented.

[0101] The device may also include: at least one network interface 312. The various components in the device are coupled together via a bus system 313. It is understood that the bus system 313 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 313 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 9 Various buses are labeled as bus system 313.

[0102] Memory 311 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM); magnetic surface memory may include magnetic disk or tape memory. Volatile memory may include random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memory 311 described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memories.

[0103] The memory 311 in this embodiment of the present invention is used to store various types of data to support the operation of the device. Examples of this data include: any computer programs used to operate on the device, such as the operating system and application programs; contact data; phone book data; messages; images; videos, etc. The operating system includes various system programs, such as the framework layer, core library layer, and driver layer, which implement various basic services and handle hardware-based tasks. Application programs may include various applications, such as media players and browsers, which implement various application services. Here, the program implementing the method of the embodiment of the present invention may be included in the application program.

[0104] Based on the same inventive concept as the above-mentioned embodiment, this embodiment further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. The computer-readable storage medium may be a magnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); or various devices including one or any combination of the above-mentioned memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc. When the computer program stored in the computer-readable storage medium is executed by a processor, the above-mentioned method is implemented. For the specific steps implemented when the computer program is executed by the processor, please refer to Figure 1 The description of the illustrated embodiment will not be repeated here.

[0105] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0106] As used herein, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion of elements other than the listed elements and may also include additional elements not specifically listed.

[0107] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A processing method, applied to a server, characterized in that: The method comprises: Receive preset instructions; Generate an asynchronous processing request according to the preset instruction; Based on the asynchronous processing request, the Excel file to be processed is asynchronously processed by using a dynamic thread pool to obtain a result file; Execute a preset process on the result file.

2. The processing method according to claim 1, characterized in that The step of generating an asynchronous processing request includes: Based on the preset instructions sent by the client, an asynchronous processing request is generated using the Java asynchronous mechanism to respond to the client.

3. The processing method according to claim 1, characterized in that Before the step of generating the asynchronous processing request, the method further includes: Determine a transmission task for an Excel file, and then execute a multi-threaded processing flow based on the task type of the transmission task; wherein, when the preset instruction is an upload instruction, the transmission task is determined to be an upload task; when the preset instruction is a download instruction, the transmission task is determined to be a download task.

4. The processing method according to claim 1, characterized in that The step of asynchronously processing the Excel file to be processed by using a dynamic thread pool includes: A dynamic thread pool is used to perform multi-threaded processing on the data content of the Excel file to obtain a result file.

5. The processing method according to claim 1, characterized in that Before the step of using a dynamic thread pool to perform multi-threaded processing on the data content of the Excel file, the method includes: Counting the number of requests for the asynchronous processing; Extracting current thread data of the dynamic thread pool; Based on the request amount and the thread data, the thread data required for concurrent execution of multiple threads is dynamically adjusted.

6. The processing method according to claim 5, characterized in that: The thread data includes a maximum number of threads and a core number of threads; and the step of dynamically adjusting the thread data required for concurrent execution of multiple threads based on the request volume and the thread data includes: Compare the request amount with the maximum number of threads and the number of core threads; When the request amount is between the maximum number of threads and the number of core threads, the thread data remains unchanged; When the number of requests exceeds the maximum number of threads, the excess ratio between the number of requests and the maximum number of threads is used to calculate the additional thread data, thereby adjusting the maximum number of threads and the number of core threads. When the request amount is less than the number of core threads, the ratio of underutilized threads is calculated and the reduction ratio is calculated to adjust the number of core threads.

7. The processing method according to claim 4, characterized in that The step of performing multi-thread processing on the data content of the Excel file by using a dynamic thread pool includes: When executing the upload task, the data content of the Excel file is divided into multiple data blocks, and each data block is uploaded in parallel using multiple threads. The uploaded data blocks are parsed, and the data blocks are merged based on the parsing results to obtain the data results; When executing the download task, multi-threading is used to perform paging query on the preset database to obtain a data set, and the data content in the data set is written into the preset worksheet to obtain the Excel file to be downloaded.

8. The processing method according to claim 7, characterized in that: The step of performing multi-thread processing on the data content of the Excel file using a dynamic thread pool also includes: Get the file stream when processing data in multiple threads; Use the preset reading and writing tools to parse or write the file stream to obtain the result file.

9. The processing method according to claim 7, characterized in that: The step of executing a preset process on the result file includes one of the following: When executing the upload task, the data result is verified. If the data result passes the verification, the data result is saved. If the data result fails the verification, the error cause is obtained, the error cause is mapped to the corresponding row information of the Excel file, and the error cause is written into the imported Excel file. When executing the download task, the obtained Excel file is sent to the client.

10. A computing device, characterized in that include: A processor and a memory storing a computer program, wherein when the processor runs the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

11. A storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images