A method and system for efficiently processing massive files based on cloud functions

Abstract

This invention relates to the field of internet technology, specifically to a method and system for efficiently processing massive files based on cloud functions. The system includes a demand acquisition and cloud function locking module that acquires the business usage requirements of cloud users and locks the cloud functions and corresponding cloud function interfaces for the business files to be processed within those requirements. This invention deeply customizes cloud functions and object storage, reducing file upload and download processing, minimizing network impact, improving file processing speed, and reducing server consumption. File processing is event-triggered, and its business characteristics are suitable for serverless technology, providing on-demand supply. During periods of high-volume file processing, cloud function instances can be automatically scheduled to meet processing demands. Developers do not need to monitor business peaks and troughs, saving idle time costs and ultimately reducing operational costs. It also enables asynchronous processing of large batches of files, meeting the needs of high-concurrency business scenarios and achieving peak-shaving scheduling.

Description

Technical Field

[0001] This invention relates to the field of Internet technology, specifically to a method and system for efficiently processing massive files based on cloud functions. Background Technology

[0002] Cloud storage provides enterprises with a storage solution that reduces the costs of purchasing, maintaining, and upgrading physical storage devices that are traditionally required. For small and medium-sized enterprises (SMEs), cloud storage can significantly reduce their IT budgets, allowing them to invest more funds in their core business.

[0003] However, while cloud storage providers have a fixed number of servers for storing data, user companies experience significant fluctuations in their business data and face a wide variety of business needs over time. Existing cloud storage technologies struggle to meet the current cloud storage requirements of enterprise users, primarily in the following aspects:

[0004] (1) A large number of corporate accountants need to upload a large number of PDF, Excel and other files. The files are large, with some large files reaching hundreds of megabytes (especially concentrated at the beginning and end of the month). A large amount of file parsing, file format conversion and file decompression processing consume a lot of server memory and has high requirements for server configuration.

[0005] (2) High concurrency, large file processing requires high-configuration servers to process, cloud providers have limited resources and cannot provide such huge resources in a short time, making server scheduling difficult and resulting in serious resource waste.

[0006] (3) In high-concurrency scenarios, user experience must also be guaranteed. Files under 10M should be processed in near real-time. Ordinary cloud host scheduling time is difficult to meet the real-time experience of users.

[0007] (4) This mode also brings complexity to host scheduling, requires matching a lot of additional system resources for processing, and also incurs high costs.

[0008] Therefore, existing cloud storage technologies have significant shortcomings. Summary of the Invention

[0009] The purpose of this invention is to provide a method and system for efficiently processing massive files based on cloud functions, so as to solve the problems mentioned in the background art.

[0010] To address the aforementioned technical problems, this invention provides the following technical solution: a method for efficiently processing massive files based on cloud functions, comprising the following steps:

[0011] Step S100: Based on cloud function development tools, business types, and cloud object storage, develop and encapsulate cloud functions with different functions, and bind each cloud function to a pre-defined cloud function interface;

[0012] Step S200: Obtain the business usage requirements of cloud users, and identify the cloud functions and corresponding cloud function interfaces corresponding to the business files to be processed in the business usage requirements of cloud users;

[0013] Step S300: Extract the business characteristics of the cloud user's business usage requirements, and combine the current cloud business processing status and historical business processing data to generate cloud function call method determination results for the pending business files in the cloud user's business usage requirements; the cloud function call method determination results include synchronous execution and asynchronous execution;

[0014] Step S400: Summarize the pending business files that are executed synchronously, call and execute the corresponding cloud functions for each pending business file in the summary result through the corresponding cloud function interface, and feed back the cloud function processing results of the corresponding pending business files to the corresponding business parties;

[0015] Step S500: Summarize the asynchronously executed pending business files, send the file processing tasks corresponding to the summarization results to RabbitMQ, and have RabbitMQ listen for the asynchronously executed cloud functions of the file processing tasks corresponding to the summarization results. RabbitMQ will then notify each business party of the cloud function processing results of the corresponding pending business files.

[0016] Preferably, the specific implementation process of step S100 includes:

[0017] Step S101: Using the cloud function development tool, develop cloud functions with different functions for different business types and cloud object storage; the cloud function functions include file compression, decompression and file conversion functions;

[0018] Step S102: Test the developed cloud function, use the cloud function to directly manipulate the object storage data, execute the test file in the object storage data through the data processing method inside the corresponding cloud function, and pass the execution result to the corresponding object storage;

[0019] Step S103: Receive the analysis results from step S102, compare the obtained analysis results with the preset analysis results corresponding to the corresponding test file. If the obtained analysis results are the same as the preset analysis results corresponding to the corresponding test file, it is determined that the corresponding cloud function function test has passed, and the corresponding cloud function is packaged and released; otherwise, it is determined that the corresponding cloud function function test has failed, the corresponding cloud function needs to be optimized, the optimized cloud function replaces the original cloud function, and jumps to step S102.

[0020] This invention utilizes cloud function technology from cloud vendors, combined with cloud object storage, to deeply customize and develop cloud function functionalities such as file compression and file conversion. The cloud functions developed using this combination of cloud functions and object storage can reduce file uploads and downloads, improving file processing efficiency. The process is developed using Python to enhance the startup and execution speed of the cloud functions, which are then deployed to cloud services.

[0021] Preferably, step S200 includes:

[0022] The cloud user's business usage requirements include each pending business file, the business type corresponding to each pending business file, and the file size; the cloud function corresponding to the pending business file in the cloud user's business usage requirements is obtained by querying the cloud function corresponding to the business type of the pending business file in the database; the corresponding cloud function interface is the cloud function interface bound to the cloud function corresponding to the pending business file in the cloud user's business usage requirements.

[0023] The cloud functions in this invention are pre-built cloud functions developed according to business needs. The cloud functions can be elastically expanded and shrunk according to the number of requested cloud function interfaces. At the same time, each business develops an independent cloud function, which makes the cloud function business simple and straightforward, the cloud function scheduling clear, the amount of business code small, and the cloud function startup speed faster.

[0024] Preferably, the specific implementation process of step S300 includes:

[0025] Step S301: Extract business characteristics from the business usage requirements of cloud users to obtain business requirement feature data, which includes the file size of the business file to be processed and the corresponding cloud function;

[0026] Step S302: Obtain the current cloud-based business processing status and historical business processing data;

[0027] Step S303: Calculate the congestion risk assessment coefficient corresponding to the i-th pending business file in the cloud user's business usage requirements. The calculation formula is as follows:

[0028] Pi=max{BV·(FR·max{Gi / VC|i∈[1,i1]}+1),BV},

[0029] Where Pi represents the congestion risk assessment coefficient corresponding to the i-th pending business file in the cloud user's business usage needs; BV represents the ratio of the sum of pending business volumes corresponding to each cloud user in the current cloud business processing state to the preset network bandwidth corresponding to the cloud server; pending business volume represents the file size of each pending business file in the corresponding business usage needs; FR represents the characteristic change coefficient composed of the sum of pending business volumes of each cloud user in the cloud server at different time points in the current cloud business processing state based on the preset unit time before the current time; VC represents the average file processing speed corresponding to each processed business file in the historical business processing data, where the file processing speed is equal to the average value of the quotient of the file size corresponding to the processed business file divided by the corresponding processing time; Gi represents the file size of the i-th pending business file in the cloud user's business usage needs; i1 represents the number of pending business files in the cloud user's business usage needs; max{} represents the operation of finding the maximum value.

[0030] During the process of obtaining FR, the sum of the pending business volume of each cloud user on the cloud server corresponding to any time point t within the preset unit time based on the current time is denoted as Gt, where t∈[0,T1] and T1 represents the preset unit time; the average value of each Gt corresponding to different values ​​of t within the preset unit time based on the current time is calculated and denoted as GVt; a set of trend change fluctuation points constructed within the preset unit time based on the current time is obtained and denoted as the trend change fluctuation point set; the trend change fluctuation point is the sum of the pending business volume of each cloud user on the cloud server corresponding to the corresponding time point; the Gt corresponding to each element in the trend change fluctuation point set is equal to GT1 and is greater than the sum of the pending business volume of each cloud user on the cloud server corresponding to the adjacent time points. The sum, or simultaneously less than the sum of the pending business volumes of each cloud user on the cloud server corresponding to the adjacent time points; GT1 represents the sum of the pending business volumes of each cloud user on the cloud server corresponding to the current time T1 within the preset unit time before the current time; obtain the element with the smallest difference between GVt and the current time interval among the elements in the trend change fluctuation point set, and record it as the trend change fluctuation reference point; calculate the difference obtained by subtracting the element with the smallest absolute value of the difference between G0 and VC, and divide it by the quotient of the time interval between the time point corresponding to the element with the smallest absolute value of the difference between G0 and VC and the current time; G0 represents the sum of the pending business volumes of each cloud user on the cloud server corresponding to the current time;

[0031] Step S304: Obtain the cloud function call method determination result corresponding to the i-th pending business file in the cloud user's business usage requirements;

[0032] If Gi≥e -Pi If N is less than or equal to 0, the cloud function call method corresponding to the i-th pending business file in the cloud user's business usage requirements is determined to be asynchronous; otherwise, the cloud function call method corresponding to the i-th pending business file in the cloud user's business usage requirements is determined to be synchronous. N is a preset threshold for the size of the pending business file.

[0033] Preferably, step S400 includes:

[0034] In the summary results of the pending business files executed synchronously, each pending business file directly calls and executes the corresponding cloud function on the cloud server. The functions of the cloud functions called by different pending business files are different.

[0035] When the cloud function processing result of the corresponding pending business file is fed back to the corresponding business party, the cloud function processing result of the corresponding pending business file is directly fed back to the cloud account by the cloud server.

[0036] Preferably, step S500 includes:

[0037] In the summary results of asynchronously executed pending business files, each pending business file is sent to RabbitMQ via RabbitMQ messages, and RabbitMQ messages are listened to in real time. Based on the listening results, the corresponding cloud functions of each pending business file are called and executed in RabbitMQ. When feeding back the processing results of the cloud functions of each pending business file to each business party, the processing results of the cloud functions of the corresponding pending business file are fed back by RabbitMQ to the cloud account in the cloud server. Each business party corresponds to a business account.

[0038] The purpose of determining the cloud function call method in this invention is to distribute the processing tasks corresponding to the business files to be processed under high concurrency of business data, i.e., synchronous execution and asynchronous execution. In this process, the synchronous execution method is to ensure the processing response speed of the corresponding business files to be processed; while the asynchronous execution method is to prevent large file business from consuming too much server resources due to long processing time and avoiding service blocking caused by large files under high concurrency. The processing tasks corresponding to the business files to be processed are peak-shaving through RabbitMQ.

[0039] A system for efficiently processing massive files based on cloud functions, the system comprising: a cloud function development and management module, a requirement acquisition and cloud function locking module, a cloud function call method analysis module, a cloud function synchronous execution management module, and a cloud function asynchronous execution management module.

[0040] The cloud function development and management module develops and encapsulates cloud functions with different functions based on cloud function development tools, business types, and cloud object storage. Each cloud function is bound to a pre-built cloud function interface.

[0041] The requirement acquisition and cloud function locking module acquires the business usage requirements of cloud users and locks the cloud functions and corresponding cloud function interfaces corresponding to the business files to be processed in the business usage requirements of cloud users.

[0042] The cloud function call method analysis module extracts the business characteristics of cloud users' business usage needs, and combines the current cloud business processing status and historical business processing data to generate cloud function call method determination results for the pending business files in the cloud users' business usage needs; the cloud function call method determination results include synchronous execution and asynchronous execution;

[0043] The cloud function synchronous execution management module summarizes the pending business files to be executed synchronously, calls and executes the corresponding cloud functions for each pending business file in the summary result through the corresponding cloud function interface, and feeds back the cloud function processing results of the corresponding pending business files to the corresponding business parties.

[0044] The cloud function asynchronous execution management module aggregates the pending business files to be executed asynchronously, and sends the file processing tasks corresponding to the aggregated results to RabbitMQ. RabbitMQ listens for the cloud functions that are executed asynchronously for the file processing tasks corresponding to the aggregated results, and notifies each business party of the cloud function processing results of the corresponding pending business files.

[0045] Preferably, the cloud function call method analysis module includes a business requirement characteristic extraction unit and a cloud function call method discrimination unit.

[0046] The business requirement feature extraction unit extracts business characteristics from the business usage requirements of cloud users.

[0047] The cloud function call method discrimination unit combines the current cloud business processing status and historical data to generate cloud function call method discrimination results for the pending business files in the cloud user's business usage requirements.

[0048] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0049] 1) This invention features deep customization of cloud functions and object storage, reducing file upload and download processing, minimizing network impact, improving file processing speed, and reducing server consumption;

[0050] 2) File processing is event-triggered, and its business characteristics are suitable for serverless technology. It provides on-demand supply, and when there is a surge in massive file processing business, cloud function instances can be automatically scheduled to meet the processing needs. Developers do not need to pay attention to business peaks and troughs, saving idle time costs and ultimately reducing operation and maintenance costs.

[0051] 3) Custom development of RabbitMQ and cloud functions to achieve asynchronous processing of large batches of files, meet the needs of high-concurrency business scenarios, and realize peak shaving scheduling function;

[0052] 4) Each business function is an independent cloud function, and cloud functions do not interfere with each other. The business functions are clear, cloud function scheduling is simple, and cloud function startup speed is faster. Attached Figure Description

[0053] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0054] Figure 1 This is a schematic diagram of the structure of a system for efficiently processing massive files based on cloud functions according to the present invention;

[0055] Figure 2 This is a schematic diagram illustrating the steps of a method for efficiently processing massive files based on cloud functions according to the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Please see Figure 1 This invention provides a technical solution: a system for efficiently processing massive files based on cloud functions, the system comprising: a cloud function development and management module, a requirement acquisition and cloud function locking module, a cloud function call method analysis module, a cloud function synchronous execution management module, and a cloud function asynchronous execution management module.

[0058] The cloud function development and management module develops and encapsulates cloud functions with different functions based on cloud function development tools, business types, and cloud object storage. Each cloud function is bound to a pre-built cloud function interface.

[0059] The requirement acquisition and cloud function locking module acquires the business usage requirements of cloud users and locks the cloud functions and corresponding cloud function interfaces corresponding to the business files to be processed in the business usage requirements of cloud users.

[0060] The cloud function call method analysis module includes a business requirement feature extraction unit and a cloud function call method discrimination unit.

[0061] The business requirement feature extraction unit extracts business characteristics from the business usage requirements of cloud users.

[0062] The cloud function call mode determination unit combines the current cloud business processing status and historical business processing data to generate cloud function call mode determination results corresponding to the pending business files in the cloud user's business usage requirements; the cloud function call mode determination results include synchronous execution and asynchronous execution.

[0063] The cloud function synchronous execution management module summarizes the pending business files to be executed synchronously, calls and executes the corresponding cloud functions for each pending business file in the summary result through the corresponding cloud function interface, and feeds back the cloud function processing results of the corresponding pending business files to the corresponding business parties.

[0064] The cloud function asynchronous execution management module aggregates the pending business files to be executed asynchronously, and sends the file processing tasks corresponding to the aggregated results to RabbitMQ. RabbitMQ listens for the cloud functions that are executed asynchronously for the file processing tasks corresponding to the aggregated results, and notifies each business party of the cloud function processing results of the corresponding pending business files.

[0065] Example 1:

[0066] Cloud functions: Real-time scheduling service resources provided by various cloud vendors. Users can develop cloud function services based on the cloud function development SDK provided by the cloud vendors. Deployment and operation are all configured on the cloud function configuration page of the cloud vendors.

[0067] Object storage: Various cloud providers offer services for storing files in various formats, charging based on the total number of files. Almost all of Cloud Accounting's files are stored on object storage systems across different clouds. Uploading and downloading files using object storage consumes significant bandwidth.

[0068] RabbitMQ: A message queue that handles asynchronous tasks and high-concurrency file processing tasks, and is responsible for scheduling file tasks.

[0069] like Figure 2 As shown, a method for efficiently processing massive files based on cloud functions is presented. This method includes the following steps:

[0070] Step S100: Based on cloud function development tools, business types, and cloud object storage, develop and encapsulate cloud functions with different functions, and bind each cloud function to a pre-defined cloud function interface;

[0071] The specific implementation process of step S100 includes:

[0072] Step S101: Using the cloud function development tool, develop cloud functions with different functions for different business types and cloud object storage; the cloud function functions include file compression, decompression and file conversion functions;

[0073] Step S102: Test the developed cloud function, use the cloud function to directly manipulate the object storage data, execute the test file in the object storage data through the data processing method inside the corresponding cloud function, and pass the execution result to the corresponding object storage;

[0074] Step S103: Receive the analysis results from step S102, compare the obtained analysis results with the preset analysis results corresponding to the corresponding test file. If the obtained analysis results are the same as the preset analysis results corresponding to the corresponding test file, it is determined that the corresponding cloud function function test has passed, and the corresponding cloud function is packaged and released; otherwise, it is determined that the corresponding cloud function function test has failed, the corresponding cloud function needs to be optimized, the optimized cloud function replaces the original cloud function, and jumps to step S102.

[0075] Step S200: Obtain the business usage requirements of cloud users, and identify the cloud functions and corresponding cloud function interfaces corresponding to the business files to be processed in the business usage requirements of cloud users;

[0076] Step S200 includes:

[0077] The cloud user's business usage requirements include each pending business file, the business type corresponding to each pending business file, and the file size; the cloud function corresponding to the pending business file in the cloud user's business usage requirements is obtained by querying the cloud function corresponding to the business type of the pending business file in the database; the corresponding cloud function interface is the cloud function interface bound to the cloud function corresponding to the pending business file in the cloud user's business usage requirements.

[0078] Step S300: Extract the business characteristics of the cloud user's business usage requirements, and combine the current cloud business processing status and historical business processing data to generate cloud function call method determination results for the pending business files in the cloud user's business usage requirements; the cloud function call method determination results include synchronous execution and asynchronous execution;

[0079] The specific implementation process of step S300 includes:

[0080] Step S301: Extract business characteristics from the business usage requirements of cloud users to obtain business requirement feature data, which includes the file size of the business file to be processed and the corresponding cloud function;

[0081] Step S302: Obtain the current cloud-based business processing status and historical business processing data;

[0082] Step S303: Calculate the congestion risk assessment coefficient corresponding to the i-th pending business file in the cloud user's business usage requirements. The calculation formula is as follows:

[0083] Pi=max{BV·(FR·max{Gi / VC|i∈[1,i1]}+1),BV},

[0084] Where Pi represents the congestion risk assessment coefficient corresponding to the i-th pending business file in the cloud user's business usage needs; BV represents the ratio of the sum of pending business volumes corresponding to each cloud user in the current cloud business processing state to the preset network bandwidth corresponding to the cloud server; pending business volume represents the file size of each pending business file in the corresponding business usage needs; FR represents the characteristic change coefficient composed of the sum of pending business volumes of each cloud user in the cloud server at different time points in the current cloud business processing state based on the preset unit time before the current time; VC represents the average file processing speed corresponding to each processed business file in the historical business processing data, where the file processing speed is equal to the average value of the quotient of the file size corresponding to the processed business file divided by the corresponding processing time; Gi represents the file size of the i-th pending business file in the cloud user's business usage needs; i1 represents the number of pending business files in the cloud user's business usage needs; max{} represents the operation of finding the maximum value.

[0085] During the process of obtaining FR, the sum of the pending business volume of each cloud user on the cloud server corresponding to any time point t within the preset unit time based on the current time is denoted as Gt, where t∈[0,T1] and T1 represents the preset unit time; the average value of each Gt corresponding to different values ​​of t within the preset unit time based on the current time is calculated and denoted as GVt; a set of trend change fluctuation points constructed within the preset unit time based on the current time is obtained and denoted as the trend change fluctuation point set; the trend change fluctuation point is the sum of the pending business volume of each cloud user on the cloud server corresponding to the corresponding time point; the Gt corresponding to each element in the trend change fluctuation point set is equal to GT1 and is greater than the sum of the pending business volume of each cloud user on the cloud server corresponding to the adjacent time points. The sum, or simultaneously less than the sum of the pending business volumes of each cloud user on the cloud server corresponding to the adjacent time points; GT1 represents the sum of the pending business volumes of each cloud user on the cloud server corresponding to the current time T1 within the preset unit time before the current time; obtain the element with the smallest difference between GVt and the current time interval among the elements in the trend change fluctuation point set, and record it as the trend change fluctuation reference point; calculate the difference obtained by subtracting the element with the smallest absolute value of the difference between G0 and VC, and divide it by the quotient of the time interval between the time point corresponding to the element with the smallest absolute value of the difference between G0 and VC and the current time; G0 represents the sum of the pending business volumes of each cloud user on the cloud server corresponding to the current time;

[0086] Step S304: Obtain the cloud function call method determination result corresponding to the i-th pending business file in the cloud user's business usage requirements;

[0087] If Gi≥e -Pi If N is less than or equal to 0, the cloud function call method corresponding to the i-th pending business file in the cloud user's business usage requirements is determined to be asynchronous; otherwise, the cloud function call method corresponding to the i-th pending business file in the cloud user's business usage requirements is determined to be synchronous. N is a preset threshold for the size of the pending business file.

[0088] In this embodiment, N represents no more than 100 images in the corresponding PDF file or a file size of less than 10 MB.

[0089] Step S400: Summarize the pending business files that are executed synchronously, call and execute the corresponding cloud functions for each pending business file in the summary result through the corresponding cloud function interface, and feed back the cloud function processing results of the corresponding pending business files to the corresponding business parties;

[0090] Step S400 includes:

[0091] In the summary results of the pending business files executed synchronously, each pending business file directly calls and executes the corresponding cloud function on the cloud server. The functions of the cloud functions called by different pending business files are different.

[0092] When the cloud function processing result of the corresponding pending business file is fed back to the corresponding business party, the cloud function processing result of the corresponding pending business file is directly fed back to the cloud account by the cloud server.

[0093] Step S500: Summarize the asynchronously executed pending business files, send the file processing tasks corresponding to the summarization results to RabbitMQ, RabbitMQ listens for the cloud functions that are executed asynchronously for the file processing tasks corresponding to the summarization results, and RabbitMQ notifies each business party of the cloud function processing results of the corresponding pending business files.

[0094] Step S500 includes:

[0095] In the summary results of asynchronously executed pending business files, each pending business file is sent to RabbitMQ via RabbitMQ messages, and RabbitMQ messages are listened to in real time. Based on the listening results, the corresponding cloud functions of each pending business file are called and executed in RabbitMQ. When feeding back the processing results of the cloud functions of each pending business file to each business party, the processing results of the cloud functions of the corresponding pending business file are fed back by RabbitMQ to the cloud account in the cloud server. Each business party corresponds to a business account.

[0096] Example 2: User data collection for invoices. The original invoice files are in PDF format, but the page needs to display images. The file system needs to convert the PDF invoices into images. On the 1st of each month, tens of millions of PDFs need to be converted into images. The specific execution steps are as follows:

[0097] Step 11: The user uses the cloud accounting system to collect invoice data (PDF). The invoice PDF is uploaded to object storage and a download URL is returned (URL is a Uniform Resource Locator, a concise representation of the location and access method of a resource that can be obtained from the Internet. It is the address of a standard resource on the Internet; every file on the Internet has a unique URL, which contains information indicating the location of the file and how the browser should handle it).

[0098] Step 12: The system sends the URL of the invoice PDF to RabbitMQ, and the business side listens to RabbitMQ and receives the URL of the PDF.

[0099] Step 13: The business side calls the file conversion cloud function, passing in the URL of the PDF. The cloud function uses the URL to operate on object storage to download the PDF file, converts the PDF into a PNG image, uploads the image to the object storage service, and sends the image URL to the business side.

[0100] Step 14: The business side receives the image URL returned by the cloud function and saves the URL to the business database;

[0101] Step 15: The user searches for invoice images on the page.

[0102] Example 3: When a user prints documents at the end of the month, multiple PDF files are merged and downloaded.

[0103] Step 21: At the end of the month, users will print financial data such as vouchers, account balances, invoices, bank statements, and accounting statements. Each company will generate multiple PDFs. Users can compress these PDFs into a single zip file according to their company. Each PDF contains more than 100 images.

[0104] Step 22: The user selects a company on the page and clicks "File Download". The business side receives the interface request, generates multiple PDF files, and uploads the multiple PDFs to object storage step by step, generating multiple URLs.

[0105] Step 23: The business side calls the file compression cloud function, passing in the URLs of multiple PDFs. The cloud function operates on the object storage to download the PDFs based on the file URLs, merges and compresses the multiple PDFs into a zip file, and then operates on the object storage to upload the zip file to the object storage and returns the URL of the zip file.

[0106] Step 24: The business side receives the zip file URL message returned by the cloud function and saves the URL to the business database;

[0107] Step 25: The user downloads the zip file from the page.

[0108] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0109] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for efficiently processing massive files based on cloud functions, characterized in that, The method includes the following steps: Step S100: Based on cloud function development tools, business types, and cloud object storage, develop and encapsulate cloud functions with different functions, and bind each cloud function to a pre-defined cloud function interface; Step S200: Obtain the business usage requirements of cloud users, and identify the cloud functions and corresponding cloud function interfaces corresponding to the business files to be processed in the business usage requirements of cloud users; Step S300: Extract the business characteristics of the cloud user's business usage requirements, and combine the current cloud business processing status and historical business processing data to generate cloud function call method determination results for the pending business files in the cloud user's business usage requirements; the cloud function call method determination results include synchronous execution and asynchronous execution; The specific implementation process of step S300 includes: Step S301: Extract business characteristics from the business usage requirements of cloud users to obtain business requirement feature data, which includes the file size of the business file to be processed and the corresponding cloud function; Step S302: Obtain the current cloud-based business processing status and historical business processing data; Step S303: Calculate the congestion risk assessment coefficient corresponding to the i-th pending business file in the cloud user's business usage requirements. The calculation formula is as follows: Pi=max{BV·(FR·max{Gi / VC|i∈[1,i1]}+1),BV}, Where Pi represents the congestion risk assessment coefficient corresponding to the i-th pending business file in the cloud user's business usage needs; BV represents the ratio of the sum of pending business volumes corresponding to each cloud user in the current cloud business processing state to the preset network bandwidth corresponding to the cloud server; pending business volume represents the file size of each pending business file in the corresponding business usage needs; FR represents the characteristic change coefficient composed of the sum of pending business volumes of each cloud user in the cloud server at different time points in the current cloud business processing state based on the preset unit time before the current time; VC represents the average file processing speed corresponding to each processed business file in the historical business processing data, where the file processing speed is equal to the average value of the quotient of the file size corresponding to the processed business file divided by the corresponding processing time; Gi represents the file size of the i-th pending business file in the cloud user's business usage needs; i1 represents the number of pending business files in the cloud user's business usage needs; max{} represents the operation of finding the maximum value. During the process of obtaining FR, the sum of the pending business volume of each cloud user on the cloud server corresponding to any time point t within the preset unit time based on the current time is denoted as Gt, where t∈[0,T1] and T1 represents the preset unit time; the average value of each Gt corresponding to different values ​​of t within the preset unit time based on the current time is calculated and denoted as GVt; a set of trend change fluctuation points constructed within the preset unit time based on the current time is obtained and denoted as the trend change fluctuation point set; the trend change fluctuation point is the sum of the pending business volume of each cloud user on the cloud server corresponding to the corresponding time point; the Gt corresponding to each element in the trend change fluctuation point set is equal to GT1 and is greater than the sum of the pending business volume of each cloud user on the cloud server corresponding to the adjacent time points. The sum, or simultaneously less than the sum of the pending business volumes of each cloud user on the cloud server corresponding to the adjacent time points; GT1 represents the sum of the pending business volumes of each cloud user on the cloud server corresponding to the current time T1 within the preset unit time before the current time; obtain the element with the smallest difference between GVt and the current time interval among the elements in the trend change fluctuation point set, and record it as the trend change fluctuation reference point; calculate the difference obtained by subtracting the element with the smallest absolute value of the difference between G0 and VC, and divide it by the quotient of the time interval between the time point corresponding to the element with the smallest absolute value of the difference between G0 and VC and the current time; G0 represents the sum of the pending business volumes of each cloud user on the cloud server corresponding to the current time; Step S304: Obtain the cloud function call method determination result corresponding to the i-th pending business file in the cloud user's business usage requirements; If Gi≥e -Pi If N is less than N, then the cloud function call method corresponding to the i-th pending business file in the cloud user's business usage requirements is determined to be asynchronous; otherwise, the cloud function call method corresponding to the i-th pending business file in the cloud user's business usage requirements is determined to be synchronous; where N is a preset threshold for the size of the pending business file. Step S400: Summarize the pending business files that are executed synchronously, call and execute the corresponding cloud functions for each pending business file in the summary result through the corresponding cloud function interface, and feed back the cloud function processing results of the corresponding pending business files to the corresponding business parties; Step S500: Summarize the asynchronously executed pending business files, send the file processing tasks corresponding to the summarization results to RabbitMQ, and have RabbitMQ listen for the asynchronously executed cloud functions of the file processing tasks corresponding to the summarization results. RabbitMQ will then notify each business party of the cloud function processing results of the corresponding pending business files.

2. The method for efficiently processing massive files based on cloud functions according to claim 1, characterized in that, The specific implementation process of step S100 includes: Step S101: Using the cloud function development tool, develop cloud functions with different functions for different business types and cloud object storage; cloud function functions include file compression, decompression, and file conversion. Step S102: Test the developed cloud function, use the cloud function to directly manipulate the object storage data, execute the test file in the object storage data through the data processing method inside the corresponding cloud function, and pass the execution result to the corresponding object storage; Step S103: Receive the analysis results from step S102, compare the obtained analysis results with the preset analysis results corresponding to the corresponding test file. If the obtained analysis results are the same as the preset analysis results corresponding to the corresponding test file, it is determined that the corresponding cloud function function test has passed, and the corresponding cloud function is packaged and released; otherwise, it is determined that the corresponding cloud function function test has failed, the corresponding cloud function needs to be optimized, the optimized cloud function replaces the original cloud function, and jumps to step S102.

3. The method for efficiently processing massive files based on cloud functions according to claim 2, characterized in that, Step S200 includes: The cloud user's business usage requirements include each pending business file, the business type corresponding to each pending business file, and the file size; the cloud function corresponding to the pending business file in the cloud user's business usage requirements is obtained by querying the cloud function corresponding to the business type of the pending business file in the database; the corresponding cloud function interface is the cloud function interface bound to the cloud function corresponding to the pending business file in the cloud user's business usage requirements.

4. The method for efficiently processing massive files based on cloud functions according to claim 1, characterized in that, Step S400 includes: In the summary results of the pending business files executed synchronously, each pending business file directly calls and executes the corresponding cloud function on the cloud server. The functions called by different pending business files have different functions. When the cloud function processing result of the corresponding pending business file is fed back to the corresponding business party, the cloud function processing result of the corresponding pending business file is directly fed back to the cloud account by the cloud server.

5. The method for efficiently processing massive files based on cloud functions according to claim 1, characterized in that, Step S500 includes: In the summary results of asynchronously executed pending business files, each pending business file is sent to RabbitMQ via RabbitMQ messages, and RabbitMQ messages are listened to in real time. Based on the listening results, the corresponding cloud functions of each pending business file are called and executed in RabbitMQ. When feeding back the processing results of the cloud functions of each pending business file to each business party, the processing results of the cloud functions of the corresponding pending business file are fed back by RabbitMQ to the cloud account in the cloud server. Each business party corresponds to a business account.

6. A system for efficiently processing massive files based on cloud functions, employing the method for efficiently processing massive files based on cloud functions as described in any one of claims 1-5, characterized in that, The system includes: a cloud function development management module, a requirement acquisition and cloud function locking module, a cloud function call method analysis module, a cloud function synchronous execution management module, and a cloud function asynchronous execution management module. The cloud function development and management module develops and encapsulates cloud functions with different functions based on cloud function development tools, business types, and cloud object storage. Each cloud function is bound to a pre-built cloud function interface. The requirement acquisition and cloud function locking module acquires the business usage requirements of cloud users and locks the cloud functions and corresponding cloud function interfaces corresponding to the business files to be processed in the business usage requirements of cloud users. The cloud function call method analysis module includes a business requirement feature extraction unit and a cloud function call method discrimination unit. The business requirement feature extraction unit extracts business characteristics from the business usage requirements of cloud users. The cloud function call method determination unit combines the current cloud business processing status and the business processing data in historical data to generate cloud function call method determination results for the pending business files in the cloud user's business usage requirements. The cloud function synchronous execution management module summarizes the pending business files to be executed synchronously, calls and executes the corresponding cloud functions for each pending business file in the summary result through the corresponding cloud function interface, and feeds back the cloud function processing results of the corresponding pending business files to the corresponding business parties. The cloud function asynchronous execution management module aggregates the pending business files to be executed asynchronously, and sends the file processing tasks corresponding to the aggregated results to RabbitMQ. RabbitMQ listens for the cloud functions that are executed asynchronously for the file processing tasks corresponding to the aggregated results, and notifies each business party of the cloud function processing results of the corresponding pending business files.

Images