Method and system for efficiently processing massive files based on cloud function

Abstract

The invention relates to the technical field of the Internet, in particular to a method and system for efficiently processing massive files based on a cloud function. A demand obtaining and cloud function locking module in the system obtains the service use demand of a cloud user; and locking a cloud function and a corresponding cloud function interface corresponding to the to-be-processed service file in the service use demand of the cloud user. According to the method, cloud functions and object storage are deeply customized, file uploading and downloading processing is reduced, network influences are reduced, the file processing speed is increased, and server consumption is reduced; file processing belongs to event triggering, business characteristics are suitable for no-service technology and on-demand supply, cloud function instances can be automatically dispatched to meet processing requirements when massive file processing business outbreaks, developers do not need to pay attention to business wave crests and wave troughs, idle time cost is saved, finally operation and maintenance cost is reduced, asynchronous processing of massive files is achieved, and the efficiency is improved. And a service high-concurrency scene is satisfied, and a peak clipping scheduling function is realized.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and specifically to a method and system for efficiently processing massive files based on cloud functions. Background Art

[0002] Cloud storage provides enterprises with a storage solution that reduces the traditional cost of purchasing, maintaining and upgrading physical storage devices. For small and medium-sized enterprises, cloud storage can significantly reduce IT budgets, allowing them to invest more funds in core businesses.

[0003] However, for cloud storage vendors, the number of servers storing data is fixed, but the business data of user enterprises to be stored fluctuates greatly at different times, and the types of business are also relatively diverse; the existing cloud storage technology is difficult to meet the cloud storage needs of current enterprise users, which is mainly reflected in the following aspects:

[0004] (1) Accountants of many companies need to upload a large number of PDF, Excel and other files. The files are large in size, and some large files are as large as several hundred megabytes (especially at the beginning and end of the month). A large number of file parsing, file format conversion, and file decompression processing consume a lot of server memory, which places high requirements on server configuration.

[0005] (2) High concurrency: Large file processing requires highly configured servers. Cloud vendors have limited resources and cannot allocate such huge resources in a short period of time, making server scheduling difficult and resulting in serious resource waste.

[0006] (3) In high-concurrency scenarios, user experience must also be guaranteed. Files below 10M must be processed in near real-time. Ordinary cloud host scheduling time is difficult to meet users' real-time experience.

[0007] (4) This model also brings complexity to host scheduling, requiring the matching of many additional system resources for processing, and also incurring high costs;

[0008] Therefore, existing cloud storage technology has major flaws. Summary of the invention

[0009] The purpose of the present invention is to provide a method and system for efficiently processing massive files based on cloud functions, so as to solve the problems raised in the above-mentioned background technology.

[0010] In order to solve the above technical problems, the present invention provides the following technical solution: a method for efficiently processing massive files based on cloud functions, the method comprising the following steps:

[0011] Step S100: Develop and encapsulate cloud functions with different functions based on cloud function development tools, business types, and cloud object storage, and bind each cloud function to a preset cloud function interface;

[0012] Step S200: Obtain the business usage requirements of the cloud user, and lock the cloud function and the corresponding cloud function interface corresponding to the business file to be processed in the business usage requirements of the cloud user;

[0013] Step S300: extracting business characteristics from the business usage requirements of the cloud user, and combining the business processing status of the cloud at the current time and the business processing data in the historical data, generating cloud function call mode determination results corresponding to the business files to be processed in the business usage requirements of the cloud user; the cloud function call mode determination results include synchronous execution and asynchronous execution;

[0014] Step S400: Summarize the business files to be processed that are executed synchronously, call and execute the corresponding cloud function through the corresponding cloud function interface for each business file to be processed in the summary result, and feed back the cloud function processing result of the corresponding business file to be processed to the corresponding business party;

[0015] Step S500: Summarize the asynchronously executed business files to be processed, and transmit the file processing tasks corresponding to the summary results to RabbitMQ. RabbitMQ monitors the cloud functions corresponding to the asynchronously executed file processing tasks corresponding to the summary results, and notifies each business party of the cloud function processing results of the corresponding business files to be processed.

[0016] Preferably, the specific implementation process of step S100 includes:

[0017] Step S101: Develop cloud functions with different functions according to business types and cloud object storage using a cloud function development tool; the cloud function functions include file compression, decompression and file conversion functions;

[0018] Step S102: testing the developed cloud function, using the cloud function to directly operate the object storage data, executing the to-be-tested file in the object storage data through the data processing method inside the corresponding cloud function, and transmitting the execution result to the corresponding object storage;

[0019] Step S103: Receive the analysis result of step S102, and compare the obtained analysis result with the preset analysis result corresponding to the corresponding file to be tested. If the obtained analysis result is the same as the preset analysis result corresponding to the corresponding file to be tested, it is determined that the functional test of the corresponding cloud function has passed, and the corresponding cloud function is packaged and released; otherwise, it is determined that the functional test of the corresponding cloud function has not passed, and the corresponding cloud function needs to be optimized, and the optimized cloud function replaces the original cloud function, and jumps to step S102.

[0020] The present invention uses cloud vendor cloud function technology, combined with cloud object storage, and deeply customizes and develops cloud function functions such as file compression and file conversion. The cloud function developed by combining cloud function and object storage can reduce file upload and download and improve file processing efficiency. In this process, Python language is used for development to improve the startup and running speed of cloud functions, and the developed cloud functions are deployed to cloud services.

[0021] Preferably, the step S200 includes:

[0022] The business usage requirements of the cloud user include each business file to be processed, the business type and file size corresponding to each business file to be processed; the cloud function corresponding to the business file to be processed in the business usage requirements of the cloud user is obtained by querying the cloud function corresponding to the business type to which the business file to be processed belongs in the database; the corresponding cloud function interface is the cloud function interface bound to the cloud function corresponding to the business file to be processed in the business usage requirements of the cloud user.

[0023] The cloud function in the present invention is a prefabricated cloud function developed according to the business. The cloud function can be elastically expanded and reduced according to the number of requested cloud function interfaces. At the same time, each business develops an independent cloud function, which makes the cloud function business single and simple, the cloud function scheduling is clear, the business code amount is small, and the cloud function startup speed is faster.

[0024] Preferably, the specific implementation process of step S300 includes:

[0025] Step S301: extracting business characteristics of business usage requirements of cloud users to obtain business requirement characteristic data, wherein the business requirement characteristic data includes the file size of the business file to be processed and the corresponding cloud function;

[0026] Step S302: Obtain the current cloud business processing status and business processing data in historical data;

[0027] Step S303: Calculate the congestion risk assessment coefficient corresponding to the ith to-be-processed business file in the business usage requirements of the cloud user, and the calculation formula is as follows:

[0028] Pi=max{BV·(FR·max{Gi / VC|i∈[1,i1]}+1),BV},

[0029] Wherein, Pi represents the congestion risk assessment coefficient corresponding to the ith pending business file in the business usage demand of the cloud user; BV represents the ratio of the sum of the pending business volumes corresponding to each cloud user in the cloud server at the current time in the cloud business processing state to the preset network bandwidth corresponding to the cloud server; the pending business volume represents the file size of each pending business file in the corresponding business usage demand; FR represents the characteristic variation coefficient composed of the sum of the corresponding pending business volumes of each cloud user of the cloud server corresponding to different time points in the preset unit time before the current time in the cloud business processing state at the current time; VC represents the file processing speed corresponding to each processed business file in the business processing data in the historical data, and the file processing speed is equal to the average value of the quotient of the file size corresponding to the corresponding processed business file divided by the corresponding processing time; Gi represents the file size of the ith pending business file in the business usage demand of the cloud user; i1 represents the number of pending business files in the business usage demand of the cloud user; max{} represents the operation of finding the maximum value;

[0030] In the process of obtaining FR, the sum of the corresponding pending business volumes of each cloud user of the cloud server corresponding to any time point t within the preset unit time before the current time is recorded as Gt, t∈[0,T1], T1 represents the preset unit time; the average value of each Gt corresponding to different values ​​of t within the preset unit time before the current time is calculated, recorded as GVt; a set constructed by trend change fluctuation points within the preset unit time before the current time is obtained, recorded as trend change fluctuation point set; the trend change fluctuation point is the sum of the corresponding pending business volumes of each cloud user of the cloud server corresponding to the corresponding time point; the Gt corresponding to the element in the trend change fluctuation point set is equal to GT1, and is greater than the sum of the corresponding pending business volumes of each cloud user of the cloud server corresponding to the adjacent time points before and after. and, or less than the sum of the corresponding pending business volumes of each cloud user of the cloud server corresponding to the adjacent time points before and after; GT1 represents the sum of the corresponding pending business volumes of each cloud user of the cloud server corresponding to the current time T1 within the preset unit time before the current time; obtain the element with the smallest time interval with the current time among the elements with the smallest difference between GVt in the trend change fluctuation point set, and record it as the trend change fluctuation reference point; calculate G0 minus the element with the smallest absolute difference between VC in the trend change fluctuation point set, and divide it by the quotient of the interval between the time point corresponding to the element with the smallest absolute difference between VC in the trend change fluctuation point set and the current time; G0 represents the sum of the corresponding pending business volumes of each cloud user of the cloud server corresponding to the current time;

[0031] Step S304: obtaining a cloud function call mode determination result corresponding to the ith to-be-processed business file in the business usage requirements of the cloud user;

[0032] If Gi≥e -Pi ·N, then the cloud function call mode corresponding to the ith pending business file in the cloud user's business usage requirements is determined to be asynchronous execution; otherwise, the cloud function call mode corresponding to the ith pending business file in the cloud user's business usage requirements is determined to be synchronous execution; N is the preset threshold value of the size of the pending business file.

[0033] Preferably, the step S400 includes:

[0034] In the summary results of the pending business files executed synchronously, each pending business file directly calls and executes the corresponding cloud function in the cloud server, and the cloud function functions called by different pending business files are different;

[0035] When the cloud function processing result of the corresponding business file to be processed is fed back to the corresponding business party, the cloud function processing result of the corresponding business file to be processed is directly fed back to the cloud account by the cloud server.

[0036] Preferably, the step S500 includes:

[0037] In the summary results of the asynchronously executed pending business files, each pending business file is sent to RabbitMQ in the form of a RabbitMQ message, and RabbitMQ messages are monitored in real time. According to the monitoring results, the corresponding cloud functions of each pending business file are called and executed in RabbitMQ; when the cloud function processing results of each pending business file are fed back to each business party, the cloud function processing results of the corresponding pending business file are fed back by RabbitMQ to the cloud account in the cloud server, and each business party corresponds to a business account.

[0038] The cloud function calling mode is determined in the present invention in order to realize the diversion of processing tasks corresponding to the business files to be processed, that is, synchronous execution and asynchronous execution, when the business data is highly concurrency. In this process, the synchronous execution mode is to ensure the processing response speed of the corresponding business files to be processed; and the asynchronous execution mode is to prevent the processing of large file businesses from taking a long time and occupying more server resources, and to avoid large files causing service congestion when the business data is highly concurrency. The processing tasks corresponding to the corresponding business files to be processed are peak-shaving processing through rabbitmq.

[0039] A system for efficiently processing massive files based on cloud functions, the system comprising: a cloud function development management module, a demand acquisition and cloud function locking module, a cloud function calling mode analysis module, a cloud function synchronous execution management module and a cloud function asynchronous execution management module.

[0040] The cloud function development management module develops and encapsulates cloud functions with different functions based on cloud function development tools, business types and cloud object storage, and each cloud function is bound to a preset cloud function interface;

[0041] The demand acquisition and cloud function locking module acquires the business usage requirements of cloud users, and locks the cloud functions and corresponding cloud function interfaces corresponding to the business files to be processed in the business usage requirements of cloud users;

[0042] The cloud function calling mode analysis module extracts the business characteristics of the business usage requirements of the cloud user, and combines the business processing status of the cloud at the current time and the business processing data in the historical data to generate the cloud function calling mode determination results corresponding to the business files to be processed in the business usage requirements of the cloud user; the cloud function calling mode determination results include synchronous execution and asynchronous execution;

[0043] The cloud function synchronous execution management module aggregates the pending business files to be processed for synchronous execution, calls and executes the corresponding cloud function through the corresponding cloud function interface for each pending business file in the aggregated result, and feeds back the cloud function processing result of the corresponding pending business file to the corresponding business party;

[0044] The cloud function asynchronous execution management module summarizes the asynchronously executed business files to be processed, and transmits the file processing tasks corresponding to the summary results to RabbitMQ. RabbitMQ monitors the cloud functions corresponding to the asynchronously executed file processing tasks corresponding to the summary results, and notifies each business party of the cloud function processing results of the corresponding business files to be processed.

[0045] Preferably, the cloud function calling mode analysis module includes a business demand feature extraction unit and a cloud function calling mode determination unit.

[0046] The business demand feature extraction unit extracts business features from the business usage requirements of cloud users;

[0047] The cloud function calling mode determination unit combines the business processing status of the cloud at the current time and the business processing data in the historical data to generate cloud function calling mode determination results corresponding to the business files to be processed in the business usage requirements of the cloud user.

[0048] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0049] 1) The cloud function and object storage of the present invention are deeply customized, which reduces the file upload and download processing, reduces the network impact, improves the file processing speed, and reduces server consumption;

[0050] 2) File processing is event-triggered, and its business characteristics are suitable for serverless technology. It is supplied on demand. When massive file processing business explodes, cloud function instances can be automatically scheduled to meet processing needs. Developers do not need to pay attention to business peaks and troughs, saving idle time costs and ultimately reducing operation and maintenance costs.

[0051] 3) Custom development of rabbitmq and cloud functions to achieve asynchronous processing of large batches of files, meet high-concurrency business scenarios, and implement peak-cutting scheduling functions;

[0052] 4) Each business is an independent cloud function. The cloud functions do not interfere with each other. The business is clear, the cloud function scheduling is simple, and the cloud function startup speed is faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0054] Figure 1 It is a structural schematic diagram of a system for efficiently processing massive files based on cloud functions of the present invention;

[0055] Figure 2 It is a schematic diagram of the steps of a method for efficiently processing massive files based on cloud functions of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] See also Figure 1 The present invention provides a technical solution: a system for efficiently processing massive files based on cloud functions, the system comprising: a cloud function development management module, a demand acquisition and cloud function locking module, a cloud function calling mode analysis module, a cloud function synchronous execution management module and a cloud function asynchronous execution management module,

[0058] The cloud function development management module develops and encapsulates cloud functions with different functions based on cloud function development tools, business types and cloud object storage, and each cloud function is bound to a preset cloud function interface;

[0059] The demand acquisition and cloud function locking module acquires the business usage requirements of cloud users, and locks the cloud functions and corresponding cloud function interfaces corresponding to the business files to be processed in the business usage requirements of cloud users;

[0060] The cloud function calling mode analysis module includes a business demand feature extraction unit and a cloud function calling mode identification unit.

[0061] The business demand feature extraction unit extracts business features from the business usage requirements of cloud users;

[0062] The cloud function calling mode determination unit combines the current cloud business processing status and the business processing data in the historical data to generate cloud function calling mode determination results corresponding to the business files to be processed in the business usage requirements of the cloud user; the cloud function calling mode determination results include synchronous execution and asynchronous execution;

[0063] The cloud function synchronous execution management module aggregates the pending business files to be processed for synchronous execution, calls and executes the corresponding cloud function through the corresponding cloud function interface for each pending business file in the aggregated result, and feeds back the cloud function processing result of the corresponding pending business file to the corresponding business party;

[0064] The cloud function asynchronous execution management module summarizes the asynchronously executed business files to be processed, and transmits the file processing tasks corresponding to the summary results to RabbitMQ. RabbitMQ monitors the cloud functions corresponding to the asynchronously executed file processing tasks corresponding to the summary results, and notifies each business party of the cloud function processing results of the corresponding business files to be processed.

[0065] This embodiment 1:

[0066] Cloud function: Real-time scheduling service resources provided by cloud vendors. Users can develop cloud function services based on the cloud function development SDK provided by cloud vendors. Deployment and operation are all configured on the cloud function configuration page of the cloud vendor.

[0067] Object storage: Each cloud vendor provides users with services to store files in various formats, charging based on the total amount of files. Almost all files in Cloud Accounting are stored on object storage in different clouds. Uploading and downloading files using object storage consumes a lot of bandwidth;

[0068] Rabbitmq: Message middleware that handles asynchronous tasks and high-concurrency file processing tasks, and is responsible for scheduling file processing tasks.

[0069] like Figure 2 As shown, a method for efficiently processing massive files based on cloud functions includes the following steps:

[0070] Step S100: Develop and encapsulate cloud functions with different functions based on cloud function development tools, business types, and cloud object storage, and bind each cloud function to a preset cloud function interface;

[0071] The specific implementation process of step S100 includes:

[0072] Step S101: Develop cloud functions with different functions according to business types and cloud object storage using a cloud function development tool; the cloud function functions include file compression, decompression and file conversion functions;

[0073] Step S102: testing the developed cloud function, using the cloud function to directly operate the object storage data, executing the to-be-tested file in the object storage data through the data processing method inside the corresponding cloud function, and transmitting the execution result to the corresponding object storage;

[0074] Step S103: Receive the analysis result of step S102, and compare the obtained analysis result with the preset analysis result corresponding to the corresponding file to be tested. If the obtained analysis result is the same as the preset analysis result corresponding to the corresponding file to be tested, it is determined that the functional test of the corresponding cloud function has passed, and the corresponding cloud function is packaged and released; otherwise, it is determined that the functional test of the corresponding cloud function has not passed, and the corresponding cloud function needs to be optimized, and the optimized cloud function replaces the original cloud function, and jumps to step S102.

[0075] Step S200: Obtain the business usage requirements of the cloud user, and lock the cloud function and the corresponding cloud function interface corresponding to the business file to be processed in the business usage requirements of the cloud user;

[0076] The step S200 includes:

[0077] The business usage requirements of the cloud user include each business file to be processed, the business type and file size corresponding to each business file to be processed; the cloud function corresponding to the business file to be processed in the business usage requirements of the cloud user is obtained by querying the cloud function corresponding to the business type to which the business file to be processed belongs in the database; the corresponding cloud function interface is the cloud function interface bound to the cloud function corresponding to the business file to be processed in the business usage requirements of the cloud user.

[0078] Step S300: extracting business characteristics from the business usage requirements of the cloud user, and combining the business processing status of the cloud at the current time and the business processing data in the historical data, generating cloud function call mode determination results corresponding to the business files to be processed in the business usage requirements of the cloud user; the cloud function call mode determination results include synchronous execution and asynchronous execution;

[0079] The specific implementation process of step S300 includes:

[0080] Step S301: extracting business characteristics of business usage requirements of cloud users to obtain business requirement characteristic data, wherein the business requirement characteristic data includes the file size of the business file to be processed and the corresponding cloud function;

[0081] Step S302: Obtain the current cloud business processing status and business processing data in historical data;

[0082] Step S303: Calculate the congestion risk assessment coefficient corresponding to the ith to-be-processed business file in the business usage requirements of the cloud user, and the calculation formula is as follows:

[0083] Pi=max{BV·(FR·max{Gi / VC|i∈[1,i1]}+1),BV},

[0084] Wherein, Pi represents the congestion risk assessment coefficient corresponding to the ith pending business file in the business usage demand of the cloud user; BV represents the ratio of the sum of the pending business volumes corresponding to each cloud user in the cloud server at the current time in the cloud business processing state to the preset network bandwidth corresponding to the cloud server; the pending business volume represents the file size of each pending business file in the corresponding business usage demand; FR represents the characteristic variation coefficient composed of the sum of the corresponding pending business volumes of each cloud user of the cloud server corresponding to different time points in the preset unit time before the current time in the cloud business processing state at the current time; VC represents the file processing speed corresponding to each processed business file in the business processing data in the historical data, and the file processing speed is equal to the average value of the quotient of the file size corresponding to the corresponding processed business file divided by the corresponding processing time; Gi represents the file size of the ith pending business file in the business usage demand of the cloud user; i1 represents the number of pending business files in the business usage demand of the cloud user; max{} represents the operation of finding the maximum value;

[0085] In the process of obtaining FR, the sum of the corresponding pending business volumes of each cloud user of the cloud server corresponding to any time point t within the preset unit time before the current time is recorded as Gt, t∈[0,T1], T1 represents the preset unit time; the average value of each Gt corresponding to different values ​​of t within the preset unit time before the current time is calculated, recorded as GVt; a set constructed by trend change fluctuation points within the preset unit time before the current time is obtained, recorded as trend change fluctuation point set; the trend change fluctuation point is the sum of the corresponding pending business volumes of each cloud user of the cloud server corresponding to the corresponding time point; the Gt corresponding to the element in the trend change fluctuation point set is equal to GT1, and is greater than the sum of the corresponding pending business volumes of each cloud user of the cloud server corresponding to the adjacent time points before and after. and, or less than the sum of the corresponding pending business volumes of each cloud user of the cloud server corresponding to the adjacent time points before and after; GT1 represents the sum of the corresponding pending business volumes of each cloud user of the cloud server corresponding to the current time T1 within the preset unit time before the current time; obtain the element with the smallest time interval with the current time among the elements with the smallest difference between GVt in the trend change fluctuation point set, and record it as the trend change fluctuation reference point; calculate G0 minus the element with the smallest absolute difference between VC in the trend change fluctuation point set, and divide it by the quotient of the interval between the time point corresponding to the element with the smallest absolute difference between VC in the trend change fluctuation point set and the current time; G0 represents the sum of the corresponding pending business volumes of each cloud user of the cloud server corresponding to the current time;

[0086] Step S304: obtaining a cloud function call mode determination result corresponding to the ith to-be-processed business file in the business usage requirements of the cloud user;

[0087] If Gi≥e -Pi ·N, then the cloud function call mode corresponding to the ith pending business file in the cloud user's business usage requirements is determined to be asynchronous execution; otherwise, the cloud function call mode corresponding to the ith pending business file in the cloud user's business usage requirements is determined to be synchronous execution; N is the preset threshold value of the size of the pending business file.

[0088] In this embodiment, N means that there are no more than 100 pictures in the corresponding pdf file or the corresponding file size is less than 10 megabytes;

[0089] Step S400: Summarize the business files to be processed that are executed synchronously, call and execute the corresponding cloud function through the corresponding cloud function interface for each business file to be processed in the summary result, and feed back the cloud function processing result of the corresponding business file to be processed to the corresponding business party;

[0090] The step S400 comprises:

[0091] In the summary results of the pending business files executed synchronously, each pending business file directly calls and executes the corresponding cloud function in the cloud server, and the cloud function functions called by different pending business files are different;

[0092] When the cloud function processing result of the corresponding business file to be processed is fed back to the corresponding business party, the cloud function processing result of the corresponding business file to be processed is directly fed back to the cloud account by the cloud server.

[0093] Step S500: aggregate the asynchronously executed pending business files, transmit the file processing tasks corresponding to the aggregated results to RabbitMQ, and RabbitMQ monitors the asynchronously executed cloud functions corresponding to the file processing tasks corresponding to the aggregated results, and informs each business party of the cloud function processing results of the corresponding pending business files by RabbitMQ;

[0094] The step S500 includes:

[0095] In the summary results of the asynchronously executed pending business files, each pending business file is sent to RabbitMQ in the form of a RabbitMQ message, and RabbitMQ messages are monitored in real time. According to the monitoring results, the corresponding cloud functions of each pending business file are called and executed in RabbitMQ; when the cloud function processing results of each pending business file are fed back to each business party, the cloud function processing results of the corresponding pending business file are fed back by RabbitMQ to the cloud account in the cloud server, and each business party corresponds to a business account.

[0096] Example 2: The user collects invoice data. The original invoice file data is PDF. The page needs to display pictures. The file system needs to convert the invoice PDF into invoice pictures. On the 1st of each month, tens of millions of PDFs need to be converted into pictures. The specific execution method steps are as follows:

[0097] Step 11. The user uses the cloud accounting system to collect the invoice data PDF data, uploads the invoice PDF to the object storage, and returns the download URL (URL is a uniform resource locator, a concise representation of the location and access method of resources available on the Internet, and is the address of a standard resource on the Internet; each file on the Internet has a unique URL, which contains information indicating the location of the file and how the browser should handle it);

[0098] Step 12: The system sends the URL of the invoice PDF to RabbitMQ. The business party monitors RabbitMQ and receives the URL of the PDF.

[0099] Step 13: The business party calls the file conversion cloud function and passes in the URL of the PDF. The cloud function uses the URL to operate the object storage to download the PDF file, convert the PDF into a PNG image, upload the image to the object storage service, and send the image URL to the business party.

[0100] Step 14: The business party receives the image URL returned by the cloud function and saves the URL to the business database;

[0101] Step 15: The user searches for the invoice image on the page.

[0102] Example 3: When a user prints a document at the end of the month, multiple PDF files are combined and downloaded;

[0103] Step 21. At the end of the month, users will print vouchers, account balances, invoices, bank receipts, accounting reports and other financial data. Each company will generate multiple PDF files. Users compress multiple PDF files into a zip file according to the company. Each PDF file contains more than 100 pictures.

[0104] Step 22: The user selects an enterprise on the page and clicks on file download. The business party receives the interface request and generates multiple PDF files. The multiple PDF files are gradually uploaded to the object storage and multiple URLs are generated.

[0105] Step 23: The business party calls the file compression cloud function and passes in the URLs of multiple PDFs. The cloud function operates the object storage to download the PDFs according to the file URLs, merges and compresses the multiple PDFs into a zip file, and then operates the object storage to upload the zip file to the object storage and returns the URL of the zip file.

[0106] Step 24: The business party receives the zip file URL message returned by the cloud function and saves the URL to the business database;

[0107] Step 25: The user downloads the zip file on the page.

[0108] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0109] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for efficiently processing massive files based on cloud functions, characterized in that: The method comprises the following steps: Step S100: Develop and encapsulate cloud functions with different functions based on cloud function development tools, business types, and cloud object storage, and bind each cloud function to a preset cloud function interface; Step S200: Obtain the business usage requirements of the cloud user, and lock the cloud function and the corresponding cloud function interface corresponding to the business file to be processed in the business usage requirements of the cloud user; Step S300: extracting business characteristics from the business usage requirements of the cloud user, and combining the business processing status of the cloud at the current time and the business processing data in the historical data, generating cloud function call mode determination results corresponding to the business files to be processed in the business usage requirements of the cloud user; the cloud function call mode determination results include synchronous execution and asynchronous execution; Step S400: Summarize the business files to be processed that are executed synchronously, call and execute the corresponding cloud function through the corresponding cloud function interface for each business file to be processed in the summary result, and feed back the cloud function processing result of the corresponding business file to be processed to the corresponding business party; Step S500: Summarize the asynchronously executed business files to be processed, and transmit the file processing tasks corresponding to the summary results to RabbitMQ. RabbitMQ monitors the cloud functions corresponding to the asynchronously executed file processing tasks corresponding to the summary results, and notifies each business party of the cloud function processing results of the corresponding business files to be processed.

2. According to claim 1, a method for efficiently processing massive files based on cloud functions is characterized in that: The specific implementation process of step S100 includes: Step S101: Develop cloud functions with different functions according to business types and cloud object storage using a cloud function development tool; the cloud function functions include file compression, decompression and file conversion functions; Step S102: testing the developed cloud function, using the cloud function to directly operate the object storage data, executing the to-be-tested file in the object storage data through the data processing method inside the corresponding cloud function, and transmitting the execution result to the corresponding object storage; Step S103: Receive the analysis result of step S102, and compare the obtained analysis result with the preset analysis result corresponding to the corresponding file to be tested. If the obtained analysis result is the same as the preset analysis result corresponding to the corresponding file to be tested, it is determined that the functional test of the corresponding cloud function has passed, and the corresponding cloud function is packaged and released; otherwise, it is determined that the functional test of the corresponding cloud function has not passed, and the corresponding cloud function needs to be optimized, and the optimized cloud function replaces the original cloud function, and jumps to step S102.

3. According to claim 2, a method for efficiently processing massive files based on cloud functions is characterized in that: The step S200 includes: The business usage requirements of the cloud user include each business file to be processed, the business type and file size corresponding to each business file to be processed; the cloud function corresponding to the business file to be processed in the business usage requirements of the cloud user is obtained by querying the cloud function corresponding to the business type to which the business file to be processed belongs in the database; the corresponding cloud function interface is the cloud function interface bound to the cloud function corresponding to the business file to be processed in the business usage requirements of the cloud user.

4. According to claim 3, a method for efficiently processing massive files based on cloud functions is characterized in that: The specific implementation process of step S300 includes: Step S301: extracting business characteristics of business usage requirements of cloud users to obtain business requirement characteristic data, wherein the business requirement characteristic data includes the file size of the business file to be processed and the corresponding cloud function; Step S302: Obtain the current cloud business processing status and business processing data in historical data; Step S303: Calculate the congestion risk assessment coefficient corresponding to the ith to-be-processed business file in the business usage requirements of the cloud user, and the calculation formula is as follows: Pi=max{BV·(FR·max{Gi / VC|i∈[1,i1]}+1),BV}, Wherein, Pi represents the congestion risk assessment coefficient corresponding to the ith pending business file in the business usage demand of the cloud user; BV represents the ratio of the sum of the pending business volumes corresponding to each cloud user in the cloud server at the current time in the cloud business processing state to the preset network bandwidth corresponding to the cloud server; the pending business volume represents the file size of each pending business file in the corresponding business usage demand; FR represents the characteristic variation coefficient composed of the sum of the corresponding pending business volumes of each cloud user of the cloud server corresponding to different time points in the preset unit time before the current time in the cloud business processing state at the current time; VC represents the file processing speed corresponding to each processed business file in the business processing data in the historical data, and the file processing speed is equal to the average value of the quotient of the file size corresponding to the corresponding processed business file divided by the corresponding processing time; Gi represents the file size of the ith pending business file in the business usage demand of the cloud user; i1 represents the number of pending business files in the business usage demand of the cloud user; max{} represents the operation of finding the maximum value; In the process of obtaining FR, the sum of the corresponding pending business volumes of each cloud user of the cloud server corresponding to any time point t within the preset unit time before the current time is recorded as Gt, t∈[0,T1], T1 represents the preset unit time; the average value of each Gt corresponding to different values ​​of t within the preset unit time before the current time is calculated, recorded as GVt; a set constructed by trend change fluctuation points within the preset unit time before the current time is obtained, recorded as trend change fluctuation point set; the trend change fluctuation point is the sum of the corresponding pending business volumes of each cloud user of the cloud server corresponding to the corresponding time point; the Gt corresponding to the element in the trend change fluctuation point set is equal to GT1, and is greater than the sum of the corresponding pending business volumes of each cloud user of the cloud server corresponding to the adjacent time points before and after. and, or less than the sum of the corresponding pending business volumes of each cloud user of the cloud server corresponding to the adjacent time points before and after; GT1 represents the sum of the corresponding pending business volumes of each cloud user of the cloud server corresponding to the current time T1 within the preset unit time before the current time; obtain the element with the smallest time interval with the current time among the elements with the smallest difference between GVt in the trend change fluctuation point set, and record it as the trend change fluctuation reference point; calculate G0 minus the element with the smallest absolute difference between VC in the trend change fluctuation point set, and divide it by the quotient of the interval between the time point corresponding to the element with the smallest absolute difference between VC in the trend change fluctuation point set and the current time; G0 represents the sum of the corresponding pending business volumes of each cloud user of the cloud server corresponding to the current time; Step S304: obtaining a cloud function call mode determination result corresponding to the ith to-be-processed business file in the business usage requirements of the cloud user; If Gi≥e -Pi ·N, it is determined that the cloud function call mode corresponding to the ith to-be-processed business file in the business usage requirements of the cloud user is asynchronously executed; otherwise, it is determined that the cloud function call mode corresponding to the ith to-be-processed business file in the business usage requirements of the cloud user is synchronously executed; and N is a preset threshold value of the size of the to-be-processed business file.

5. According to claim 4, a method for efficiently processing massive files based on cloud functions is characterized in that: The step S400 comprises: In the summary results of the pending business files executed synchronously, each pending business file directly calls and executes the corresponding cloud function in the cloud server, and the cloud function functions called by different pending business files are different; When the cloud function processing result of the corresponding business file to be processed is fed back to the corresponding business party, the cloud function processing result of the corresponding business file to be processed is directly fed back to the cloud account by the cloud server.

6. According to claim 4, a method for efficiently processing massive files based on cloud functions is characterized in that: The step S500 includes: In the summary results of the asynchronously executed pending business files, each pending business file is sent to RabbitMQ in the form of a RabbitMQ message, and RabbitMQ messages are monitored in real time. According to the monitoring results, the corresponding cloud functions of each pending business file are called and executed in RabbitMQ; when the cloud function processing results of each pending business file are fed back to each business party, the cloud function processing results of the corresponding pending business file are fed back by RabbitMQ to the cloud account in the cloud server, and each business party corresponds to a business account.

7. A system for efficiently processing massive files based on cloud functions, characterized in that: The system includes: a cloud function development management module, a demand acquisition and cloud function locking module, a cloud function calling mode analysis module, a cloud function synchronous execution management module and a cloud function asynchronous execution management module. The cloud function development management module develops and encapsulates cloud functions with different functions based on cloud function development tools, business types and cloud object storage, and each cloud function is bound to a preset cloud function interface; The demand acquisition and cloud function locking module acquires the business usage requirements of cloud users, and locks the cloud functions and corresponding cloud function interfaces corresponding to the business files to be processed in the business usage requirements of cloud users; The cloud function calling mode analysis module extracts the business characteristics of the business usage requirements of the cloud user, and combines the business processing status of the cloud at the current time and the business processing data in the historical data to generate the cloud function calling mode determination results corresponding to the business files to be processed in the business usage requirements of the cloud user; the cloud function calling mode determination results include synchronous execution and asynchronous execution; The cloud function synchronous execution management module aggregates the pending business files to be processed for synchronous execution, calls and executes the corresponding cloud function through the corresponding cloud function interface for each pending business file in the aggregated result, and feeds back the cloud function processing result of the corresponding pending business file to the corresponding business party; The cloud function asynchronous execution management module summarizes the asynchronously executed business files to be processed, and transmits the file processing tasks corresponding to the summary results to RabbitMQ. RabbitMQ monitors the cloud functions corresponding to the asynchronously executed file processing tasks corresponding to the summary results, and notifies each business party of the cloud function processing results of the corresponding business files to be processed.

8. A system for efficiently processing massive files based on cloud functions according to claim 7, characterized in that: The cloud function calling mode analysis module includes a business demand feature extraction unit and a cloud function calling mode identification unit. The business demand feature extraction unit extracts business features from the business usage requirements of cloud users; The cloud function calling mode determination unit combines the business processing status of the cloud at the current time and the business processing data in the historical data to generate cloud function calling mode determination results corresponding to the business files to be processed in the business usage requirements of the cloud user.

Images