Large-batch general file uploading method and system

Abstract

The invention provides a method and a system for uploading large-batch general files. The method comprises the following steps of: 1, packaging the files into messages and sending the messages to a packaged message queue; 2, consuming the messages in the packaged message queue through a packaging service, processing, downloading and storing a file, packaging the processed file into messages, and sending the messages to an uploaded message queue; 3, consuming the messages in the message uploading queue through an uploading service, and uploading the file; and 4, packaging the cloud path returned after uploading into a message body, and recording the message body into a distributed file storage database. According to the method, the kafka and rabbitMQ message queue components are adopted to decouple and upload the files, so that the problem that time and labor are wasted when a large number of files are manually uploaded to the cloud is solved, repeated operation is avoided, the working efficiency is improved, and the high-concurrency requirement is met.

Description

Technical Field

[0001] This invention relates to the field of computer science and technology, and more specifically, to a method and system for uploading large quantities of general-purpose files. Background Technology

[0002] The background of the mass file upload method includes the tiered processing and uploading of large batches of files under different business needs, file types, and usage scenarios, aiming to improve the stability of cloud service uploads in high-concurrency scenarios. During the mass file upload process, by being compatible with different types and prioritizing the processing of high-priority files, the system resource utilization and the timeliness of critical business processes are improved.

[0003] Currently, traditional file upload methods often rely on a specific file submission order, which has the following drawbacks: First, uploading files sequentially according to the submission order lacks hierarchical processing of files, especially prioritizing high-level attachments. Second, when uploading a large number of files to the cloud simultaneously, upload efficiency is poor in high-concurrency scenarios due to objective factors such as network and server limitations.

[0004] Patent application CN119025480A discloses a standardized processing method and system for multi-format files, including: Step S1: Obtaining file information; classifying and transferring files based on the file information to obtain classified attachments; Step S2: Determining whether the classified attachments need to be replaced based on their type; if yes, replacing the file with an HTML file, formatting the HTML file, and saving it to a local file; if no, converting the file into a file stream, performing a unified conversion, and saving it to a local file. However, this patent cannot completely solve the existing technical problems, nor can it meet the needs of this invention. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for large-scale universal file uploads.

[0006] The method for bulk universal file upload provided by the present invention includes: Step 1: Encapsulate the file into a message and send it to the packaged message queue; Step 2: Consume messages in the packaging message queue through the packaging service, process, download and save the files, and encapsulate the processed files into messages and send them to the upload message queue; Step 3: Consume messages in the upload message queue through the upload service to upload the file; Step 4: Encapsulate the cloud path returned after uploading into a message body and record it in the distributed file storage database.

[0007] Preferably, in the packaged message queue, message priority is set according to the urgency of file upload, with a higher priority value indicating a higher urgency. The priority values ​​are dynamically configured between 1 and 10, including: setting re-uploaded files to the highest priority of 10, setting business-critical files to priorities of 5 to 9, and setting low-time-sensitivity files to priorities of 1 to 4.

[0008] Preferably, when the packaged service consumes messages, it uses a strategy pattern to perform parameter validation, content parsing, and compression on the files in the message, and saves the processed files locally. The strategy pattern includes: a strategy interface, which defines a unified method for processing file content; concrete strategy classes, which implement the strategy interface and provide different content processing logic; and a context class, which holds the strategy interface object and dynamically switches between concrete strategies to execute file content processing. File parameter validation includes: parsing the message structure using key-value pair retrieval to obtain the list of required fields; verifying the completeness of the required fields; and returning an error message and recording it in the relational database if the validation fails. File content parsing involves using regular expressions to extract the file path from the message and then using that path to obtain the file content.

[0009] Preferably, the file upload step includes: uploading the file to the cloud via the Object Service Storage SDK; obtaining the cloud file path after successful upload; deleting the local file; and encapsulating the cloud path into a message body and recording it in a relational database and a downstream message queue.

[0010] Preferably, the distributed file storage database is MongoDB, used to provide file path services to external users; Log points are embedded at key process nodes, including file acquisition, file processing, and file upload. The embedded logs are encapsulated as entities and stored in a relational database.

[0011] The mass general-purpose file upload system provided by the present invention includes: Module M1: Encapsulates files into messages and sends them to the packaged message queue; Module M2: Consumes messages in the packaging message queue through the packaging service, processes, downloads, and saves files, and encapsulates the processed files into messages to send to the upload message queue; Module M3: Consumes messages in the upload message queue through the upload service to upload files; Module M4: Encapsulates the cloud path returned after uploading into a message body and records it in the distributed file storage database.

[0012] Preferably, in the packaged message queue, message priority is set according to the urgency of file upload, with a higher priority value indicating a higher urgency. The priority values ​​are dynamically configured between 1 and 10, including: setting re-uploaded files to the highest priority of 10, setting business-critical files to priorities of 5 to 9, and setting low-time-sensitivity files to priorities of 1 to 4.

[0013] Preferably, when the packaged service consumes messages, it uses a strategy pattern to perform parameter validation, content parsing, and compression on the files in the message, and saves the processed files locally. The strategy pattern includes: a strategy interface, which defines a unified method for processing file content; concrete strategy classes, which implement the strategy interface and provide different content processing logic; and a context class, which holds the strategy interface object and dynamically switches between concrete strategies to execute file content processing. File parameter validation includes: parsing the message structure using key-value pair retrieval to obtain the list of required fields; verifying the completeness of the required fields; and returning an error message and recording it in the relational database if the validation fails. File content parsing includes: using regular expressions to extract the file path from the message, and then obtaining the file content through that path.

[0014] Preferably, the file upload step includes: uploading the file to the cloud via the Object Service Storage SDK; obtaining the cloud file path after successful upload; deleting the local file; and encapsulating the cloud path into a message body and recording it in a relational database and a downstream message queue.

[0015] Preferably, the distributed file storage database is MongoDB, used to provide file path services to external users; Log points are embedded at key process nodes, including file acquisition, file processing, and file upload. The embedded logs are encapsulated as entities and stored in a relational database.

[0016] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention uses Kafka and RabbitMQ message queue components to decouple and upload files, which solves the problem of time-consuming and laborious manual upload of large batches of files to the cloud, avoids repetitive operations, improves work efficiency, and meets the needs of high concurrency; (2) The present invention adopts the strategy pattern to meet the unified upload requirements of different businesses, different file types and different usage scenarios; the strategy pattern makes the execution method independent of the client using the method, thereby realizing the polymorphic strategy selection to realize different business logic and enhance the scalability of the business. (3) RabbitMQ itself supports setting message priority. This invention uses the RabbitMQ message queue component to set message priority. Messages with higher priority are consumed first. This feature is used to classify the priority range of queue messages, thereby ensuring that messages with higher priority are consumed and processed first. Attached Figure Description

[0017] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A flowchart illustrating one embodiment of the present invention; Figure 2 The flowchart is described as another embodiment of the present invention. Detailed Implementation

[0018] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0019] Example 1 This invention provides a method for uploading large batches of general-purpose files, comprising: first, encapsulating the files into messages and sending them to a packaging message queue; second, a packaging service consuming messages to process, download, and save the files, and then sending the processed files back to the upload message queue as messages; third, an upload service consuming messages and uploading the files; and finally, encapsulating the cloud path returned after the upload into a message body conforming to the message queue requirements and recording it in a distributed file storage database (MongoDB), through which the service is provided externally.

[0020] This method includes the following steps: Step 1: The upstream provider encapsulates the message into a parsable message format as required and sends it to the RabbitMQ message queue. Simultaneously, message priorities are set based on the urgency of the file upload; higher urgency results in a higher priority value. Specifically, when a file needs to be re-uploaded due to parsing errors or upload failures, it is set to the highest priority (priority=10) for priority upload; files belonging to business-critical categories (such as real-time incremental files) are set to a higher priority (priority=5~9); files belonging to low-time-sensitivity categories (such as historical archive files, backup files, etc.) are set to the lowest priority (priority=1~4). In practical applications, priority values ​​can be dynamically configured between 1 and 10. The system prioritizes consuming high-priority messages, ensuring that important files can be uploaded more quickly.

[0021] Step 2: The packaging service consumes messages according to their priority in the message queue using the strategy pattern in behavioral design patterns. It performs standardized operations such as parameter validation, content parsing, and compression on the files in the messages, and saves the processed files locally, generating local file paths.

[0022] Step 2 includes the following steps: Step 2.1: Utilize the priority feature of the message queue to sort the messages according to their priority and retrieve the message at the top of the list.

[0023] Step 2.2: The system performs key-value pair type comparison on the message using entity classes and obtains the required fields based on analysis methods. Specifically, the original message is a JSON format string. The system uses a key-value pair retrieval method based on a predefined field list, parsing the message structure by field name (key) to obtain a list of all required fields that need to be validated. Subsequently, the system performs parameter validation on the required fields, determining whether each required field is transmitted in the message; if any field is missing, the validation fails and an error message is returned. After all required fields pass validation, the system continues to parse the message content, extracting the corresponding value for each required field and filling these values ​​into a predefined data structure. Finally, the system performs entity initialization, mapping the parsed and validated JSON string to Java entity class objects to facilitate subsequent file processing and upload operations.

[0024] Step 2.3: Parse the message entity to obtain the path of the corresponding file, and parse the file content using the obtained file path; Step 2.4: Standardize the file content using the strategy pattern from behavioral design patterns to achieve flexible reuse and dynamic expansion of processing logic. The strategy pattern includes a strategy interface, concrete strategy classes, and a context class. The strategy interface defines a unified method for processing file content; the concrete strategy classes implement the strategy interface, providing different content processing logic, such as filtering HTML content, parsing and filling JSON content, and can return different file formats, such as HTML, zlib compressed format, or JSON format; the context class holds the strategy interface object, allowing dynamic switching of concrete strategies by setting the strategy object and calling strategy methods to execute file content processing, thereby achieving flexible and controllable standardized processing.

[0025] Step 2.5: Save the processed file locally and generate the local file path.

[0026] Step 3: After encapsulating the local file path into a message body conforming to the upload message format, send it to the RabbitMQ upload message queue, and simultaneously add log points to key nodes. These key nodes refer to critical processes such as file acquisition, file processing, and file upload, used to track the status and record errors during the file upload process. Once file processing is complete, the log points are encapsulated into entities and stored in a relational database, providing file upload retry and logging capabilities.

[0027] Step 4: The upload service consumes messages from the upload message queue, obtains the local path of the file in the message, retrieves the file content, uploads it, and then encapsulates the returned cloud path into a message body that meets the requirements of the message queue, and records it in the relational database and distributes it to the downstream message queue.

[0028] Step 4 includes the following steps: Step 4.1: Utilize the priority feature of the message queue to sort the messages according to their priority and retrieve the message at the top of the list.

[0029] Step 4.2: The message is compared using key-value pairs through the entity class. Then, the required fields are obtained through analysis methods, and parameter validation is performed. After validation, the message is parsed to obtain all fields, and the message entity is initialized.

[0030] Step 4.3: Parse the message entity to obtain the path of the corresponding file, and then obtain the file content through the file path; Step 4.4: Upload the file to the cloud using the Object Service Storage SDK. After successful upload, obtain the returned cloud file path. Step 4.5: Delete the local file, encapsulate the returned cloud path into a message body that meets the requirements of the message queue, and record the message body in the MySQL relational database and distribute it to the downstream Kafka message queue.

[0031] Step 5: Downstream storage services consume the message queue, save the uploaded path to the distributed file storage database, and provide services to the outside world through this database.

[0032] These steps detail a system and method for processing and storing attachments, covering key technologies such as message processing, priority management, file operations, and data storage.

[0033] This invention solves the following technical problems: 1) Solve the problem of unstable upload efficiency in high-concurrency file upload scenarios: When uploading files in batches to object storage, factors such as network and server may cause some files to fail to upload, affecting the overall efficiency; 2) Solving the priority issue in file uploads: During batch file uploads, files are uploaded to the cloud strictly in the order they are submitted. There is a lack of file hierarchical processing, and more important files cannot be uploaded first when there are a large number of files. 3) Addressing the issue of non-universal file upload methods: File upload methods lack reusability across different business operations, file types, and usage scenarios, impacting upload efficiency. There is a lack of a universal file upload method that can be applied and extended to various business applications.

[0034] Example 2: In response to the problems existing in the relevant technologies, such as Figure 1 As shown, the following describes the steps of a specific implementation method: Step 1: Send the message to the RabbitMQ packaged message queue and set its priority; Step 2: Consume messages in the packaged message queue, obtain the message file path, standardize the file, compress the file and save it locally, and generate the local file path; Step 2 includes the following steps: Step 2.1: First, the packaging service retrieves the message at the top of the message queue based on the priority of the messages in the message queue; Step 2.2: Compare the key-value pairs of the obtained messages using entity classes, and extract the file paths from the messages based on regular expressions; Step 2.3: Step 2.3 is implemented as described in patent CN202411514646.1. The file content is obtained, and parameter verification is performed on the file. If the verification passes, the strategy pattern from behavioral design patterns is applied to standardize the file content. If the verification fails, the failure record is stored in a relational database. Step 2.4: Save the compressed file to the local disk and generate a local path; Step 3: Encapsulate the message body and send the message to the RabbitMQ upload message queue; Step 4: Consume messages in the upload message queue, obtain the local path of the file in the message, retrieve the file content through the local path, upload the file content, and return the cloud file path if the upload is successful; if the upload fails, store the failure record in a MySQL relational database; encapsulate the cloud file path into a message body, record it in the MySQL relational database and distribute it to the downstream Kafka message queue; Step 4 includes the following steps: Step 4.1: First, the upload service retrieves the message at the top of the message queue based on the priority of the messages in the message queue; Step 4.2: Compare the key-value pairs of the obtained messages using entity classes, and extract the file paths from the messages based on regular expressions; Step 4.3: Obtain the file content through the file path, upload the file to the cloud through the Object Service Storage SDK, and obtain the returned cloud file path after successful upload; Step 4.4: Delete the local file, encapsulate the message body, store it in a MySQL relational database, and distribute it to the downstream Kafka message queue.

[0035] Step 5: Consume the Kafka message queue, save the uploaded path to the MongoDB distributed file storage database, and provide file path services to the outside world through MongoDB.

[0036] Example 3: Based on the above file upload method using RabbitMQ, such as Figure 2 As shown, this application also provides a file upload method for retrieving file messages from a Kafka message queue, including: Step 1: Retrieve messages from Kafka, parse them, encapsulate the parsed messages according to a specific message format, set message queue priorities based on file importance, and send the messages to the RabbitMQ message queue.

[0037] Step 2: Send the message to the RabbitMQ packaged message queue and set its priority; Step 3: Consume messages in the packaged message queue, obtain the message file path, standardize the file, compress the file and save it locally, and generate the local file path; Step 3 includes the following steps: Step 3.1: First, the packaging service retrieves the message at the top of the message queue based on the priority of the messages in the message queue; Step 3.2: Compare the key-value pairs of the obtained messages using entity classes, and extract the file paths from the messages based on regular expressions; Step 3.3: Step 3.3 is implemented as described in patent CN202411514646.1. The file content is obtained, and parameter verification is performed on the file. If the verification passes, the strategy pattern in behavioral design patterns is applied to standardize the file content. If the verification fails, the failure record is stored in a relational database. Step 3.4: Save the compressed file to the local disk and generate a local path; Step 4: Encapsulate the message body and send the message to the RabbitMQ upload message queue; Step 5: Consume messages in the upload message queue, obtain the local path of the file in the message, retrieve the file content through the local path, upload the file content, and return the cloud file path if the upload is successful; if the upload fails, store the failure record in a MySQL relational database; encapsulate the cloud file path into a message body, record it in the MySQL relational database and distribute it to the downstream Kafka message queue; Step 4 includes the following steps: Step 5.1: First, the upload service retrieves the message at the top of the message queue based on the priority of the messages in the message queue; Step 5.2: Compare the key-value pairs of the obtained messages using entity classes, and extract the file paths from the messages based on regular expressions; Step 5.3: Obtain the file content through the file path, upload the file to the cloud through the Object Service Storage SDK, and obtain the returned cloud file path after successful upload; Step 5.4: Delete the local file, encapsulate the message body, store it in a MySQL relational database, and distribute it to the downstream Kafka message queue.

[0038] Step 6: Consume the Kafka message queue, save the uploaded path to the MongoDB distributed file storage database, and provide file path services to the outside world through MongoDB.

[0039] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0040] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for large-scale general file upload, characterized in that, include: Step 1: Encapsulate the file into a message and send it to the packaged message queue; Step 2: Consume messages in the packaging message queue through the packaging service, process, download and save the files, and encapsulate the processed files into messages and send them to the upload message queue; Step 3: Consume messages in the upload message queue through the upload service to upload the file; Step 4: Encapsulate the cloud path returned after uploading into a message body and record it in the distributed file storage database.

2. The method for uploading large quantities of general-purpose files according to claim 1, characterized in that, In the packaged message queue, message priorities are set according to the urgency of file uploads, with higher priority values ​​indicating higher urgency. The priority values ​​are dynamically configured between 1 and 10, including: setting re-uploaded files to the highest priority of 10, setting business-critical files to priorities of 5 to 9, and setting low-time-sensitivity files to priorities of 1 to 4.

3. The method for uploading large quantities of general-purpose files according to claim 1, characterized in that, When the packaged service consumes messages, it uses the strategy pattern to perform parameter validation, content parsing, and compression on the files in the message, and saves the processed files locally. The strategy pattern includes: a strategy interface, which defines a unified method for processing file content; concrete strategy classes, which implement the strategy interface and provide different content processing logic; and a context class, which holds the strategy interface object and dynamically switches between concrete strategies to execute file content processing. File parameter validation includes: parsing the message structure using key-value pair retrieval to obtain the list of required fields; verifying the completeness of the required fields; and returning an error message and recording it in the relational database if the validation fails. File content parsing involves using regular expressions to extract the file path from the message and then using that path to obtain the file content.

4. The method for uploading large quantities of general-purpose files according to claim 1, characterized in that, The file upload steps include: uploading the file to the cloud via the Object Service Storage SDK; obtaining the cloud file path after successful upload; deleting the local file; and encapsulating the cloud path into a message body and recording it in a relational database and downstream message queue.

5. The method for uploading large quantities of general-purpose files according to claim 1, characterized in that, The distributed file storage database is MongoDB, which is used to provide file path services to external users; Log points are embedded at key process nodes, including file acquisition, file processing, and file upload. The embedded logs are encapsulated as entities and stored in a relational database.

6. A large-scale general-purpose file upload system, characterized in that, include: Module M1: Encapsulates files into messages and sends them to the packaged message queue; Module M2: Consumes messages in the packaging message queue through the packaging service, processes, downloads, and saves files, and encapsulates the processed files into messages to send to the upload message queue; Module M3: Consumes messages in the upload message queue through the upload service to upload files; Module M4: Encapsulates the cloud path returned after uploading into a message body and records it in the distributed file storage database.

7. The mass general-purpose file upload system according to claim 6, characterized in that, In the packaged message queue, message priorities are set according to the urgency of file uploads, with higher priority values ​​indicating higher urgency. The priority values ​​are dynamically configured between 6 and 10, including: setting re-uploaded files to the highest priority of 10, setting business-critical files to priorities of 5 to 9, and setting low-time-sensitivity files to priorities of 1 to 4.

8. The mass general-purpose file upload system according to claim 6, characterized in that, When the packaged service consumes messages, it uses the strategy pattern to perform parameter validation, content parsing, and compression on the files in the message, and saves the processed files locally. The strategy pattern includes: a strategy interface, which defines a unified method for processing file content; concrete strategy classes, which implement the strategy interface and provide different content processing logic; and a context class, which holds the strategy interface object and dynamically switches between concrete strategies to execute file content processing. File parameter validation includes: parsing the message structure using key-value pair retrieval to obtain the list of required fields; verifying the completeness of the required fields; and returning an error message and recording it in the relational database if the validation fails. File content parsing involves using regular expressions to extract the file path from the message and then using that path to obtain the file content.

9. The mass general-purpose file upload system according to claim 6, characterized in that, The file upload steps include: uploading the file to the cloud via the Object Service Storage SDK; obtaining the cloud file path after successful upload; deleting the local file; and encapsulating the cloud path into a message body and recording it in a relational database and downstream message queue.

10. The mass general-purpose file upload system according to claim 6, characterized in that, The distributed file storage database is MongoDB, which is used to provide file path services to external users; Log points are embedded at key process nodes, including file acquisition, file processing, and file upload. The embedded logs are encapsulated as entities and stored in a relational database.

Images