A lightweight universal file uploading system capable of achieving second transmission

Abstract

The application discloses a kind of lightweight universal file uploading systems that can realize second transmission, it is related to computer data transmission and storage technical field.The system includes system program module, database module and object storage module.The application provides file uploading and task management function by HTTP interface through system program module, and contains upload task generation, simple upload and second transmission upload unit;Database module is composed of file uploading task table and file storage table, for recording task and file information;Object storage module is used as independent permanent storage medium.By uploading voucher, stateless interface call is realized, and distributed session or cache is not needed;Second transmission is realized by unified task model and file characteristic value duplication detection.The whole only relies on two kinds of basic components of database and object storage, and the architecture is lightweight, deployment is simplified, the practicability of lightweight universal file uploading is improved, and the problem of low practicability of lightweight universal file uploading in the prior art is solved.

Description

Technical Field

[0001] This invention relates to the field of computer data transmission and storage technology, and in particular to a lightweight, universal file upload system that can achieve instant file transfer. Background Technology

[0002] First, the user triggers a file selection dialog box through the interactive interface of a webpage or application. After selecting a local file, the front-end verification mechanism is triggered. For example, JavaScript is used to perform preliminary verification of the file type and size, and the file metadata (such as MIME type and size attribute) is read through the HTML5 File API to achieve fast feedback and prevent invalid transmission.

[0003] Next, the file enters the preparation stage. To improve transmission efficiency and reliability, modern technology often introduces chunking processing, which involves cutting large files into fixed-length (e.g., 1MB) data blocks and using hash algorithms such as MD5 or SHA-256 to calculate a unique file identifier for the server to verify file integrity. The core transmission stage is conducted via HTTP / HTTPS protocols, which can use standard form uploads or more efficient asynchronous uploads. When the front end sends the file chunks in binary stream form, either block by block or concurrently, to the preset server interface through APIs such as Fetch API or dedicated libraries, load balancing and CDN (Content Delivery Network) acceleration are incorporated. Requests are routed to the optimal server node through a distributed gateway to improve throughput.

[0004] After receiving the data, the server first performs a security scan, then temporarily stores the fragments in a temporary directory. Once all fragments arrive, the server triggers a merge operation to restore the original file and re-verifies its hash value against the client's to ensure data integrity. Finally, the file is persistently stored in the target location, such as the local disk, and a unique access URL (Uniform Resource Locator) is generated and returned to the client, completing the entire lightweight and general-purpose file upload loop.

[0005] For example, a file upload method and system disclosed in patent application CN115412547A includes: responding to a fragmentation request sent by a front-end service, calling a distributed storage service deployed on a storage server to enable the distributed storage service to perform fragmentation operations and return storage information, sending the storage information to the front-end service so that the front-end service can upload each fragment of the file to be stored to the storage server according to the fragmentation information and storage address links; responding to a merge request sent by the front-end service, calling the distributed storage service to enable the distributed storage service to perform a merge operation on the stored fragments and return the merge result; and sending the merge result to the front-end service so that the front-end service can display the merge result.

[0006] For example, the file upload method and apparatus of patent application CN115865894B includes: when an object to be uploaded is detected in the file upload tag, calling a target plugin to determine the type of the object to be uploaded; when the object to be uploaded is a network address, calling the target plugin to obtain the network file stored in the network address, wherein the target plugin includes a pre-loaded first dynamic scripting language program, the first dynamic scripting language program being used to obtain the network file stored in the network address; and uploading the network file to a database in a backend server.

[0007] The above-mentioned technology has at least the following technical problems:

[0008] In existing technologies, complex system architectures are commonly used to achieve file deduplication and fast uploads. This leads to numerous prominent problems in deployment and practical application. Specifically, existing systems typically require the introduction of many mature external components such as distributed message queues, independent file deduplication systems, and distributed caching. This results in redundant system architecture, numerous deployment units, high resource consumption, and significant operational costs and stability risks. Furthermore, clients must integrate dedicated SDKs (Software Development Kits) that are tightly bound to the system, leading to strong code invasiveness, high adaptation costs, and poor portability when migrating between different platforms. Simultaneously, the system relies heavily on external sessions or caches when handling upload task states. This stateful design not only increases system complexity but also severely restricts the system's stateless scalability, making it difficult to adapt to high-availability and elastic scaling scenarios, resulting in low practicality for lightweight, general-purpose file uploads. Summary of the Invention

[0009] To address the technical problem of low practicality in existing lightweight general-purpose file upload technologies, this invention provides a lightweight general-purpose file upload system capable of achieving instant file transfer. The technical solution is as follows:

[0010] On one hand, a lightweight and general-purpose file upload system capable of instant file transfer is provided. This system includes a system program module, a database module, and an object storage module. The system program module provides a set of HTTP interfaces to implement file upload logic and task management logic. This module includes an upload task generation unit, a simple upload unit, and an instant upload unit. The database module records file upload tasks and file storage information and consists of two logical data tables: a file upload task table and a file storage table. The object storage module serves as an independent permanent storage medium for file content. The upload credentials enable stateless interface calls, eliminating the need for the server to rely on distributed sessions or caching components to maintain task state. Furthermore, by abstracting simple upload and instant upload into a unified file upload task model and using file feature values ​​for deduplication in the file storage table, the instant file transfer function is achieved. Ultimately, the overall architecture relies only on the database module and the object storage module as the two basic components.

[0011] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0012] 1. Through a refined modular design, core dependencies are reduced to only two basic components: database and object storage. No distributed message queues, independent deduplication systems, or caching components are needed. At the database level, only two logical tables are required to support complete file and task management, minimizing deployment units and significantly reducing the system's consumption of server computing, memory, and storage resources. This simplified architecture directly leads to standardized and lightweight deployment processes, significantly reducing configuration complexity, operational difficulty, and overall costs in private or independent deployment scenarios.

[0013] 2. By adopting a stateless design based on the HTTP standard interface and passing task context through self-contained upload credentials, the server does not need to rely on distributed sessions or caching to maintain state. Clients do not need to integrate any dedicated SDKs; the entire upload process can be completed through regular network requests, greatly reducing integration costs and code invasiveness, and inherently possessing cross-platform and portability advantages. This stateless architecture also provides the system with excellent horizontal scalability. When facing high concurrency requests, elastic scaling can be achieved by simply adding service instances, effectively improving the overall availability and stability of the system.

[0014] 3. By constructing a unified file upload task model, simple uploads and instant uploads are logically abstracted and integrated, and file feature values ​​are used for efficient deduplication in the file storage table. This mechanism ensures that duplicate files are accurately identified during upload and achieves "instant upload" by creating new task records instead of repeatedly transmitting file data, greatly saving network bandwidth consumption, especially in large file scenarios where uploads are completed in seconds. Simultaneously, this mechanism avoids redundant data in object storage from the source, improving storage space utilization and achieving a dual optimization of transmission efficiency and storage economy. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the structure of a lightweight, universal file upload system that can achieve instant file transfer, provided by an embodiment of the present invention;

[0017] Figure 2 This is a flowchart illustrating the unified upload credential generation process of a lightweight, universal file upload system capable of instant file transfer, provided by an embodiment of the present invention.

[0018] Figure 3 This is a simplified upload flowchart of a lightweight, universal file upload system that enables instant file transfer, provided by an embodiment of the present invention.

[0019] Figure 4 This is a flowchart illustrating the upload process of a lightweight, universal file upload system that enables instant file transfer, as provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0021] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0022] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0023] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0024] This invention provides a lightweight, universal file upload system capable of achieving instant file transfer, such as... Figure 1 The diagram illustrates the structure of a lightweight, general-purpose file upload system capable of instant file transfer. The system includes a system program module, a database module, and an object storage module. The system program module provides a set of HTTP interfaces to implement file upload and task management logic. It includes an upload task generation unit, a simple upload unit, and an instant upload unit. The database module records file upload tasks and file storage information and consists of two logical data tables: a file upload task table and a file storage table. The object storage module serves as an independent permanent storage medium for file content. The upload credentials enable stateless interface calls, eliminating the need for distributed session or caching components on the server side to maintain task state. Furthermore, by abstracting simple upload and instant upload into a unified file upload task model and using file feature values ​​for deduplication in the file storage table, the instant file transfer function is achieved. Ultimately, the overall architecture relies solely on the database module and the object storage module.

[0025] In this embodiment, the system consists of three core components: a system program module, a database module, and an object storage module. The system program module provides a set of standard HTTP interfaces to implement file upload and task management logic. Internally, it includes an upload task generation unit, a simple upload unit, and a fast upload unit, used to generate upload credentials, process regular file stream uploads, and determine fast uploads based on file feature values, respectively. The database module records file upload tasks and file storage information, consisting of only two logical data tables: a file upload task table and a file storage table, which manage the task lifecycle and the mapping relationship between file features and storage paths, respectively. The object storage module serves as an independent permanent storage medium for file content, responsible for the persistent storage of actual files. By unifying simple uploads and instant uploads into a single file upload task model, and using file feature values ​​for deduplication in the file storage table, the system achieves extreme simplification of architecture, lightweight deployment, and low barrier to entry for client integration, relying only on two basic components: database and object storage. At the same time, by leveraging a stateless upload credential mechanism, it eliminates the dependence on distributed session or caching components, significantly improving the system's scalability, stability, and upload efficiency. It also demonstrates a significant "instant upload" advantage and storage economy, especially in large file scenarios.

[0026] Furthermore, the upload task generation unit is specifically used for: receiving an upload initialization request initiated by the client, the request including at least a filename; responding to the request, generating a globally unique file upload task identifier; summarizing and encrypting at least one of the file upload task identifier, filename, client device identifier, and business source information to form a self-contained upload credential; and returning the upload credential to the client for the client to carry in subsequent upload or instant upload operations.

[0027] It needs to be explained that, such as Figure 2 The diagram shows a unified upload credential generation flowchart for a lightweight, universal file upload system capable of instant upload, provided by an embodiment of the present invention. The specific process is as follows: After the client initiates an upload initialization request, the upload task generation unit receives request information including at least the filename, then generates a globally unique file upload task identifier. The task ID, filename, and client device identifier are then aggregated and encrypted to form a self-contained secure upload credential, which is returned to the client. After obtaining this credential, the client can use it to execute subsequent simple upload processes or instant upload processes.

[0028] In this embodiment, the upload task generation unit, as a core component of the system program module, has the following specific functions: receiving an upload initialization request initiated by the client, which includes at least a filename; responding to the request to generate a globally unique file upload task identifier; encrypting the identifier along with parameters such as the filename, client device identifier, and business source information to form a self-contained upload credential; and returning the credential to the client for subsequent operations. This design achieves complete statelessness of interface calls through encrypted and encapsulated task credentials, eliminating the need for the server to maintain session state or rely on distributed caching components. This ensures data transmission security and significantly reduces client integration complexity, allowing for full-process access with only standard HTTP requests. This lays a key technical foundation for lightweight system deployment and high-availability expansion.

[0029] Furthermore, the simple upload unit is specifically used for: receiving an upload request initiated by a client, the parameters of which are the upload credential and the binary stream data of the file; parsing the upload credential, decrypting and restoring the file upload task identifier and other context information therein; querying the file upload task table according to the file upload task identifier to verify whether the task is the first submission; after successful verification, creating a task record in the file upload task table and marking its status as initialized; uploading the received file binary stream data to the object storage module; after successful upload, calculating the feature value of the file, recording the feature value in the current file upload task record, and querying the file storage table according to the feature value; if there is no record matching the feature value in the file storage table, inserting a new file record in the file storage table and updating the status of the current file upload task to completed.

[0030] It needs to be explained that, such as Figure 3 The diagram shows a simplified upload flowchart of a lightweight, universal file upload system capable of instant upload, provided by an embodiment of the present invention. The specific process is as follows: After the client initiates a request carrying upload credentials and a file stream, the simple upload unit first parses and decrypts the upload credentials, restoring the task ID and context information. Then, it queries the file upload task table to verify if the task is being submitted for the first time. If the task is duplicated, it is considered abnormal; if it is the first submission, a task record is created and its status is marked as "initialized." Next, the file binary stream data is uploaded to the object storage module. After obtaining the returned global access address, the file feature value is calculated, and the file storage table is queried to verify if the feature value exists. If it does not exist, a new file record is inserted into the file storage table. Finally, the file upload task status is updated to "completed," completing the entire upload process.

[0031] In this embodiment, the simple upload unit serves as the core functional unit for the system to implement regular file uploads. Its complete workflow includes: receiving a transmission request from a client carrying an upload credential and binary file stream data; parsing and decrypting the upload credential to restore the file upload task identifier and task context; performing a uniqueness check on the file upload task table based on the task identifier to ensure that the task is the first submission; establishing a task record and setting it to the initial state after successful verification; uploading the file stream data to the object storage module; calculating the file feature value and backfilling it into the task record after the upload is complete, while querying the file storage table based on the feature value; if no duplicate files are found, creating a new file storage record, and finally updating the task status to completed. This unit decouples business logic and transmission logic through a credential parsing mechanism, and establishes system-level file deduplication capabilities through feature value calculation and comparison. This ensures the integrity and security of the file transmission process and fundamentally avoids redundant occupation of storage resources, providing key technical support for the system to achieve a lightweight architecture and high-efficiency storage management.

[0032] Furthermore, the instant upload unit is specifically used for: receiving an instant upload request initiated by a client, wherein the parameters of the request are an upload credential containing file feature values, but without carrying binary stream data of the file; parsing the upload credential, decrypting and restoring the file feature values ​​and file upload task identifier therein; querying the file upload task table according to the file upload task identifier to verify whether the task is submitted for the first time; after the verification is successful, querying the file storage table according to the restored file feature values; if a record matching the file feature values ​​is found in the file storage table, a new file upload task record is directly created in the file upload task table, and the status of the task is directly marked as completed, thereby achieving the instant upload effect without transmitting file data.

[0033] It needs to be explained that, such as Figure 4 The diagram shows a flowchart of a lightweight, universal file upload system capable of instant upload, provided by an embodiment of the present invention. The specific process is as follows: After the client initiates an instant upload request with an upload credential that does not contain a file stream, the instant upload unit first parses and decrypts the upload credential, restoring the file feature value and task ID. Then, it queries the file upload task table to verify if the task is the first submission. If the task is duplicated, it is considered abnormal. If it is the first submission, it continues to query the file storage table to verify if the feature value exists. If the feature value does not exist, it returns empty data to the client, allowing the client to switch to a simple upload process. If the feature value exists, it directly creates a new task record in the file upload task table and marks the status as "completed," achieving successful instant upload with zero transmission overhead.

[0034] In this embodiment, the upload unit, as the core functional module for achieving ultra-fast file transfer, has the following complete workflow: receiving a client-initiated instant transfer request containing only file feature value upload credentials and no file binary stream data; parsing and decrypting the upload credentials to restore key information such as file feature values ​​and file upload task identifiers; performing validity verification in the file upload task table based on the task identifier to ensure that the task has not been submitted repeatedly; after successful verification, performing precise matching and querying in the file storage table based on the restored file feature values; when a file record with the same feature value is confirmed, the system directly creates a new task record in the file upload task table and marks its status as completed, thereby completely skipping the physical file transfer stage to achieve instant transfer. This mechanism, through intelligent feature value comparison and task status reconstruction, completely eliminates the network bandwidth consumption caused by duplicate file transfers while ensuring data consistency, enabling large file uploads to achieve millisecond-level response. This not only significantly improves system transmission efficiency and user experience but also significantly reduces the write load and storage redundancy of object storage, forming a highly efficient and energy-saving intelligent file transfer solution.

[0035] Furthermore, the file upload task table is used to record the entire lifecycle information centered on the task. Its fields include at least the following: a unique identifier for the file upload task, serving as the business primary key of the table; file characteristic values; the upload task type, used to distinguish between simple uploads and instant uploads; the upload task status, used to identify whether the task is in the initialization, in progress, or completed state; at least one of the following: file name, file size, file extension, and file category; and at least one of the following: client device identifier, client operating system platform, and business source module.

[0036] In this embodiment, the file upload task table, as the core data table of the system database module, plays a crucial role in recording the entire lifecycle information of file upload tasks. Its table structure design includes the following core fields: a unique identifier for the file upload task serves as the business primary key to ensure global uniqueness; file characteristic values ​​are used to establish a relationship with the file storage table; the upload task type field clearly distinguishes between simple upload and instant upload modes; the upload task status field fully tracks the entire process of the task from initialization, in progress to completion; it also records basic file attributes such as file name, file size, extension, and file category, as well as contextual information such as client device identifier, operating system platform, and business source module. This table, through its refined field design, achieves complete metadata management for upload tasks, providing data-level support for a unified task model and creating a unified task processing framework for the system through the standardization of task types and statuses. Its collaborative working mode with the file storage table enables the system to maintain a minimalist architecture while implementing complex task status tracking, business source analysis, and file deduplication logic, providing a solid data foundation for the efficient operation and maintenance management of the system.

[0037] Furthermore, the file storage table is used to establish a unique mapping relationship between file feature values ​​and physical storage locations. The fields it contains are used to record at least: the file feature value, which serves as the business primary key of the table and has a uniqueness constraint; the actual storage path of the file in the object storage module; and the file size, wherein the same file feature value in the file storage table uniquely corresponds to one stored actual file.

[0038] In this embodiment, the file storage table, as a core component of the system's database module, is specifically designed to establish a unique mapping between file feature values ​​and physical storage locations. Its table structure includes the following key fields: the file feature value serves as the business primary key and has a unique constraint, ensuring that each file feature value is unique within the system; the actual storage path of the file in the object storage module, precisely pointing to the file's physical storage location; and basic attribute information such as file size. Through this design, the file storage table achieves a precise mapping where the same file feature value uniquely corresponds to a single stored actual file. This innovative design forms the data foundation for the system's deduplication mechanism, providing authoritative deduplication criteria for the instant file transfer function through the strong association between feature values ​​and storage paths. This mapping mechanism not only ensures the uniqueness and consistency of file storage but also fundamentally avoids duplicate files occupying storage space, significantly improving storage space utilization. Simultaneously, the collaborative work between this table and the file upload task table enables the system to achieve efficient file retrieval and management while maintaining a minimalist architecture, providing crucial technical support for the system's lightweight deployment and high-performance operation.

[0039] Furthermore, the file upload task table and the file storage table are logically associated through the file feature value; the same file feature value has at most one record in the file storage table, while it can correspond to multiple file upload task records in the file upload task table, thereby realizing the mapping relationship between multiple upload task records based on the same file and a single file storage record, supporting file deduplication and instant upload functions.

[0040] In this embodiment, the file upload task table and the file storage table are logically associated through file feature values, forming an efficient one-to-many mapping relationship: in the file storage table, each file feature value has at most one record, uniquely corresponding to one stored actual file; while in the file upload task table, the same file feature value can correspond to multiple upload task records, accurately recording multiple upload behaviors based on the same file. This carefully designed association mechanism constitutes the data foundation of the system's core deduplication architecture. Through the separate management of task records and file entities, the business context and state trajectory of each upload task are fully preserved, while ensuring the uniqueness of physical file storage. This design directly supports the implementation of the instant upload function. When a file with the same feature value is detected to already exist, the system only needs to create a new task record without repeatedly transmitting the stored file. This not only significantly improves upload efficiency and reduces network bandwidth consumption but also significantly optimizes storage space utilization, achieving a perfect balance between business traceability and storage economy.

[0041] Furthermore, the object storage module is an external general storage service independent of the system program module and the database module; the system program module communicates with the object storage module through the standard application programming interface provided by the object storage module to realize file uploading, storage and management.

[0042] In this embodiment, the object storage module serves as an external general-purpose storage service independent of the program module and database module. It communicates and interacts with the system program module through a standardized application programming interface, undertaking the functions of uploading, storing, and managing all entity file data. This architectural design decouples business logic, metadata management, and entity file storage, enabling the system to fully leverage the high reliability, high availability, and elastic scalability of mature object storage services. The independent deployment of this module not only avoids the complexity and maintenance costs of building a self-built file storage system but also maintains a highly concise overall system architecture, allowing focus solely on the implementation of core upload business logic. Simultaneously, it provides professional-grade storage security and near-unlimited storage expansion potential, offering a solid storage foundation for the stable operation of the system in various deployment environments.

[0043] Furthermore, in the workflow of the simple upload unit, the binary stream data of the file is directly uploaded to the object storage module, and the object storage module generates and returns the global access address of the file; the storage path field recorded in the file storage table is the unique identifier of the file in the object storage module or the global access address.

[0044] In this embodiment, during the workflow of the simple upload unit, the file's binary stream data is directly transmitted to the object storage module via a standardized interface. This module then generates and returns the file's global access address. Simultaneously, the storage path field recorded in the file storage table precisely corresponds to this global access address or the file's unique identifier in the object storage. This mechanism achieves perfect decoupling between physical file storage and logical management, ensuring both the professionalism and reliability of file storage while establishing a strong correlation between file characteristics and actual storage locations through path mapping. This design not only simplifies the file addressing process and improves access efficiency but also provides crucial data support for the system to achieve file deduplication and instant transfer functions. Furthermore, it significantly reduces the complexity of storage management, allowing the system to focus on business logic processing without needing to concern itself with underlying storage details.

[0045] Furthermore, when the instant upload unit is working, it only operates the database module and does not interact with the object storage module in any way that involves file data transmission. After the instant upload logic verification is passed, it reuses the storage path that has been recorded in the file storage table and associated with the existing file, thereby avoiding repeated write operations to the object storage module.

[0046] In this embodiment, the core feature of the instant upload unit during operation is that it only operates on the database module, completely avoiding any interaction with the object storage module involving file data transfer. Once the instant upload logic is verified through file feature value comparison, the system directly reuses the storage path information already existing in the file storage table, associated with files of the same feature value, thereby completely avoiding duplicate write operations to the object storage module. This mechanism achieves extreme efficiency in file-level deduplication through precise metadata management and intelligent path reuse. It not only simplifies the upload process of duplicate files from physical transmission to logical recording, significantly improving system response speed to the millisecond level, but also effectively reduces the write pressure and network bandwidth consumption of the object storage service. Simultaneously, this design significantly improves storage space utilization by avoiding storage redundancy, achieving dual optimization of transmission efficiency and resource utilization while ensuring data consistency. This provides key technical guarantees for the stable operation and cost control of the system in high-concurrency scenarios.

[0047] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0048] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0049] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0050] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0051] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0052] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0053] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0054] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0055] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0056] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0057] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A lightweight universal file uploading system capable of achieving P2SP, characterized in that, The system program module, the database module and the object storage module are included: The system program module is used for providing a set of HTTP interfaces to realize file uploading logic and task management logic, and the system program module includes an uploading task generation unit, a simple uploading unit and a second uploading unit; The database module is used for recording file uploading tasks and file storage information, and is composed of two logical data tables, i.e., a file uploading task table and a file storage table; The object storage module is used as an independent file content permanent storage medium, wherein statelessness of interface calling is realized through the uploading credential, and the server side does not need to rely on a distributed session or a cache component to maintain a task state; and the simple uploading and the second uploading are abstracted into a unified file uploading task model, and file feature values are used to perform duplication checking in the file storage table, so that the second uploading function of the file is realized; finally, only the database module and the object storage module are relied on in the overall architecture; The uploading task generation unit is specifically used for: receiving an uploading initialization request initiated by a client, the request at least including a file name; generating a globally unique file uploading task identifier in response to the request; encrypting and collecting at least one of the file uploading task identifier, the file name, a client device identifier and business source information to form a self-contained uploading credential; returning the uploading credential to the client for carrying in subsequent uploading or second uploading operations; The simple uploading unit is specifically used for: receiving an uploading request initiated by the client, the request parameter being the uploading credential and file binary stream data; decrypting and restoring the file uploading task identifier and other context information in the uploading credential; querying the file uploading task table according to the file uploading task identifier to check whether the task is submitted for the first time; after verification, creating a task record in the file uploading task table and marking the state of the task record as initialization; uploading the received file binary stream data to the object storage module; after successful uploading, calculating a feature value of the file, recording the feature value in the file uploading task record, and querying the file storage table according to the feature value; if there is no record matched with the feature value in the file storage table, inserting a new file record in the file storage table, and updating the state of the file uploading task to completed; The second uploading unit is specifically used for: receiving a second uploading request initiated by the client, the request parameter being an uploading credential including a file feature value, and not carrying file binary stream data; decrypting and restoring the file feature value and the file uploading task identifier in the uploading credential; querying the file uploading task table according to the file uploading task identifier to check whether the task is submitted for the first time; after verification, querying the file storage table according to the restored file feature value; If a record matching the file characteristic value is found in the file storage table, a new file upload task record is created in the file upload task table, and the status of the task is directly marked as completed, thereby realizing the second transmission effect without transmitting file data; The file upload task table is used to record the whole life cycle information centered on the task, and contains fields for recording at least: The file upload task unique identifier, as the business primary key of the table; the file characteristic value; the upload task type, used to distinguish simple upload and second transmission upload; the upload task status, used to identify the task in the initialization, in progress or completed state; at least one of the file name, file size, file extension and file classification; at least one of the client device identifier, client operating system platform and business source module; The file storage table is used to establish a unique mapping relationship between the file characteristic value and the physical storage location, and contains fields for recording at least: The file characteristic value, as the business primary key of the table and having a unique constraint; the actual storage path of the file in the object storage module; the file size, wherein the same file characteristic value of the file storage table uniquely corresponds to an already stored actual file; The file upload task table and the file storage table are logically associated through the file characteristic value; The same file characteristic value at most exists one record in the file storage table, while in the file upload task table, it can correspond to multiple file upload task records, thereby realizing the mapping relationship between the multiple upload task records based on the same file and the single file storage record, supporting file deduplication and second transmission function; The object storage module is an external general storage service independent of the system program module and the database module; The system program module communicates with the object storage module through the standard application programming interface provided by the object storage module to realize the upload, storage and management of files; In the workflow of the simple upload unit, the file binary stream data is directly uploaded to the object storage module, and the object storage module generates and returns the global access address of the file; The storage path field recorded in the file storage table is the unique identifier or the global access address of the file in the object storage module; The second transmission upload unit operates only the database module when working, and does not interact with the object storage module in any file data transmission related operation; When the second transmission logic verification passes, the storage path associated with the existing file recorded in the file storage table is reused, thereby avoiding repeated write operations to the object storage module.

Images