Resource file storage method based on Fastdfs technology lateral extension storage and related method

Abstract

The invention discloses a resource file storage method based on Fastdfs (Fastdfs) technology lateral extension storage and a related method. The method comprises the following steps: receiving a resource file uploaded by a client, and reading an interface path and a service type of the resource file based on a file management micro-service; according to an interface path and a service type of the resource file, selecting a corresponding storage group in the corresponding Fastdfs and writing the selected storage group in the corresponding Fastdfs into the resource file; generating resource file metadata corresponding to the resource file, and writing the resource file metadata into a database; judging whether the resource file and the resource file metadata are written successfully or not; and if yes, returning the file information to the client. According to the method, one set or multiple sets of Fastdfs are designed, each set of Fastdfs uses multiple storage groups, and the multiple sets of Fastdfs are used together in combination with an independent service Springboot, so that the storage space is expanded, the storage groups are self-defined to classify resource files of different services, distributed storage of large-data-volume file data is realized, and the resource file metadata is scanned at regular time, so that the resource file data can be stored in a large-data-volume manner. And regularly migrating the original file in the Fastdfs to release the storage space.

Description

Technical Field

[0001] The present application relates to a resource file storage method and related methods based on Fastdfs technology for horizontally expanding storage. Background Art

[0002] In some long-term projects, such as projects with various modules such as online voting, questionnaires, and statistical analysis reports, functions such as exporting files and uploading related resource files will use a large amount of Fastdfs storage space, involving many file types, such as doc, doc, xlsx, pdf, jpg, png, mp4, and mp3, as well as file uploads of image, audio, and video types. With the increase in project operation time, the increase in functions, and the long-term accumulation of resources such as attachments, the resource storage capacity has reached the PB level, and the daily increment has reached the 30GB level. With the long-term project operation, the storage space of the existing architecture storage solution has reached its limit. Summary of the invention

[0003] In order to better realize the storage and migration of resource files in long-term project operations, the embodiment of the present application provides a resource file storage method and related methods for horizontally expanding storage based on Fastdfs technology.

[0004] In a first aspect, an embodiment of the present application provides a resource file storage method for horizontally expanding storage based on Fastdfs technology, the method comprising:

[0005] Receive the resource file uploaded by the client, and read the interface path and business type of the resource file based on the file management microservice;

[0006] According to the interface path and business type of the resource file, select the corresponding storage group in Fastdfs to write the resource file;

[0007] Generate resource file metadata corresponding to the resource file and write it into the database;

[0008] Determine whether the resource file and the resource file metadata are both written successfully;

[0009] If yes, return the file information to the client.

[0010] In an optional implementation of the embodiment of the present application, a correspondence between an interface path and each storage group in each Fastdfs is preset; a correspondence between a specific interface path and each Fastdfs is preset; and a correspondence between each Fastdfs and a business type of a resource file is preset by a preset identifier; wherein the Fastdfs includes at least one storage group;

[0011] The step of selecting a corresponding storage group in Fastdfs to write the resource file according to the interface path and the service type of the resource file includes:

[0012] Determine whether the interface path of the resource file is a specific interface path;

[0013] If yes, select Fastdfs corresponding to the specific interface path, and write the resource file into the storage group corresponding to the interface path;

[0014] If not, determining whether the service type of the resource file is empty or does not carry a preset identifier;

[0015] If yes, select the default Fastdfs and write the resource file into the storage group corresponding to the interface path;

[0016] If not, select Fastdfs corresponding to the preset identifier, and write the resource file into the storage group corresponding to the interface path.

[0017] In an optional implementation of the embodiment of the present application, it also includes:

[0018] Based on the timing microservice, each resource file to be archived is determined according to the creation time of each resource file metadata at every preset time period;

[0019] Based on the archiving microservice, each resource file to be archived in Fastdfs is sent to the preset network attached storage.

[0020] In an optional implementation of the embodiment of the present application, the timing microservice is based on, at every preset time period, determining each resource file to be archived according to the creation time of each resource file metadata in the database, including:

[0021] Based on the timing microservice, at every preset time period, according to the time interval between the creation time of each resource file metadata in the database and the current time, determine whether the time interval is greater than the preset time interval;

[0022] If so, the status flag of the corresponding resource file metadata is modified to be archived, and the resource file corresponding to the resource file metadata is determined as the resource file to be archived.

[0023] In an optional implementation of the embodiment of the present application, after sending each resource file to be archived in Fastdfs to a preset network attached storage, it also includes:

[0024] Return the storage address of each resource file to be archived in the preset network attached storage to the file management microservice;

[0025] Based on the file management microservice, the storage address is written into the corresponding resource file metadata of the database, and the status identifier of the resource file metadata is changed from to be archived to archived.

[0026] In a second aspect, an embodiment of the present application provides a resource file query method based on the horizontal expansion storage of Fastdfs technology, the method comprising:

[0027] According to the query conditions input by the client, based on the file management microservice, determine the resource file metadata corresponding to the query conditions in the database;

[0028] Determine whether the status indicator in the metadata of the resource file is archived;

[0029] If yes, calling the independent service through http request to determine the resource file corresponding to the query condition in the preset network attached storage;

[0030] If not, search the corresponding storage group in Fastdfs according to the business type in the metadata of the resource file to determine the resource file corresponding to the query condition;

[0031] The file byte stream of the resource file is returned to the client.

[0032] In an optional implementation of the embodiment of the present application, determining the resource file metadata corresponding to the query condition in the database based on the file management service according to the query condition input by the client includes:

[0033] According to the query conditions input by the client, based on the file management microservice, determine whether there is resource file metadata corresponding to the query conditions in the database;

[0034] If so, determining the resource file metadata;

[0035] If not, a query failure message is returned to the client.

[0036] In a third aspect, an embodiment of the present application provides a resource file management method for horizontally expanding storage based on Fastdfs technology, the method comprising:

[0037] Receive the resource file uploaded by the client, and read the interface path and business type of the resource file based on the file management microservice;

[0038] According to the interface path and business type of the resource file, select the corresponding storage group in Fastdfs to write the resource file;

[0039] Generate resource file metadata corresponding to the resource file and write it into the database;

[0040] Based on the timing microservice, each resource file to be archived is determined according to the creation time of the metadata of each resource file in the database at every preset time period;

[0041] Based on the archiving microservice, each resource file to be archived in Fastdfs is sent to the preset network attached storage, and the identification status in the metadata of the corresponding resource file is modified to archived;

[0042] According to the query conditions input by the client, based on the file management microservice, determine the resource file metadata corresponding to the query conditions in the database;

[0043] Determine whether the status indicator in the metadata of the resource file is archived;

[0044] If yes, calling the independent service through http request to determine the resource file corresponding to the query condition in the preset network attached storage;

[0045] If not, search the corresponding storage group in Fastdfs according to the business type in the metadata of the resource file to determine the resource file corresponding to the query condition;

[0046] The file byte stream of the resource file is returned to the client.

[0047] In a fourth aspect, an embodiment of the present application provides a resource file uploading device for horizontally expanding storage based on Fastdfs technology, the device comprising:

[0048] The first receiving module is used to receive the resource file uploaded by the client and read the interface path and business type of the resource file based on the file management microservice;

[0049] A first writing module, used to select a corresponding storage group in Fastdfs to write the resource file according to the interface path and business type of the resource file;

[0050] A second writing module is used to generate resource file metadata corresponding to the resource file and write it into a database;

[0051] A first judgment module, used to judge whether the resource file and the resource file metadata are both written successfully;

[0052] The first returning module is used to return the file information to the client if both the resource file and the resource file metadata are written successfully.

[0053] In a fifth aspect, an embodiment of the present application provides a resource file query device based on the horizontal expansion storage of Fastdfs technology, the device comprising:

[0054] The first query module is used to determine the resource file metadata corresponding to the query condition in the database based on the file management microservice according to the query condition input by the client;

[0055] A second judgment module is used to judge whether the status mark in the metadata of the resource file is archived;

[0056] A first determination module is used to call an independent service through an http request to determine a resource file corresponding to the query condition in a preset network attached storage if the status in the metadata of the resource file is archived;

[0057] A second determination module is used to search for a corresponding storage group in Fastdfs according to the business type in the metadata of the resource file if the status identifier in the metadata of the resource file has not been archived, and determine the resource file corresponding to the query condition;

[0058] The second returning module is used to return the file byte stream of the resource file to the client.

[0059] In a sixth aspect, an embodiment of the present application provides a resource file management device for horizontally expanding storage based on Fastdfs technology, the device comprising:

[0060] The second receiving module is used to receive the resource file uploaded by the client and read the interface path and business type of the resource file based on the file management microservice;

[0061] The third writing module is used to select a corresponding storage group in Fastdfs to write the resource file according to the interface path and business type of the resource file;

[0062] A fourth writing module, used to generate resource file metadata corresponding to the resource file and write it into a database;

[0063] The third determination module is used to determine each resource file to be archived according to the creation time of each resource file metadata in the database at every preset time period based on the timing microservice;

[0064] The status modification module is used to send each resource file to be archived in Fastdfs to the preset network attached storage based on the archiving microservice, and modify the identification status in the metadata of the corresponding resource file to archived;

[0065] The second query module is used to determine the resource file metadata corresponding to the query condition in the database based on the query condition input by the client and the file management microservice;

[0066] A third judgment module is used to judge whether the status mark in the metadata of the resource file is archived;

[0067] A fourth determination module is used to call an independent service through an http request to determine a resource file corresponding to the query condition in a preset network attached storage if the status in the metadata of the resource file is archived;

[0068] A fifth determination module, for searching a corresponding storage group in Fastdfs according to a business type in the metadata of the resource file if the status in the metadata of the resource file is not archived, and determining a resource file corresponding to the query condition;

[0069] The third returning module is used to return the file byte stream of the resource file to the client.

[0070] In the seventh aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the resource file storage method for horizontally expanding storage based on the Fastdfs technology as described above, and / or, the resource file query method for horizontally expanding storage based on the Fastdfs technology, and / or, the resource file management method for horizontally expanding storage based on the Fastdfs technology.

[0071] In an eighth aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the resource file storage method for horizontally expanding storage based on the Fastdfs technology as described above, and / or, the resource file query method for horizontally expanding storage based on the Fastdfs technology, and / or, the resource file management method for horizontally expanding storage based on the Fastdfs technology.

[0072] In the ninth aspect, an embodiment of the present application provides a computer program product comprising instructions. When the computer program product is run on a computer device, the computer device executes the resource file storage method based on the horizontal expansion storage of the Fastdfs technology as described above, and / or the resource file query method based on the horizontal expansion storage of the Fastdfs technology, and / or the resource file management method based on the horizontal expansion storage of the Fastdfs technology.

[0073] In the tenth aspect, an embodiment of the present application provides a chip, the chip including a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run a computer program or instruction to implement the resource file storage method based on the horizontal expansion storage of Fastdfs technology as mentioned above, and / or, the resource file query method based on the horizontal expansion storage of Fastdfs technology, and / or, the resource file management method based on the horizontal expansion storage of Fastdfs technology.

[0074] The beneficial effects of the above technical solution provided by the embodiment of the present application include at least:

[0075] The resource file storage method based on the horizontal expansion storage of Fastdfs technology provided in the embodiment of the present application is to select the corresponding Fastdfs and storage group for writing the resource file uploaded by the client according to its business type and interface path, and generate the resource file metadata corresponding to the resource file, write it into the database, and finally, after the resource file and the resource file metadata are successfully written, return the file information to the client. This method expands the storage space by designing one or more sets of Fastdfs, each set of Fastdfs uses multiple groups of storage groups, and combines with the independent service Springboot to use multiple sets of Fastdfs together, and customizes the storage group to classify resource files of different businesses, so as to realize the distributed storage of large amounts of file data; by regularly scanning the resource file metadata, the original files in Fastdfs are regularly migrated to release storage space, which solves the problem of manual migration intervention after the Fastdfs storage is full of capacity under long-term project operation, and the problem that the resource files cannot be previewed again after being archived.

[0076] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings.

[0077] The technical solution of the present application is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The accompanying drawings are used to provide a further understanding of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings:

[0079] Figure 1 A schematic diagram of the steps of a resource file storage method for horizontally expanding storage based on Fastdfs technology provided in an embodiment of the present application;

[0080] Figure 2 A schematic diagram of the process of uploading resource files provided in the embodiment of the present application;

[0081] Figure 3 A schematic diagram of the steps of a resource file query method for horizontally expanding storage based on Fastdfs technology provided in an embodiment of the present application;

[0082] Figure 4 A schematic diagram of a resource file query process provided in an embodiment of the present application;

[0083] Figure 5 A schematic diagram of the steps of a resource file management method for horizontally expanding storage based on Fastdfs technology provided in an embodiment of the present application;

[0084] Figure 6 A schematic diagram of the overall process of uploading, archiving and querying resource files provided in an embodiment of the present application;

[0085] Figure 7 A schematic diagram of the structure of a resource file storage device for horizontally expanding storage based on Fastdfs technology provided in an embodiment of the present application;

[0086] Figure 8 A schematic diagram of the structure of a resource file query device for horizontally expanding storage based on Fastdfs technology provided in an embodiment of the present application;

[0087] Fig. 9 A schematic diagram of the structure of a resource file management device for horizontally expanding storage based on Fastdfs technology provided in an embodiment of the present application. DETAILED DESCRIPTION

[0088] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0089] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0090] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0091] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0092] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0093] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0094] It should be understood that the size of the serial numbers of the steps in the following embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0095] In order to illustrate the technical solution of the present application, a specific embodiment is provided below for illustration.

[0096] The inventor found that in the prior art, when a long-term project is in operation, the upload of a large number of resource files occupies storage space, and the storage space of the storage solution of the existing architecture can easily reach the limit. At this time, manual migration intervention is required to release the storage space, and the migrated resource files cannot be previewed again, which does not meet the inventor's expectations. Based on this, the inventor has made this application after further research and development, providing a resource file storage method and related methods based on Fastdfs technology for horizontal expansion storage.

[0097] Embodiment 1

[0098] The present application embodiment provides a resource file storage method based on Fastdfs technology to horizontally expand storage, referring to Figure 1 As shown, the method includes:

[0099] S101: receiving a resource file uploaded by a client, and reading an interface path and a service type of the resource file based on a file management microservice.

[0100] In the embodiment of the present application, in order to achieve the expansion and migration of file storage, the inventor has designed a set of one or more sets of Fastdfs horizontal expansion with one or more storage groups (groups) in each set to classify and store resource files. It can be designed that about 3 businesses use a set of Fastdfs, and cooperate with the independent service Springboot to regularly archive resource files to achieve regular cleaning of Fastdfs service storage space. Among them, Springboot includes 3 microservices, namely the main service of file management (File-manager, hereinafter referred to as file management microservice), the service of execution plan (task-scheduling, hereinafter referred to as timing microservice) and the service of file archiving (filing-up, hereinafter referred to as archiving microservice).

[0101] In the embodiment of the present application, the client uploads the resource file through the pre-set function page. When receiving the resource file to be stored uploaded by the client, the resource file information is read through the file management microservice to obtain the interface path of the resource file and the interface parameter business type (businessType, hereinafter referred to as business type). Among them, the business type includes three states, namely, the business type is empty, the business type does not carry the preset identifier, and the business type carries the preset identifier.

[0102] S102: According to the interface path and service type of the resource file, a corresponding storage group in Fastdfs is selected to write the resource file.

[0103] In the embodiment of the present application, the correspondence between the interface path and each storage group in each Fastdfs is preset; the correspondence between a specific interface path and each Fastdfs is preset; and, by a preset identifier, the correspondence between each Fastdfs and the business type of the resource file is preset; wherein the Fastdfs includes at least one storage group;

[0104] The step of selecting a corresponding storage group in Fastdfs to write the resource file according to the interface path and the service type of the resource file includes:

[0105] Determine whether the interface path of the resource file is a specific interface path;

[0106] If yes, select Fastdfs corresponding to the specific interface path, and write the resource file into the storage group corresponding to the interface path;

[0107] If not, determining whether the service type of the resource file is empty or does not carry a preset identifier;

[0108] If yes, select the default Fastdfs and write the resource file into the storage group corresponding to the interface path;

[0109] If not, select Fastdfs corresponding to the preset identifier, and write the resource file into the storage group corresponding to the interface path.

[0110] In the embodiment of the present application, the inventor pre-binds different Fastdfs and their storage groups corresponding to different businesses in properties, wherein each Fastdfs includes at least one storage group, and a default Fastdfs is preset.

[0111] The specific binding rules are as follows:

[0112] 1. Set the corresponding relationship between the interface path and each storage group in each Fastdfs to ensure that when any set of Fastdfs storage resource files is selected, the corresponding storage group can be determined according to the interface path of the resource file to write the resource file;

[0113] 2. Set the corresponding relationship between a specific interface path and each Fastdfs;

[0114] 3. Through the preset identifier, set the corresponding relationship between each Fastdfs and the business type of the resource file. When the preset identifier exists, determine the Fastdfs corresponding to the business type of the resource file according to the preset identifier. When the preset identifier does not exist, store it in the preset default Fastdfs.

[0115] In an embodiment of the present application, after confirming the interface path and business type of the resource file to be uploaded in S101, it is first determined whether the interface path is a specific interface path that has been pre-bound. If so, the corresponding Fastdfs is directly selected, and the resource file is written according to the storage group corresponding to the interface path; if not, it is determined whether the business type is empty. If the business type is empty, the preset default Fastdfs is selected, and the resource file is written according to the storage group corresponding to the interface path; if the business type is not empty, it is determined whether the business type carries a preset identifier. If it does not carry the preset identifier, the preset default Fastdfs is selected, and the resource file is written according to the storage group corresponding to the interface path; if it carries a preset identifier, the Fastdfs corresponding to the preset identifier is selected, and the resource file is written according to the storage group corresponding to the interface path.

[0116] S103: Generate resource file metadata corresponding to the resource file and write it into a database.

[0117] In the embodiment of the present application, when the resource file is written into the corresponding storage group in the corresponding Fastdfs, the resource file metadata corresponding to the resource file is generated. Usually, the resource file is a graphic image, document, table, ppt, video and audio, etc., and the resource file metadata is the file information produced by the java program and the Fastdfs server component. The resource file metadata may include the primary key ID, file name, file path, file group, file size, file type, creator and creation time, etc.

[0118] After the resource file metadata is generated, it is stored in a preset persistent storage database, such as an Oracle database.

[0119] S104: Determine whether the resource file and the resource file metadata are both written successfully, if so, execute S105, if not, execute S106.

[0120] S105: Return the file information to the client.

[0121] S106: Return error information.

[0122] In the embodiment of the present application, in the entire file upload process, the resource file is considered to be uploaded successfully only when both the resource file and the resource file metadata are successfully written. Therefore, it is necessary to determine whether both the resource file and the resource file metadata are successfully written. If so, the relevant file information stored in the file is returned to the client. If not, it means that the upload failed, an exception is thrown, and an error message is returned.

[0123] In the present application embodiment, Figure 2As shown, it is a flowchart of resource file upload. When the user end starts to upload resource files, the file information of the resource files, including the interface path and business type, is read through the file management microservice; then, it is determined whether the interface path is a specific interface path. If it is a specific interface path, a specific lightweight distributed storage is selected, that is, a predetermined corresponding Fastdfs is selected; if it is not a specific interface path, the business type of the resource file is continued to be judged to determine whether the business type exists and is 1, that is, whether the business type is not empty and carries a preset identifier. If so, a specific lightweight distributed storage is selected. If not, the default lightweight distributed storage is selected, that is, the preset default Fastdfs is selected; after selecting the corresponding lightweight distributed storage, the storage group corresponding to the interface path of the resource file is determined, and the resource file is stored according to the business; after the storage is completed, it is determined whether the file is uploaded to the server, that is, whether the resource file is stored successfully. If so, the generated resource file metadata is written into the persistent storage database. After writing successfully, the entire upload process ends normally. If either the resource file or the resource file metadata is not written successfully, the overall upload process ends, an exception is thrown, and an error message is returned.

[0124] In the embodiment of the present application, managing resource files in Fastdfs also includes:

[0125] Based on the timing microservice, each resource file to be archived is determined according to the creation time of each resource file metadata at every preset time period;

[0126] Based on the archiving microservice, each resource file to be archived in Fastdfs is sent to the preset network attached storage.

[0127] In the embodiment of the present application, based on the timing microservice, the Oracle database table is scanned at every preset period to find the corresponding resource file whose metadata creation time (create_time) is greater than the preset time interval from the current time, and the archiving microservice in the independent service Springboot is used to migrate the resource file to the preset network attached storage (nas) to achieve centralized storage and archiving of cold and old data. Among them, the preset period can be set to execute once at 3 am every day, and the preset time interval can be set to 93 days. When designing the Fastdfs service, the resource files can be designed to store up to about 3 months of resource files. Then, resource files that are more than 93 days after uploading need to be migrated to the preset network attached storage.

[0128] In the embodiment of the present application, the timing microservice is based on, at every preset time period, determining each resource file to be archived according to the creation time of each resource file metadata in the database, specifically including:

[0129] Based on the timing microservice, at every preset time period, according to the time interval between the creation time of each resource file metadata in the database and the current time, determine whether the time interval is greater than the preset time interval;

[0130] If so, the status flag of the corresponding resource file metadata is modified to be archived, and the resource file corresponding to the resource file metadata is determined as the resource file to be archived.

[0131] In the embodiment of the present application, after sending each resource file to be archived in Fastdfs to the preset network attached storage, it also includes:

[0132] Return the storage address of each resource file to be archived in the preset network attached storage to the file management microservice;

[0133] Based on the file management microservice, the storage address is written into the corresponding resource file metadata of the database, and the status identifier of the resource file metadata is changed from to be archived to archived.

[0134] After scanning the metadata of the resource file whose creation time is greater than the preset time interval from the current time, the status identifier (filing_status) in the resource file metadata is changed from "unarchived" to "to be archived", and the resource file corresponding to the resource file metadata is determined as the resource file to be archived. The resource file to be archived is migrated to the preset network attached storage through the archiving microservice. After the archiving of the resource file is completed, the storage address of the archived resource file in the preset network attached storage is returned to the file management microservice. Through the file management microservice, the storage address is written into the corresponding resource file metadata of the database, and the status identifier of the resource file metadata is changed to archived.

[0135] After archiving is completed, the Fastdfs storage path of the resource file, the business type of the file, the storage group to which the storage belongs, the file name, file type, file status identifier and file archiving address (nas_location) are all used as file information to identify a resource file, which can be used to quickly find the storage method and storage path of the corresponding resource file, thereby quickly obtaining and returning the file byte stream.

[0136] Embodiment 2

[0137] Based on the same inventive concept, the embodiment of the present application also provides a resource file query method based on the horizontal expansion storage of Fastdfs technology, referring to Figure 3 As shown, the method includes:

[0138] S201: According to the query conditions input by the client, based on the file management microservice, determine the resource file metadata corresponding to the query conditions in the database.

[0139] In the embodiment of the present application, determining the resource file metadata corresponding to the query condition in the database based on the file management service according to the query condition input by the client specifically includes:

[0140] According to the query conditions input by the client, based on the file management microservice, determine whether there is resource file metadata corresponding to the query conditions in the database;

[0141] If so, determining the resource file metadata;

[0142] If not, a query failure message is returned to the client.

[0143] In an embodiment of the present application, when a client queries a resource file, it first enters the query conditions and sends a query request to the file management microservice. The file management microservice queries the Oracle database through the query conditions whether there is corresponding resource file metadata. If so, it means that the corresponding resource file metadata has been found. If not, it means that the corresponding resource file metadata cannot be found according to the query conditions, and a query exception occurs, and query failure information can be returned to the client.

[0144] S202: Determine whether the status identifier in the metadata of the resource file is archived, if so, execute S203, if not, execute S204.

[0145] S203: calling an independent service through an http request to determine a resource file corresponding to the query condition in a preset network attached storage.

[0146] S204: searching for a corresponding storage group in Fastdfs according to the business type in the metadata of the resource file, and determining a resource file corresponding to the query condition.

[0147] In the embodiment of the present application, after finding the resource file metadata corresponding to the query condition, it is possible to determine whether the corresponding resource file is stored in Fastdfs or in the preset network attached storage according to the status displayed by the status identifier in the resource file metadata. If the status identifier is "archived", it means that the corresponding resource file is stored in the preset network attached storage, and an independent service is called through an http request to determine whether the corresponding resource file exists in the preset network attached storage. If not, the query fails and the query is directly exited; if the status identifier is "unarchived", it means that the corresponding resource file is stored in Fastdfs, and according to the file information recorded in the resource file metadata, a specific set of Fastdfs is located through the interface path or business type, and the specific storage group is located through the file storage group information (storage_group) contained in the file information, and then it is determined whether the corresponding resource file exists therein. If not, the query fails and the query is directly exited.

[0148] S205: Return the file byte stream of the resource file to the client.

[0149] In the embodiment of the present application, after the resource file corresponding to the query condition is determined through the query method of S101 to S104, the data in the resource file is returned to the client in the format of a file byte stream, thus completing the entire query process.

[0150] In the present application embodiment, Figure 4 As shown, it is a flowchart of resource file query. The client enters the query conditions for query. After the query starts, it enters the file management microservice to query whether the persistent database file information exists, that is, to determine whether the resource file metadata corresponding to the query conditions exists in the database. If not, it means that the query is abnormal; if it exists, then determine whether the resource file is archived according to the status identifier of the resource file metadata. If the resource file has been archived, query whether there is a corresponding resource file in the archived resource file of the preset network attached storage. If it exists, the query is successful, and the file byte stream of the resource file is returned to the client, and the process ends. If it does not exist, it means that the query is abnormal; if the resource file is not archived, query the lightweight distributed storage. In one or more sets of lightweight distributed storages, determine which set of Fastdfs storage service is used according to the file information in the resource file metadata, further determine whether there is a specified storage group, determine which storage group in Fastdfs the resource file is stored, and then determine whether there is a resource file corresponding to the query conditions in the storage group. If it exists, the query is successful, and the file byte stream of the resource file is returned to the client, and the process ends. If it does not exist, it means that the query is abnormal.

[0151] Embodiment 3

[0152] Based on the same inventive concept, the embodiment of the present application also provides a resource file management method based on the horizontal expansion storage of Fastdfs technology, referring to Figure 5 As shown, the method includes:

[0153] S301: Receive a resource file uploaded by a client, and read the interface path and business type of the resource file based on the file management microservice.

[0154] S302: According to the interface path and service type of the resource file, a corresponding storage group in Fastdfs is selected to write the resource file.

[0155] S303: Generate resource file metadata corresponding to the resource file and write it into a database.

[0156] S304: Based on the timing microservice, at every preset time period, each resource file to be archived is determined according to the creation time of the metadata of each resource file in the database.

[0157] S305: Based on the archiving microservice, each resource file to be archived in Fastdfs is sent to the preset network attached storage, and the identification status in the metadata of the corresponding resource file is modified to archived.

[0158] S306: According to the query condition input by the client, based on the file management microservice, determine the resource file metadata corresponding to the query condition in the database.

[0159] S307: Determine whether the status indicator in the metadata of the resource file is archived, if so, execute step S308, if not, execute step S309.

[0160] S308: calling an independent service through an http request to determine a resource file corresponding to the query condition in a preset network attached storage.

[0161] S309: Searching for a corresponding storage group in Fastdfs according to the business type in the metadata of the resource file, and determining a resource file corresponding to the query condition.

[0162] S310: Return the file byte stream of the resource file to the client.

[0163] In the embodiment of the present application, the specific implementation process of the above steps S301 to S305 can refer to the detailed description of resource file uploading and archiving in steps S101 to S106 in Example 1; the specific implementation process of the above steps S306 to S310 can refer to the detailed description of resource file query in steps S201 to S205 in Example 2. The repeated parts of the above processes will not be repeated here.

[0164] In the present application embodiment, Figure 6 As shown in the figure, it is a schematic diagram of the overall process of resource file uploading, archiving and querying. The arrow lines of the same type in the figure represent the process of the same work. The entire workflow includes the browser or APP application client, agent, file management microservice cluster (that is, the above-mentioned file management microservice), Fastdfs, database, archiving microservice, timing microservice and preset network attached storage.

[0165] Figure 6 The upload process of resource files in the file management microservice cluster includes uploading resource files from the client through the proxy to the file management microservice cluster, writing the resource files into the corresponding storage groups in the corresponding Fastdfs through the file management microservice cluster, and writing the generated resource file metadata into the database to complete the upload process;

[0166] Figure 6 The archiving process of resource files in the system includes: the scheduled microservice runs the metadata of resource files older than 93 days in the database at 3 a.m. every day through the file management microservice cluster, and changes their status to "to be archived"; the file management microservice cluster takes out the resource files to be archived in Fastdfs, actively calls the archive transmission interface to enter the archive microservice, and archives the resource files to be archived into the preset network attached storage nas through the archive microservice;

[0167] Figure 6 The query process of resource files in the file management microservice cluster includes the client sending query conditions through the proxy to enter the file management microservice cluster, judging the corresponding resource file metadata in the database through the file management microservice cluster, judging whether the corresponding resource file is archived, and if archived, calling the archive microservice in the independent service through an http request, determining the resource file corresponding to the query condition in the preset network attached storage, and returning its file byte stream to the client. If it is not archived, determining the Fastdfs and storage group storing the resource file through the file information in the resource file metadata, finding the resource file and returning the file byte stream of the resource file to the client to complete the resource file query.

[0168] Embodiment 4

[0169] Based on the same inventive concept, the embodiment of the present application also provides a resource file upload device based on the horizontal expansion storage of Fastdfs technology, referring to Figure 7 As shown, the device comprises:

[0170] The first receiving module 101 is used to receive the resource file uploaded by the client, and read the interface path and business type of the resource file based on the file management microservice;

[0171] The first writing module 102 is used to select a corresponding storage group in Fastdfs to write the resource file according to the interface path and business type of the resource file;

[0172] The second writing module 103 is used to generate resource file metadata corresponding to the resource file and write it into the database;

[0173] A first judgment module 104 is used to judge whether the resource file and the resource file metadata are both written successfully;

[0174] The first returning module 105 is configured to return file information to the client if both the resource file and the resource file metadata are written successfully.

[0175] Embodiment 5

[0176] Based on the same inventive concept, the embodiment of the present application also provides a resource file query device based on the horizontal expansion storage of Fastdfs technology, referring to Figure 8 As shown, the device comprises:

[0177] The first query module 201 is used to determine the resource file metadata corresponding to the query condition in the database based on the file management microservice according to the query condition input by the client;

[0178] The second judgment module 202 is used to judge whether the status mark in the metadata of the resource file is archived;

[0179] The first determination module 203 is used to call an independent service through an http request to determine the resource file corresponding to the query condition in the preset network attached storage if the status in the metadata of the resource file is archived;

[0180] The second determination module 204 is used to search for a corresponding storage group in Fastdfs according to the business type in the metadata of the resource file if the status identifier in the metadata of the resource file has not been archived, and determine the resource file corresponding to the query condition;

[0181] The second returning module 205 is used to return the file byte stream of the resource file to the client.

[0182] Embodiment 6

[0183] Based on the same inventive concept, the embodiment of the present application also provides a resource file management device based on the horizontal expansion storage of Fastdfs technology, referring to Fig. 9 As shown, the device comprises:

[0184] The second receiving module 301 is used to receive the resource file uploaded by the client, and read the interface path and business type of the resource file based on the file management microservice;

[0185] The third writing module 302 is used to select a corresponding storage group in Fastdfs to write the resource file according to the interface path and business type of the resource file;

[0186] The fourth writing module 303 is used to generate resource file metadata corresponding to the resource file and write it into the database;

[0187] The third determination module 304 is used to determine each resource file to be archived according to the creation time of each resource file metadata in the database at every preset time period based on the timing microservice;

[0188] The status modification module 305 is used to send each resource file to be archived in Fastdfs to a preset network attached storage based on the archiving microservice, and modify the identification status in the metadata of the corresponding resource file to archived;

[0189] The second query module 306 is used to determine the resource file metadata corresponding to the query condition in the database based on the file management microservice according to the query condition input by the client;

[0190] The third judgment module 307 is used to judge whether the status mark in the metadata of the resource file is archived;

[0191] The fourth determination module 308 is configured to, if the status in the metadata of the resource file is archived, call an independent service through an http request to determine the resource file corresponding to the query condition in the preset network attached storage;

[0192] The fifth determination module 309 is used to search for a corresponding storage group in Fastdfs according to the business type in the metadata of the resource file if the status in the metadata of the resource file is not archived, and determine the resource file corresponding to the query condition;

[0193] The third returning module 310 is used to return the file byte stream of the resource file to the client.

[0194] Embodiment 7

[0195] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a resource file storage method for horizontally expanding storage based on the Fastdfs technology as described in the above-mentioned embodiment one, and / or a resource file query method for horizontally expanding storage based on the Fastdfs technology as described in the above-mentioned embodiment two, and / or a resource file management method for horizontally expanding storage based on the Fastdfs technology as described in the above-mentioned embodiment three.

[0196] Embodiment 8

[0197] Based on the same inventive concept, an embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a resource file storage method for horizontally expanding storage based on the Fastdfs technology as described in the above embodiment one, and / or a resource file query method for horizontally expanding storage based on the Fastdfs technology as described in the above embodiment two, and / or a resource file management method for horizontally expanding storage based on the Fastdfs technology as described in the above embodiment three.

[0198] Embodiment 9

[0199] Based on the same inventive concept, an embodiment of the present application also provides a computer program product comprising instructions. When the computer program product is run on a computer device, the computer device executes the resource file storage method for horizontally expanding storage based on Fastdfs technology as described in the above embodiment one, and / or the resource file query method for horizontally expanding storage based on Fastdfs technology as described in the above embodiment two, and / or the resource file management method for horizontally expanding storage based on Fastdfs technology as described in the above embodiment three.

[0200] Embodiment 10

[0201] Based on the same inventive concept, an embodiment of the present application also provides a chip, the chip includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run a computer program or instruction to implement the resource file storage method based on the horizontal expansion storage of Fastdfs technology as described in the above embodiment one, and / or, the resource file query method based on the horizontal expansion storage of Fastdfs technology as described in the above embodiment two, and / or, the resource file management method based on the horizontal expansion storage of Fastdfs technology as described in the above embodiment three.

[0202] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0203] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0204] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0205] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0206] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A resource file storage method for horizontally expanding storage based on Fastdfs technology, characterized in that: include: Receive the resource file uploaded by the client, and read the interface path and business type of the resource file based on the file management microservice; According to the interface path and business type of the resource file, select the corresponding storage group in Fastdfs to write the resource file; Generate resource file metadata corresponding to the resource file and write it into the database; Determine whether the resource file and the resource file metadata are both written successfully; If yes, return the file information to the client.

2. The method according to claim 1, characterized in that Presetting the corresponding relationship between the interface path and each storage group in each Fastdfs; Presetting the corresponding relationship between a specific interface path and each Fastdfs; and, by presetting an identifier, presetting the corresponding relationship between each Fastdfs and the business type of the resource file; wherein the Fastdfs includes at least one storage group; The step of selecting a corresponding storage group in Fastdfs to write the resource file according to the interface path and the service type of the resource file includes: Determine whether the interface path of the resource file is a specific interface path; If yes, select Fastdfs corresponding to the specific interface path, and write the resource file into the storage group corresponding to the interface path; If not, determining whether the service type of the resource file is empty or does not carry a preset identifier; If yes, select the default Fastdfs and write the resource file into the storage group corresponding to the interface path; If not, select Fastdfs corresponding to the preset identifier, and write the resource file into the storage group corresponding to the interface path.

3. The method according to claim 1, characterized in that Also includes: Based on the timing microservice, each resource file to be archived is determined according to the creation time of each resource file metadata at every preset time period; Based on the archiving microservice, each resource file to be archived in Fastdfs is sent to the preset network attached storage.

4. The method according to claim 3, characterized in that The timing-based microservice determines each resource file to be archived according to the creation time of each resource file metadata in the database at every preset time period, including: Based on the timing microservice, at every preset time period, according to the time interval between the creation time of each resource file metadata in the database and the current time, determine whether the time interval is greater than the preset time interval; If so, the status flag of the corresponding resource file metadata is modified to be archived, and the resource file corresponding to the resource file metadata is determined as the resource file to be archived.

5. The method according to claim 4, characterized in that After sending each resource file to be archived in Fastdfs to the preset network attached storage, it also includes: Return the storage address of each resource file to be archived in the preset network attached storage to the file management microservice; Based on the file management microservice, the storage address is written into the corresponding resource file metadata of the database, and the status identifier of the resource file metadata is changed from to be archived to archived.

6. A resource file query method based on horizontally expanded storage of Fastdfs technology, characterized in that: include: According to the query conditions input by the client, based on the file management microservice, determine the resource file metadata corresponding to the query conditions in the database; Determine whether the status indicator in the metadata of the resource file is archived; If yes, calling the independent service through http request to determine the resource file corresponding to the query condition in the preset network attached storage; If not, search the corresponding storage group in Fastdfs according to the business type in the metadata of the resource file to determine the resource file corresponding to the query condition; The file byte stream of the resource file is returned to the client.

7. The method according to claim 6, characterized in that The step of determining, based on the file management service and according to the query condition input by the client, the resource file metadata corresponding to the query condition in the database includes: According to the query conditions input by the client, based on the file management microservice, determine whether there is resource file metadata corresponding to the query conditions in the database; If so, determining the resource file metadata; If not, a query failure message is returned to the client.

8. A resource file management method for horizontally expanding storage based on Fastdfs technology, characterized in that: include: Receive the resource file uploaded by the client, and read the interface path and business type of the resource file based on the file management microservice; According to the interface path and business type of the resource file, select the corresponding storage group in Fastdfs to write the resource file; Generate resource file metadata corresponding to the resource file and write it into the database; Based on the timing microservice, each resource file to be archived is determined according to the creation time of the metadata of each resource file in the database at every preset time period; Based on the archiving microservice, each resource file to be archived in Fastdfs is sent to the preset network attached storage, and the identification status in the metadata of the corresponding resource file is modified to archived; According to the query conditions input by the client, based on the file management microservice, determine the resource file metadata corresponding to the query conditions in the database; Determine whether the status indicator in the metadata of the resource file is archived; If yes, calling the independent service through http request to determine the resource file corresponding to the query condition in the preset network attached storage; If not, search the corresponding storage group in Fastdfs according to the business type in the metadata of the resource file to determine the resource file corresponding to the query condition; The file byte stream of the resource file is returned to the client.

9. A resource file uploading device for horizontally expanding storage based on Fastdfs technology, characterized in that: include: The first receiving module is used to receive the resource file uploaded by the client and read the interface path and business type of the resource file based on the file management microservice; A first writing module, used to select a corresponding storage group in Fastdfs to write the resource file according to the interface path and business type of the resource file; A second writing module is used to generate resource file metadata corresponding to the resource file and write it into a database; A first judgment module, used to judge whether the resource file and the resource file metadata are both written successfully; The first returning module is used to return the file information to the client if both the resource file and the resource file metadata are written successfully.

10. A resource file query device based on Fastdfs technology for horizontally expanding storage, characterized in that: include: The first query module is used to determine the resource file metadata corresponding to the query condition in the database based on the file management microservice according to the query condition input by the client; A second judgment module is used to judge whether the status mark in the metadata of the resource file is archived; A first determination module is used to call an independent service through an http request to determine a resource file corresponding to the query condition in a preset network attached storage if the status in the metadata of the resource file is archived; A second determination module is used to search for a corresponding storage group in Fastdfs according to the business type in the metadata of the resource file if the status identifier in the metadata of the resource file has not been archived, and determine the resource file corresponding to the query condition; The second returning module is used to return the file byte stream of the resource file to the client.

11. A resource file management device for horizontally expanding storage based on Fastdfs technology, characterized in that: include: The second receiving module is used to receive the resource file uploaded by the client and read the interface path and business type of the resource file based on the file management microservice; The third writing module is used to select a corresponding storage group in Fastdfs to write the resource file according to the interface path and business type of the resource file; A fourth writing module, used to generate resource file metadata corresponding to the resource file and write it into a database; The third determination module is used to determine each resource file to be archived according to the creation time of each resource file metadata in the database at every preset time period based on the timing microservice; The status modification module is used to send each resource file to be archived in Fastdfs to the preset network attached storage based on the archiving microservice, and modify the identification status in the metadata of the corresponding resource file to archived; The second query module is used to determine the resource file metadata corresponding to the query condition in the database based on the query condition input by the client and the file management microservice; A third judgment module is used to judge whether the status mark in the metadata of the resource file is archived; A fourth determination module is used to call an independent service through an http request to determine a resource file corresponding to the query condition in a preset network attached storage if the status in the metadata of the resource file is archived; A fifth determination module, for searching a corresponding storage group in Fastdfs according to a business type in the metadata of the resource file if the status in the metadata of the resource file is not archived, and determining a resource file corresponding to the query condition; The third returning module is used to return the file byte stream of the resource file to the client.

12. A computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal, the terminal executes a resource file storage method based on the Fastdfs technology for horizontally scalable storage as described in any one of claims 1 to 5, and / or a resource file query method based on the Fastdfs technology for horizontally scalable storage as described in any one of claims 6 to 7, and / or a resource file management method based on the Fastdfs technology for horizontally scalable storage as described in claim 8.

13. A computer device, characterized in that: It includes a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements a resource file storage method based on the Fastdfs technology for horizontally scalable storage as described in any one of claims 1 to 5, and / or a resource file query method based on the Fastdfs technology for horizontally scalable storage as described in any one of claims 6 to 7, and / or a resource file management method based on the Fastdfs technology for horizontally scalable storage as described in claim 8.

14. A computer program product comprising instructions, which, when executed on a computer device, enables the computer device to execute a resource file storage method based on the Fastdfs technology for horizontally scalable storage as described in any one of claims 1 to 5, and / or a resource file query method based on the Fastdfs technology for horizontally scalable storage as described in any one of claims 6 to 7, and / or a resource file management method based on the Fastdfs technology for horizontally scalable storage as described in claim 8.

Images