A method, device, electronic device and medium for generating a list file

By scanning and generating manifest files on the cloud server, the problem of inefficient acquisition of manifest files in public cloud object storage is solved, and the method of directly obtaining manifest files from the target storage space is realized, which improves efficiency.

CN113010478BActive Publication Date: 2024-11-01BEIJING KINGSOFT CLOUD NETWORK TECH CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202110276066.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-03-15
Publication Date
2024-11-01
Estimated Expiration
2041-03-15

AI Technical Summary

Technical Problem

In public cloud object storage, obtaining manifest files is less efficient, mainly because the file list information involves a large number of files, and it takes a long time to obtain the metadata of each file one by one.

Method used

By scanning the file metadata information within the preset scan range in the source storage space on the cloud server, obtaining the contents of the preset inventory field, generating a manifest file, and storing it in the target storage space, including generating a description file to record the scanning time Stamp, field, file name, file size, check value and other information.

Benefits of technology

It improves the efficiency of obtaining manifest files, allowing users to directly obtain manifest files from the target storage space, without having to obtain metadata one by one through the head interface, reducing the need for manual sorting of files.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113010478B_ABST
    Figure CN113010478B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention provides a method, device, electronic device and medium for generating a manifest file, and relates to the field of cloud storage technology. The method comprises: scanning metadata information of files within a preset scanning range in a source storage space, obtaining field content corresponding to a preset manifest field of each file from the metadata information of the scanned files, generating a manifest file with the field content corresponding to the preset manifest field of the obtained files, and then storing the manifest file in a target storage space. The method can improve the efficiency of obtaining manifest files.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of cloud storage technology, and in particular to a method, device, electronic device and medium for generating a list file. Background Art

[0002] Public cloud object storage is a distributed data storage method with massive data, unlimited expansion and low storage cost. It realizes the reading and writing of files in public cloud object storage based on Hypertext Transfer Protocol (HTTP), and can well realize cross-platform data sharing.

[0003] Currently, most big data platforms support public cloud object storage as a data source and use public cloud object storage for big data backup. The tens of thousands of files in public cloud object storage make it difficult for users to count and analyze files. Currently, users can obtain file list information through the public cloud file list interface, and then obtain the metadata of each file in the file list information one by one through the head interface, thereby generating a list file including the metadata information of each file. Because the file list information involves a large number of files, it takes a long time to obtain the metadata of each file one by one, resulting in low efficiency for users to obtain list files. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a method, device, electronic device and medium for generating a list file, so as to improve the efficiency of obtaining the list file. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for generating a manifest file, the method being applied to a cloud server, the method comprising:

[0006] Scan metadata information of files within a preset scanning range in the source storage space;

[0007] Obtaining field content corresponding to a preset list field of each file from metadata information of the scanned file;

[0008] Generate a manifest file using the field contents corresponding to the preset manifest fields of the acquired file;

[0009] The manifest file is stored in the target storage space.

[0010] In a possible implementation, after generating a manifest file from the manifest contents corresponding to the preset manifest fields of all the acquired files, the method further includes:

[0011] Generate a description file of the manifest file, the description file including any one or more of a timestamp of starting to scan the source storage space, fields included in the manifest file, a file name of the manifest file, a file size of the manifest file, a checksum of the manifest file, an identifier of the source storage space, an identifier of the target storage space, and a manifest version;

[0012] The description file of the manifest file is stored in the target storage space.

[0013] In a possible implementation, before generating the description file of the manifest file, the method further includes:

[0014] The file name of the manifest file is generated according to the timestamp of the scanning end time.

[0015] In a possible implementation, the metadata information within a preset scanning range in the scanning source storage space includes:

[0016] Periodically scanning metadata information within a preset scanning range in the source storage space with a preset scanning time interval as a scanning period;

[0017] The step of generating a manifest file from the field contents corresponding to the preset manifest fields of the acquired file includes:

[0018] Generate a manifest file based on the field contents corresponding to the preset manifest fields of the files acquired in the current scanning cycle.

[0019] In a possible implementation, the metadata information of the files within the preset scanning range in the scanning source storage space includes:

[0020] If the preset scanning range is the source storage space, scanning metadata information of all files in the source storage space;

[0021] If the preset scanning range is files with a specified prefix in the source storage space, metadata information of files with the specified prefix in the source storage space is scanned.

[0022] In a possible implementation, the method further includes:

[0023] Receive the source storage space, preset scanning range, target storage space and preset scanning time interval set by the user;

[0024] The configurable list fields are output, and in response to a user's selection operation on the configurable list fields, the configurable list fields selected by the user and the default list fields are used as the preset list fields.

[0025] In a second aspect, an embodiment of the present application provides a device for generating a manifest file, the device being applied to a cloud server, the device comprising:

[0026] A scanning module, used to scan metadata information of files within a preset scanning range in the source storage space;

[0027] An acquisition module, used to acquire field content corresponding to a preset list field of each file from metadata information of the scanned file;

[0028] A generation module, used to generate a manifest file from the field contents corresponding to the preset manifest fields of the acquired file;

[0029] The storage module is used to store the list file in the target storage space.

[0030] In a possible implementation, the generating module is further used to generate a description file of the manifest file, the description file including any one or more of a timestamp for starting to scan the source storage space, fields included in the manifest file, a file name of the manifest file, a file size of the manifest file, a checksum of the manifest file, an identifier of the source storage space, an identifier of the target storage space, and a manifest version;

[0031] The storage module is further used to store the description file of the list file in the target storage space.

[0032] In a possible implementation, the generating module is further configured to generate a file name of the manifest file according to a timestamp of a scanning end time.

[0033] In a possible implementation, the scanning module is specifically used to:

[0034] Periodically scanning metadata information within a preset scanning range in the source storage space with a preset scanning time interval as a scanning period;

[0035] The step of generating a manifest file from the field contents corresponding to the preset manifest fields of the acquired file includes:

[0036] Generate a manifest file based on the field contents corresponding to the preset manifest fields of the files acquired in the current scanning cycle.

[0037] In a possible implementation, the scanning module is specifically used to:

[0038] If the preset scanning range is the source storage space, scanning metadata information of all files in the source storage space;

[0039] If the preset scanning range is files with a specified prefix in the source storage space, metadata information of files with the specified prefix in the source storage space is scanned.

[0040] In a possible implementation manner, the device further includes:

[0041] A receiving module, used to receive a source storage space, a preset scanning range, a target storage space and a preset scanning time interval set by a user;

[0042] The setting module is used to output the configurable list fields, and in response to the user's selection operation on the configurable list fields, use the configurable list fields and default list fields selected by the user as the preset list fields.

[0043] In a third aspect, an embodiment of the present invention further provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0044] Memory, used to store computer programs;

[0045] The processor is used to implement any of the above-mentioned steps of the method for generating the manifest file when executing the program stored in the memory.

[0046] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method for generating a manifest file described in the first aspect is implemented.

[0047] In a fifth aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method for generating a manifest file described in the first aspect above.

[0048] By adopting the above technical solution, the cloud server can scan the metadata information of files within the preset scanning range in the source storage space, and obtain the field content corresponding to the preset list field of each file from the scanned source data information, and then generate a list file with the field content corresponding to the preset list field of the obtained file, and store the list file in the target storage space, so that users can directly obtain the list file from the target storage space without having to obtain the metadata of each file one by one through the head interface, and without manually organizing the files, which can improve the efficiency of obtaining the list file.

[0049] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A flowchart of a method for generating a list file provided in an embodiment of the present application;

[0052] Figure 2 A flowchart of another method for generating a list file provided in an embodiment of the present application;

[0053] Figure 3 An exemplary schematic diagram of a list file provided in an embodiment of the present application;

[0054] Figure 4 A schematic diagram of the structure of a device for generating a list file provided in an embodiment of the present application;

[0055] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] In order to improve the efficiency of obtaining the list file, the embodiment of the present application provides a method for generating a list file, which is applied to a cloud server, such as Figure 1 As shown, the method includes:

[0058] S101, scanning metadata information of files within a preset scanning range in a source storage space.

[0059] The source storage space may be a portion of storage space in a public cloud, for example, a storage space that a user who uses a public cloud object storage service has permission to use.

[0060] S102: Obtain field content corresponding to a preset list field of each file from metadata information of the scanned file.

[0061] The preset list fields can be set according to actual needs. The metadata information of the file includes the field contents of all fields of the file. After scanning the metadata information of the file, the field contents corresponding to the preset list fields can be filtered out.

[0062] For example, if the metadata information of a file includes the field contents of 100 fields, and the number of preset list fields is 20, the field contents corresponding to the 20 preset list fields of each file can be obtained from the source data information of the file.

[0063] S103: Generate a manifest file using the field content corresponding to the preset manifest field of the acquired file.

[0064] As an example, the manifest file may be in a comma-separated values ​​(CSV) format, and the manifest file includes field contents corresponding to preset manifest fields of the acquired file.

[0065] S104: Store the list file in the target storage space.

[0066] It is understandable that after the manifest file is stored in the target storage space, the user can obtain the manifest file from the target storage space as needed.

[0067] By adopting the above technical solution, the cloud server can scan the metadata information of files within the preset scanning range in the source storage space, and obtain the field content corresponding to the preset list field of each file from the scanned source data information, and then generate a list file with the field content corresponding to the preset list field of the obtained file, and store the list file in the target storage space, so that users can directly obtain the list file from the target storage space without having to obtain the metadata of each file one by one through the head interface, and without manually organizing the files, which can improve the efficiency of obtaining the list file.

[0068] In the embodiments of the present application, Figure 2 As shown, after generating a manifest file by obtaining the field content corresponding to the preset manifest field of the file in S103, the method further includes:

[0069] S105: Generate a description file of the manifest file.

[0070] The description file includes any one or more of the timestamp of starting to scan the source storage space, the fields included in the manifest file, the file name of the manifest file, the file size of the manifest file, the checksum of the manifest file, the identifier of the source storage space, the identifier of the target storage space, and the manifest version.

[0071] Optionally, the description file of the manifest file may specifically be a manifest file, and the manifest file includes manifest.json and manifest.checksum, wherein manifest.checksum is a checksum of the manifest file.

[0072] manifest.json may include the timestamp of starting to scan the source storage space, the fields included in the manifest file, the file name of the manifest file, the file size of the manifest file, the identifier of the source storage space, the identifier of the target storage space, and the manifest version.

[0073] The manifest.json specifically includes the following fields:

[0074] creationTimestamp: the timestamp of starting to scan the source storage space, which is the timestamp created in the epoch date format;

[0075] destination storage space: the identifier of the target storage space;

[0076] fileSchema: the fields contained in the manifest file;

[0077] files: contains the file name, file size and checksum of the manifest file. The checksum can be a message digest algorithm (MD5) checksum.

[0078] Source storage space: the identifier of the source storage space;

[0079] version: manifest version.

[0080] S106: Store the description file of the manifest file in the target storage space.

[0081] By adopting this method, after generating the manifest file, the cloud server can also generate a description file of the manifest file, and store the description file of the manifest file in the target storage space, so that the user can search for the information of the required manifest file through the description file of the manifest file, and then quickly obtain the required manifest file, further improving the efficiency of obtaining the manifest file.

[0082] In one embodiment of the present application, the cloud server may periodically generate a list file, that is, the metadata information of the files within a preset scanning range in the above S101 scanning source storage space, which may be implemented as follows:

[0083] The preset scanning time interval is used as the scanning period, and the metadata information within the preset scanning range in the source storage space is periodically scanned.

[0084] The preset scanning time interval may be set based on actual needs, for example, may be set to one week, or may be set to other time periods.

[0085] Taking the preset scanning time interval as one week as an example, the cloud server can start scanning the metadata information within the preset scanning range in the source storage space at the same time every week, such as 8 am every Monday.

[0086] Accordingly, the above S103, generating a manifest file from the field contents corresponding to the preset manifest fields of the acquired file, can be implemented as follows:

[0087] Generate a manifest file based on the field contents corresponding to the preset manifest fields of the files acquired in the current scanning cycle.

[0088] If the preset time interval is one week, and the scan starts at 8:00 a.m. on Monday every week, then the current scan period is from 8:00 a.m. on Monday this week to 8:00 a.m. on Monday next week. In other words, the cloud server can generate a list file through the source storage space every week.

[0089] After generating the manifest file and before generating the description file of the manifest file, the cloud server may also generate a file name of the manifest file according to the timestamp of the scanning end time.

[0090] In one implementation, the cloud server may use the timestamp of the scan end time as a part of the file name of the manifest file.

[0091] If the cloud server generates a list file for the source storage space every week, these list files all include the scan end time, and then the list files can be stored in the target storage space in order from the earliest to the latest timestamps included in the list files, so that users can view them easily.

[0092] By adopting this method, the metadata information of files within a preset scanning range in the source storage space is periodically scanned, and a list file is generated based on the content scanned from the source storage space in each scanning cycle, thereby realizing automatic organization of the list files and facilitating users to understand the file status in different scanning cycles.

[0093] In another embodiment of the present invention, in order to generate a manifest file, the rules for generating the manifest file need to be set in advance. The cloud server can receive the source storage space, preset scanning range, target storage space and preset scanning time interval set by the user; and output configurable manifest fields, and in response to the user's selection operation on the configurable manifest fields, use the configurable manifest fields and default manifest fields selected by the user as preset manifest fields.

[0094] Optionally, after the user logs in to the cloud server through an account, the object storage console of the cloud server can display the storage space information that the account has permission to use, and the user can select the source storage space and the target storage space as needed, and set the preset scanning range and preset scanning time interval in the source storage space. The source storage space can be the storage bucket space in the cloud server, and the preset scanning range can be the entire source storage space, or files with a specified prefix in the source storage space.

[0095] In addition, the object storage console of the cloud server can output and display configurable inventory fields and default inventory fields. The default inventory fields are fields that need to be scanned by default, and the user does not need to select them. Configurable inventory fields are fields that the user can choose whether to scan. The cloud server can recognize the user's selection operation on the configurable inventory fields, and then use the configurable inventory fields and default inventory fields selected by the user as the above-mentioned preset inventory fields.

[0096] As an example, the default list fields and the configurable list fields are specifically shown in Table 1, and the optional fields in "Whether to support optional configuration" in Table 1 are configurable list fields.

[0097] Table 1

[0098]

[0099]

[0100] As an example, if the preset list fields set by the user include fields except the last field in Table 1, the list file obtained by scanning is as follows: Figure 3 shown.

[0101] Figure 3 Each column in corresponds to a field in Table 1, for example, Figure 3 Column A is the name of the source storage space where the inventory task is executed, column B is the name of the file in the storage space, column C is the version ID of the file, column D is the file size, column E is the last modification time of the file, column F is the Etag created when the file is generated, column G is the storage type of the file, column H is used to mark whether the file is generated through shard upload, column I is used to indicate whether the file is encrypted, and column J is used to indicate whether the file is the latest version.

[0102] It can be seen that by using the above method, the user sets the scanning range, scanning time interval and the preset list fields to be scanned, and the cloud server can automatically scan the files within the preset scanning range in the source storage space and automatically generate a list file. There is no need for the user to manually organize the list files, which can improve the efficiency of the user in obtaining the list files.

[0103] It can be known from the above embodiments that the preset scanning range may be the entire source storage space or files with a specified prefix in the source storage space. Accordingly, the above S101, scanning metadata information of files within the preset scanning range in the source storage space, may include the following two implementation methods:

[0104] Method 1: If the preset scanning range is the source storage space, the metadata information of all files in the source storage space is scanned.

[0105] Method 2: If the preset scanning range is files with a specified prefix in the source storage space, the metadata information of the files with the specified prefix in the source storage space is scanned.

[0106] For example, if the specified prefix is ​​"123", it means that the files corresponding to the file names starting with "123" in the storage space need to be scanned, such as: 123.txt, 12345.mp4, 123tec.ts, 123 / qaz / wsx.jpg, etc.

[0107] In an embodiment of the present application, in order to improve scanning efficiency, if the number of files within a preset scanning range in the source storage space is greater than a preset number threshold, multiple threads are used to scan metadata information of files with different prefixes within the preset scanning range.

[0108] Optionally, as an example, the preset quantity threshold may be 1 billion.

[0109] After using multiple threads to scan the metadata information of files with different prefixes within the preset scanning range, the scanning results of each thread can be spliced ​​into a list file. Then, the list file and the description file of the list file are transferred to the target storage space in the form of object storage, so that the list file and the description file of the list file can be persistently stored on the disk, that is, the list file and the description file of the list file are written to the hard disk of the cloud server. If there is a need for auditing in the future, it is convenient for users to obtain the list file.

[0110] Corresponding to the above method embodiment, the present application embodiment also provides a device for generating a manifest file, such as Figure 4 As shown, the device is applied to a cloud server, and the device includes:

[0111] Scanning module 401, used to scan metadata information of files within a preset scanning range in the source storage space;

[0112] The acquisition module 402 is used to acquire the field content corresponding to the preset list field of each file from the metadata information of the scanned file;

[0113] A generating module 403 is used to generate a manifest file from the field content corresponding to the preset manifest field of the acquired file;

[0114] The storage module 404 is used to store the list file in the target storage space.

[0115] Optionally, the generating module 403 is further used to generate a description file of the manifest file, the description file including any one or more of a timestamp of starting to scan the source storage space, fields included in the manifest file, a file name of the manifest file, a file size of the manifest file, a checksum of the manifest file, an identifier of the source storage space, an identifier of the target storage space, and a manifest version;

[0116] The storage module 404 is further configured to store the description file of the manifest file in the target storage space.

[0117] Optionally, the generating module 403 is further configured to generate a file name of the list file according to a timestamp of the scanning end time.

[0118] Optionally, the scanning module 401 is specifically used for:

[0119] The metadata information within the preset scanning range in the source storage space is periodically scanned with a preset scanning time interval as the scanning period;

[0120] Generate a manifest file with the field contents corresponding to the preset manifest fields of the acquired file, including:

[0121] Generate a manifest file based on the field contents corresponding to the preset manifest fields of the files acquired in the current scanning cycle.

[0122] Optionally, the scanning module 401 is specifically used for:

[0123] If the preset scanning scope is the source storage space, the metadata information of all files in the source storage space is scanned;

[0124] If the preset scanning scope is files with a specified prefix in the source storage space, metadata information of files with the specified prefix in the source storage space is scanned.

[0125] Optionally, the device further comprises:

[0126] A receiving module, used to receive a source storage space, a preset scanning range, a target storage space and a preset scanning time interval set by a user;

[0127] The setting module is used to output the configurable list fields, and in response to the user's selection operation on the configurable list fields, use the configurable list fields and the default list fields selected by the user as preset list fields.

[0128] By adopting the above technical solution, the cloud server can scan the metadata information of files within the preset scanning range in the source storage space, and obtain the field content corresponding to the preset list field of each file from the scanned source data information, and then generate a list file with the field content corresponding to the preset list field of the obtained file, and store the list file in the target storage space, so that users can directly obtain the list file from the target storage space without having to obtain the metadata of each file one by one through the head interface, and without manually organizing the files, which can improve the efficiency of obtaining the list file.

[0129] The embodiment of the present invention further provides an electronic device, such as Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0130] Memory 503, used for storing computer programs;

[0131] The processor 501 is used to implement the method steps in the above method embodiment when executing the program stored in the memory 503.

[0132] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0133] The communication interface is used for communication between the above electronic device and other devices.

[0134] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0135] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0136] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods for generating a manifest file are implemented.

[0137] In another embodiment of the present invention, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to execute any method for generating a manifest file in the above embodiments.

[0138] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0139] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0140] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0141] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for generating a manifest file, characterized in that: The method is applied to a cloud server, and the method comprises: Scan metadata information of files within a preset scanning range in the source storage space; Obtaining field content corresponding to a preset list field of each file from metadata information of the scanned file; Generate a manifest file using the field contents corresponding to the preset manifest fields of the acquired file; Storing the manifest file in a target storage space; The metadata information of the files within the preset scanning range in the scanning source storage space includes: If the number of files within the preset scanning range in the source storage space is greater than a preset number threshold, multiple threads are used to respectively scan metadata information of files with different prefixes within the preset scanning range; The step of generating a manifest file from the field contents corresponding to the preset manifest fields of the acquired file includes: The field contents corresponding to the preset list fields of the files scanned by each thread are concatenated into a list file; Before generating the description file of the manifest file, the method further includes: The file name of the manifest file is generated according to the timestamp of the scanning end time.

2. The method according to claim 1, characterized in that: After generating a manifest file from the manifest contents corresponding to the preset manifest fields of all the acquired files, the method further includes: Generate a description file of the manifest file, the description file including any one or more of a timestamp of starting to scan the source storage space, fields included in the manifest file, a file name of the manifest file, a file size of the manifest file, a checksum of the manifest file, an identifier of the source storage space, an identifier of the target storage space, and a manifest version; The description file of the manifest file is stored in the target storage space.

3. The method according to claim 1 or 2, characterized in that: The metadata information within the preset scanning range in the scanning source storage space includes: Periodically scanning metadata information within a preset scanning range in the source storage space with a preset scanning time interval as a scanning period; The step of generating a manifest file from the field contents corresponding to the preset manifest fields of the acquired file includes: Generate a manifest file based on the field contents corresponding to the preset manifest fields of the files acquired in the current scanning cycle.

4. The method according to claim 1, characterized in that: The metadata information of the files within the preset scanning range in the scanning source storage space includes: If the preset scanning range is the source storage space, scanning metadata information of all files in the source storage space; If the preset scanning range is files with a specified prefix in the source storage space, metadata information of files with the specified prefix in the source storage space is scanned.

5. The method according to claim 3, characterized in that: The method further comprises: Receive the source storage space, preset scanning range, target storage space and preset scanning time interval set by the user; The configurable list fields are output, and in response to a user's selection operation on the configurable list fields, the configurable list fields selected by the user and the default list fields are used as the preset list fields.

6. A device for generating a manifest file, characterized in that: The device is applied to a cloud server, and the device includes: A scanning module, used to scan metadata information of files within a preset scanning range in the source storage space; An acquisition module, used to acquire field content corresponding to a preset list field of each file from metadata information of the scanned file; A generation module, used to generate a manifest file from the field contents corresponding to the preset manifest fields of the acquired file; A storage module, used for storing the list file in a target storage space; The scanning module is specifically configured to respectively scan metadata information of files with different prefixes within the preset scanning range using multiple threads if the number of files within the preset scanning range in the source storage space is greater than a preset number threshold; The generating module is specifically used to splice the field contents corresponding to the preset list fields of the files scanned by each thread into a list file; The generating module is further used to generate a file name of the list file according to the timestamp of the scanning end time.

7. The device according to claim 6, characterized in that The generating module is further used to generate a description file of the manifest file, wherein the description file includes any one or more of a timestamp for starting to scan the source storage space, fields included in the manifest file, a file name of the manifest file, a file size of the manifest file, a checksum of the manifest file, an identifier of the source storage space, an identifier of the target storage space, and a manifest version; The storage module is further used to store the description file of the list file in the target storage space.

8. The device according to claim 6 or 7, characterized in that The scanning module is specifically used for: Periodically scanning metadata information within a preset scanning range in the source storage space with a preset scanning time interval as a scanning period; The step of generating a manifest file from the field contents corresponding to the preset manifest fields of the acquired file includes: Generate a manifest file based on the field contents corresponding to the preset manifest fields of the files acquired in the current scanning cycle.

9. The device according to claim 6, characterized in that The scanning module is specifically used for: If the preset scanning range is the source storage space, scanning metadata information of all files in the source storage space; If the preset scanning range is files with a specified prefix in the source storage space, metadata information of files with the specified prefix in the source storage space is scanned.

10. The device according to claim 8, characterized in that The device also includes: A receiving module, used to receive a source storage space, a preset scanning range, a target storage space and a preset scanning time interval set by a user; The setting module is used to output the configurable list fields, and in response to the user's selection operation on the configurable list fields, use the configurable list fields and default list fields selected by the user as the preset list fields.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-5 when executing a program stored in a memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Citation Information

Patent Citations

  • Multi-service form file processing system and method

    CN101552976A

  • Data storage method and device based on distributed objects

    CN111104404A

  • Supply chain intelligent image comparison system and method

    CN111160091A

  • JAVA-based project source code scanning analysis method and system, and storage medium

    CN112416787A

  • Data processing method and device, storage medium and electronic equipment

    CN112464255A