File management method and device based on Redis

Abstract

The invention belongs to the technical field of information application, and particularly discloses a file management method and device based on Redis, and the method comprises the steps that a file is published, a subsystem uploads the file and stores the file in a Redis database in a slicing mode, and file information is published to a channel; message subscription and index maintenance: receiving file information by a sub-system subscription channel, and updating a local file index table; and file asking and checking: asking the file blocks by the subsystem according to the file index table, and integrating and checking the file integrity. A publishing and subscribing mechanism of the Redis database serves as a core, service nodes do not need to be newly added, use of resources is reduced, the method is a lightweight file management mode, file consistency of subsystems under a distributed architecture is guaranteed, file blocks are cached through the Redis database, a plurality of systems are allowed to acquire and upload a plurality of files at the same time, and the file management efficiency is improved. The file transmission efficiency is guaranteed, and the file exclusive problem is solved.

Description

Technical Field

[0001] The present invention relates to the field of information application technology, and in particular to a Redis-based file management method and device. Background Art

[0002] With the rapid development of the Internet, the demand for distributed systems is increasing. In distributed systems, it is sometimes inevitable to share some files directly between subsystems in order to collaborate. However, each subsystem has its own memory and processing power. How to transfer files between systems is a difficult point.

[0003] Traditional file management methods, such as FTP and SMB, will encounter performance bottlenecks and poor scalability in a distributed system architecture. Building a distributed file system to manage files requires multiple servers to ensure file reliability and availability, which is too heavyweight for some smaller distributed systems.

[0004] In view of this, the present invention proposes a file management method and device based on Redis. Summary of the invention

[0005] The purpose of the present invention is to provide a file management method and device based on Redis, to implement a file management method in a distributed system, and to ensure the consistency, reliability and scalability of files.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a file management device based on Redis, wherein the distributed system includes multiple subsystems and a Redis database, and the Redis database includes a first module, a second module, and a third module;

[0008] In the first module, the subsystem uploads files and stores slices in the Redis database, and publishes file information to the channel;

[0009] In the second module, the subsystem subscribes to the channel to receive file information and updates the local file index table. The file information includes the file name, the number of file blocks, the file size and the file MD5 code;

[0010] In the third module, the subsystem retrieves file blocks according to the file index table, integrates and verifies the file integrity.

[0011] As a preferred technical solution of the first aspect of the present invention, the first module includes a file uploading unit and a channel publishing unit;

[0012] File upload unit: each subsystem performs file upload operations and saves them to the Redis database.

[0013] Channel publishing unit, after uploading the file, each subsystem publishes the file information in its own channel.

[0014] As a preferred technical solution of the first aspect of the present invention, during the file upload operation, the file is sliced ​​to determine the file bytes, with the file name and file index number as the primary key and the file byte stream as the value.

[0015] As a preferred technical solution of the first aspect of the present invention, the file name and the number of file blocks can be used as primary keys to search for file content; the file size and the file MD5 code are used by the subscriber to verify whether the file is correct.

[0016] As a preferred technical solution of the first aspect of the present invention, the second module includes a message subscription unit and an index maintenance unit;

[0017] Message subscription unit: each subsystem subscribes to the corresponding subsystem channel according to its own business needs. The number of subsystem channels includes but is not limited to one. After the new file information is released in the corresponding subsystem channel, the file index table maintenance work is carried out in the first time;

[0018] The index maintenance unit parses and obtains new file information and saves the file information into its own file index table.

[0019] As a preferred technical solution of the first aspect of the present invention, the third module includes a file retrieval unit and a file verification unit;

[0020] File retrieval unit, each subsystem searches its own file index table when using the corresponding file;

[0021] The file verification unit calculates the size of the complete file and the file MD5 code, and compares them with the values ​​recorded in the file index table. If they are inconsistent, the file retrieval is considered to have failed. The block number and block size recorded in the file retrieval unit are verified, the problem block is found by comparison, and it is retrieved again. If they are consistent, the entire file transfer process is completed.

[0022] As a preferred technical solution of the first aspect of the present invention, the logic for obtaining the complete file is:

[0023] According to the content in the file index table, the file name + file sequence number is used as the primary key to obtain file blocks from the database in sequence. Each time a block is obtained, the block number and block size are recorded. Finally, the file blocks are integrated in the system to form a complete file.

[0024] In a second aspect, the present invention provides a file management method based on Redis, based on the implementation of the first aspect, comprising the following steps:

[0025] File publishing: The subsystem uploads files and stores them in slices in the Redis database, and publishes file information to the channel;

[0026] Message subscription and index maintenance: the subsystem subscribes to the channel to receive file information and updates the local file index table;

[0027] File retrieval and verification,the subsystem retrieves file blocks according to the file index table,,integrates and verifies the file integrity.

[0028] In a third aspect, the present invention provides an electronic device comprising: a processor and a memory, wherein the memory stores a computer program that can be called by the processor;

[0029] The processor executes the first aspect by calling a computer program stored in the memory.

[0030] In a fourth aspect, the present invention provides a computer-readable storage medium storing instructions, which, when executed on a computer, enable the computer to execute the first aspect.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The present invention takes the publish-subscribe mechanism of the Redis database as the core, does not require new service nodes, reduces resource usage, is a lightweight file management method, ensures file consistency of each subsystem under a distributed architecture, and uses the Redis database to cache file blocks, allowing multiple systems to retrieve and upload multiple files at the same time, thereby ensuring the efficiency of file transmission and solving the problem of file exclusivity. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a structural framework diagram of a file management device based on Redis of the present invention;

[0034] Figure 2 It is a flow chart of the file management method based on Redis of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] In the description of the present invention, it should be noted that the terms "vertical", "up", "down", "horizontal", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0037] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0038] In order to reduce the use of server resources, the present invention proposes to use the existing Redis database and the publish-subscribe function of the Redis database to implement a file management method under a distributed system to ensure the consistency, reliability and scalability of the files. The following part of the existing content is used to assist in understanding the patent application concept and technical solution of the present invention application, which does not necessarily belong to the prior art of the present invention application. In the absence of clear evidence that the above content has been disclosed on the filing date of the present invention application, the above background technology should not be used to evaluate the novelty and creativity of the present application.

[0039] 1. Distributed Architecture

[0040] Distributed architecture is a computer system design in which multiple components are distributed across different computers that communicate and collaborate over a network. The goal of this architecture is to improve the performance, reliability, and scalability of the system. In a distributed architecture, each component (also called a node) can operate independently, which makes the system more flexible and reliable. If a node fails, the other nodes can still continue to work, which improves the reliability of the system. In addition, since each node can operate independently, the performance and scalability of the system are also improved.

[0041] 2. Redis database

[0042] Redis database is an open source in-memory database that can be used as a database, cache, and message middleware. It supports multiple data structures, such as strings, hash tables, lists, sets, ordered sets, etc. The features of Redis database include:

[0043] Fast speed: The data of the Redis database is stored in memory, so the reading and writing speed is very fast.

[0044] Support for a variety of data structures: Redis database supports a variety of data structures, such as strings, hash tables, lists, sets, ordered sets, etc.

[0045] Persistence: The Redis database supports data persistence and can save data to disk to prevent data loss.

[0046] High availability: The Redis database supports master-slave replication and sentinel mechanisms, which can improve the availability of the system.

[0047] In a distributed system, the Redis database can be used as a cache to improve system performance. At the same time, the publish-subscribe mechanism of the Redis database can be used to implement message publishing and subscription, which is used to decouple the various components of the system.

[0048] 3. File Slicing

[0049] File slicing is a technique that divides a large file into multiple smaller parts. This technique is often used to handle large files so that they can be stored, transferred, and processed more efficiently. Each file slice usually has a fixed size, which makes it easier to manage and manipulate these files.

[0050] Example 1

[0051] like Figure 1 As shown, this embodiment provides a file management device based on Redis, the distributed system includes multiple subsystems and a Redis database, the Redis database includes a first module, a second module and a third module, and the modules are connected by wire or wirelessly.

[0052] In the first module, the publisher uploads and uploads file information in the corresponding subsystem file;

[0053] Specifically, the first module includes a file uploading unit and a channel publishing unit;

[0054] In the file upload unit, each subsystem performs file upload operations. When uploading, the file is sliced ​​into 1024 bytes, and the file name and file index number are used as the primary key, and the file byte stream is used as the value, and saved to the Redis database.

[0055] Channel publishing unit: After uploading the file, each subsystem publishes the file information in its own channel. The file information includes the file name, number of file blocks, file size and file MD5 code. The file name and file block number can be used as the primary key to find the file content. The file size and file MD5 code are used by the subscriber to verify whether the file is correct.

[0056] In the second module, the subscriber subscribes to messages and maintains his own file index table;

[0057] Specifically, the second module includes a message subscription unit and an index maintenance unit;

[0058] Message subscription unit: each subsystem subscribes to the corresponding subsystem channel according to its own business needs. The number of subsystem channels includes but is not limited to one. After new file information is released in the corresponding subsystem channel, the file index table maintenance work is carried out as soon as possible.

[0059] The index maintenance unit parses and obtains new file information, including the file name, number of file blocks, file length, and file MD5 code, and saves the file information to its own file index table:

[0060] In the third module, the subsystem obtains file blocks according to the file index table, integrates and verifies the file integrity;

[0061] Specifically, the third module includes a file retrieval unit and a file verification unit;

[0062] The file retrieval unit, when each subsystem uses the corresponding file, searches its own file index table. According to the content in the file index table, the file name + file sequence number is used as the primary key to obtain file blocks from the database in sequence. Each time a block is obtained, the block number and block size are recorded. Finally, the file blocks are integrated in the system to form a complete file.

[0063] The file verification unit calculates the size of the complete file and the file MD5 code of the file, and compares them with the values ​​recorded in the file index table. If they are inconsistent, it is considered that the file retrieval has failed, and the block number and block size recorded in the file retrieval unit are verified. The problem block is found by comparison and then re-retrieved. If they are consistent, the entire file transfer process is completed.

[0064] In summary, this embodiment is implemented based on the existing Redis database, without the need to add new service nodes, reducing the use of resources. At the same time, the high availability and high performance of the Redis database can also ensure the consistency and reliability of file management under the distributed architecture. In addition, the increase in the number of file blocks can be dealt with by adding Redis database nodes in the later stage, which has good scalability. In other words, this implementation is based on the high performance, high availability, distribution and other characteristics of the Redis database and the publish-subscribe function to achieve file management through a lightweight solution.

[0065] In a distributed system, each subsystem is both a publisher and a subscriber. As a publisher, it maintains its own subscription channel, which is subscribed by other subsystems. Whenever a file is uploaded, the file slices are saved in the Redis database as key-value pairs, with the file name + file number as the key value and the file content as the value. After the upload is completed, a message about the new file upload is published to the subscription channel. As a subscriber, subscribe to the channel you want, and maintain a corresponding file index list based on the channel content. This list contains the attributes of the file, such as file name, file number, file size, etc. When you need to obtain a file, you can use the index list to obtain the file name and file number, and use the file name + file number as the key value to obtain the corresponding file content.

[0066] Example 2

[0067] like Figure 2 As shown, please refer to the description of Example 1 for the parts not described in detail in this embodiment. This embodiment provides a Redis-based file management method, which is applied in a distributed system. The distributed system includes multiple subsystems and a Redis database. In this embodiment, the subsystem uses three subsystems ABC as application scenarios to demonstrate the file transfer process between subsystems. The three subsystems ABC each maintain their own file information channels in the Redis database. Subsystem A subscribes to the channel of subsystem B, subsystem B subscribes to the channel of subsystem A, and subsystem C subscribes to the channel of subsystem A and the channel of subsystem B. Subsystem A uploads a file, and the other two systems retrieve the file.

[0068] Step A: First, the three subsystems ABC act as publishers and maintain a channel named A file information, B file information and C file information in the first module respectively. As subscribers, subsystem A subscribes to B file information, subsystem B subscribes to A file information, and subsystem C subscribes to A file information and B file information.

[0069] Step B: In the file upload unit of the first module, subsystem A uploads file A with a file size of 20M, and slices the file into 20 blocks of 1024K each. With A file 1, A file 2...A file 20 as the primary key, the 20 file blocks are saved to the Redis database in sequence. In the channel publishing unit, subsystem A publishes a message to the A file channel, including file name: A file, file block number: 20, file size: 20M, file Md5: 0959cf94a16248f0ac950b16dec30b07.

[0070] Step C: Subsystem B obtains the file information published by subsystem A in file A in the message subscription unit of module 2. Save the message to its own file index table. Record subsystem: A, file name: file A, file block number: 20, file size: 20M, file Md5: 0959cf94a16248f0ac950b16dec30b07. After obtaining the message, system C performs the same operation as subsystem B.

[0071] Step D: Subsystem B requests files from subsystem A in the file retrieval unit of module 3. First, search the file index table maintained by itself to find all files under subsystem A. Second, compare the file names, find the file named file A, and obtain the number of file blocks, which is 20; use file A1, file A2...file A20 as the primary key, and obtain the file blocks of file A in sequence. Each time a block is obtained, record the block number and size in the errata list.

[0072] Step E: After obtaining the complete file, subsystem B integrates the files and compares the file size recorded in the file index table with the file MD5 code. If the file size and the file MD5 code are consistent, it is considered that the file retrieval is successful, the errata list is deleted, and the file retrieval is completed. If they are inconsistent, the errata list recorded in step D is compared to find which block is missing and the file block is read again.

[0073] Example 3

[0074] Based on the description of Example 2, it can be seen that when the subsystems access each other, the file blocks are sequentially requested in a single-threaded manner. In actual use, since the file blocks are stored in different Key values, the subsystems can adopt a multi-threaded manner to simultaneously request different file blocks of the same file, or request different file blocks from different files;

[0075] When uploading, the file provider slices the uploaded file into blocks to obtain file information and saves it in the Redis database. This method breaks the file from a large whole into multiple small file blocks, which naturally supports the file acquirer to obtain multiple file blocks at the same time at a smaller granularity and finally assemble them into a large file.

[0076] For example, in the file index table of subsystem A, there is a file M, including file block 1, file block 2, file block 3, file block 4, and file block 5. The file name M combined with the 5 file block numbers corresponds to the five Redis primary keys in the Redis database.

[0077] When subsystem A obtains a file, it can start five threads at the same time to obtain five file blocks respectively, and assemble them uniformly after the acquisition is completed. Subsystem A does not need to send requests one by one and then read the file information in sequence, which greatly speeds up the file acquisition speed.

[0078] Embodiment 4: For the parts not described in detail in this embodiment, please refer to the description of Embodiment 1. This embodiment provides an electronic device, including: a processor and a memory, wherein the memory stores a computer program that can be called by the processor;

[0079] The processor executes the above-mentioned edge computing underwater target detection method by calling the computer program stored in the memory.

[0080] Embodiment 5: This embodiment provides a computer-readable storage medium storing instructions. When the instructions are executed on a computer, the computer executes the above-mentioned edge computing method for underwater target detection.

[0081] It should also be understood by those skilled in the art that the various exemplary devices, modules, units and steps described in conjunction with this patent disclosure can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, a general description has been given of the functions of various illustrative systems, modules, units and steps. Whether such functions are implemented as software or hardware depends on the specific application and the design constraints imposed on the entire system. Those skilled in the art can implement the functions in various ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope disclosed in the embodiments of the present invention.

[0082] The various exemplary devices, modules, units, and steps described in conjunction with the disclosure herein may be implemented or performed using the following components designed to perform such functions: a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. A general purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP, and / or any other such configuration.

[0083] The steps of the method or algorithm described in conjunction with the disclosure herein may be directly embodied in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0084] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0085] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A file management device based on Redis, characterized in that: The distributed system includes multiple subsystems and a Redis database, wherein the Redis database includes a first module, a second module and a third module; In the first module, the subsystem uploads files and stores slices in the Redis database, and publishes file information to the channel; In the second module, the subsystem subscribes to the channel to receive file information and updates the local file index table. The file information includes the file name, the number of file blocks, the file size and the file MD5 code; In the third module, the subsystem retrieves file blocks according to the file index table, integrates and verifies the file integrity.

2. A file management device based on Redis according to claim 1, characterized in that: The first module includes a file uploading unit and a channel publishing unit; File upload unit: each subsystem performs file upload operations and saves them to the Redis database. Channel publishing unit, after uploading the file, each subsystem publishes the file information in its own channel.

3. A Redis-based file management device according to claim 2, characterized in that: When uploading a file, the file is sliced ​​to determine the file bytes, with the file name and file index number as the primary key and the file byte stream as the value.

4. A Redis-based file management device according to claim 1, characterized in that: The file name and file block number can be used as primary keys to search for file content; the file size and file MD5 code are used by subscribers to verify whether the file is correct.

5. A Redis-based file management device according to claim 1, characterized in that: The second module includes a message subscription unit and an index maintenance unit; Message subscription unit: each subsystem subscribes to the corresponding subsystem channel according to its own business needs. The number of subsystem channels includes but is not limited to one. After the new file information is released in the corresponding subsystem channel, the file index table maintenance work is carried out in the first time; The index maintenance unit parses and obtains new file information and saves the file information into its own file index table.

6. A Redis-based file management device according to claim 1, characterized in that: The third module includes a file retrieval unit and a file verification unit; File retrieval unit, each subsystem searches its own file index table when using the corresponding file; The file verification unit calculates the size of the complete file and the file MD5 code, and compares them with the values ​​recorded in the file index table. If they are inconsistent, the file retrieval is considered to have failed. The block number and block size recorded in the file retrieval unit are verified, the problem block is found by comparison, and it is retrieved again. If they are consistent, the entire file transfer process is completed.

7. A Redis-based file management device according to claim 1, characterized in that: The logic for obtaining the complete file is as follows: According to the content in the file index table, the file name + file sequence number is used as the primary key to obtain file blocks from the database in sequence. Each time a block is obtained, the block number and block size are recorded. Finally, the file blocks are integrated in the system to form a complete file.

8. A Redis-based file management method, based on the implementation of a Redis-based file management device according to any one of claims 1 to 7, characterized in that: The following steps are involved: File publishing: The subsystem uploads files and stores slices in the Redis database, and publishes file information to the channel; Message subscription and index maintenance: the subsystem subscribes to the channel to receive file information and updates the local file index table; File retrieval and verification,the subsystem retrieves file blocks according to the file index table,,integrates and verifies the file integrity.

9. An electronic device comprising: include: A processor and a memory, wherein the memory stores a computer program that can be called by the processor; The processor executes the Redis-based file management device described in any one of claims 1 to 7 by calling the computer program stored in the memory.

10. A computer-readable storage medium storing instructions, which, when executed on a computer, enable the computer to execute a Redis-based file management device as described in any one of claims 1 to 7.

Images