Picture reading system and method based on MinIO distributed storage on mobile terminal APP

Abstract

The invention provides a picture reading system and method based on MinIO distributed storage on a mobile terminal APP. The picture reading system comprises a MinIO cluster, a client, a front end and a gateway. And the MinIO cluster is deployed on a plurality of server nodes by adopting a distributed architecture. Picture data interaction is carried out between the client APP and the MinIO cluster through the gateway and the front end. According to the technical scheme, picture preprocessing and MinIO distributed storage are innovatively combined, picture data are optimized at the source, and the storage and reading efficiency is improved. A unique Hash-based data distribution strategy is designed, uniform distribution and load balance of data are ensured, and the hotspot problem is effectively avoided. According to the method, a high-efficiency metadata management and caching mechanism is established, the picture retrieval and reading time is greatly shortened, the picture reading experience of a user is improved, and especially in a high-concurrency access scene, quick and smooth picture display can be ensured.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a picture reading system and method based on MinIO distributed storage on a mobile terminal APP. Background Art

[0002] With the rapid development of the Internet and digital technology, the scale of image data has exploded. Traditional image storage and viewing methods often face many challenges, such as limited storage capacity, slow reading speed, and low data security.

[0003] In the prior art, common image storage solutions include centralized storage and partially distributed storage. When facing large-scale image data, centralized storage architecture is prone to performance bottlenecks and poor scalability. As a result, storage performance and stability face challenges in high concurrent access scenarios. Such systems often have the following problems: a single node failure in centralized storage may cause the entire system to be unavailable; storage capacity is limited and difficult to expand; image loading speed is slow and user experience is poor. The existing image browsing system has a significantly increased response time during concurrent access, which affects the user experience. Although some distributed storage solutions have solved the scalability problem to a certain extent, they still have shortcomings in data reading efficiency and data consistency. When some existing distributed storage systems process a large number of small files, metadata management is complex, resulting in high reading latency; in addition, some systems are not optimized in data distribution and redundancy strategies, which may cause storage space waste or increase the risk of data loss. Summary of the invention

[0004] In order to overcome the above technical defects, the first aspect of the present invention provides a picture reading system based on MinIO distributed storage on a mobile terminal APP, which includes: a MinIO cluster, a client, a gateway and a front-end.

[0005] The MinIO cluster is deployed on multiple server nodes using a distributed architecture, and each node is responsible for a part of the data storage task. Each server is divided into several drives, i.e., disks for storing data. All object data in MinIO will be stored in the drives. A group of drives are regularly assembled to form a set, which is the smallest unit for storing a complete object. Distributed deployment automatically divides one or more sets according to the size of the cluster, and the drives in each set are distributed in different locations. Each server node includes a data storage module, a metadata management module, and multiple MinIO servers. The metadata management module performs a hash based on the name of the image, and then maps it to a specified set, performs metadata management and indexing on the image data, uses the Reed-Solomon algorithm to optimize redundancy, and encodes the object into several data blocks and check blocks. This method ensures that the data can be evenly distributed on all sets, and each image has a unique index KEY directly associated with the metadata index; the MinIO server is used to store the metadata of the image using Bucket as a container, and the data between each Bucket is isolated from each other; the data storage module is used to store all the images stored by the multiple MinIO servers through physical and logical divisions, and to share data between multiple MinIO servers; the data storage module uses a NAS (Network Attached Storage) server, and is used to store all the images stored by the multiple MinIO servers through physical and logical divisions, and to share data between multiple MinIO servers.

[0006] The client includes a client APP and a local cache layer; the client APP is used to interact with the MinIO cluster for image data, including pre-processing local images before uploading them to the MinIO cluster, and / or sending image retrieval requests to the MinIO cluster; the local cache layer is used to store hot image data and load images that users want to view in advance;

[0007] The gateway is used to perform access control and encryption and decryption on the interactive data between the client and the front-end. IDS and WAF are deployed for intrusion prevention monitoring and WEB security monitoring. At the same time, Nginx is deployed for HTTP reverse proxy from the Internet to the private cloud, and a global policy gateway is defined that is completely unrelated to specific back-end business applications and services. The traffic gateway focuses on global API management strategies, such as global traffic monitoring, logging, global current limiting, blacklist and whitelist control, and load balancing of access requests to business systems.

[0008] The front-end is deployed in the cloud computer room and is used for data interaction between the gateway and the MinIO cluster. It is also used to rename images, manage image ownership, and record image URLs.

[0009] The mobile terminal in this application includes but is not limited to mobile phones, etc.

[0010] Furthermore, the metadata includes the name of the picture, a hash value of the picture name, size, creation time, storage location and picture type.

[0011] Furthermore, the method for the client APP to pre-process the local image includes compression and format conversion.

[0012] Furthermore, when uploading an image, MinIO automatically generates a special URL (object URL), which directly points to the image stored in MinIO.

[0013] Furthermore, when reading an image, after the MinIO server receives a preview request, it will find the corresponding file based on the information in the URL and perform a preview process. During the preview process, the server will call the image processing library to generate a thumbnail. After the preview process is completed, the server will return the preview result to the client. After receiving the preview result, the client will display it in the browser for the user to view. Furthermore, the MinIO cluster further includes a data protection module, which divides the data into fragments, expands and encodes the redundant data blocks, and stores them in different locations, such as disks, storage nodes, or other geographical locations. And it can quickly recover in the event of a failure to ensure that data is not lost.

[0014] The second aspect of the present application provides a method for viewing pictures based on the above-mentioned picture viewing system based on MinIO distributed storage on a mobile terminal APP, which includes:

[0015] Step S1: The client APP sends a request for image access;

[0016] Step S2: The client searches the local cache layer; if the local cache layer hits the image, the data is returned; if the local cache layer does not hit the image, the client sends an image reading request to the MinIO cluster that pre-stores the image data through the gateway;

[0017] Step S3: The load balancer of the MinIO cluster distributes each image access request to one of the MinIO servers. The metadata management module collects the metadata of the meta-images scattered in each disk according to the index KEY and performs preview processing. During the preview processing, the server calls the image processing library to generate thumbnails.

[0018] Step S4: The MinIO server sends the returned image data to the client's browser through the gateway for the user to view.

[0019] Furthermore, in step S4, when reading the picture, the gateway controls the bandwidth and concurrency of data transmission according to the network status and request priority of the client.

[0020] Furthermore, the method of pre-storing image data in the MinIO cluster includes:

[0021] Step W1: The client APP pre-processes the local images to be uploaded to the MinIO cluster;

[0022] Step W2: The gateway encrypts and decrypts the pre-processed image data;

[0023] Step W3: The MinIO metadata management module adopts a hash-based distributed storage strategy, hashes the image name, and then evenly distributes the image data and saves it to multiple server nodes to ensure balanced data storage.

[0024] Furthermore, in step W1, the method in which the client APP pre-processes the local image includes compression and format conversion.

[0025] Furthermore, the above-mentioned picture browsing method further comprises step W4: the data backup module of the MinIO cluster uses a near-synchronous mechanism to back up the picture data and can quickly restore it in case of a failure to ensure that the data is not lost.

[0026] Compared with the prior art, the above technical solution has the following beneficial effects:

[0027] The present invention proposes a method for quickly browsing pictures based on MinIO distributed storage on a mobile terminal APP. MinIO is a high-performance, distributed object storage service (object storage server), which is compatible with the Amazon S3 cloud storage service interface and is intended to be deployed in a private cloud, data center or public cloud environment. It can be easily installed and deployed on various operating systems, including Linux, Windows and macOS, and supports containerized deployment methods such as Docker and Kubernetes. MinIO can achieve extremely high read and write performance on standard hardware, especially when processing a large number of small files. It supports horizontal expansion, and can linearly increase storage capacity and throughput by adding more nodes. The distributed architecture design allows data to be shared between multiple nodes, and each node can run independently, improving the reliability and fault tolerance of the system. At the same time, built-in data encryption and security functions are provided, including authentication, authorization and audit logs, to ensure data security and privacy. By sharing an access_key and secret_key, services are built on multiple servers, and data is scattered on multiple (greater than 4, no upper limit) disks, providing a relatively powerful data redundancy mechanism. The technical solution of this application innovatively combines image preprocessing with MinIO distributed storage to optimize image data at the source and improve storage and reading efficiency. A unique hash-based data distribution strategy is designed to ensure uniform distribution and load balancing of data, effectively avoiding hot spot problems. An efficient metadata management and caching mechanism is established, which greatly shortens the time for image retrieval and reading. The reading speed of images is significantly improved. Compared with the traditional image conversion to Base64 solution, the reading delay is reduced by 82%. The storage space utilization is optimized. Taking a certain bank's business development APP as an example, IOS saves 42% of storage space, and Android can set the compression ratio according to parameters, saving 38% of storage space in a certain bank's business development APP. The scalability and stability of the system are enhanced, and it can easily cope with the growth of large-scale image data. It improves the user's image reading experience, especially in high-concurrency access scenarios, and can ensure fast and smooth image display. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is the system architecture diagram of the picture reading system based on MinIO distributed storage on the mobile terminal APP of this application;

[0029] Figure 2 This is a diagram of MinIO storage architecture;

[0030] Figure 3 This is a schematic diagram of the image data interaction principle between the client and the MinIO cluster;

[0031] Figure 4 This is the image data interaction flow chart between the client and the MinIO cluster (image upload process);

[0032] Figure 5 This is a flowchart of the image data interaction between the client and the MinIO cluster (image retrieval process). DETAILED DESCRIPTION

[0033] The advantages of the present invention are further described below in conjunction with the accompanying drawings and specific embodiments. Those skilled in the art should understand that the following specific description is illustrative rather than restrictive, and should not be used to limit the scope of protection of the present invention.

[0034] like Figure 1 As shown, this embodiment provides a picture reading system based on MinIO distributed storage on a mobile terminal APP, which includes a MinIO cluster, a client, a front-end and a gateway.

[0035] like Figure 2As shown, the MinIO cluster is deployed on multiple server nodes using a distributed architecture, and each node is responsible for a part of the data storage task. Each server is divided into several drives, i.e., disks for storing data. All object data in MinIO will be stored in the drives. A group of drives are regularly assembled to form a set, which is the smallest unit for storing a complete object. Distributed deployment automatically divides one or more sets according to the size of the cluster, and the drives in each set are distributed in different locations. Each server node includes a data storage module, a metadata management module, and multiple MinIO servers. The metadata management module performs a hash based on the name of the image, and then maps it to a specified set, performs metadata management and indexing on the image data, uses the Reed-Solomon algorithm to optimize redundancy, and encodes the object into several data blocks and check blocks. This method theoretically ensures that the data can be evenly distributed on all sets, and each image has a unique index KEY, which is directly associated with the metadata index, effectively improving the metadata reorganization efficiency, and thus improving the image reading efficiency; the MinIO server is used to store the metadata of the image using Bucket as a container, and the data between each Bucket is isolated from each other; the data storage module is used to store all the images stored by the multiple MinIO servers through physical and logical divisions, and to share data between multiple MinIO servers; the data storage module uses a NAS (Network Attached Storage) server, which is used to store all the images stored by the multiple MinIO servers through physical and logical divisions, and to share data between multiple MinIO servers. The data backup module uses a near-synchronous mechanism to back up the image data, and can quickly recover in the event of a failure to ensure that the data is not lost.

[0036] The client includes a client APP and a local cache layer; the client APP is used to interact with the MinIO cluster for image data, including pre-processing local images before uploading them to the MinIO cluster, or sending image retrieval requests to the MinIO cluster; the local cache layer is used to save hot image data and load images that users want to view in advance.

[0037] The gateway is used to control access and encrypt and decrypt the interactive data between the client and the front-end. Deploy IDS and WAF for intrusion prevention monitoring and WEB security monitoring. At the same time, deploy Nginx for HTTP reverse proxy from the Internet to the private cloud, and define a global policy gateway that is completely unrelated to specific back-end business applications and services. The traffic gateway focuses on global API management strategies, such as global traffic monitoring, logging, global current limiting, blacklist and whitelist control, and load balancing of access requests to business systems.

[0038] The front-end is deployed in the cloud computer room and is used for data interaction between the gateway and the MinIO cluster; it is also used for image renaming, managing image ownership and recording image URLs.

[0039] like Figure 3 As shown in the figure, the image data interaction process between the client and the MinIO cluster includes the following two scenarios:

[0040] 1. The client uploads the image to the MinIO cluster

[0041] like Figure 4 As shown in the figure, the process of uploading an image from a client to a MinIO cluster includes the following steps:

[0042] Step W1: The client APP pre-processes the local images to be uploaded to the MinIO cluster.

[0043] Before uploading the image to MinIO storage, the APP compresses and converts the image to reduce the storage space of the image and improves the reading efficiency. For example, the method in which the client APP preprocesses the local image includes compression and format conversion.

[0044] Step W2: The gateway encrypts and decrypts the pre-processed image data.

[0045] Step W3: The MinIO metadata management module adopts a hash-based distributed storage strategy, hashes the image name, and then evenly distributes and saves the image data to multiple server nodes to ensure balanced storage of the data.

[0046] Step W4: MinIO cluster near-synchronous mechanism backs up the image data and can quickly recover in case of failure to ensure that data is not lost. MinIO cluster data protection module divides the data into fragments, expands and encodes the redundant data blocks, and stores them in different locations, such as disks, storage nodes or other geographical locations. It can also quickly recover in case of failure to ensure that data is not lost.

[0047] 2. The client retrieves images from the MinIO cluster

[0048] like Figure 5 As shown in the figure, the method for the client to retrieve images from the MinIO cluster includes the following steps:

[0049] Step S1: The client APP sends a request for image access.

[0050] Step S2: The client searches the local cache layer: If the local cache layer hits the image, the data is returned; if the local cache layer does not hit the image, the client sends an image browsing request to the MinIO cluster that pre-stores the image data through the gateway.

[0051] Step S3: The load balancer of the MinIO cluster distributes each image retrieval request to one of the MinIO servers. The metadata management module collects the meta-image metadata scattered on each disk according to the index KEY and performs preview processing. During the processing, the server calls the image processing library to generate thumbnails.

[0052] Step S4: The MinIO server sends the returned image data to the client's browser through the gateway for the user to view. When reading the image, the gateway dynamically adjusts the bandwidth and concurrency of data transmission according to the client's network status and request priority to achieve fast reading. Set up caches on the client and server respectively to cache frequently accessed image data to reduce repeated reading of storage nodes. Due to the requirements of some industries for data security, data cache can be used on demand. It is also possible to predict the images that the user may view next by analyzing user behavior data and load them into the local cache in advance.

[0053] In this embodiment, the client that uploads the image to the MinIO cluster and the client that retrieves the image from the MinIO cluster may be the same client or different clients.

[0054] It should be noted that the embodiments of the present invention have better practicability and do not impose any form of limitation on the present invention. Any technician familiar with the field may use the technical content disclosed above to change or modify it into an equivalent effective embodiment. However, any modification or equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention, as long as it does not deviate from the content of the technical solution of the present invention, are still within the scope of the technical solution of the present invention.

Claims

1. A picture reading system based on MinIO distributed storage on a mobile terminal APP, characterized in that: include: MinIO cluster, client, gateway and front-end; The MinIO cluster is deployed on multiple server nodes using a distributed architecture. Each node is responsible for a part of the data storage task. Each server is divided into several drives, i.e., disks for storing data. All object data in MinIO will be stored in the drives. A group of drives are regularly assembled to form a set, i.e., the smallest unit for storing a complete object. The distributed deployment automatically divides one or more sets according to the cluster size, and the drives in each set are distributed in different locations. Each server node includes a data storage module, a metadata management module, and multiple MinIO servers. The metadata management module performs hashing according to the name of the image, and then maps it to a specified set, performs metadata management and indexing on the image data, uses the Reed-Solomon algorithm to optimize redundancy, encodes the object into several data blocks and check blocks, so that the data is evenly distributed on all sets, and each image has a unique index KEY directly associated with the metadata index. The MinIO server is used to store the metadata of the image using Bucket as a container, and the data between each Bucket is isolated from each other. The data storage module uses a NAS server, which is used to store all the images stored by the multiple MinIO servers through physical and logical divisions, and provides data sharing between multiple MinIO servers. The client includes a client APP and a local cache layer; the client APP is used to interact with the MinIO cluster for image data, including pre-processing the local image before uploading it to the MinIO cluster, and / or sending an image retrieval request to the MinIO cluster; The local cache layer is used to store hotspot image data and load images that users want to view in advance; The gateway is used to perform access control and encryption and decryption on the interactive data between the client and the front-end; The gateway deploys IDS and WAF to perform intrusion prevention monitoring and WEB security monitoring, and deploys Nginx for HTTP reverse proxy from the Internet to the private cloud, defining a global policy gateway that is completely unrelated to specific backend business applications and services; Traffic gateway focuses on global API management strategies; The front-end is deployed in the cloud computer room and is used for data interaction between the gateway and the MinIO cluster; it is also used for image renaming, managing image ownership and recording image URLs.

2. The picture viewing system based on MinIO distributed storage on the mobile terminal APP as claimed in claim 1 is characterized in that: The metadata includes the name of the picture, a hash value of the picture name, size, creation time, storage location, and picture type.

3. The picture viewing system based on MinIO distributed storage on the mobile terminal APP as claimed in claim 1 is characterized in that: When uploading an image, MinIO automatically generates an object URL that directly points to the image stored in MinIO.

4. The picture viewing system based on MinIO distributed storage on the mobile terminal APP as claimed in claim 3 is characterized in that: When reading an image, after the MinIO server receives a preview request, it will find the corresponding file based on the information in the URL and perform a preview process. During the preview process, the server will call the image processing library to generate a thumbnail. After the preview process is completed, the server will return the preview result to the client. After receiving the preview result, the client will display it in the browser for the user to view.

5. The picture viewing system based on MinIO distributed storage on the mobile terminal APP as claimed in claim 1 is characterized in that: The MinIO cluster further includes a data protection module, which divides the data into fragments, expands and encodes the redundant data blocks, and stores them in different locations. It can also quickly recover in the event of a failure to ensure that data is not lost.

6. A method for viewing pictures based on a MinIO distributed storage picture viewing system on a mobile terminal APP according to any one of claims 1 to 5, characterized in that: include: Step S1: The client APP sends a request for image access; Step S2: The client searches for the local cache layer; If the local cache layer hits the image, the data is returned. If the local cache layer does not hit the image, the client sends an image reading request to the MinIO cluster that pre-stores the image data through the gateway. Step S3: The load balancer of the MinIO cluster distributes each image access request to one of the MinIO servers. The metadata management module collects the metadata of the meta-images scattered in each disk according to the index KEY and performs preview processing. During the preview processing, the server calls the image processing library to generate thumbnails. Step S4: The MinIO server sends the returned image data to the client's browser through the gateway for the user to view.

7. The picture viewing method according to claim 6, characterized in that: In step S4, when reading the picture, the gateway controls the bandwidth and concurrency of data transmission according to the network status and request priority of the client.

8. The picture viewing method according to claim 6, characterized in that: Methods for pre-storing image data in a MinIO cluster include: Step W1: The client APP pre-processes the local images to be uploaded to the MinIO cluster; Step W2: The gateway encrypts and decrypts the pre-processed image data; Step W3: The MinIO metadata management module adopts a hash-based distributed storage strategy, hashes the image name, and then evenly distributes and saves the image data to multiple server nodes to ensure balanced storage of the data.

9. The picture viewing method according to claim 8, characterized in that: In step W1, the client APP pre-processes the local image by compression and format conversion.

10. The picture viewing method according to claim 8, characterized in that: Further including step W4: MinIO cluster near synchronization mechanism backs up the image data and can quickly restore it in the event of a failure to ensure that data is not lost.

Images