Image viewing system and method based on MinIO distributed storage for mobile terminal APP

Abstract

This invention provides an image viewing system and method based on MinIO distributed storage for mobile terminal apps. The image viewing system includes a MinIO cluster, a client, a front-end, and a gateway. The MinIO cluster is deployed on multiple server nodes using a distributed architecture. The client app interacts with the MinIO cluster through the gateway and the front-end. This application innovatively combines image preprocessing with MinIO distributed storage, optimizing image data at the source and improving storage and retrieval efficiency. A unique hash-based data distribution strategy is designed to ensure uniform data distribution and load balancing, effectively avoiding hotspot issues. An efficient metadata management and caching mechanism is established, greatly shortening image retrieval and retrieval time and improving the user's image viewing experience, especially in high-concurrency access scenarios, ensuring fast and smooth image display.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, and in particular to an image viewing system and method based on MinIO distributed storage for mobile terminal apps. Background Technology

[0002] With the rapid development of the internet and digital technologies, the scale of image data has exploded. Traditional methods of image storage and viewing often face numerous challenges, such as limited storage capacity, slow reading speed, and low data security.

[0003] In existing technologies, common image storage solutions include centralized storage and partially distributed storage. Centralized storage architectures are prone to performance bottlenecks and poor scalability when dealing with large-scale image data. This leads to challenges in storage performance and stability under high-concurrency access scenarios. Such systems often suffer from the following problems: a single node failure in centralized storage can render the entire system unavailable; storage capacity is limited and difficult to expand; image loading speed is slow, resulting in a poor user experience. Existing image browsing systems experience significantly increased response time under concurrent access, impacting user experience. While some distributed storage solutions address scalability issues to some extent, they still have shortcomings in data read efficiency and data consistency. Some existing distributed storage systems suffer from complex metadata management when handling a large number of small files, leading to high read latency; furthermore, some systems lack optimized data distribution and redundancy strategies, potentially resulting in wasted storage space or an increased risk of data loss. Summary of the Invention

[0004] To overcome the above-mentioned technical defects, the first aspect of the present invention provides an image viewing system based on MinIO distributed storage on a mobile terminal APP, which includes: a MinIO cluster, a client, a gateway, and a front-end.

[0005] The MinIO cluster employs a distributed architecture deployed across multiple server nodes, with each node responsible for a portion of data storage. Each server is divided into several drives, which are the disks used to store data. All object data in MinIO is stored in these drives. A set is formed by a set of drives, which is the smallest unit for storing a complete object. The distributed deployment automatically creates one or more sets based on the cluster size, with drives distributed across different locations within each set. Each server node includes a data storage module, a metadata management module, and multiple MinIO servers. The metadata management module hashes the image name and maps it to a designated set. It manages and indexes image data, using the Reed-Solomon algorithm to optimize redundancy. Objects are encoded into several data blocks and checksum blocks, ensuring data is evenly distributed across all sets. Each image has a unique index key directly associated with the metadata index. The MinIO server stores image metadata in buckets, with data isolated between buckets. The data storage module stores all images stored by the multiple MinIO servers through physical and logical partitioning, facilitating data sharing among the servers. The data storage module uses a NAS (Network Attached Storage) server, employing physical and logical partitioning to store all images stored by the multiple MinIO servers, enabling data sharing among them.

[0006] The client includes a client APP and a local cache layer; the client APP is used to interact with the MinIO cluster to exchange image data, including preprocessing local images before uploading them to the MinIO cluster, and / or sending image retrieval requests to the MinIO cluster; the local cache layer is used to store hot image data and preload images that users want to view.

[0007] The gateway is used for access control and encryption / decryption of data exchanged between clients and front-end systems. It deploys IDS and WAF for intrusion prevention and web security monitoring, and Nginx for HTTP reverse proxying from the internet to the private cloud. A global policy gateway is defined, completely independent of specific backend business applications and services. The traffic gateway focuses on global API management policies, such as global traffic monitoring, logging, global rate limiting, blacklist / whitelist control, and load balancing of access requests to business systems.

[0008] The front-end, deployed in a cloud data center, is used for data interaction between the gateway and the MinIO cluster. It is also used for image renaming, managing image ownership, and recording image URLs.

[0009] The mobile terminal in this application includes, but is not limited to, mobile phones.

[0010] Furthermore, the metadata includes the image name, the hash value of the image name, size, creation time, storage location, and image type.

[0011] Furthermore, the method for preprocessing local images by the client APP includes compression and format conversion.

[0012] Furthermore, when uploading an image, MinIO automatically generates a special URL (object URL) that directly points to the image stored in MinIO.

[0013] Furthermore, when reading images, the MinIO server, upon receiving a preview request, locates the corresponding file based on the information in the URL and performs preview processing. During preview processing, the server calls an image processing library to generate thumbnails. After preview processing is complete, the server returns the preview result to the client. Upon receiving the preview result, the client displays it in its browser for the user to view. Furthermore, the MinIO cluster includes a data protection module. This module segments data into fragments, expands and encodes redundant data blocks, and stores them in different locations, such as disks, storage nodes, or other geographical locations. It can also quickly recover in the event of a failure, ensuring no data loss.

[0014] The second aspect of this application provides an image viewing method based on the aforementioned image viewing system using MinIO distributed storage on a mobile terminal APP, comprising:

[0015] Step S1: The client APP sends an image retrieval request;

[0016] Step S2: The client searches the local cache layer; if the local cache layer matches the image, it returns the data; if the local cache layer does not match the image, the client sends an image viewing request to the MinIO cluster that has pre-stored the image data through the gateway.

[0017] Step S3: The load balancer of the MinIO cluster distributes each image retrieval request to one of the MinIO servers. The metadata management module collects the metadata of the image scattered on each disk according to the index key and performs preview processing. During the preview processing, the server calls the image processing library to generate thumbnails.

[0018] Step S4: The MinIO server sends the returned image data to the client's browser through the gateway for the user to view.

[0019] Furthermore, in step S4, when reading an image, the gateway controls the bandwidth and concurrency of data transmission based on the client's network conditions and request priority.

[0020] Furthermore, methods for pre-storing image data in a MinIO cluster include:

[0021] Step W1: The client APP preprocesses the local images to be uploaded to the MinIO cluster;

[0022] Step W2: The gateway encrypts and decrypts the preprocessed image data;

[0023] Step W3: The MinIO metadata management module adopts a hash-based distributed storage strategy. It hashes the image name and then distributes the image data evenly across multiple server nodes to ensure balanced data storage.

[0024] Furthermore, in step W1, the method by which the client APP preprocesses local images includes compression and format conversion.

[0025] Furthermore, the above image viewing method further includes step W4: the MinIO cluster's data backup module uses a near-synchronization mechanism to back up the image data and can quickly recover in the event of a failure, ensuring that the data is not lost.

[0026] Compared with existing technologies, the above technical solution has the following advantages:

[0027] This invention proposes a method for fast image viewing on mobile terminal apps based on MinIO distributed storage. MinIO is a high-performance, distributed object storage service compatible with the Amazon S3 cloud storage service interface, designed for deployment in private cloud, data center, or public cloud environments. It can be easily installed and deployed on various operating systems, including Linux, Windows, and macOS, and supports containerized deployment methods such as Docker and Kubernetes. MinIO achieves extremely high read and write performance on standard hardware, especially excelling when handling large numbers of small files. It supports horizontal scaling, allowing storage capacity and throughput to increase linearly by adding more nodes. Its distributed architecture design allows multiple nodes to share data, while each node can operate independently, improving system reliability and fault tolerance. It also provides built-in data encryption and security features, including authentication, authorization, and audit logs, ensuring data security and privacy. By sharing a single access_key and secret_key, services can be built on multiple servers, with data distributed across multiple disks (more than four, with no upper limit), providing a robust data redundancy mechanism. This application's technical solution innovatively combines image preprocessing with MinIO distributed storage, optimizing image data at the source and improving storage and retrieval efficiency. A unique hash-based data distribution strategy is designed to ensure uniform data distribution and load balancing, effectively avoiding hotspot issues. An efficient metadata management and caching mechanism is established, significantly shortening image retrieval and retrieval time. Image retrieval speed is significantly improved; compared to traditional image-to-Base64 conversion schemes, read latency is reduced by 82%. Storage space utilization is optimized; for example, in a bank's business application, iOS saves 42% of storage space, and Android, with adjustable compression ratios, saves 38% of storage space in the same application. The system's scalability and stability are enhanced, easily handling large-scale image data growth. The user's image viewing experience is improved, especially in high-concurrency access scenarios, ensuring fast and smooth image display. Attached Figure Description

[0028] Figure 1 This is a system architecture diagram of the image viewing system based on MinIO distributed storage on a mobile terminal APP according to this application;

[0029] Figure 2 This is a schematic diagram of the MinIO storage architecture;

[0030] Figure 3 A schematic diagram illustrating the principle of image data interaction between the client and the MinIO cluster;

[0031] Figure 4 A flowchart illustrating the image data interaction process between the client and the MinIO cluster (image upload process);

[0032] Figure 5 This is a flowchart illustrating the image data interaction process between the client and the MinIO cluster (image retrieval process). Detailed Implementation

[0033] The advantages of the present invention are further illustrated below with reference to the accompanying drawings and specific embodiments. Those skilled in the art should understand that the following detailed description is illustrative rather than restrictive and should not be construed as limiting the scope of protection of the present invention.

[0034] like Figure 1 As shown, this embodiment provides an image viewing system based on MinIO distributed storage on a mobile terminal APP, which includes a MinIO cluster, a client, a front-end, and a gateway.

[0035] like Figure 2As shown, the MinIO cluster adopts a distributed architecture deployed across multiple server nodes, with each node responsible for a portion of data storage tasks. Each server is divided into several drives, which are the disks for storing data. All object data in MinIO is stored in these drives. A set is formed by a set of drives, which is the smallest unit for storing a complete object. The distributed deployment automatically divides the cluster into one or more sets based on the cluster size, and the drives in each set are distributed in different locations. Each server node includes a data storage module, a metadata management module, and multiple MinIO servers. The metadata management module hashes the image names and maps them to a designated set. It manages and indexes image data, using the Reed-Solomon algorithm to optimize redundancy. Objects are encoded into several data blocks and checksum blocks, theoretically ensuring data is evenly distributed across all sets. Each image has a unique index key directly associated with the metadata index, effectively improving metadata reassembly efficiency and thus image retrieval efficiency. The MinIO server stores image metadata in Buckets, with data isolated between each Bucket. The data storage module uses physical and logical partitioning to store all images stored by the multiple MinIO servers, facilitating data sharing among them. The data storage module employs a NAS (Network Attached Storage) server, using physical and logical partitioning to store all images stored by the multiple MinIO servers, also facilitating data sharing among them. The data backup module uses a near-synchronization mechanism to back up image data and quickly recover in case of failure, ensuring no data loss.

[0036] The client includes a client APP and a local cache layer; the client APP is used to interact with the MinIO cluster for image data, including preprocessing local images before uploading them to the MinIO cluster, or sending image retrieval requests to the MinIO cluster; the local cache layer is used to store hot image data and preload images that users want to view.

[0037] The gateway is used for access control and encryption / decryption of data exchanged between clients and front-end systems. It deploys IDS and WAF for intrusion prevention and web security monitoring, and Nginx for HTTP reverse proxying from the internet to the private cloud. It defines a global policy gateway that is completely independent of specific backend business applications and services. The traffic gateway focuses on global API management policies, such as global traffic monitoring, logging, global rate limiting, blacklist / whitelist control, and load balancing of access requests to business systems.

[0038] The front-end is deployed in a cloud data center and is used for data interaction between the gateway and the MinIO cluster; it is also used for image renaming, managing image ownership, and recording image URLs.

[0039] like Figure 3 As shown, the image data interaction process between the client and the MinIO cluster includes the following two scenarios:

[0040] 1. The client uploads the image to the MinIO cluster.

[0041] like Figure 4 As shown, the process of a client uploading an image to a MinIO cluster includes the following steps:

[0042] Step W1: The client APP preprocesses the local images to be uploaded to the MinIO cluster.

[0043] Before uploading images to MinIO storage, the app compresses and converts the images to reduce storage space and improve retrieval efficiency. For example, the client app's preprocessing method for local images includes compression and format conversion.

[0044] Step W2: The gateway encrypts and decrypts the preprocessed image data.

[0045] Step W3: The MinIO metadata management module adopts a hash-based distributed storage strategy. It hashes the image name and then distributes the image data evenly across multiple server nodes to ensure balanced data storage.

[0046] Step W4: The MinIO cluster near-synchronization mechanism backs up image data and enables rapid recovery in the event of a failure, ensuring no data loss. The MinIO cluster data protection module segments the data into fragments, expands and encodes redundant data blocks, and stores them in different locations, such as disks, storage nodes, or other geographical locations. It also enables rapid recovery in the event of a failure, ensuring no data loss.

[0047] II. Client retrieves images from the MinIO cluster

[0048] like Figure 5 As shown, the method for a client to retrieve images from a MinIO cluster includes the following steps:

[0049] Step S1: The client APP sends an image retrieval request.

[0050] Step S2: The client searches the local cache layer: If the local cache layer matches the image, it returns the data; otherwise, the client sends an image viewing request to the MinIO cluster that has pre-stored the image data through the gateway.

[0051] Step S3: The load balancer of the MinIO cluster distributes each image retrieval request to one of the MinIO servers. The metadata management module collects the metadata of the images scattered on each disk according to the index key and performs preview processing. During the processing, the server calls the image processing library to generate thumbnails.

[0052] Step S4: The MinIO server sends the returned image data to the client's browser via the gateway for user viewing. When reading images, the gateway dynamically adjusts the data transmission bandwidth and concurrency based on the client's network conditions and request priority to achieve fast viewing. Caches are set up on both the client and server sides to cache frequently accessed image data, reducing repeated reads from storage nodes. Due to data security requirements in some industries, data caching can be used on demand. Furthermore, by analyzing user behavior data, the system can predict the images the user might view next and pre-load them into the local cache.

[0053] In this embodiment, the client that uploads the image to the MinIO cluster and the client that retrieves the image from the MinIO cluster can be the same client or different clients.

[0054] It should be noted that the embodiments of the present invention have better implementability and are not intended to limit the present invention in any way. Any person skilled in the art may use the above-disclosed technical content to change or modify it into equivalent effective embodiments. However, any modifications or equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A picture viewing system based on MinIO distributed storage for mobile terminal APP, characterized in that, include: MinIO cluster, client, gateway, and front-end; The MinIO cluster employs a distributed architecture deployed across multiple server nodes. Each node is responsible for a portion of data storage tasks. Each server is divided into several drives, which are disks for storing data. All object data in MinIO is stored in drives. A set is formed by a set of drives, which is the smallest unit for storing a complete object. The distributed deployment automatically divides the cluster into one or more sets based on the cluster size, with drives distributed in different locations within each set. Each server node includes a data storage module, a metadata management module, and multiple MinIO servers. The metadata management module hashes the image name and maps it to a designated set, managing and indexing the image data. It uses the Reed-Solomon algorithm to optimize redundancy, encoding objects into several data blocks and checksum blocks, thus ensuring that the data is evenly distributed across all sets. Each image has a unique index key directly associated with the metadata index. The MinIO servers use buckets as containers to store image metadata, with data isolated between buckets. The data storage module uses a NAS server, which, through physical and logical partitioning, stores all images stored by the multiple MinIO servers and allows data sharing among them. The client includes a client APP and a local cache layer; the client APP is used to interact with the MinIO cluster to exchange image data, including preprocessing the local image before uploading it to the MinIO cluster, and / or sending an image retrieval request to the MinIO cluster. The local cache layer is used to store hot image data and preload images that the user wants to view; The gateway is used to perform access control and encryption / decryption of interactive data between the client and the front-end. The gateway deploys IDS and WAF for intrusion prevention and web security monitoring, and deploys Nginx for HTTP reverse proxy from the Internet to the private cloud, defining a global policy gateway that is completely independent of specific backend business applications and services. Traffic gateways focus on global API management strategies; The front-end is deployed in a cloud data center and is used for data interaction between the gateway and the MinIO cluster; it is also used for image renaming, managing image ownership, and recording image URLs.

2. The image viewing system based on MinIO distributed storage on a mobile terminal APP as described in claim 1, characterized in that, The metadata includes the image name, the hash value of the image name, the size, the creation time, the storage location, and the image type.

3. The image viewing system based on MinIO distributed storage on a mobile terminal APP as described in claim 1, characterized in that, When uploading an image, MinIO automatically generates an object URL that points directly to the image stored in MinIO.

4. The image viewing system based on MinIO distributed storage on a mobile terminal APP as described in claim 3, characterized in that, When reading an image, the MinIO server receives a preview request, finds the corresponding file based on the information in the URL, and performs preview processing. During the preview processing, the server calls an image processing library to generate a thumbnail. After the preview processing is complete, the server returns the preview result to the client. After receiving the preview result, the client displays it in the browser for the user to view.

5. The image viewing system based on MinIO distributed storage on a mobile terminal APP as described in claim 1, characterized in that, The MinIO cluster further includes a data protection module, which segments the data into fragments, expands and encodes redundant data blocks, stores them in different locations, and can quickly recover in the event of a failure, ensuring that no data is lost.

6. A method for viewing images in a MinIO-based distributed storage image viewing system on a mobile terminal APP, based on any one of claims 1 to 5, characterized in that, include: Step S1: The client APP sends an image retrieval request; Step S2: The client searches the local cache layer; If the local cache layer finds the image, it returns the data; otherwise, the client sends an image viewing request to the MinIO cluster that has pre-stored the image data through the gateway. Step S3: The load balancer of the MinIO cluster distributes each image retrieval request to one of the MinIO servers. The metadata management module collects the metadata of the image scattered on each disk according to the index key and performs preview processing. During the preview processing, the server calls the image processing library to generate thumbnails. Step S4: The MinIO server sends the returned image data to the client's browser through the gateway for the user to view.

7. The image viewing method as described in claim 6, characterized in that, In step S4, when reading images, the gateway controls the bandwidth and concurrency of data transmission based on the client's network conditions and request priority.

8. The image viewing method as described in claim 6, characterized in that, Methods for pre-storing image data in a MinIO cluster include: Step W1: The client APP preprocesses the local images to be uploaded to the MinIO cluster; Step W2: The gateway encrypts and decrypts the preprocessed image data; Step W3: The MinIO metadata management module adopts a hash-based distributed storage strategy. It hashes the image name and then distributes the image data evenly across multiple server nodes to ensure balanced data storage.

9. The image viewing method as described in claim 8, characterized in that, In step W1, the method by which the client APP preprocesses local images includes compression and format conversion.

10. The image viewing method as described in claim 8, characterized in that, Further steps include W4: The MinIO cluster near-synchronization mechanism backs up image data and can quickly recover in the event of a failure, ensuring that no data is lost.

Images