Distributed thumbnail hierarchical storage method and system based on imgproxy and distributed thumbnail hierarchical acquisition method and system based on imgproxy
Through the distributed thumbnail hierarchical storage and acquisition method based on imgproxy, load balancing and hierarchical storage services are used to quickly generate and access thumbnails without saving the original picture, solving the risk of large-scale storage resource occupation and resource leakage Problem, improving development efficiency and response speed.
Patent Information
- Application Number
- CN202410896469.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-05
- Publication Date
- 2024-12-06
AI Technical Summary
The prior art usually needs to save the original picture when generating thumbnails, resulting in large consumption of storage resources, increasing operation and maintenance costs, and there is a risk of resource leakage. At the same time, it cannot meet certain business conditions and requires changing the project source code.
Using the distributed thumbnail hierarchical storage and acquisition method based on imgproxy, the load balancing service and hierarchical storage service can realize the rapid online generation and access of thumbnails that only save the original image. The imgproxy server is used to adjust and convert online images, combining RAM, SSD and HHD-level hierarchical storage enables fast access and cache acceleration.
It realizes that without intruding into the project source code, reduces the risk of resource leakage, improves project development efficiency, and speeds up the response speed of thumbnail resources through hierarchical storage and reduces the load pressure of imgproxy services.
Smart Images

Figure CN119089060A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of IT and software development technology, and specifically relates to a distributed thumbnail hierarchical storage and acquisition method and system based on imgproxy Background Art
[0002] Currently, most technologies provide thumbnails to speed up access when the original image is too large. Figure 1 The general practice is to convert the image into a thumbnail and save it when uploading it, discarding the original image, or saving the original image and thumbnail at the same time. The problems with these methods are that some will cause the loss of the original image, some images have a fixed size and format, and cannot meet certain business conditions; they occupy a large amount of storage resources, or use external resources and external services, which increases the risk of operation and maintenance costs and resource leakage, and requires changes to the project source code. Summary of the invention
[0003] In view of the existing technical problems, the present invention provides a distributed thumbnail hierarchical storage and acquisition method based on imgproxy, which can realize the rapid generation of thumbnails of original images online and real-time access while only saving the original images, and can be locally deployed independently without invading the project source code and requiring no major modifications, thereby achieving rapid access to images, reducing storage space occupancy, reducing the risk of resource leakage, and improving project development efficiency.
[0004] The technical solution of the present invention is as follows: The distributed thumbnail hierarchical storage and acquisition method based on imgproxy includes the following steps: 1) The client requests to access image resources, and the request reaches the load balancing service module; 2) The load balancing service module determines whether the current request has been cached. If the resource has been cached, it will access the hierarchical storage service module to obtain the cached resource and return it. If it has not been cached, it will proceed to the next step; 3) The load balancing service module reverse proxies the request to the corresponding imgproxy server according to the load balancing strategy; 4) The imgproxy server accesses the image resource server where the original image is located according to the original image address in the request; 5) The image resource server returns the original image to the imgproxy server; 6) The imgproxy server processes the original image based on the acquired original image and the image processing parameters or preset values in the request URL to obtain a thumbnail, and returns it to the load balancing service module; 7) After the load balancing service module obtains the thumbnail, it sends the resource to the hierarchical storage service module for caching and returns the thumbnail to the client; 8) The client obtains the thumbnail.
[0005] To further explain the present invention, when the client initiates a request to access an image resource and the load balancing service module determines that the requested resource is not cached, the request is forwarded to the imgproxy server according to the load balancing strategy; the imgproxy server obtains the original image according to the original image address carried in the request, crops it into a thumbnail of a specified size according to the request parameters, and returns the processed thumbnail to the load balancing service module; the load balancing service module determines that the thumbnail resource responded to is not cached, and while returning the thumbnail to the front end, sends the resource to the hierarchical storage service module cache; the hierarchical storage service module caches the thumbnail resource to the specified storage type according to the hierarchical strategy.
[0006] To further explain the present invention, when a client initiates a request to access an image resource and the load balancing service module determines that the requested resource has been cached, it requests the cached thumbnail resource from the hierarchical storage service module; the hierarchical storage service module determines the level of storage the current resource is in based on the hierarchical storage strategy, takes out the resource from the corresponding level of storage space, returns it to the load balancing service module, and adjusts the storage level of the resource based on the hierarchical storage strategy and migrates it to the corresponding location; after obtaining the resource, the load balancing service module determines that the resource has been cached and directly returns the resource to the front end.
[0007] To further explain the present invention, the hierarchical storage service module is used to cache thumbnail resources. When a cache exists, directly obtaining them from the cache can speed up the response speed, including RAM level, SSD level and HHD level. By registering each request for thumbnail resources, the resources are stored hierarchically at each level according to different parameters.
[0008] To further explain the present invention, the load balancing service module requests to enter the hierarchical cache, first determining whether the current request is a new resource request. If it is a new resource, the resource is cached into the RAM level cache; if it is a cached resource request, the level of the requested resource is determined. If it is RAM level storage, the resource is directly obtained from the storage and returned to the load balancing service module; if it is SSD or HHD level storage, while returning the resource, the priority of the current resource is adjusted according to the storage policy and migrated to the corresponding storage location.
[0009] To further explain the present invention, the hierarchical storage will perform periodic polling to determine whether all resources match the corresponding storage level based on the storage policy. If there are mismatched resources, they will be allocated to the specified storage level based on the storage policy. If new resources are placed in RAM-level storage, if the response frequency does not meet the standard within a certain period of time, they will be migrated to SSD-level storage. If the SSD-level resources are large in size and have a low response frequency for a long time, they will be migrated to HHD-level storage.
[0010] Further explanation of the present invention, the use of hash algorithm in the load balancing service and the hierarchical storage service can improve the balance of load balancing and the hit rate of cache, and can also facilitate unified management; the configuration algorithm is as follows: ( $request_uri ~* "^ / proxy / (.*) / (.*)" ); According to the different parameters after / proxy / ** in the URL, the hash is performed to balance the traffic to different imgproxy servers. It is also used to set the cache key of the hierarchical storage service to distinguish the caches of different imgproxy servers. In this way, the proxy of an imgproxy server corresponds to different cache hash key spaces, which is convenient for managing and refreshing the cache. Hash key: The hash key value customized by the load balancing service is used to hit the corresponding server when the load balancing strategy is used and is used as the key for storage in the hierarchical storage service.
[0011] To further illustrate the present invention, the process of converting the original image to a thumbnail is as follows: adding the parameter resize:auto:400:400:0 / plain / {image address}@png in the access connection, the imgproxy server obtains the original image according to the requested image address, and then crops the original image into a png image with a resolution of 400*400 according to parameters such as auto:400:400:0 and @png, and returns a response, thereby achieving the purpose of converting the original image into a thumbnail online.
[0012] As a result, a more reasonable and efficient online thumbnail acquisition and cache acceleration service can be achieved. Compared with ordinary methods, the load pressure on imgproxy is greatly reduced, and the response speed of thumbnail resources can be accelerated through hierarchical storage services.
[0013] A distributed thumbnail hierarchical storage and acquisition system based on imgproxy, comprising: imgproxy server: for resizing and converting online images; Load balancing service module: used to provide load balancing services to smooth out peak traffic and fill valleys, so as to balance the traffic and pressure of each imgproxy service; at the same time, it determines whether the requested resources have been cached in the hierarchical storage service through strategies, and directly obtains the cached resources from the hierarchical storage service to speed up the resource response speed. At the same time, it can also significantly reduce the pressure on the imgproxy service and achieve the purpose of quickly accessing the original image thumbnails.
[0014] Hierarchical storage service module: used to store data on appropriate storage devices according to storage resources such as resource size and access frequency; Image resource server: used to store original images.
[0015] Beneficial effects of the present invention: Through this method, it is possible to save only the original image, quickly generate the thumbnail of the original image online and access it in real time, and it can be deployed locally and independently without invading the project source code and making many modifications, so as to achieve fast access to images, reduce storage space usage, reduce the risk of resource leakage, and improve project development efficiency. By using the characteristics of imgproxy to adjust and convert online images, it supports online parameter transfer to adjust the image size and format, and can flexibly obtain the required image format on the front end to expand the scope of application.
[0016] The load balancing service is used to balance the traffic and pressure of each imgproxy service. At the same time, the policy is used to determine whether the requested resource has been cached in the hierarchical storage service, and the cached resource is directly obtained from the hierarchical storage service to speed up the resource response speed. At the same time, it can also significantly reduce the pressure on the imgproxy service and achieve the purpose of quickly accessing the original image thumbnail. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION
[0018] The present invention will be further described below in conjunction with specific embodiments. Example
[0019] The distributed thumbnail hierarchical storage and acquisition method based on imgproxy includes the following steps: 1) The client requests to access image resources, and the request reaches the load balancing service module; 2) The load balancing service module determines whether the current request has been cached. If the resource has been cached, it will access the hierarchical storage service module to obtain the cached resource and return it. If it has not been cached, it will proceed to the next step; 3) The load balancing service module reverse proxies the request to the corresponding imgproxy server according to the load balancing strategy; 4) The imgproxy server accesses the image resource server where the original image is located according to the original image address in the request; 5) The image resource server returns the original image to the imgproxy server; 6) The imgproxy server processes the original image based on the acquired original image and the image processing parameters or preset values in the request URL to obtain a thumbnail, and returns it to the load balancing service module; 7) After the load balancing service module obtains the thumbnail, it sends the resource to the hierarchical storage service module for caching and returns the thumbnail to the client; 8) The client obtains the thumbnail.
[0020] When the client initiates a request to access image resources, and the load balancing service module determines that the requested resource is not cached, it forwards the request to the imgproxy server according to the load balancing strategy; the imgproxy server obtains the original image according to the original image address carried in the request, crops it into a thumbnail of the specified size according to the request parameters, and returns the processed thumbnail to the load balancing service module; the load balancing service module determines that the thumbnail resource responded to is not cached, and returns the thumbnail to the front end while sending the resource to the hierarchical storage service module cache; the hierarchical storage service module caches the thumbnail resource to the specified storage type according to the hierarchical strategy.
[0021] When the client initiates a request to access image resources and the load balancing service module determines that the requested resource has been cached, it requests the cached thumbnail resource from the hierarchical storage service module; the hierarchical storage service module determines the storage level of the current resource based on the hierarchical storage strategy, takes out the resource from the corresponding level of storage space, returns it to the load balancing service module, and adjusts the storage level of this resource according to the hierarchical storage strategy and migrates it to the corresponding location; after obtaining the resource, the load balancing service module determines that this resource has been cached and directly returns the resource to the front end.
[0022] The hierarchical storage service module is used to cache thumbnail resources. If there is a cache, directly obtaining from the cache can speed up the response speed, including RAM level, SSD level and HHD level. By registering each request for thumbnail resources, resources are hierarchically stored in various levels according to different parameters. Resources with many requests and small volumes will be stored at the RAM level; resources with medium requests and medium volumes will be stored at the SSD level; resources with fewer requests and larger volumes will be stored at the HHD level.
[0023] RAM level is memory type storage, with the fastest read and write speeds. It caches resources in memory and is suitable for high-frequency, small-volume resources, but power outages can cause resource loss. SSD level is solid-state hard disk type storage, with fast read and write speeds and the ability to persist data, but the unit storage space cost is high. HHD level is mechanical hard disk type storage, with average read and write speeds, the ability to persist data, low unit storage space cost, and low risk of data loss.
[0024] The load balancing service module requests to enter the hierarchical cache, first determining whether the current request is a new resource request. If it is a new resource, the resource is cached into the RAM level cache; if it is a cached resource request, the level of the requested resource is determined. If it is RAM level storage, the resource is directly obtained from the storage and returned to the load balancing service module; if it is SSD or HHD level storage, while returning the resource, the priority of the current resource is adjusted according to the storage policy and migrated to the corresponding storage location.
[0025] The hierarchical storage will perform periodic polling to determine whether all resources match the corresponding storage level based on the storage policy. If there are mismatched resources, they will be allocated to the specified storage level based on the storage policy. If new resources are placed in RAM-level storage and the response frequency does not meet the standard within a certain period of time, they will be migrated to SSD-level storage. If the SSD-level resources are large in size and have a low response frequency for a long time, they will be migrated to HHD-level storage.
[0026] The use of hash algorithms in the load balancing service and the hierarchical storage service can improve the balance of load balancing and the hit rate of cache, and can also facilitate unified management; the configuration algorithm is as follows: ( $request_uri ~* "^ / proxy / (.*) / (.*)" ); According to the different parameters after URL / proxy / **, the traffic is balanced to different imgproxy servers. It is also used to set the cache key of the hierarchical storage service to distinguish the caches of different imgproxy servers. In this way, the proxy of an imgproxy server corresponds to different cache hash key spaces, which is convenient for managing and refreshing the cache. Hash key: The hash key value customized by the load balancing service is used to hit the corresponding server when the load balancing strategy is used and is used as the key for storage in the hierarchical storage service.
[0027] The process of converting the original image to a thumbnail is as follows: adding the parameter resize:auto:400:400:0 / plain / {image address}@png in the access connection, the imgproxy server obtains the original image according to the requested image address, and then crops the original image into a png image with a resolution of 400*400 according to the parameters such as auto:400:400:0 and @png, and returns a response, thereby achieving the purpose of converting the original image into a thumbnail online.
[0028] As a result, a more reasonable and efficient online thumbnail acquisition and cache acceleration service can be achieved. Compared with ordinary methods, the load pressure on imgproxy is greatly reduced, and the response speed of thumbnail resources can be accelerated through hierarchical storage services.
[0029] A distributed thumbnail hierarchical storage and acquisition system based on imgproxy, comprising: imgproxy server: for resizing and converting online images; Load balancing service module: used to provide load balancing services to smooth out peak traffic and fill valleys, so as to balance the traffic and pressure of each imgproxy service; at the same time, it determines whether the requested resources have been cached in the hierarchical storage service through strategies, and directly obtains the cached resources from the hierarchical storage service to speed up the resource response speed. At the same time, it can also significantly reduce the pressure on the imgproxy service and achieve the purpose of quickly accessing the original image thumbnails.
[0030] Hierarchical storage service module: used to store data on appropriate storage devices according to storage resources such as resource size and access frequency; Image resource server: used to store original images.
Claims
1. A distributed thumbnail hierarchical storage and acquisition method based on imgproxy, characterized in that: The following steps are involved: 1) The client requests to access image resources, and the request reaches the load balancing service module; 2) The load balancing service module determines whether the current request has been cached. If the resource has been cached, it will access the hierarchical storage service module to obtain the cached resource and return it. If it has not been cached, it will proceed to the next step; 3) The load balancing service module reverse proxies the request to the corresponding imgproxy server according to the load balancing strategy; 4) The imgproxy server accesses the image resource server where the original image is located according to the original image address in the request; 5) The image resource server returns the original image to the imgproxy server; 6) The imgproxy server processes the original image according to the obtained original image, adds the request URL fragment processing parameter or the preset value, and obtains a thumbnail, and returns it to the load balancing service module; 7) After the load balancing service module obtains the thumbnail, it sends the resource to the hierarchical storage service module for caching and returns the thumbnail to the client; 8) The client obtains the thumbnail.
2. The distributed thumbnail hierarchical storage and acquisition method based on imgproxy according to claim 1 is characterized in that: When a client initiates a request to access an image resource, and the load balancing service module determines that the requested resource is not cached, it forwards the request to the imgproxy server according to the load balancing strategy; The imgproxy server obtains the original image according to the original image address carried in the request, crops it into a thumbnail of the specified size according to the request parameters, and returns the processed thumbnail to the load balancing service module; The load balancing service module determines that the thumbnail resource responded to is not cached, returns the thumbnail to the front end, and sends the resource to the hierarchical storage service module cache; The hierarchical storage service module caches thumbnail resources to the specified storage type according to the hierarchical strategy.
3. The distributed thumbnail hierarchical storage and acquisition method based on imgproxy according to claim 1 is characterized in that: When the client initiates a request to access image resources, and the load balancing service module determines that the requested resource has been cached, it requests the cached thumbnail resource from the hierarchical storage service module; The hierarchical storage service module determines which level of storage the current resource is in according to the hierarchical storage strategy, takes out the resource from the corresponding level of storage space, returns it to the load balancing service module, and adjusts the storage level of this resource according to the hierarchical storage strategy and migrates it to the corresponding location; After the load balancing service module obtains the resource, it determines that the resource has been cached and directly returns the resource to the front end.
4. The distributed thumbnail hierarchical storage and acquisition method based on imgproxy according to claim 1 is characterized in that: The hierarchical storage service module is used to cache thumbnail resources. When there is a cache, directly obtaining them from the cache can speed up the response speed, including RAM level, SSD level and HHD level. By registering each request for thumbnail resources, the resources are hierarchically stored at each level according to different parameters.
5. The distributed thumbnail hierarchical storage and acquisition method based on imgproxy according to claim 4 is characterized in that: The load balancing service module requests to enter the hierarchical cache, first determine whether the current request is a new resource request, if it is a new resource, cache the resource into the RAM level cache; if it is a cached resource request, determine the level of the requested resource, if it is RAM level storage, directly obtain the resource from the storage and return it to the load balancing service module; If it is SSD or HHD level storage, while returning the resources, the priority of the current resources will be adjusted according to the storage policy and migrated to the corresponding storage location.
6. The distributed thumbnail hierarchical storage and acquisition method based on imgproxy according to claim 5 is characterized in that: The hierarchical storage will perform periodic polling to determine whether all resources match the corresponding storage level based on the storage policy. If there are mismatched resources, they will be allocated to the specified storage level based on the storage policy. If new resources are placed in RAM-level storage and the response frequency does not meet the standard within a certain period of time, they will be migrated to SSD-level storage. If the SSD-level resources are large in size and have a low response frequency for a long time, they will be migrated to HHD-level storage.
7. The distributed thumbnail hierarchical storage and acquisition method based on imgproxy according to claim 1 is characterized in that: The use of hash algorithms in the load balancing service and the hierarchical storage service can improve the balance of load balancing and the hit rate of cache, and can also facilitate unified management; the configuration algorithm is as follows: ( $request_uri ~* "^ / proxy / (.*) / (.*)" ); Hash the parameters after / proxy / ** in the URL to balance the traffic to different imgproxy servers. It is also used to set the cache key of the hierarchical storage service to distinguish the caches of different imgproxy servers.
8. The distributed thumbnail hierarchical storage and acquisition method based on imgproxy according to claim 1 is characterized in that: The process of converting the original image to a thumbnail is as follows: adding the parameter resize:auto:400:400:0 / plain / {image address}@png in the access connection, the imgproxy server obtains the original image according to the requested image address, and then crops the original image into a png image with a resolution of 400*400 according to the parameters such as auto:400:400:0 and @png, and returns a response, thereby achieving the purpose of converting the original image into a thumbnail online.
9. A distributed thumbnail hierarchical storage and acquisition system based on imgproxy, characterized by: include: imgproxy server: for resizing and converting online images; Load balancing service module: used to provide load balancing services to smooth out peak traffic and fill valleys, so as to balance the traffic and pressure of each imgproxy service; at the same time, it determines whether the requested resources have been cached in the hierarchical storage service through strategies, and directly obtains the cached resources from the hierarchical storage service to speed up the resource response speed. At the same time, it can also significantly reduce the pressure on the imgproxy service and achieve the purpose of quickly accessing the original image thumbnails.
10. Hierarchical storage service module: used to store data on appropriate storage devices according to storage resources such as resource size and access frequency; Image resource server: used to store original images.
Citation Information
Patent Citations
Picture caching processing method and system, and server.
CN108055302A
Electronic device, image processing method and storage medium
CN108933814A
