Image storage gateway system supporting near data processing

Abstract

The invention discloses an image storage gateway system supporting near data processing, and relates to the technical field of data storage. The system comprises a safety management module, an adaptation module and a plurality of adapters. The security management module comprises a transmission encryption unit, a calling authentication unit, a mapping conversion unit and a parameter verification unit; the adaptation module comprises an adapter management module, a mapping engine module, an algorithm management module, a calculation engine module, a data interface module, a data metering module and a load calculation module; and the adapter is used for converting the data format of the image to realize communication between different networks. According to the invention, encryption, authentication, mapping conversion and verification are carried out on the to-be-stored image, and whether the image has storage qualification and legality is judged, so that the concealment of data protection of the image storage gateway is ensured, and the security level of the system is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of data storage, and in particular relates to an image storage gateway system supporting near data processing. Background Art

[0002] With the development of information technologies such as the Internet, cloud computing, and big data, more and more applications rely on the storage and processing of massive data, such as intelligent monitoring, e-commerce, geographic information, etc. These applications all require the storage and retrieval of massive images. Many high-performance computing applications require a large amount of data transmission between computing nodes and storage nodes, and the explosive growth of image data has increased the overhead of data movement, reducing system performance. Summary of the invention

[0003] The purpose of the present invention is to provide an image storage gateway system that supports near data processing. By encrypting, authenticating, mapping, converting and verifying the images to be stored, and judging the transmission and storage of the images through load calculation, the problem of low efficiency of existing image data transmission and reduced system performance is solved.

[0004] In order to solve the above technical problems, the present invention is achieved through the following technical solutions:

[0005] The present invention is an image storage gateway system supporting near data processing, comprising a security management module, an adapter module, and multiple adapters;

[0006] The security management module includes a transmission encryption unit, a call authentication unit, a mapping conversion unit and a parameter verification unit; the transmission encryption unit is used to encrypt the input and output image data using RSA and SM2; the call authentication unit is used to determine whether the user has the right to store the image; the mapping conversion unit is used to configure access rules on the gateway system to realize request mapping and conversion; the parameter verification unit is used to verify the legitimacy of the parameters;

[0007] The adapter module includes an adapter management module, a mapping engine module, an algorithm management module, a computing engine module, a data interface module, a data metering module and a load calculation module; the adapter management module is used to configure different storage platform clients; the mapping engine is used to synchronize algorithm parameters between different storage platforms; the algorithm management module is used to store and interconnect parameters between heterogeneous storage platforms based on the same algorithm; the computing engine module is used to process input images and input results; the data interface module is used to; the data metering module is used to meter the stored image data; the load calculation module is used to calculate the load of the image storage gateway;

[0008] The adapter is used to convert the data format of the image to achieve communication between different networks.

[0009] As a preferred technical solution, the specific working process of the transmission encryption unit is as follows:

[0010] Step S1, parameter and file check: check the input image path and output path to confirm that the file exists and the output path is valid;

[0011] Step S2, image processing: deleting the existing encrypted image and randomly generating an AES key for encrypting the image data;

[0012] Step S3, generate encrypted file header: create a file header including a random AES key, original image length and hash value;

[0013] Step S4, AES encryption of image data: using the AES key to encrypt the original image file, and appending the encrypted data to the output file containing the file header;

[0014] Step S5, processing the image file: adding the padding data to the encrypted data;

[0015] Step S6, position search and data insertion: find a position in the image suitable for inserting encrypted data;

[0016] Step S7, image copying and cleaning: copying the modified image to the final path output.

[0017] As a preferred technical solution, the call authentication unit distributes keys to the docked system through a storage gateway. The API caller generates a signature through a key pair and uses the signature to make a request. After receiving the request, the API gateway verifies the signature. If the signatures are consistent, storage is allowed, otherwise the request is rejected.

[0018] As a preferred technical solution, the mapping conversion unit is configured by the developer on the image storage gateway, and the development program for actually implementing the business is hidden from the developer in the external request and the actual backend request.

[0019] As a preferred technical solution, the adapter management module includes a node management unit and a resource management unit; the node management unit is used to manage nodes of different storage platforms, and supports the creation, editing, deletion, display and search query of storage platform nodes; the storage platform node information includes name information, description information, port information and platform information; the resource management unit is used to realize the management of various resources between storage platforms.

[0020] As a preferred technical solution, the load calculation formula of the load calculation module for the image storage gateway is as follows:

[0021]

[0022] In the formula, φ(t) represents the exponentially weighted moving average, t represents time, Y(t) represents the gateway performance measurement value, n represents the collection time, α represents the weight coefficient of the historical measurement value, E represents the load parameter, and E n Indicates the load parameter synthesis.

[0023] As a preferred technical solution, the weight ratio of the image storage gateway is calculated as follows:

[0024]

[0025] In the formula, R is the weight ratio of the image storage gateway, k is the weight coefficient of the image storage gateway, and F t is the traffic of the image storage gateway at time t, and D is the distance from the network attack source traffic passing through multiple image storage gateways to the attacked one.

[0026] As a preferred technical solution, the image storage gateway is also provided with a data cache library; the data cache library uses a high-speed storage medium, and the image data will be stored in the data cache library when passing through the image storage gateway. After the image data is sent, the data cache library will delete the image data to release buffer space.

[0027] The present invention has the following beneficial effects:

[0028] (1) The present invention encrypts, authenticates, maps, converts and verifies the images to be stored, determines whether the images are eligible for storage and their legality, ensures the confidentiality of data protection of the image storage gateway, and improves the security level of the system.

[0029] (2) The present invention uses an adapter to adjust the configuration between different storage platforms, synchronize the algorithm parameters between the platforms, calculate the load of the gateway, and determine the transmission and storage efficiency of the image through the load calculation. At the same time, it cleans up the image data after transmission to release buffer space and improve the transmission efficiency.

[0030] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0032] Figure 1 A schematic diagram of the structure of an image storage gateway system supporting near data processing according to the present invention;

[0033] Figure 2 This is the specific workflow diagram of the transmission encryption unit. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0036] In order to make the purpose, technical solutions and advantages of this application more clear, the following Figure 1 It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0037] See also Figure 1 As shown, the present invention is an image storage gateway system supporting near data processing, including a security management module, an adapter module, and multiple adapters;

[0038] The security management module includes a transmission encryption unit, a call authentication unit, a mapping conversion unit and a parameter verification unit; the transmission encryption unit is used to encrypt the input and output image data using RSA and SM2; the call authentication unit is used to determine whether the user has the right to store images; the mapping conversion unit is used to configure access rules on the gateway system to realize request mapping and conversion; the parameter verification unit is used to verify the legitimacy of the parameters;

[0039] The adapter module includes an adapter management module, a mapping engine module, an algorithm management module, a computing engine module, a data interface module, a data metering module and a load calculation module; the adapter management module is used to configure different storage platform clients; the mapping engine is used to synchronize algorithm parameters between different storage platforms; the algorithm management module is used to store and interconnect parameters between heterogeneous storage platforms based on the same algorithm; the computing engine module is used to process the input images and input the results; the data interface module is used to; the data metering module is used to meter the stored image data; the load calculation module is used to calculate the load of the image storage gateway;

[0040] The adapter is used to convert the data format of the image to achieve communication between different networks.

[0041] The JPEG format is a widely used image format that supports efficient image compression. A JPEG file consists of multiple parts, including a file header, image data, and optional metadata segments (such as EXIF ​​information); the JPEG standard also allows custom comment segments to be added to the file, which usually do not affect the display of the image.

[0042] The encrypted data can be embedded in several different locations within a JPEG image, such as a custom comment segment or other non-image data. This embedding method takes advantage of the flexibility of the JPEG format, allowing additional information to be stored without interfering with the image itself.

[0043] When encryption data is embedded in a JPEG file, it is stored separately from the image content itself and has no effect on normal viewing of the image. The JPEG image looks the same as before encryption because the encryption data is stored in non-visual parts of the image, such as the metadata area or comment section of the file.

[0044] Concealment and security of encrypted data: By embedding encrypted data into JPEG files, a covert encryption method is achieved. This method not only protects the security of data, but also makes the existence of encrypted data difficult to notice. For most users, they may not be able to recognize that JPEG images contain encrypted data, thus increasing the concealment of data protection.

[0045] Furthermore, even if someone noticed that the size or structure of a JPEG file was different from a normal JPEG image, without the correct key, there would be no way to decrypt or understand the embedded data.

[0046] See also Figure 2 As shown, the specific workflow of the transmission encryption unit is as follows:

[0047] Step S1, parameter and file check: check the input image path and output path to confirm that the file exists and the output path is valid;

[0048] Step S2, image processing: deleting the existing encrypted image and randomly generating an AES key for encrypting the image data;

[0049] Step S3, generate encrypted file header: create a file header including a random AES key, original image length and hash value;

[0050] Step S4, AES encryption of image data: using the AES key to encrypt the original image file, and appending the encrypted data to the output file containing the file header;

[0051] Step S5, processing the image file: adding the padding data to the encrypted data;

[0052] Step S6, position search and data insertion: find a position in the image suitable for inserting encrypted data;

[0053] Step S7, image copying and cleaning: copying the modified image to the final path output.

[0054] The authentication unit is called to distribute keys to the docked system through the storage gateway. The API caller generates a signature through the key pair and uses the signature to make a request. After receiving the request, the API gateway verifies the signature. If the signatures are consistent, storage is allowed, otherwise the request is rejected. Authentication refers to verifying whether the user has the right to access the business system, and is also the most common way to protect the security of the gateway interface. The interfaces registered in the gateway are authenticated and authorized to achieve business scenarios with fine-grained permission management.

[0055] The mapping conversion unit is configured by the developer on the image storage gateway. In external requests and actual backend requests, the developer's development program for actually implementing the business is hidden, avoiding security risks caused by the client directly requesting the business backend.

[0056] The parameter verification unit ensures the security and robustness of the system. Data verification is essential. By verifying the legitimacy of the parameters, we can avoid the problem of business system errors caused by abnormal transmission parameters. The image storage gateway supports configuring verification rules for request parameters. If the developer passes image parameters that do not comply with the rules, the request will be intercepted by the image storage gateway and will not be forwarded to the backend.

[0057] The adapter management module includes a node management unit and a resource management unit; the node management unit is used to manage nodes of different storage platforms, and supports the creation, editing, deletion, display and search query of storage platform nodes; the storage platform node information includes name information, description information, port information and platform information; the resource management unit is used to realize the management of various resources between storage platforms.

[0058] The load calculation module is used to ensure that the gateway meets the execution conditions of the attack blocking strategy. The load calculation formula for the image storage gateway is as follows:

[0059]

[0060] In the formula, φ(t) represents the exponentially weighted moving average, t represents time, Y(t) represents the gateway performance measurement value, n represents the collection time, α represents the weight coefficient of the historical measurement value, E represents the load parameter, and E n Indicates the load parameter synthesis.

[0061] The network load of the gateway is high in a certain period of time, but the blocking weight calculation result is low, which will cause errors in the entire calculation result. This will cause the network load to be too high and affect the network computing performance. Calculating the historical weight parameters through the exponentially weighted moving average algorithm can reduce the error rate. The calculation can obtain the load situation, and the gateway's own performance and load parameters determine whether the gateway can issue policy instructions. The weight ratio of the image storage gateway is determined according to the actual gateway traffic. If the gateway weight value obtained is smaller, it is more suitable for executing network attack blocking policy instructions.

[0062] The weight ratio of the image storage gateway is calculated as follows:

[0063]

[0064] In the formula, R is the weight ratio of the image storage gateway, k is the weight coefficient of the image storage gateway, and F t is the traffic of the image storage gateway at time t, and D is the distance from the network attack source traffic passing through multiple image storage gateways to the attacked one.

[0065] The image storage gateway is also equipped with a data cache library; the data cache library uses high-speed storage media, and the image data will be stored in the data cache library when passing through the image storage gateway. After the image data is sent, the data cache library will delete the image data to release buffer space.

[0066] It is worth noting that in the above system embodiment, the various units included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0067] In addition, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiments can be completed by instructing related hardware through a program, and the corresponding program can be stored in a computer-readable storage medium.

[0068] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. An image storage gateway system supporting near data processing, comprising a security management module, an adapter module, and multiple adapters, characterized in that: The security management module includes a transmission encryption unit, a call authentication unit, a mapping conversion unit and a parameter verification unit; the transmission encryption unit is used to encrypt the input and output image data using RSA and SM2; the call authentication unit is used to determine whether the user has the right to store the image; the mapping conversion unit is used to configure access rules on the gateway system to realize request mapping and conversion; the parameter verification unit is used to verify the legitimacy of the parameters; The adaptation module includes an adapter management module, a mapping engine module, an algorithm management module, a calculation engine module, a data interface module, a data metering module and a load calculation module; The adapter management module is used to configure different storage platform clients; the mapping engine is used to synchronize algorithm parameters between different storage platforms; The algorithm management module is used for parameter storage and interconnection between heterogeneous storage platforms according to the same algorithm; the computing engine module is used for processing the input images and inputting the results; The data interface module is used for: the data metering module is used for metering the stored image data; the load calculation module is used for calculating the load of the image storage gateway; The adapter is used to convert the data format of the image to achieve communication between different networks.

2. The image storage gateway system supporting near data processing according to claim 1, characterized in that: The specific working process of the transmission encryption unit is as follows: Step S1, parameter and file check: check the input image path and output path to confirm that the file exists and the output path is valid; Step S2, image processing: deleting the existing encrypted image and randomly generating an AES key for encrypting the image data; Step S3, generate encrypted file header: create a file header including a random AES key, original image length and hash value; Step S4, AES encryption of image data: using the AES key to encrypt the original image file, and appending the encrypted data to the output file containing the file header; Step S5, processing the image file: adding the padding data to the encrypted data; Step S6, position search and data insertion: find a position in the image suitable for inserting encrypted data; Step S7, image copying and cleaning: copying the modified image to the final path output.

3. The image storage gateway system supporting near data processing according to claim 1, characterized in that: The call authentication unit distributes the key to the docked system through the storage gateway. The API caller generates a signature through the key pair and uses the signature to make a request. After receiving the request, the API gateway verifies the signature. If the signatures are consistent, storage is allowed, otherwise the request is rejected.

4. The image storage gateway system supporting near data processing according to claim 1, characterized in that: The mapping conversion unit is configured by the developer on the image storage gateway, and is hidden from the developer's development program for actually implementing the business in external requests and actual backend requests.

5. The image storage gateway system supporting near data processing according to claim 1, characterized in that: The adapter management module includes a node management unit and a resource management unit; The node management unit is used to manage nodes of different storage platforms and supports the creation, editing, deletion, display and search query of storage platform nodes; the storage platform node information includes name information, description information, port information and platform information; The resource management unit is used to manage various types of resources between storage platforms.

6. The image storage gateway system supporting near data processing according to claim 1, characterized in that: The load calculation formula of the load calculation module for the image storage gateway is as follows: In the formula, φ(t) represents the exponentially weighted moving average, t represents time, Y(t) represents the gateway performance measurement value, n represents the collection time, α represents the weight coefficient of the historical measurement value, E represents the load parameter, and E n Indicates the load parameter synthesis.

7. The image storage gateway system supporting near data processing according to claim 6, characterized in that: The weight ratio of the image storage gateway is calculated as follows: In the formula, R is the weight ratio of the image storage gateway, k is the weight coefficient of the image storage gateway, and F t is the traffic of the image storage gateway at time t, and D is the distance from the network attack source traffic passing through multiple image storage gateways to the attacked target.

8. The image storage gateway system supporting near data processing according to claim 1, characterized in that: The image storage gateway is also provided with a data cache library; the data cache library uses a high-speed storage medium, and the picture data will be stored in the data cache library when passing through the image storage gateway. After the image data is sent, the data cache library will delete the image data to release buffer space.

Images