A multi-modal data management method for computing power network

Abstract

This invention discloses a multimodal data management method for computing power networks. First, a multimodal similarity-preserving hash algorithm based on latent alignment is designed, using text modalities as intermediate modalities. Through adversarial networks, the common representation of input data and corresponding text modal data is learned to generate multimodal data labels. Then, semantic relevance-preserving labels are extracted from the multimodal data using computing power network nodes to ensure the association between cross-modal similarity data. Subsequently, semantic feature extraction based on generative adversarial networks is used to extract shared feature vectors and deduplicate multimodal data similarity, significantly saving storage space. Finally, data recovery is performed based on latent diffusion technology, using the computing power network to restore the semantic similarity from shared feature vectors to the original data. This method can significantly save storage space and transmission bandwidth, effectively maintaining data availability while improving the efficiency of multimodal data management.

Description

Technical Field

[0001] This invention relates to the field of network data management technology, and in particular to a multimodal data management method for computing power networks. Background Technology

[0002] With the continuous development of technologies such as the Internet and mobile communications, information technology has been integrated into social production and life. The diversified media characteristics and collaborative working modes of Internet of Things (IoT) devices have placed higher demands on data management capabilities and data processing speed. At the same time, in recent years, emerging technologies such as digital twins and virtual reality have gradually risen, highlighting the growing need for flexible data management and efficient computing power support.

[0003] Limited by factors such as power consumption and cost, existing data acquisition equipment is typically lightweight, resulting in limited storage and computing capabilities, posing challenges to local data management. Mainstream data management solutions effectively reduce the storage burden on data acquisition equipment by outsourcing the storage of collected data. Existing solutions often rely on cloud storage for synchronous management of massive amounts of data, but this centralized data management method generally suffers from limitations in scale and vulnerability to single points of failure, failing to meet the demands of large-scale data processing. Computing power, as the foundation of the information society, has become a fundamental and core driving force for the development of the digital economy, directly impacting its speed and determining the level of social intelligence development. Computing networks, as a new technology, can allocate and flexibly schedule computing and storage resources on demand across the cloud, network, and edge according to business needs, meeting the demands of large-scale data processing and computationally intensive tasks, providing efficient, secure, and flexible services to various industries. This distributed storage not only effectively resists single-point-of-failure attacks but also effectively compensates for the storage space limitations of cloud storage.

[0004] On the other hand, with the rapid development of multimedia devices, IoT data is exhibiting diversified characteristics. Various types of IoT sensors often work together, generating a large amount of cross-modal data containing the same semantic information. For example, in industrial IoT scenarios, monitoring videos and equipment operation logs may record the same operational information. The same information appears repeatedly across multiple modalities, and this cross-modal data with identical semantic information occupies a significant amount of storage space. Data deduplication techniques can effectively solve the data redundancy problem by retaining only one copy of the data and deleting other duplicate copies. However, because data in different modalities has different forms of representation and composition, existing data deduplication methods struggle to directly compare the similarity between different modalities, and can only operate on single-modal data, making them unsuitable for real-world scenarios with multimodal data.

[0005] Furthermore, since users lose physical control over their data during edge computing and cloud storage, data security is a fundamental user requirement. Encryption technology aims to provide semantic security for plaintext data, making ciphertext data indistinguishable, which increases the difficulty of data management. While methods such as key transmission can encrypt identical data into identical ciphertext, this still cannot meet the requirement of deduplication of similar data ciphertexts. Therefore, it is necessary to consider how to break down information barriers between users without transmitting keys, achieving secure multimodal data fusion management, while simultaneously analyzing the relationships between multimodal data and designing a universal encryption mechanism to ensure data relevance. Summary of the Invention

[0006] The problem to be solved by this invention is to provide a multimodal data management method for computing power networks. By analyzing the correlation between multimodal data, a general encryption mechanism to ensure data correlation is designed, thereby achieving secure multimodal data fusion management.

[0007] This invention adopts the following technical solution: a multimodal data management method for computing power networks, comprising the following steps:

[0008] S1. The data owner initiates a data upload request. The nearest computing network node calculates the shared feature vector and semantic relevance retention label of the multimodal data and uploads them to the storage server.

[0009] S2. The storage server retains tags based on the received semantic relevance, compares the tags with existing data tags, determines whether semantically similar multimodal data has been uploaded, and returns the tag comparison result; the data owner jointly calculates the encryption key based on the data tags and the tag comparison result;

[0010] S3. The data owner performs local cross-modal data deduplication based on the label comparison results, encrypts the shared feature vector, and uploads the deduplicated shared feature vector ciphertext to the storage server for storage management.

[0011] S4. When a data user initiates a data query request, the storage server verifies the user's ownership and returns the verification result to the data user.

[0012] S5. After the data accessor verifies ownership and obtains the ciphertext of the shared feature vector, the shared feature vector is decrypted, and the original data is recovered with the assistance of the nearest computing power network node to obtain the original data.

[0013] Preferably, in step S1, the computing network nodes calculate the shared feature vectors and semantic relevance preserved labels of the multimodal data, as follows:

[0014] S1.1 Generate arbitrary modal input data Corresponding text modality ,Will and As input to generative adversarial networks;

[0015] S1.2 The generative adversarial network uses the Unet network as the feature learning network to extract input data. Corresponding text modality Actual feature vectors and ;

[0016] S1.3, Extract the actual feature vector and As input, a generative adversarial network generator is used to generate pseudo-feature vectors that correspond to the real modalities. generate and generate ;

[0017] S1.4, respectively < , > and < , As input to the generative adversarial network discriminator, the Unet network parameters are adjusted based on the discrimination results. Steps S1.2 and S1.3 are repeated to generate shared feature vectors for similar data across modalities.

[0018] S1.5, until the discriminator determines. and For being similar, and and Similarly, the generative adversarial network learns and obtains input data. Shared feature vectors ,in ;

[0019] S1.6, Input data Shared feature vectors As input, calculate the semantic relevance preserved labels of the input data. .

[0020] Preferably, the method for determining the semantically similar multimodal data in step S2 is as follows: the storage server retains the received semantically related labels. , and the data labels of existing data w Perform Hamming distance calculation and compare the result with a predetermined threshold number. A comparison is made, and the calculated result is not greater than a predetermined threshold number. When that happens, the input data is judged. The data is similar to the existing data w.

[0021] Furthermore, the data owner jointly calculates the encryption key based on the data tag and the tag comparison result, as follows:

[0022] S2.1 If no similar data exists, the data owner uses the system public parameter Para and the data label. As input, the encryption key Key for the data is generated by the fuzz extractor:

[0023] ;

[0024] in, For system common parameters, As a key pair As a key auxiliary parameter;

[0025] S2.2 If similar data exists on the storage server The storage server returns key auxiliary parameters to the data owner. Users will and data tags As input, the key for data regeneration is performed:

[0026] ;

[0027] in, This is a key regeneration algorithm.

[0028] Preferably, in step S3, the data owner performs local cross-modal data deduplication based on the label comparison results and encrypts the shared feature vector, as follows:

[0029] S3.1 The data owner retains the label based on semantic relevance. Based on the comparison results of similar data, multimodal data deduplication is performed based on semantic relevance;

[0030] S3.2 If no similar data exists on the edge nodes, the data owner encrypts the shared feature vector based on the encryption key, and outputs the ciphertext of the shared feature vector. The data owner retains the data label locally. and key pair ciphertext sharing feature vectors Upload to the storage server for storage management;

[0031] S3.3 If similar data already exists on the edge node, the data owner performs local deduplication on the data and retains the data label. and key pair .

[0032] Preferably, in step S4, when a data access user initiates a data access request, the storage server verifies the ownership of the data access user, as follows:

[0033] S4.1 If the verification passes, the storage server uses the data tag to query the shared feature vector and returns the encrypted shared feature vector to the data accessor. ;

[0034] S4.2 If the verification fails, the storage server rejects the data access request from the user.

[0035] Preferably, in step S5, the data visitor obtains the raw data, as follows:

[0036] S5.1, The data owner uses the key pair For ciphertext with shared feature vectors Decrypt to obtain the plaintext of the shared feature vector. ;

[0037] S5.2 The data owner will decrypt the obtained shared feature vector plaintext. The data is sent to the nearest computing network node. The computing network node uses a potential diffusion model based on the UNet network to recover the original data from the feature vector and then sends the original data to the data visitor.

[0038] The present invention also provides: an electronic device, comprising:

[0039] One or more processors;

[0040] A storage device on which one or more programs are stored;

[0041] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-described multimodal data management methods for computing power networks.

[0042] The present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in any of the above-mentioned multimodal data management methods for computing power networks.

[0043] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0044] 1. The multimodal data management method of this invention relies on the designed multimodal data latent alignment technology to achieve semantic-based cross-modal data similarity comparison. By designing a multimodal similarity-preserving hash algorithm based on latent alignment, the text modality is used as an intermediate modality, and the input data and its corresponding text modality are subjected to adversarial network learning, thereby learning the common representation of the input data and its corresponding text modality data. This ensures that different modal data achieves a unified mapping from high-dimensional data modality to low-dimensional Hamming space, and generates multimodal data labels that preserve semantic relevance, thereby realizing unified retrieval of multimodal data.

[0045] 2. The multimodal data management method of this invention utilizes a computing network to extract semantically relevant tags from multimodal data. The parallel operation greatly satisfies the real-time requirements of IoT data. Furthermore, the distributed nature of the computing network fully preserves the interactivity and collaboration characteristics between IoT devices in real-world scenarios, further ensuring the correlation of cross-modal similarity data.

[0046] 3. The multimodal data management method of the present invention uses a multimodal data secure deduplication algorithm and a multimodal data semantic feature extraction model based on generative adversarial networks to achieve shared feature vector extraction based on semantic relevance and deduplication of multimodal data, thereby improving the efficiency of multimodal data management and saving storage space.

[0047] 4. The multimodal data management method of this invention, through a data recovery method based on latent diffusion technology, ensures the usability of cross-modal data after deduplication based on similarity, providing comprehensive data support for subsequent data analysis. This method can greatly save storage space and transmission bandwidth, effectively maintaining data availability while improving the efficiency of multimodal data management. Attached Figure Description

[0048] Figure 1 This is a flowchart of the multimodal data management method for computing power networks according to the present invention;

[0049] Figure 2 This is a flowchart illustrating the steps of the multimodal data management method for computing power networks according to the present invention.

[0050] Figure 3 This invention provides a model diagram for shared feature extraction and semantic relevance retention label generation.

[0051] Figure 4 This is a diagram of the data recovery model based on potential diffusion in this invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the application will be further described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in this invention. All non-innovative embodiments based on these embodiments by other researchers in the art are within the protection scope of this invention. Furthermore, the step numbers in the embodiments of this invention are only set for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0053] In one embodiment of the present invention, a multimodal data fusion management method for computing power networks is provided, such as... Figure 1 and Figure 2 As shown, it includes the following steps:

[0054] S1. The data owner initiates a data upload request. The nearest computing network node uses the following method to calculate the shared feature vector and semantic relevance retention label of the multimodal data and uploads them to the storage server CSP.

[0055] S2. The storage server CSP retains tags based on the received semantic relevance. Determine whether semantically similar multimodal data has been uploaded and return the label comparison results. The data owner calculates the encryption key together with the data labels and the label comparison results.

[0056] S3. The data owner performs local cross-modal data deduplication based on the label comparison results, encrypts the shared feature vector, and uploads the deduplicated shared feature vector ciphertext to the storage server for storage management.

[0057] S4. When a data user initiates a data query request, the storage server (CSP) performs the following actions: If user ownership verification is successful, the CSP uses the data tag to query the shared feature vector and returns the encrypted shared feature vector to the queryer.

[0058] S5. After the data accessor verifies ownership and obtains the ciphertext of the shared feature vector, they decrypt the shared feature vector. The original data was then recovered with the assistance of nearby computing network nodes, and the original data was finally obtained.

[0059] The main parameters involved in this embodiment are defined in the following table:

[0060] Table 1. Symbols and their definitions

[0061]

[0062] Specifically, such as Figure 3 As shown, step S1 includes the following sub-steps:

[0063] S1.1, Input Data Corresponding text modality and will and As input to generative adversarial networks.

[0064] S1.2 For input data of any modality, the generative adversarial network uses the UNet network as the feature learning network to extract the input data. Corresponding text modality Actual feature vectors and .

[0065] S1.3, Extract the actual feature vector and As input, the generator of the adversarial generative network generates pseudo-feature vectors that are as realistic as possible for the corresponding modality. generate and generate .

[0066] S1.4, respectively < , > and < , The discriminator of the generative adversarial network is used as input, and the discriminant results are used to adjust the parameters of the Unet network. Steps S1.2 and S1.3 are repeated to achieve the goal of generating shared feature vectors that are as similar as possible for similar data across modalities.

[0067] S1.5, until the discriminator determines. and For being similar, and and Similarly, once the generative adversarial network finishes learning, it obtains the input data. Shared feature vectors ,in .

[0068] S1.6, Input data Shared feature vectors As input, the semantic relevance preserved labels of the input data are calculated, specifically as follows: .

[0069] Specifically, step S2 includes the following sub-steps:

[0070] S2.1 After an IoT data upload request arrives, the storage server (CSP) will retain the received semantic relevance tags. , with existing data labels Perform Hamming distance calculation and apply it to a predetermined threshold number. Compare;

[0071] when If the two sets of data are similar, then they are considered to be similar.

[0072] S2.2 If no similar data exists, the data owner will use a fuzz extractor to generate the encryption key for the data:

[0073] ;

[0074] Key-Gen is the key generation algorithm. These are common system parameters. , Including randomly selected prime number p-order bilinear pairing groups and And the generator g of G and a target collision-resistant hash function This is used to generate the necessary metadata;

[0075] The algorithm uses system common parameters and data tags As input, a random string is obtained through a fuzz extractor. and auxiliary parameters Then calculate: , ;Will As a key pair As a key auxiliary parameter;

[0076] S2.3 If similar data exists on the storage server Then the CSP returns the key auxiliary parameters to the data owner. Users will and data tags As input for key regeneration process Output key pair .

[0077] Specifically, step S3 includes the following sub-steps:

[0078] S3.1 The data owner retains the label based on semantic relevance. The similar data comparison results are used to perform a deduplication operation.

[0079] S3.2 If no similar data exists on the edge nodes, the data owner uses the encryption key to encrypt the shared feature vector. To share feature vectors and public key As input, output ciphertext ;

[0080] Specifically, for a given shared feature vector The algorithm first randomly selects Then calculate in sequence: , , ;

[0081] in, The shared feature vector ciphertext is obtained. .

[0082] Ultimately, the data owner retains the data tags themselves. and key pair and will share the ciphertext of the feature vector Upload to CSP for storage management.

[0083] S3.3 If similar data already exists on the edge node, the data owner should directly perform local deduplication on the data and retain the data label. and key pair .

[0084] Specifically, step S4 includes the following sub-steps:

[0085] S4.1 When a data user initiates a data access request, the CSP first verifies the data user's ownership and then feeds back the result to the data analyst.

[0086] S4.2 If the verification passes, CSP returns the shared feature vector ciphertext to the data owner. .

[0087] S4.3 If the verification fails, the CSP will reject the data access request from the user.

[0088] Specifically, step S5 includes the following sub-steps:

[0089] S5.1, The data owner uses the key pair Ciphertext of shared feature vectors Decrypt to obtain the plaintext of the shared feature vector. .

[0090] S5.2 The data owner will decrypt the obtained shared feature vector plaintext. The data is sent to the nearest computing network node, which uses a potential diffusion model based on the UNet network to recover the original data from the feature vector.

[0091] In this embodiment of the invention, an electronic device is also provided, comprising: one or more processors; a storage device storing one or more programs thereon; when the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal data fusion management method for computing power networks described in any of the above embodiments.

[0092] In this embodiment of the invention, a computer-readable storage medium is also provided, on which a computer program is stored. When the program is executed by a processor, it implements the steps of any of the multimodal data fusion management methods for computing power networks described above.

[0093] In summary, this invention provides a multimodal data fusion management method for computing power networks. It relies on a designed multimodal data latent alignment technology to achieve semantic-based cross-modal data similarity comparison. Furthermore, it designs a multimodal similarity-preserving hash algorithm based on latent alignment, using text modalities as intermediate modalities. The input data and its corresponding text modalities are then subjected to adversarial network learning to learn the common representation of the input data and its corresponding text modal data. This ensures a unified mapping of multimodal data from high-dimensional data modalities to low-dimensional Hamming space, and generates semantically relevant multimodal data labels accordingly, enabling unified retrieval of multimodal data.

[0094] Subsequently, semantic relevance preservation label extraction is performed on multimodal data using a computing network. Its ample computing resources and flexible, parallelizable resource scheduling greatly meet the real-time requirements of IoT data. Furthermore, the distributed nature of the computing network fully preserves the interactivity and collaboration between IoT devices in real-world scenarios, further ensuring the correlation between cross-modal similarity data.

[0095] Based on this, the present invention also designs a multimodal data secure deduplication algorithm, which utilizes a multimodal data semantic feature extraction model based on generative adversarial networks to achieve shared feature vector extraction that maintains semantic relevance and deduplication of multimodal data similarity, thereby improving the efficiency of multimodal data management and saving storage space.

[0096] Finally, this invention constructs a data recovery model based on latent diffusion technology to ensure the usability of cross-modal data after deduplication, providing comprehensive data support for subsequent data analysis. The method of this invention can significantly save storage space and transmission bandwidth, effectively maintaining data availability while improving the efficiency of multimodal data management.

[0097] It should be noted that this invention achieves multimodal data fusion management and storage space saving by retaining only one shared feature vector for multimodal semantically similar data. When necessary, it can also use a diffusion model for data recovery and fusion generation, providing sufficient data resources for subsequent data analysis.

[0098] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal data management method for computing power networks, characterized in that, Includes the following steps: S1. The data owner initiates a data upload request. The nearest computing network node calculates the shared feature vector and semantic relevance retention label of the multimodal data and uploads them to the storage server. S2. The storage server retains tags based on the received semantic relevance, compares the tags with existing data tags, determines whether semantically similar multimodal data has been uploaded, and returns the tag comparison result. The data owner jointly calculates the encryption key based on the data tags and the tag comparison results; S3. The data owner performs local cross-modal data deduplication based on the label comparison results, encrypts the shared feature vector, and uploads the deduplicated shared feature vector ciphertext to the storage server for storage management. S4. When a data user initiates a data query request, the storage server verifies the user's ownership and returns the verification result to the data user. S5. After the data accessor verifies ownership and obtains the ciphertext of the shared feature vector, the shared feature vector is decrypted, and the original data is recovered with the assistance of the nearest computing power network node to obtain the original data. Step S1 utilizes computing network nodes to calculate the shared feature vectors and semantic relevance preserved labels for multimodal data, as follows: S1.1 Generate arbitrary modal input data Corresponding text modality ,Will and As input to generative adversarial networks; S1.2 The generative adversarial network uses the Unet network as the feature learning network to extract input data. Corresponding text modality Actual feature vectors and ; S1.3, Extract the actual feature vector and As input, a generative adversarial network generator is used to generate pseudo-feature vectors that correspond to the real modalities. generate and generate ; S1.4, respectively < , > and < , As input to the generative adversarial network discriminator, the Unet network parameters are adjusted based on the discrimination results. Steps S1.2 and S1.3 are repeated to generate shared feature vectors for similar data across modalities. S1.5, until the discriminator determines. and Similar, and and Similarly, the generative adversarial network learns and obtains input data. Shared feature vectors ,in ; S1.6, Input data Shared feature vectors As input, calculate the semantic relevance preserved labels of the input data. Specifically: ； in, It is a locality-sensitive hash function.

2. The multimodal data management method for computing power networks according to claim 1, characterized in that, The method for determining semantically similar multimodal data in step S2 is as follows: the storage server retains the received semantically relevant labels. , and the data labels of existing data w Perform Hamming distance calculation and compare the result with a predetermined threshold number. When comparing, When that happens, the input data is judged. The data w is similar to the existing data; where Ham is the formula for calculating Hamming distance and dis is the result of calculating Hamming distance.

3. The multimodal data management method for computing power networks according to claim 1, characterized in that, Step S2: The data owner calculates the encryption key together with the data tag and the tag comparison result, as follows: S2.1 If no similar data exists, the data owner uses the system public parameter Para and the data label. As input, the encryption key for the data is generated using a fuzz extractor: ； Key-Gen is the key generation algorithm. , The system's common parameters include a randomly selected p-order bilinear pairing group. and And the generator g of G and a target collision-resistant hash function This is used to generate the necessary metadata; The fuzz extractor outputs a random string. and an auxiliary parameter ,calculate: ； ； Will As a key pair As a key auxiliary parameter; S2.2 If similar data exists on the storage server The storage server returns key auxiliary parameters to the data owner. Users will and data tags As input, the key for data regeneration is performed: 。 4. The multimodal data management method for computing power networks according to claim 3, characterized in that, Step S3: The data owner performs local cross-modal data deduplication based on the label comparison results and encrypts the shared feature vectors, as follows: S3.1 The data owner retains the label based on semantic relevance. Based on the comparison results of similar data, multimodal data deduplication is performed based on semantic relevance; S3.2 If no similar data exists on the edge nodes, the data owner encrypts the shared feature vector based on the encryption key, and outputs the ciphertext of the shared feature vector. The data owner retains the data label locally. and key pair ciphertext sharing feature vectors Upload to the storage server for storage management; S3.3 If similar data already exists on the edge node, the data owner performs local deduplication on the data and retains the data label. and key pair .

5. The multimodal data management method for computing power networks according to claim 4, characterized in that, Step S3.2 shared feature vector ciphertext To share feature vectors and public key As input, output ciphertext : ； Wherein, ENC represents the encryption algorithm; For a given shared feature vector Random selection As auxiliary parameters, they are calculated sequentially: ， ， ； in, The shared feature vector ciphertext is obtained. .

6. The multimodal data management method for computing power networks according to claim 5, characterized in that, Step S4: When a data access user initiates a data access request, the storage server verifies the data access user's ownership, as follows: S4.1 If the verification passes, the storage server uses the data tag to query the shared feature vector and returns the encrypted shared feature vector to the data accessor. ; S4.2 If the verification fails, the storage server rejects the data access request from the user.

7. The multimodal data management method for computing power networks according to claim 6, characterized in that, Step S5: The data visitor obtains the raw data, as follows: S5.1, The data owner uses the key pair For ciphertext with shared feature vectors Decrypt to obtain the plaintext of the shared feature vector. ; S5.2 The data owner will decrypt the obtained shared feature vector plaintext. The data is sent to the nearest computing network node. The computing network node uses a potential diffusion model based on the UNet network to recover the original data from the feature vector and then sends the original data to the data visitor.

8. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal data management method for computing power networks as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the multimodal data management method for computing power networks as described in any one of claims 1 to 7.

Images