Multi-modal data management method oriented to computing power network

Abstract

The invention discloses a computing power network-oriented multi-modal data management method, which comprises the following steps of: firstly, designing a potential alignment-based multi-modal similarity preserving Hash algorithm, taking a text modal as an intermediate modal, learning common representation of input data and corresponding text modal data through an adversarial network, generating a multi-modal data label, and carrying out multi-modal data management on the multi-modal data label; semantic correlation retention label extraction is carried out on the multi-modal data by means of computing power network nodes, and the incidence relation of cross-modal similarity data is ensured; then, multi-modal data semantic feature extraction based on the generative adversarial network is utilized to perform shared feature vector extraction and multi-modal data similarity de-duplication, so that the storage space is greatly saved; and finally, performing data recovery based on a latent diffusion technology, and realizing semantic similarity recovery from shared feature vectors to original data by means of a computing power network. According to the method, the storage space and the transmission bandwidth can be saved to a great extent, and the data availability is effectively maintained while the multi-modal data management efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of network data management, and in particular to a multimodal data management method for computing power networks. Background Art

[0002] With the continuous development of technologies such as the Internet and mobile communications, information technology has been integrated into social production and life. The diversified media characteristics and collaborative working mode of IoT devices have put forward higher requirements for data management capabilities and data processing speed. At the same time, emerging technologies such as digital twins and virtual reality have gradually emerged in recent years, and the demand for flexible data management and efficient computing support has become increasingly prominent.

[0003] Limited by factors such as power consumption and cost, existing data acquisition equipment usually has lightweight features, which makes its storage and computing capabilities limited, bringing challenges to local data management. Mainstream data management solutions can effectively reduce the storage burden of data acquisition equipment by outsourcing the collected data. Existing solutions usually use cloud storage to achieve synchronous management of massive data, but this centralized data management method generally has defects such as limited scale and difficulty in resisting single point failures, and cannot cope with the processing needs of large-scale data. As the foundation of the information society, computing power has become the basic and core development power of the digital economy, directly affecting the speed of digital economic development and determining the development height of social intelligence. As a new technology, computing power network can allocate and flexibly schedule computing resources and storage resources between cloud, network and edge according to business needs to meet the needs of large-scale data processing and computing-intensive tasks, and provide efficient, secure and flexible services for all walks of life. This distributed storage can not only effectively resist single point failure attacks, but also effectively make up for the lack of storage space in cloud storage.

[0004] On the other hand, with the rapid development of multimedia devices, IoT data presents diversified characteristics. Various types of IoT sensors usually work together and generate a large amount of cross-modal data containing the same semantic information. For example, in the industrial IoT scenario, monitoring videos and equipment operation logs may record the same work information. The same information is repeated in multiple modalities, and these cross-modal data with the same semantic information occupy a lot of storage space. Data deduplication technology can keep only one piece of data and delete other duplicate copies, effectively solving the problem of data redundancy. However, since data of different modalities have different forms of expression and composition, it is difficult for existing data deduplication methods to directly compare the similarities between data of different modalities. They can only operate on data of a single modality and cannot be directly applied to real-world scenarios of multimodal data.

[0005] In addition, since users lose physical control over their data during edge computing and cloud storage, data security is a basic requirement of users. The goal of encryption technology is to provide semantic security for plaintext data, making ciphertext data indistinguishable, which increases the difficulty of data management. Although the same data can be encrypted into the same ciphertext through key transfer and other methods, it still cannot meet the needs of ciphertext deduplication of similar data. To this end, it is also necessary to consider how to break the information barriers between users without transferring keys, realize secure multimodal data fusion management, analyze the correlation between multimodal data, and design a general encryption mechanism to ensure data relevance. Summary of the invention

[0006] The problem to be solved by the present invention is to provide a multimodal data management method for computing power networks, analyze the correlation between multimodal data, design a universal encryption mechanism to ensure data correlation, and realize secure multimodal data fusion management.

[0007] The present invention adopts the following technical solution: a multimodal data management method for computing power network, comprising the following steps:

[0008] S1. The data owner initiates a data upload request. The nearest computing network node calculates the shared feature vectors and semantic relevance retention labels of the multimodal data and uploads them to the storage server.

[0009] S2, the storage server retains the label according to the received semantic relevance, compares the label with the existing data label, determines whether multimodal data with similar semantics has been uploaded, and returns the label comparison result; the data owner jointly calculates the encryption key according to the data label and the label comparison result;

[0010] S3: The data owner performs local cross-modal data deduplication based on the label comparison results, encrypts the shared feature vector, and uploads the deduplicated shared feature vector ciphertext to the storage server for storage management;

[0011] S4. When the data accessor initiates a data query request, the storage server verifies the user ownership of the data accessor and feeds back the verification result to the data accessor;

[0012] S5. After the data accessor passes the ownership verification and obtains the shared feature vector ciphertext, the shared feature vector is decrypted, and the original data is restored with the assistance of the nearest computing power network node to obtain the original data.

[0013] Preferably, in step S1, the computing network node calculates the shared feature vectors and semantic relevance retention labels of the multimodal data in the following manner:

[0014] S1.1. Generate arbitrary modal input data wi The corresponding text mode w T , w i and w T As input to the Generative Adversarial Network;

[0015] S1.2, the generative adversarial network uses the Unet network as a feature learning network to extract the input data w i The corresponding text mode w T The actual eigenvector of and

[0016] S1.3. The actual feature vector extracted and As input, a pseudo feature vector of the real corresponding modality is generated through the generative adversarial network generator. generate and generate

[0017] S1.4, respectively and As the input of the generative adversarial network discriminator, adjust the Unet network parameters according to the discriminant results, repeat steps S1.2 and S1.3 to generate shared feature vectors for similar data across modalities;

[0018] S1.5, until the discriminator determines and is similar, and and Similarly, complete the generative adversarial network learning and obtain the input data w i The shared eigenvector of in

[0019] S1.6. Input data w i The shared eigenvector of As input, we compute semantic relevance preserving labels for the input data.

[0020] Preferably, the semantically similar multimodal data in step S2 is judged by: the storage server retains the received semantic relevance label The data tag Tag of the existing data w w ∈{T} performs Hamming distance calculation and compares the calculation result with the predetermined threshold value d. If the calculation result is not greater than the predetermined threshold value d, the input data w is judged to be i The data is similar to the existing data w.

[0021] Furthermore, the data owner jointly calculates the encryption key based on the data label and the label comparison result, as follows:

[0022] S2.1. If there is no similar data, the data owner uses the system public parameter Para and data label As input, the encryption key Key of the data is generated by the fuzzy extractor:

[0023]

[0024] Among them, Para is the system common parameter. As a key pair, As key auxiliary parameters;

[0025] S2.2. If there is similar data w on the storage server j , the storage server returns the key auxiliary parameters to the data owner The user will and data labels As input, rekey the data:

[0026]

[0027] Among them, Key-ReGen is the key regeneration algorithm.

[0028] Preferably, in step S3, the data owner performs local cross-modal data deduplication according to the label comparison result and encrypts the shared feature vector in the following manner:

[0029] S3.1. Data owners retain labels based on semantic relevance Based on the similar data comparison results, multimodal data deduplication is performed based on semantic relevance;

[0030] S3.2. If there is no similar data on the edge node, the data owner encrypts the shared feature vector based on the encryption key and outputs the shared feature vector ciphertext Data owners keep data labels locally and key pair Encrypt the shared feature vector Upload to the storage server for storage management;

[0031] S3.3. If similar data already exists on the edge node, the data owner performs local deduplication on the data and retains the data label and key pair

[0032] Preferably, in step S4, when the data accessor initiates a data access request, the storage server performs ownership verification on the data accessor in the following manner:

[0033] S4.1. If the verification is successful, the storage server uses the data tag to query the shared feature vector and returns the shared feature vector ciphertext to the data accessor.

[0034] S4.2. If the verification fails, the storage server rejects the access request of the data accessor.

[0035] Preferably, in step S5, the data accessor obtains the original data in the following manner:

[0036] S5.1. Data owner uses key pair Encrypt the shared feature vector Decrypt and obtain the shared feature vector plaintext z wi ;

[0037] S5.2. The data owner decrypts the shared feature vector plaintext It is sent to the nearest computing network node. The computing network node uses the potential diffusion model based on the UNet network to restore the feature vector to the original data, and then sends the original data to the data accessor.

[0038] The technical solution of the present invention also provides: an electronic device, comprising:

[0039] one or more processors;

[0040] a storage device having one or more programs stored thereon;

[0041] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-mentioned multimodal data management methods for computing power networks.

[0042] The technical solution of the present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in any of the above-mentioned multimodal data management methods for computing power networks are implemented.

[0043] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0044] 1. The multimodal data management method of the present invention relies on the designed multimodal data potential alignment technology to realize semantic-based cross-modal data similarity comparison. By designing a multimodal similarity-preserving hash algorithm based on potential alignment, the text modality is used as the intermediate modality, and the input data and its corresponding text modality are subjected to adversarial network learning, so as to learn the common representation of the input data and its corresponding text modality data, ensure that different modal data realize the unified mapping of multimodal data from high-dimensional data modality to low-dimensional Hamming space, and generate multimodal data labels that retain semantic relevance based on this, so as to realize unified retrieval of multimodal data.

[0045] 2. The multimodal data management method of the present invention uses the computing power network to extract semantic relevance-preserving labels from multimodal data, and the parallel operation can meet the real-time requirements of IoT data to a great extent. In addition, the computing power network fully retains the interactive and collaborative characteristics between IoT devices in real scenarios due to its distributed characteristics, further ensuring the correlation relationship between cross-modal similarity data.

[0046] 3. The multimodal data management method of the present invention, through a multimodal data security deduplication algorithm, utilizes a multimodal data semantic feature extraction model based on a generative adversarial network to achieve shared feature vector extraction based on semantic relevance and multimodal data deduplication, thereby improving the efficiency of multimodal data management and saving storage space.

[0047] 4. The multimodal data management method of the present invention ensures the availability of cross-modal data after similarity removal through a data recovery method based on latent diffusion technology, providing comprehensive data support for subsequent data analysis. This method can save storage space and transmission bandwidth to a great extent, effectively maintaining data availability while improving the efficiency of multimodal data management. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flow chart of the multimodal data management method for computing power network of the present invention;

[0049] Figure 2 This is a flowchart of the steps of the multimodal data management method for computing power network of the present invention;

[0050] Figure 3 Generate a model graph for shared feature extraction and semantic relevance-preserving labels for the present invention;

[0051] Figure 4 This is a data recovery model diagram based on potential diffusion of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the application is further elaborated in detail below in conjunction with the accompanying drawings. The described embodiments are only a part of the embodiments involved in the present invention. All non-innovative embodiments of other researchers in the field on this embodiment belong to the protection scope of the present invention. At the same time, for the step numbering in the embodiment of the present invention, it is only set for the convenience of explanation, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0053] In one embodiment of the present invention, a multimodal data fusion management method for a computing network is provided. Figure 1 and Figure 2 As shown, the following steps are included:

[0054] S1. The data owner initiates a data upload request. The nearest computing network node uses the following method to calculate the shared feature vectors and semantic relevance retention labels of the multimodal data and uploads them to the storage server CSP.

[0055] S2, the storage server CSP retains the label according to the received semantic relevance It determines whether multimodal data with semantic similarity has been uploaded and returns the label comparison result. The data owner jointly calculates the encryption key based on the data label and the label comparison result.

[0056] S3. The data owner performs local cross-modal data deduplication operations based on the label comparison results, encrypts the shared feature vectors, and uploads the deduplicated shared feature vector ciphertext to the storage server for storage management.

[0057] S4: When a data accessor initiates a data query request, the storage server CSP performs PoW on the data accessor. w User ownership verification, if the verification is successful, CSP uses the data tag to query the shared feature vector and returns the shared feature vector ciphertext to the queryer.

[0058] S5. After the data accessor passes the ownership verification and obtains the shared feature vector ciphertext, the shared feature vector is decrypted The original data is restored with the assistance of the nearest computing power network nodes to finally obtain the original data.

[0059] The main parameters involved in this embodiment are defined in the following table:

[0060] Table 1 Symbols and their definitions

[0061]

[0062] Specifically, Figure 3 As shown, step S1 includes the following sub-steps:

[0063] S1.1, input data w i The corresponding text mode w T , and w i and w T As input to the generative adversarial network.

[0064] S1.2. For input data of any modality, the generative adversarial network uses the Unet network as the feature learning network to extract the input data w i The corresponding text mode w T The actual eigenvector of and

[0065] S1.3. The actual feature vector extracted and As input, the generator of the adversarial generative network generates a pseudo feature vector of the corresponding modality that is as realistic as possible. generate and generate

[0066] S1.4, respectively and As the input of the discriminator of the generative adversarial network, and the discrimination result is used to adjust the parameters of the Unet network, and steps S1.2 and S1.3 are repeated to achieve the purpose of generating shared feature vectors that are as similar as possible for similar data across modalities.

[0067] S1.5, until the discriminator determines and is similar, and and Similarly, the generative adversarial network learning ends, and the input data w is obtained i The shared eigenvector of in

[0068] S1.6. Input data w i The shared eigenvector of As input, the semantic relevance preserving labels of the input data are calculated, specifically:

[0069] Specifically, step S2 includes the following sub-steps:

[0070] S2.1. After the IoT data upload request arrives, the storage server CSP will retain the received semantic relevance label With existing data tags Tagw ∈{T} performs Hamming distance calculation and compares it with the given threshold d;

[0071] when , then the two data are considered to be similar data.

[0072] S2.2. If there is no similar data, the data owner uses the fuzzy extractor to generate the encryption key for the data:

[0073]

[0074] Among them, Key-Gen is the key generation algorithm, Para is the system public parameter, Para=(G,G T ,p,g,x 1 ,x 2 ,x 3 ,H), Para includes randomly selected prime number p-order bilinear pairing groups (G,G T ) and x 1 ,x 2 ,x 3 ∈G, and the generator g of G and a target collision-resistant hash function H 1 :G 2 →Z p , used to generate necessary metadata;

[0075] The algorithm uses system public parameters Para and data labels As input, a random string is obtained through the fuzzy extractor and auxiliary parameters Then calculate: Will As a key pair, As key auxiliary parameters;

[0076] S2.3. If there is similar data w on the storage server j , then CSP returns the key auxiliary parameters to the data owner The user will and data labels As input for the key regeneration process Key-

[0077] Output key pair

[0078] Specifically, step S3 includes the following sub-steps:

[0079] S3.1. Data owners retain labels based on semantic relevance The deduplication operation is performed based on the similar data comparison results.

[0080] S3.2. If there is no similar data on the edge node, the data owner uses the encryption key to encrypt the shared feature vector. Shared feature vector and the public key As input, output ciphertext

[0081] Specifically, for a given shared feature vector The algorithm first randomly selects Then calculate in sequence:

[0082] in, Get the shared feature vector ciphertext

[0083] Ultimately, data owners retain data labels themselves and key pair And the shared feature vector ciphertext Upload to CSP for storage management.

[0084] S3.3. If similar data already exists on the edge node, the data owner directly performs local deduplication operations on the data and retains the data label and key pair

[0085] Specifically, step S4 includes the following sub-steps:

[0086] S4.1. When a data accessor initiates a data access request, the CSP first verifies the ownership of the data accessor and feeds back the result to the data analyst.

[0087] S4.2. If the verification is successful, the CSP returns the shared feature vector ciphertext to the data owner

[0088] S4.3. If the verification fails, the CSP rejects the data accessor’s access request.

[0089] Specifically, step S5 includes the following sub-steps:

[0090] S5.1. Data owner uses key pair Encrypt the shared feature vector Decrypt and obtain the shared feature vector plaintext z wi .

[0091] S5.2. The data owner decrypts the shared feature vector plaintext It is sent to the nearest computing network node, which uses the potential diffusion model based on the UNet network to realize the recovery process from feature vector to original data.

[0092] In an embodiment of the present invention, an electronic device is also provided, including: one or more processors; a storage device on which one or more programs are stored; when the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal data fusion management method for a computing power network described in any of the above embodiments.

[0093] In an embodiment of the present invention, a computer-readable storage medium is further provided, on which a computer program is stored. When the program is executed by a processor, the steps of any one of the multimodal data fusion management methods for a computing power network in the above-mentioned embodiments are implemented.

[0094] In summary, the present invention provides a multimodal data fusion management method for computing power networks, which relies on the designed multimodal data potential alignment technology to realize semantic-based cross-modal data similarity comparison, and designs a multimodal similarity-preserving hash algorithm based on potential alignment. The text modality is used as the intermediate modality, and the input data and its corresponding text modality are subjected to adversarial network learning, so as to learn the common representation of the input data and its corresponding text modality data, ensure that different modal data realize the unified mapping of multimodal data from high-dimensional data modality to low-dimensional Hamming space, and generate multimodal data labels that retain semantic relevance based on this, so as to realize unified retrieval of multimodal data.

[0095] Subsequently, the computing network is used to extract semantically relevant labels from multimodal data. Its sufficient computing resources and flexible and parallel resource scheduling meet the real-time requirements of IoT data to a great extent. In addition, the distributed nature of the computing network fully preserves the interactive and collaborative characteristics between IoT devices in real scenarios, further ensuring the correlation between cross-modal similarity data.

[0096] On this basis, the present invention also designs a multimodal data security deduplication algorithm, which utilizes a multimodal data semantic feature extraction model based on a generative adversarial network to achieve shared feature vector extraction that maintains semantic relevance and multimodal data similarity deduplication, thereby improving multimodal data management efficiency and saving storage space.

[0097] Finally, the present invention constructs a data recovery model based on latent diffusion technology to ensure the availability of cross-modal data after similarity deduplication, providing comprehensive data support for subsequent data analysis. The method of the present invention can save storage space and transmission bandwidth to a great extent, effectively maintaining data availability while improving the efficiency of multimodal data management.

[0098] It should be noted that the present invention realizes multimodal data fusion management and storage space saving by retaining only one shared feature vector for multimodal semantically similar data. When necessary, the diffusion model can be used for data recovery and fusion generation to provide sufficient data resources for subsequent data analysis.

[0099] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multimodal data management method for computing power network, characterized in that: The steps include: S1. The data owner initiates a data upload request. The nearest computing network node calculates the shared feature vectors and semantic relevance retention labels of the multimodal data and uploads them to the storage server. S2, the storage server compares the received semantically relevant reserved labels with the existing data labels, determines whether semantically similar multimodal data has been uploaded, and returns the label comparison result; The data owner jointly calculates the encryption key based on the data label and the label comparison result; S3: The data owner performs local cross-modal data deduplication based on the label comparison results, encrypts the shared feature vector, and uploads the deduplicated shared feature vector ciphertext to the storage server for storage management; S4. When the data accessor initiates a data query request, the storage server verifies the user ownership of the data accessor and feeds back the verification result to the data accessor; S5. After the data accessor passes the ownership verification and obtains the shared feature vector ciphertext, the shared feature vector is decrypted, and the original data is restored with the assistance of the nearest computing power network node to obtain the original data.

2. The multimodal data management method for computing power network according to claim 1, characterized in that: Step S1 uses computing network nodes to calculate the shared feature vectors and semantic relevance retention labels of multimodal data as follows: S1.

1. Generate arbitrary modal input data w i The corresponding text mode w T , w i and w T As input to the Generative Adversarial Network; S1.2, the generative adversarial network uses the Unet network as a feature learning network to extract the input data w i The corresponding text mode w T The actual eigenvector of and S1.

3. The actual feature vector extracted and As input, a pseudo feature vector of the real corresponding modality is generated through the generative adversarial network generator. generate and generate S1.4, respectively and As the input of the generative adversarial network discriminator, adjust the Unet network parameters according to the discriminant results, repeat steps S1.2 and S1.3 to generate shared feature vectors for similar data across modalities; S1.5, until the discriminator determines and Similar, and and Similarly, complete the generative adversarial network learning and obtain the input data w i The shared eigenvector of in S1.

6. Input data w i The shared eigenvector of As input, we compute semantic relevance preserving labels for the input data. Specifically: Among them, SimHash is a locality sensitive hash function.

3. The multimodal data management method for computing power network according to claim 2 is characterized in that: The semantically similar multimodal data in step S2 is judged by: the storage server retains the received semantic relevance label The data tag Tag of the existing data w w ∈{T} to calculate the Hamming distance and compare the calculation result with the predetermined threshold value d. When , the input data w is judged i and the existing data w are similar data; Ham is the Hamming distance calculation formula, and dis is the Hamming distance calculation result.

4. The multimodal data management method for computing power network according to claim 2, characterized in that: Step S2: The data owner jointly calculates the encryption key based on the data label and the label comparison result, as follows: S2.

1. If there is no similar data, the data owner uses the system public parameter Para and data label As input, generate the encryption key of the data through the fuzzy extractor: Among them, Key-Gen is the key generation algorithm, Para=(G,G T ,p,g,x1,x2,x3,H), Para is the system public parameter, including the randomly selected prime p-order bilinear pairing group (G,G T ) and x1,x2,x3∈G, as well as the generator g of G and a target collision-resistant hash function H1:G 2 →Z p , used to generate necessary metadata; The fuzzy extractor outputs a random string R wi and an auxiliary parameter calculate: Will As a key pair, As key auxiliary parameters; S2.

2. If there is similar data w on the storage server j , the storage server returns the key auxiliary parameters to the data owner The user will and data labels As input, rekey the data:

5. The multimodal data management method for computing power network according to claim 4 is characterized in that: Step S3: The data owner performs local cross-modal data deduplication based on the label comparison results and encrypts the shared feature vectors as follows: S3.

1. Data owners retain labels based on semantic relevance Based on the similar data comparison results, multimodal data deduplication is performed based on semantic relevance; S3.

2. If there is no similar data on the edge node, the data owner encrypts the shared feature vector based on the encryption key and outputs the shared feature vector ciphertext Data owners keep data labels locally and key pair Encrypt the shared feature vector Upload to the storage server for storage management; S3.

3. If similar data already exists on the edge node, the data owner performs local deduplication on the data and retains the data label and key pair 6. The multimodal data management method for computing power network according to claim 5, characterized in that: The shared feature vector ciphertext described in step S3.2 Shared feature vector and the public key As input, output ciphertext Among them, ENC represents the encryption algorithm; For a given shared eigenvector Random Selection As auxiliary parameters, calculate in sequence: c1=g r , in, Get the shared feature vector ciphertext 7. The multimodal data management method for computing power network according to claim 6, characterized in that: Step S4: When a data accessor initiates a data access request, the storage server verifies the ownership of the data accessor in the following manner: S4.

1. If the verification is successful, the storage server uses the data tag to query the shared feature vector and returns the shared feature vector ciphertext to the data accessor. S4.

2. If the verification fails, the storage server rejects the access request of the data accessor.

8. The multimodal data management method for computing power network according to claim 7, characterized in that: Step S5: The data accessor obtains the original data in the following way: S5.

1. Data owner uses key pair Encrypt the shared feature vector Decrypt and obtain the shared feature vector plaintext S5.

2. The data owner decrypts the shared feature vector plaintext It is sent to the nearest computing network node. The computing network node uses the potential diffusion model based on the UNet network to restore the feature vector to the original data, and then sends the original data to the data accessor.

9. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal data management method for computing power networks as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the steps in the multimodal data management method for a computing power network described in any one of claims 1 to 8 are implemented.

Images