Federal cross-modal feature alignment-based security privacy protection method, apparatus and device, and medium

Abstract

The invention relates to a security privacy protection method and device based on federal cross-modal feature alignment, equipment and a medium, and the method comprises the steps: a first client carries out the fragmentation processing of a first model parameter variation according to a model mask, so as to generate a first model parameter variation quantum set and a second model parameter variation quantum set, the first client performs XOR operation on the second model parameter change quantum set by adopting a first random key to generate an encryption parameter, encrypts the first random key by adopting a server public key to generate an encryption key, and sends the encryption parameter and the encryption key to a second client; and the second client carries out XOR operation on the encryption parameter and the parameter variation of the third model to generate a hybrid model, and the first client and the second client upload the hybrid parameter, the encryption key and the parameter variation of the second model in the hybrid model to the server. According to the method, privacy protection of cross-modal feature alignment can be realized, and meanwhile, the training efficiency and the model performance are ensured.

Description

Technical Field

[0001] This application relates to the field of network security, and in particular, to a security and privacy protection method based on federated cross-modal feature alignment, a corresponding device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Modal feature alignment is a core task in multimodal learning, aiming to map data of different modalities (such as text, images, audio, video, etc.) into a unified semantic space, so as to achieve information association, retrieval, and reasoning between different modalities. This technology is widely used in fields such as image-text retrieval, cross-modal generation, and video question answering. For example, in the image-text retrieval task, the model needs to align the features of images and text, so that the distance between the image and text describing the same content in the semantic space is closer, while the distance between the modal features of different contents is farther.

[0003] A cross-modal feature alignment model usually consists of two parts: a single-modal feature extraction module and an inter-modal fusion module. The single-modal feature extraction module extracts semantic features of a single modality through architectures such as convolutional neural networks (CNNs), Transformers, etc., while the inter-modal fusion module uses attention mechanisms or contrast learning methods to achieve interaction and unified modeling of features between modalities.

[0004] Due to the complexity and diversity of multimodal data, cross-modal feature alignment requires large-scale data for training in order to learn the complex associations and alignment rules between different modalities. However, in practical applications, the training of such models faces two important limitations: on the one hand, a large amount of multimodal data is scattered in different institutions or devices, and due to data privacy and legal regulations, institutions cannot directly share data; on the other hand, multimodal data itself contains rich sensitive information (such as medical images, user behavior logs), and direct transmission and centralized storage pose great privacy risks.

[0005] Traditional centralized training methods are difficult to meet the actual needs of cross-modal feature alignment, and federated learning provides an effective solution for cross-modal feature alignment. Through distributed collaborative training, federated learning allows each institution or device to retain data locally and only share model parameters or gradient information, thus completing the optimization of the global model while ensuring data privacy.

[0006] Although federated learning has shown great potential in privacy protection and distributed collaboration, it still faces severe privacy and security challenges in cross-modal feature alignment scenarios. First, the parameters of the single-modal feature extraction module are highly correlated with the original data. For example, the weights of the image convolutional layer may reflect the structural features of the image, and the parameters of the text embedding layer may leak semantic information. This makes directly sharing these parameters likely to lead to privacy leakage. Second, the complexity and parameter scale of the model in cross-modal tasks make it difficult to directly apply traditional privacy protection methods (such as differential privacy and homomorphic encryption). Differential privacy will introduce noise and reduce the alignment accuracy between modalities. The computational overhead of homomorphic encryption is too high to meet the needs of resource-constrained devices.

[0007] As described above, in the prior art, the parameters of the single-modal feature extraction module are highly correlated with the original data. For example, the weights of the image convolutional layer may reflect the structural features of the image, and the parameters of the text embedding layer may leak semantic information. This makes directly sharing these parameters likely to lead to privacy leakage and other problems. In consideration of solving this problem, the applicant has made corresponding explorations. Summary of the Invention

[0008] The purpose of this application is to solve the above problems and provide a secure privacy protection method, corresponding device, electronic device, and computer-readable storage medium based on federated cross-modal feature alignment.

[0009] To meet the various purposes of this application, the following technical solutions are adopted:

[0010] A secure privacy protection method based on federated cross-modal feature alignment proposed for one of the purposes of this application includes:

[0011] Determine the single-modal feature extraction model and the cross-modal feature alignment model to be trained in the server, divide the parameters of the single-modal feature extraction model into multiple shards, and generate a server public key and a server private key. Among them, each shard contains part of the parameters of the single-modal feature extraction model;

[0012] The first client obtains the multiple shards and the server public key, and trains the single-modal feature extraction model and the cross-modal feature alignment model to determine the first model parameter change amount corresponding to the single-modal feature extraction model and the second model parameter change amount corresponding to the cross-modal feature alignment model;

[0013] For the single-modal feature extraction model, the first client establishes a cooperative relationship with the second client, generates a model mask through a preset key exchange protocol, and generates a first random key. Among them, the model mask is a bit string with a length of k, and each bit in the bit string corresponds to each shard one by one;

[0014] The first client performs sharding processing on the first model parameter variation according to the model mask to generate a first model parameter variation subset and a second model parameter variation subset. The first client performs an exclusive OR operation on the second model parameter variation subset using the first random key to generate an encryption parameter, encrypts the first random key using the server public key to generate an encryption key, and sends the encryption parameter and the encryption key to the second client;

[0015] The second client performs an exclusive OR operation on the encryption parameter and the third model parameter variation to generate a mixed model. The first client and the second client upload the mixed parameters, the encryption key, and the second model parameter variation in the mixed model to the server to complete the security and privacy protection based on federated cross-modal feature alignment.

[0016] Optionally, the single-modal feature extraction model is denoted as w modal , and the cross-modal feature alignment model is denoted as w attention , and the multiple shards are denoted as The server public key is denoted as pk s , and the server private key is denoted as sk s ;

[0017] The first client is denoted as c i , the first model parameter variation is denoted as Δw modal，i , and the second model parameter variation is denoted as Δw attention，i ;

[0018] The second client is denoted as c j , the model mask is denoted as m, and the first random key is denoted as r i ;

[0019] The first model parameter variation subset is denoted as The second model parameter variation subset is denoted as

[0020] The third model parameter variation is denoted as The encryption parameter is denoted as The encryption key is denoted as Enc(r i , pk S ), and the mixed model is denoted as The mixed parameters are denoted as

[0021] Optionally, the step in which the first client performs sharding processing on the first model parameter variation according to the model mask to generate a first model parameter variation subset and a second model parameter variation subset includes:

[0022] The first client c i Performs sharding processing on the first model parameter variation according to the model mask m to generate two parts of parameter updates, including a first model parameter variation subset And a second model parameter variation subset Wherein, the first model parameter variation subset Represents the result of setting all shards corresponding to bit 0 in the model mask to zero, and the second model parameter variation subset Represents the result of setting all shards corresponding to bit 1 in the model mask to zero.

[0023] Optionally, the step in which the first client performs an exclusive-or operation on the second model parameter variation subset using the first random key to generate encrypted parameters, encrypts the first random key using the server public key to generate an encryption key, and sends the encrypted parameters and the encryption key to the second client includes:

[0024] The first client c i Uses the first random key r i To perform an exclusive-or operation on the second model parameter variation subset To generate encrypted parameters Meanwhile, uses the server public key pk s To encrypt the first random key r i To generate an encryption key Enc(r i , pk S ), and sends the encrypted parameters And the encryption key Enc(r i , pk S ) to the second client.

[0025] Optionally, the step in which the second client performs an exclusive-or operation on the encrypted parameters and the third model parameter variation to generate a hybrid model includes:

[0026] The second client c j Performs an exclusive-or operation on the encrypted parameters And the third model parameter variation To generate a hybrid model, wherein the third model parameter variation represents the model parameter variation updated after local training of the single-modal feature extraction model in the second client, and the hybrid model is expressed as:

[0027]

[0028] Optionally, the step in which the first client and the second client upload the hybrid parameters, the encryption key, and the change amount of the second model parameters in the hybrid model to the server includes:

[0029] The server receives the hybrid parameters in the hybrid model uploaded by the first client and the second client The encryption key Enc(r i , pk S ), and the change amount of the second model parameters Δw attention，i . Using the server private key sk s to decrypt the encryption key Enc(r i , pk S ) to obtain the first random key r i , and decrypting the hybrid parameters through an exclusive OR operation to recover the model update, which is expressed as:

[0030]

[0031] The server respectively aggregates the updated unimodal feature extraction models and the updated cross-modal feature alignment models of all clients to determine a first aggregation result and a second aggregation result. Among them, the first aggregation result is expressed as w modal ←w modal +∑(Δw j ) mix , and the second aggregation result is determined by aggregating the change amount Δw of the second model parameters attention，i using an importance weighting algorithm or a robust aggregation algorithm.

[0032] Optionally, the change amount of the first model parameters characterizes the change amount of the model parameters updated after local training of the unimodal feature extraction model in the first client, the change amount of the second model parameters characterizes the change amount of the model parameters updated after local training of the cross-modal feature alignment model in the first client, and the key exchange protocol is the Diffie-Hellman key exchange protocol.

[0033] A security and privacy protection device based on federated cross-modal feature alignment provided to meet another object of the present application includes:

[0034] ​The model parameter sharding module is configured to determine the single-modal feature extraction model and the cross-modal feature alignment model to be trained in the server, divide the parameters of the single-modal feature extraction model into multiple shards, and generate a server public key and a server private key, where each shard contains partial parameters of the single-modal feature extraction model;

[0035] The parameter variation determination module is configured to obtain, by a first client, the multiple shards and the server public key, and train the single-modal feature extraction model and the cross-modal feature alignment model to determine a first model parameter variation corresponding to the single-modal feature extraction model and a second model parameter variation corresponding to the cross-modal feature alignment model;

[0036] The model mask determination module is configured to, for the single-modal feature extraction model, establish a cooperation relationship between the first client and a second client, generate a model mask through a preset key exchange protocol, and generate a first random key, where the model mask is a bit string of length k, and each bit in the bit string corresponds to each shard one by one;

[0037] The encryption key determination module is configured to, according to the model mask, the first client perform sharding processing on the first model parameter variation to generate a first model parameter variation subset and a second model parameter variation subset, the first client perform an exclusive OR operation on the second model parameter variation subset using the first random key to generate an encryption parameter, encrypt the first random key using the server public key to generate an encryption key, and send the encryption parameter and the encryption key to the second client;

[0038] The security and privacy protection module is configured to perform an exclusive OR operation on the encryption parameter and a third model parameter variation by the second client to generate a mixed model, and the first client and the second client upload the mixed parameters, the encryption key, and the second model parameter variation in the mixed model to the server to complete the security and privacy protection based on federated cross-modal feature alignment.

[0039] An electronic device provided for another object of the present application includes a central processing unit and a memory, and the central processing unit is configured to call and run a computer program stored in the memory to execute the steps of the security and privacy protection method based on federated cross-modal feature alignment of the present application.

[0040] A computer-readable storage medium provided to meet another object of the present application stores a computer program implemented based on the security and privacy protection method for federal cross-modal feature alignment in the form of computer-readable instructions. When the computer program is called and run by a computer, it executes the steps included in the corresponding method.

[0041] Compared with the prior art, in the prior art, the parameters of the single-modal feature extraction module are highly correlated with the original data. For example, the weights of the image convolution layer may reflect the structural features of the image, and the parameters of the text embedding layer may leak semantic information, which may lead to problems such as privacy leakage when directly sharing these parameters. The present application includes, but is not limited to, the following beneficial effects:

[0042] First, the present application protects the privacy of the single-modal feature extraction model through the sharding exchange protocol, ensuring that the server cannot recover the complete model parameters and significantly reducing the risk of privacy leakage;

[0043] Second, compared with complex privacy protection methods such as homomorphic encryption and secure multi-party computation, the present application adopts a simple sharding and exclusive-or encryption strategy, achieving privacy protection while significantly reducing the computational and communication overhead.

[0044] Third, the present application optimizes the global performance of the cross-modal feature fusion model through the importance weighting and robust aggregation strategies, which is particularly suitable for multi-modal collaboration scenarios that require high performance and privacy protection;

[0045] In summary, the present application provides an efficient and privacy-friendly solution for cross-modal tasks through the sharding exchange protocol and the federated learning framework; in response to the privacy protection requirements of the single-modal feature extraction model, the present application divides the model parameters into multiple shards and generates masks by the client to control the exchange of shards. The clients exchange some shards through exclusive-or encryption, and at the same time encrypt and protect the unexchanged shards. The cooperating parties cannot know the complete model parameters of the exchanging party, and the server directly aggregates only the mixed model updates without knowing the masks and the cooperation relationship, ensuring that the aggregation result is the same as that without sharding exchange, thus ensuring the correctness and effectiveness of the model while protecting privacy;

[0046] For the cross-modal feature fusion model, the present application adopts the importance weighting or robust aggregation strategy to further improve the performance and robustness of the model. This method avoids the high computational overhead of traditional privacy protection methods (such as homomorphic encryption and secure multi-party computation), significantly reduces the communication and computational burden, and can be widely applied to scenarios such as medical data analysis, intelligent device collaboration, and financial data modeling; the present application can achieve privacy protection for cross-modal feature alignment while ensuring the training efficiency and model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description of embodiments in conjunction with the accompanying drawings, where:

[0048] Figure 1 It is a schematic flowchart of a security and privacy protection method based on federated cross-modal feature alignment in an embodiment of the present application;

[0049] Figure 2 It is an exemplary network architecture adopted by a security and privacy protection system based on federated cross-modal feature alignment in an embodiment of the present application;

[0050] Figure 3 It is a schematic diagram of a two-party secure model shard exchange protocol between clients in an embodiment of the present application;

[0051] Figure 4 It is a principle block diagram of a security and privacy protection device based on federated cross-modal feature alignment in an embodiment of the present application;

[0052] Figure 5 It is a schematic structural diagram of a computer device in an embodiment of the present application. Detailed Embodiments

[0053] Embodiments of the present application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary only for explaining the present application and should not be construed as limiting the present application.

[0054] Those skilled in the art of the present technology can understand that unless specifically stated otherwise, the singular forms "a", "an", "the", and "said" used herein may also include the plural forms. It should be further understood that the term "including" used in the specification of the present application means the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is "connected" or "coupled" to another element, it can be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any unit and all combinations of one or more related listed items.

[0055] Those skilled in the art can understand that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as the general understanding of those of ordinary skill in the art to which this application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have a meaning consistent with the meaning in the context of the prior art, and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0056] Those skilled in the art can understand that the "client", "terminal", and "terminal device" used herein include both devices with a wireless signal receiver that only has the ability to receive and no ability to transmit, and devices with receiving and transmitting hardware that can perform two-way communication on a two-way communication link. Such devices can include: cellular or other communication devices such as personal computers, tablet computers, etc., which have a single-line display or a multi-line display or a cellular or other communication device without a multi-line display; PCS (Personal Communications Service), which can combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant), which can include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar, and / or GPS (Global Positioning System) receiver; conventional laptop and / or palm-top computers or other devices, which are conventional laptop and / or palm-top computers or other devices with and / or including a radio frequency receiver. The "client", "terminal", and "terminal device" used herein can be portable, transportable, installed in a vehicle (air, sea, and / or land), or suitable for and / or configured to operate locally, and / or operate in a distributed manner at any other location on the earth and / or in space. The "client", "terminal", and "terminal device" used herein can also be a communication terminal, an Internet access terminal, a music / video playback terminal, such as a PDA, MID (Mobile Internet Device), and / or a mobile phone with music / video playback functions, and can also be devices such as a smart TV, a set-top box, etc.

[0057] The hardware referred to by names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components disclosed by the von Neumann principle, including a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. A computer program is stored in its memory, and the central processing unit loads the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input / output devices to complete specific functions.

[0058] It should be noted that the concept of "server" in this application can similarly be extended to apply to the case of a server cluster. According to the network deployment principles understood by those skilled in the art, the various servers should be logically divided. Physically, these servers can either be independent of each other but can be called through an interface, or can be integrated into a single physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility and should not be restricted by this when implementing the network deployment method of this application.

[0059] One or several technical features of this application, unless expressly specified, can either be deployed on the server and accessed by the client remotely invoking the online service interface provided by the server, or can be directly deployed and run on the client for access.

[0060] The neural network models cited or potentially cited in this application, unless expressly specified, can either be deployed on a remote server and remotely invoked on the client, or can be directly invoked on the client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence can be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive consumption of the client's hardware operating resources.

[0061] All kinds of data involved in this application, unless expressly specified, can either be remotely stored on the server or stored on the local terminal device, as long as it is suitable for being invoked by the technical solution of this application.

[0062] Those skilled in the art should be aware that although the various methods of this application are described based on the same concept and thus show commonality with each other, unless otherwise specified, these methods can be executed independently. Similarly, for the various embodiments disclosed in this application, they are all proposed based on the same inventive concept. Therefore, for concepts with the same expression, as well as concepts that are only appropriately transformed for convenience although the concept expressions are different, they should be equivalently understood.

[0063] For each embodiment to be disclosed in this application, unless explicitly stated to be mutually exclusive, the relevant technical features involved in each embodiment can be cross - combined to flexibly construct new embodiments, as long as such combination does not deviate from the creative spirit of this application and can meet the needs in the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.

[0064] Please refer to Figure 1 、 Figure 2 and Figure 3 , in one embodiment of the security and privacy protection method based on federated cross - modal feature alignment of this application, it includes:

[0065] Step S10: Determine the single - modal feature extraction model and the cross - modal feature alignment model to be trained in the server, divide the parameters of the single - modal feature extraction model into multiple shards, and generate a server public key and a server private key, where each shard contains part of the parameters of the single - modal feature extraction model;

[0066] The security and privacy protection system based on federated cross - modal feature alignment in the terminal device can respond to the security and privacy protection instruction, determine the single - modal feature extraction model and the cross - modal feature alignment model to be trained in the server, divide the parameters of the single - modal feature extraction model into multiple shards, and generate a server public key and a server private key, where each shard contains part of the parameters of the single - modal feature extraction model;

[0067] In some embodiments, the single - modal feature extraction model refers to a model for extracting features for a single modality (such as images, text, or speech). A modality generally refers to the form or type of data, such as images, videos, audio, text, etc. During the process of single - modal processing, the model extracts meaningful feature representations from data of a specific modality, and these features are used for subsequent tasks (such as classification, retrieval, generation, etc.). The cross - modal feature alignment model is a model for feature alignment or mapping between different modalities. Different modalities (such as text and images) usually have differences in information expression and structure. Therefore, how to establish an effective association or alignment between different modalities has become an important research direction. The cross - modal alignment model ensures that features of different modalities can be effectively matched or associated in the same feature space by mapping the features.

[0068] In some embodiments, the single - modal feature extraction model is denoted as w modal , the cross - modal feature alignment model is denoted as w attention , the multiple shards are denoted as The server public key is denoted as pk s The server private key is denoted as sks ; The first client is denoted as c i , and the change amount of the first model parameters is denoted as Δw modal，i , and the change amount of the second model parameters is denoted as Δw attention，i ; The second client is denoted as c j , the model mask is denoted as m, and the first random key is denoted as r i ; The subset of the change amount of the first model parameters is denoted as The subset of the change amount of the second model parameters is denoted as The change amount of the third model parameters is denoted as The encryption parameter is denoted as The encryption key is denoted as Enc(r i , pk S ), and the hybrid model is denoted as The hybrid parameter is denoted as

[0069] Specifically, the server sets the model structure to be trained, which includes one or more single-modal feature extraction models w modal and cross-modal feature alignment model w attention , divides the parameters of the single-modal feature extraction model w modal into multiple shards, and generates a public-private key pair {pk s , sk s}, where pk s represents the server public key, and sk s represents the server private key. Each shard contains a part of the parameters of the single-modal feature extraction model w modal , and the complete model can be restored after all shards are combined.

[0070] Step S20: The first client obtains the multiple shards and the server public key, and trains the single-modal feature extraction model and the cross-modal feature alignment model to determine the change amount of the first model parameters corresponding to the single-modal feature extraction model and the change amount of the second model parameters corresponding to the cross-modal feature alignment model;

[0071] In the server, determine the single-modal feature extraction model and the cross-modal feature alignment model to be trained. After dividing the parameters of the single-modal feature extraction model into multiple shards and generating the server public key and server private key, the first client obtains the multiple shards and the server public key, and trains the single-modal feature extraction model and the cross-modal feature alignment model to determine the first model parameter change amount corresponding to the single-modal feature extraction model and the second model parameter change amount corresponding to the cross-modal feature alignment model; wherein, the first model parameter change amount represents the model parameter change amount updated after local training of the single-modal feature extraction model in the first client, and the second model parameter change amount represents the model parameter change amount updated after local training of the cross-modal feature alignment model in the first client.

[0072] Specifically, the first client c i obtains the single-modal feature extraction model w modal and the cross-modal feature alignment model w attention , after obtaining the multiple shards and the server public key, performs training using its own data locally to obtain the first model parameter change amount Δw corresponding to the single-modal feature extraction model modal，i and the second model parameter change amount Δw attention corresponding to the cross-modal feature alignment model w attention，i . Since the single-modal feature extraction model is directly related to the original data and has a greater privacy threat, this application performs privacy protection on the single-modal feature extraction model w modal .

[0073] Step S30: For the single-modal feature extraction model, the first client establishes a cooperative relationship with the second client, and generates a model mask and a first random key through a preset key exchange protocol, wherein the model mask is a bit string of length k, and each bit in the bit string corresponds to each shard one by one;

[0074] After the first client obtains the multiple shards and the server public key, and trains the single-modal feature extraction model and the cross-modal feature alignment model to determine the first model parameter change amount corresponding to the single-modal feature extraction model and the second model parameter change amount corresponding to the cross-modal feature alignment model, for the single-modal feature extraction model, the first client establishes a cooperative relationship with the second client, and generates a model mask and a first random key through a preset key exchange protocol, wherein the model mask is a bit string of length k, and each bit in the bit string corresponds to each shard one by one; wherein, the key exchange protocol is the Diffie-Hellman key exchange protocol.

[0075] Specifically, the first client c i Randomly matches or pre-agrees on a trusted second client (cooperating party) c j Pre-agrees on the cooperation relationship, and the specific information of the cooperation relationship will not be known to the server. In each round of training, both parties use the Diffie-Hellman key exchange protocol to generate a model mask m, which is a bit string of length k. Among them, each bit corresponds to each shard in a plurality of shards, and then, the first client c i Generates a random key r of the same length as the model parameters i .

[0076] Step S40: The first client performs sharding processing on the first model parameter change amount according to the model mask to generate a first model parameter change quantum set and a second model parameter change quantum set. The first client performs an exclusive OR operation on the second model parameter change quantum set using the first random key to generate an encryption parameter, and encrypts the first random key using the server public key to generate an encryption key, and sends the encryption parameter and the encryption key to the second client;

[0077] For the unimodal feature extraction model, after the first client and the second client establish a cooperation relationship, generate a model mask through a preset key exchange protocol, and generate a first random key, the first client performs sharding processing on the first model parameter change amount according to the model mask to generate a first model parameter change quantum set and a second model parameter change quantum set. The first client performs an exclusive OR operation on the second model parameter change quantum set using the first random key to generate an encryption parameter, and encrypts the first random key using the server public key to generate an encryption key, and sends the encryption parameter and the encryption key to the second client;

[0078] In some embodiments, the step in which the first client performs sharding processing on the first model parameter change amount according to the model mask to generate a first model parameter change quantum set and a second model parameter change quantum set includes:

[0079] The first client c i Performs sharding processing on the first model parameter change amount according to the model mask m to generate two parts of parameter updates, which include a first model parameter change quantum set and a second model parameter change quantum set Wherein, the first model parameter change quantum set Characterize the result of setting all the shards corresponding to the bits with value 0 in the model mask to zero, and the subset of the second model parameter change quantum Characterize the result of setting all the shards corresponding to the bits with value 1 in the model mask to zero.

[0080] In a further embodiment, the step that the first client performs an exclusive OR operation on the subset of the second model parameter change quantum with the first random key to generate an encryption parameter, encrypts the first random key with the server public key to generate an encryption key, and sends the encryption parameter and the encryption key to the second client includes:

[0081] The first client c i Uses the first random key r i To perform an exclusive OR operation on the subset of the second model parameter change quantum To generate an encryption parameter At the same time, uses the server public key pk s To encrypt the first random key r i To generate an encryption key Enc(r i , pk S ), and sends the encryption parameter And the encryption key Enc(r i , pk S ) to the second client (cooperating party) c j .

[0082] Step S50: The second client performs an exclusive OR operation on the encryption parameter and the third model parameter change amount to generate a hybrid model, and the first client and the second client upload the hybrid parameters, the encryption key, and the second model parameter change amount in the hybrid model to the server to complete the security and privacy protection based on federated cross-modal feature alignment.

[0083] After the first client performs sharding processing on the first model parameter change amount according to the model mask to generate a subset of the first model parameter change quantum and a subset of the second model parameter change quantum, the first client performs an exclusive OR operation on the subset of the second model parameter change quantum with the first random key to generate an encryption parameter, encrypts the first random key with the server public key to generate an encryption key, and sends the encryption parameter and the encryption key to the second client, the second client performs an exclusive OR operation on the encryption parameter and the third model parameter change amount to generate a hybrid model, and the first client and the second client upload the hybrid parameters, the encryption key, and the second model parameter change amount in the hybrid model to the server to complete the security and privacy protection based on federated cross-modal feature alignment.

[0084] In some embodiments, the step of the second client performing an exclusive OR operation on the encryption parameter and the third model parameter variation to generate a hybrid model includes:

[0085] The second client c j performs an exclusive OR operation on the encryption parameter and the third model parameter variation to generate a hybrid model, where the third model parameter variation represents the variation of the model parameters updated after local training of the unimodal feature extraction model in the second client, and the hybrid model is expressed as:

[0086]

[0087] In this process, the second client (cooperating party) cannot obtain the plaintext data of the original shard because the shard has been encrypted by the first random key r i and the key can only be decrypted by the server private key.

[0088] In a further embodiment, the step of the first client and the second client uploading the hybrid parameters, the encryption key, and the second model parameter variation in the hybrid model to the server includes:

[0089] The server receives the hybrid parameters in the hybrid model uploaded by the first client and the second client the encryption key Enc(r i , pk S ) and the second model parameter variation Δw attention，i , and uses the server private key sk s to decrypt the encryption key Enc(r i , pk S ) to obtain the first random key r i , and decrypts the hybrid parameters through an exclusive OR operation to restore the model update, which is expressed as:

[0090]

[0091] The server aggregates the updated unimodal feature extraction models and the updated cross-modal feature alignment models of all clients respectively to determine a first aggregation result and a second aggregation result, where the first aggregation result is expressed as w modal ←w modal +∑(Δw j ) mix, the second aggregation result is determined by aggregating the second model parameter change Δw using an importance weighted algorithm or a robust aggregation algorithm attention，i through aggregation.

[0092] Specifically, after the first client and the second client upload the hybrid parameters in the hybrid model the encryption key Enc(r i , pk S ) and the second model parameter change Δw attention，i to the server, the server receives the hybrid parameters in the hybrid model uploaded by the first client and the second client the encryption key Enc(r i , pk S ) and the second model parameter change Δw attention，i . Using the server private key sk s to decrypt the encryption key Enc(r i , pk S ) to obtain the first random key r i , and decrypting the hybrid parameters through an XOR operation to restore the model update, which is expressed as:

[0093]

[0094] The server aggregates the updated single-modal feature extraction models and the updated cross-modal feature alignment models of all clients respectively to determine the first aggregation result and the second aggregation result. Among them, the first aggregation result is expressed as w modal ←w modal +∑(Δw j ) mix . The sharding exchange strategy ensures that the result after exchange is the same as that before exchange. At the same time, since the server cannot know the mask and cooperation relationship, it cannot reconstruct the original model, ensuring privacy protection. The second aggregation result is determined by aggregating the second model parameter change Δw attention，i through aggregation to further improve the model performance and robustness.

[0095] Please refer to Figure 4, A security and privacy protection device based on federated cross-modal feature alignment provided to meet one of the purposes of this application, including a model parameter sharding module 1100, a parameter change amount determination module 1200, a model mask determination module 1300, an encryption key determination module 1400, and a security and privacy protection module 1500. Among them, the model parameter sharding module 1100 is configured to determine a single-modal feature extraction model and a cross-modal feature alignment model to be trained in the server, divide the parameters of the single-modal feature extraction model into multiple shards, and generate a server public key and a server private key, where each shard contains part of the parameters of the single-modal feature extraction model; the parameter change amount determination module 1200 is configured to the first client obtains the multiple shards and the server public key, trains the single-modal feature extraction model and the cross-modal feature alignment model to determine a first model parameter change amount corresponding to the single-modal feature extraction model and a second model parameter change amount corresponding to the cross-modal feature alignment model; the model mask determination module 1300 is configured to for the single-modal feature extraction model, the first client establishes a cooperative relationship with the second client, generates a model mask through a preset key exchange protocol, and generates a first random key, where the model mask is a bit string with a length of k, and each bit in the bit string corresponds to each shard one by one; the encryption key determination module 1400 is configured to the first client performs sharding processing on the first model parameter change amount according to the model mask to generate a first model parameter change amount subset and a second model parameter change amount subset, the first client performs an exclusive OR operation on the second model parameter change amount subset with the first random key to generate an encrypted parameter, encrypts the first random key with the server public key to generate an encryption key, and sends the encrypted parameter and the encryption key to the second client; the security and privacy protection module 1500 is configured to the second client performs an exclusive OR operation on the encrypted parameter and a third model parameter change amount to generate a mixed model, and the first client and the second client upload the mixed parameters, the encryption key, and the second model parameter change amount in the mixed model to the server to complete the security and privacy protection based on federated cross-modal feature alignment.

[0096] Based on any embodiment of this application, please refer to Figure 5 , Another embodiment of this application further provides an electronic device, which can be implemented by a computer device, such as Figure 5As shown, it is a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected through a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database can store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a security and privacy protection method based on federated cross-modal feature alignment. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device can store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the security and privacy protection method based on federated cross-modal feature alignment of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art can understand that Figure 5 The structure shown in the figure is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0097] In this embodiment, the processor is used to execute Figure 4 the specific functions of each module in the figure. The memory stores the program code and various types of data required to execute the above modules. The network interface is used for data transmission between the user terminal and the server. The memory in this embodiment stores the program code and data required to execute all modules in the security and privacy protection device based on federated cross-modal feature alignment of the present application. The server can call the program code and data of the server to execute the functions of all modules.

[0098] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors are caused to execute the steps of the security and privacy protection method based on federated cross-modal feature alignment described in any embodiment of the present application.

[0099] The present application also provides a computer program product, including a computer program / instructions. When the computer program / instructions are executed by one or more processors, the steps of the security and privacy protection method based on federated cross-modal feature alignment described in any embodiment of the present application are implemented.

[0100] Those of ordinary skill in the art can understand that all or part of the processes in the above-described embodiments of the method of the present application can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-described methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0101] The above are only some embodiments of the present application. It should be noted that for those of ordinary skill in the art of the present technology, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A security and privacy protection method based on federated cross-modal feature alignment, characterized in that: include: Determine the unimodal feature extraction model and the cross-modal feature alignment model to be trained on the server, divide the parameters of the unimodal feature extraction model into multiple slices, and generate a server public key and a server private key, wherein each slice contains part of the parameters of the unimodal feature extraction model; The first client obtains the multiple shards and the server public key, and trains the unimodal feature extraction model and the cross-modal feature alignment model to determine a first model parameter change corresponding to the unimodal feature extraction model and a second model parameter change corresponding to the cross-modal feature alignment model; For the single-modal feature extraction model, the first client establishes a cooperative relationship with the second client, and generates a model mask and a first random key through a preset key exchange protocol, wherein the model mask is a bit string with a length of k, and each bit in the bit string corresponds to each shard one by one; The first client performs slicing processing on the first model parameter variation according to the model mask to generate a first model parameter variation sub-set and a second model parameter variation sub-set, the first client performs an XOR operation on the second model parameter variation sub-set using the first random key to generate an encryption parameter, encrypts the first random key using the server public key to generate an encryption key, and sends the encryption parameter and the encryption key to the second client; The second client performs an XOR operation on the encryption parameter and the third model parameter change to generate a hybrid model. The first client and the second client upload the hybrid parameters in the hybrid model, the encryption key and the second model parameter change to the server to complete security and privacy protection based on federated cross-modal feature alignment.

2. The security and privacy protection method based on federated cross-modal feature alignment according to claim 1 is characterized in that: The unimodal feature extraction model is represented as w modal , the cross-modal feature alignment model is represented as w attention , the multiple slices are represented as The server public key is represented by pk s , the server private key is represented by sk s ; The first client is denoted as c i , the first model parameter change is expressed as Δw modal，i , the second model parameter change is expressed as Δw attention，i ; The second client is denoted as c j , the model mask is denoted as m, and the first random key is denoted as r i ; The first model parameter variation subset is expressed as The second model parameter change sub-set is expressed as The third model parameter variation is expressed as The encryption parameters are expressed as The encryption key is represented as Enc(r i , pk S ), the mixed model is expressed as The mixing parameter is expressed as 3. The security and privacy protection method based on federated cross-modal feature alignment according to claim 2 is characterized in that: The first client performs slicing processing on the first model parameter variation according to the model mask to generate a first model parameter variation sub-set and a second model parameter variation sub-set, including: The first client c i The first model parameter change amount is sliced ​​according to the model mask m to generate a two-part parameter update, which includes a first model parameter change sub-set And the second model parameter change quantum set Among them, the first model parameter change quantum set represents the result of setting all the slices corresponding to the bit 0 in the model mask to zero, and the second model parameter change quantum set Represents the result of setting all the slices corresponding to the bit 1 in the model mask to zero.

4. The security and privacy protection method based on federated cross-modal feature alignment according to claim 2 is characterized in that: The first client uses the first random key to perform an XOR operation on the second model parameter variation subset to generate an encryption parameter, uses the server public key to encrypt the first random key to generate an encryption key, and sends the encryption parameter and the encryption key to the second client, including: The first client c i Using the first random key r i The second model parameter variation subset Perform an XOR operation to generate encryption parameters At the same time, the server public key pk is used s For the first random key r i Encryption is performed to generate the encryption key Enc(r i , pk S ), and the encryption parameters And the encryption key Enc(r i , pk S ) is sent to the second client.

5. The security and privacy protection method based on federated cross-modal feature alignment according to claim 2 is characterized in that: The step of the second client performing an XOR operation on the encryption parameter and the third model parameter variation to generate a hybrid model includes: The second client c j The encryption parameters The third model parameter changes An XOR operation is performed to generate a hybrid model, wherein the third model parameter change represents the model parameter change of the single-modal feature extraction model updated after local training in the second client, and the hybrid model is expressed as:

6. The security and privacy protection method based on federated cross-modal feature alignment according to any one of claims 1 to 5, characterized in that: The step of the first client and the second client uploading the mixed parameters in the mixed model, the encryption key and the second model parameter change amount to the server includes: The server receives the mixed parameters in the mixed model uploaded by the first client and the second client. The encryption key Enc(r i , pk S ) and the second model parameter change Δw attention，i , using the server private key sk s The encryption key Enc(r i , pk S ) is decrypted to obtain the first random key r i , and decrypt the mixed parameter by XOR operation To restore the model update, it is expressed as: The server aggregates the updated unimodal feature extraction models and the updated cross-modal feature alignment models of all clients to determine a first aggregation result and a second aggregation result, wherein the first aggregation result is represented by w modal ←w modal +∑(Δw j ) mix The second aggregation result is obtained by using an importance weighted algorithm or a robust aggregation algorithm to calculate the change Δw of the second model parameter. attention，i Determined by aggregation.

7. The security and privacy protection method based on federated cross-modal feature alignment according to any one of claims 1 to 5, characterized in that: The first model parameter change represents the model parameter change of the unimodal feature extraction model updated after local training in the first client, the second model parameter change represents the model parameter change of the cross-modal feature alignment model updated after local training in the first client, and the key exchange protocol is the Diffie-Hellman key exchange protocol.

8. A security and privacy protection device based on federated cross-modal feature alignment, characterized in that: include: A model parameter sharding module, configured to determine the unimodal feature extraction model and the cross-modal feature alignment model to be trained in the server, divide the parameters of the unimodal feature extraction model into multiple shards, and generate a server public key and a server private key, wherein each shard contains part of the parameters of the unimodal feature extraction model; A parameter change determination module is configured to obtain the multiple shards and the server public key for the first client, and train the unimodal feature extraction model and the cross-modal feature alignment model to determine a first model parameter change corresponding to the unimodal feature extraction model and a second model parameter change corresponding to the cross-modal feature alignment model; A model mask determination module is configured to establish a cooperative relationship between the first client and the second client for the unimodal feature extraction model, and generate a model mask through a preset key exchange protocol, and generate a first random key, wherein the model mask is a bit string with a length of k, and each bit in the bit string corresponds to each shard one by one; an encryption key determination module, configured such that the first client performs slicing processing on the first model parameter variation according to the model mask to generate a first model parameter variation sub-set and a second model parameter variation sub-set, the first client performs an XOR operation on the second model parameter variation sub-set using the first random key to generate an encryption parameter, encrypts the first random key using the server public key to generate an encryption key, and sends the encryption parameter and the encryption key to the second client; The security and privacy protection module is configured such that the second client performs an XOR operation on the encryption parameter and the change in the third model parameter to generate a hybrid model, and the first client and the second client upload the hybrid parameters in the hybrid model, the encryption key and the change in the second model parameter to the server to complete the security and privacy protection based on federated cross-modal feature alignment.

9. An electronic device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images