A multi-terminal collaborative federated online cross-media matching method and system

Abstract

The present invention belongs to the field of vision and artificial intelligence technology and provides a multi-terminal collaborative federated online cross-media matching method and system. Its technical solution is as follows: extracting the features of each modality of multiple media resources, mapping each modal feature into a common subspace to obtain a common representation; performing online clustering based on the common representation, combining the cluster centers obtained by online clustering into local prototypes, encrypting the local prototypes and local hash models to obtain local encrypted prototypes and local encrypted models; combining the local encrypted models and local encrypted prototypes with a global model to obtain a global encrypted model and a global encrypted prototype; training a local cross-modal retrieval model based on the global model, the global encrypted model, and the global encrypted prototype to obtain a trained cross-modal retrieval model; and generating corresponding resource matching results based on the trained cross-modal model. This method greatly reduces the model bias problem caused by the non-independent and identically distributed problem of local data on different clients.

Description

Technical Field

[0001] The present invention belongs to the field of vision and artificial intelligence technology, and in particular relates to a multi-terminal collaborative federated online cross-media matching method and system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the continuous development of the Internet of Things (IoT), artificial intelligence (AI), and data privacy protection technologies, privacy protection in multi-device collaboration has become a key research and application area. In multi-device, multi-node collaborative scenarios such as smart homes, smart cities, the Internet of Things (IoT), and the Internet of Vehicles (IoV), data collection, transmission, and analysis must balance efficient collaboration with privacy and security.

[0004] A range of robotic systems, from sweeping robots in smart homes to robot dogs for outdoor operations and drone swarms for low-altitude operations, have seen widespread adoption and development in recent years. These systems integrate a variety of technologies, including artificial intelligence, sensors, robotic navigation systems, and the Internet of Things. For example, different users can control a sweeping robot via voice commands to clean a designated area. The robot uses cameras to collect environmental data (such as room images and furniture placement) for navigation and recognition. However, these images may contain sensitive user data (such as home furnishings and personal belongings), so private data cannot be centrally collected or uploaded to the cloud for processing. Inspired by federated learning, different users' sweeping robots can collaboratively train a voice-image cross-media matching model through federated learning. This model enables flexible understanding of voice commands and subsequent robot navigation, positioning, and path planning. This improves the overall accuracy of the sweeping robot and provides an efficient solution for household cleaning. However, the private data of different users' robots is often non-IID. Directly training local models and then aggregating them into a global model can lead to severe model bias. Therefore, designing a method to effectively mitigate the model bias caused by the disparity in the distribution of local private multimedia data in distributed scenarios has become an unresolved problem. With federated learning, client-side private models need to be uploaded to the cloud for aggregation and then updated to the client. This process may involve privacy risks such as model parameter leakage. Therefore, designing a solution for aggregation in an encrypted state is crucial.

[0005] Federated learning is a machine learning model that enables collaborative training without compromising privacy and maintaining data decentralization. Existing cross-media matching methods are mostly integrated with the Federated Averaging (FedAvg) framework. While the FedAvg algorithm protects user data privacy, it still presents the risk of model inversion attacks, where attackers can obtain information about participating devices by analyzing model updates. In scenarios with multiple client robots, using the FedAvg algorithm to process robots with different data distributions can degrade the performance of the trained model, with poor performance in certain categories, leading to bias in the global model. Summary of the Invention

[0006] In order to solve at least one technical problem existing in the above-mentioned background technology, the present invention provides a multi-terminal collaborative federated online cross-media matching method and system, which combines federated learning and online cross-modal retrieval for the first time. While satisfying the local cross-media retrieval model of the operating robot to process the streaming information data in the room, it realizes federated aggregation on the cloud server, and also realizes the protection of data privacy in the user's room.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A first aspect of the present invention provides a multi-terminal collaborative federated online cross-media matching method, comprising the following steps:

[0009] Perform feature extraction on the acquired multiple media resources to obtain corresponding modal features, and map each modal feature into a common subspace to obtain a common representation;

[0010] Perform online clustering based on the common representation, combine the cluster centers obtained by online clustering into local prototypes, encrypt the local prototypes to obtain local encrypted prototypes; encrypt the local cross-modal retrieval model to obtain a local encrypted model;

[0011] The global encrypted model and global encrypted prototype are obtained by aggregating the local encrypted model and the local encrypted prototype. The encrypted global model is decrypted and used as the local cross-modal retrieval model for this round. The decrypted encrypted global prototype is used to guide the generation of the hash code of the local cross-modal retrieval model. After multiple rounds of training, the final cross-modal retrieval model is obtained.

[0012] Based on the trained cross-modal model, a resource matching result corresponding to the media resource to be matched is generated.

[0013] Furthermore, the BFV homomorphic encryption method is adopted when homomorphically encrypting the local prototype and the local hash model.

[0014] Furthermore, when performing homomorphic encryption on the local prototype and the local cross-modal retrieval model, secure encryption parameters are selected, including the modulus, the size of the polynomial degree and the noise parameter, to generate the public key pk , using the public key pk Local prototype Encryption using public key pk Weighting the local hash model Encryption is performed.

[0015] Furthermore, the aggregation formula of the global encryption model is:

[0016] ,

[0017] in, express t The global encryption model of round time, express t+ Global encryption model for 1 round, is a positive number that controls the degree of deviation between the local model and the global model. , and express t+ 1 round of peace t Local encryption model for rounds, Indicates the number of samples of newly arrived data for each client;

[0018] The aggregation formula of the global encryption prototype is:

[0019] ,

[0020] in, express t+ The global encryption prototype for 1 round, is the local encrypted prototype of the K-th client, and K is the total number of clients. Furthermore, the objective function of each round of local cross-modal retrieval model training is:

[0021] ,

[0022] ,

[0023] ,

[0024] ,

[0025] ,

[0026] in, is the set of variables to be optimized, is the modal balance parameter, Represents the basis vectors of different modes, is the feature-label association matrix, and is the hash code corresponding to round t-1 and round t, is a hash function, is the offset correction matrix, It's the Hadamard label. and is the modal feature representation corresponding to the t-1th round and the tth round, and are the new labels corresponding to rounds t-1 and t, represents the projection matrix, Represents the global prototype, Represents the prototype relationship of the samples, obtained from the local client Kmeans clustering process, Represents the client prototype, Represents the sample prototype relationship, and is the balance parameter, Represents the similarity relationship matrix between new and old data, Represents the similarity relationship matrix within the new data, represents the weight of the global model, Represents the mapping between new sample features and hash codes, represents the identity matrix, and represents the original one-hot labels corresponding to round t-1 and round t, Represents the total number of samples arriving at time t.

[0027] Furthermore, the overall objective function of multiple rounds of training for the cross-modal retrieval model is:

[0028] ,

[0029] in, Indicates the k The number of samples of clients, Indicates the total number of client samples participating in the aggregation, represents the client-side local cross-modal objective function, Indicates the k local cross-modal hash model weights, Represents the global model weight aggregated from multiple client model weights, is a positive number that controls the degree of deviation between the local model and the global model.

[0030] A second aspect of the present invention provides a multi-terminal collaborative federated online cross-media matching system, comprising:

[0031] The model encryption module is used to extract features from the acquired multiple media resources to obtain corresponding features of each modality, map the features of each modality into a common subspace to obtain a common representation; perform online clustering based on the common representation, combine the cluster centers obtained by online clustering into a local prototype, encrypt the local prototype to obtain a local encrypted prototype; and encrypt the local cross-modal retrieval model to obtain a local encrypted model.

[0032] The encryption aggregation module is used to aggregate the local encryption model and the local encryption prototype to obtain the global encryption model and global encryption prototype. The encrypted global model is decrypted and used as the local cross-modal retrieval model for this round. The decrypted encrypted global prototype is used to guide the generation of the hash code of the local cross-modal retrieval model. After multiple rounds of training, the final cross-modal retrieval model is obtained.

[0033] Online cross-media matching is used to generate resource matching results corresponding to the media resources to be matched based on the trained cross-modal model.

[0034] A third aspect of the present invention provides a computer-readable storage medium.

[0035] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the multi-terminal collaborative federated online cross-media matching method as described above.

[0036] A fourth aspect of the present invention provides a computer device.

[0037] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the multi-terminal collaborative federated online cross-media matching method described above are implemented.

[0038] A fifth aspect of the present invention provides a computer device.

[0039] A program product includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the multi-terminal collaborative federated online cross-media matching method described above are implemented.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. This invention introduces an online cross-media matching hashing method for local cross-media matching model training, which is more suitable for multi-client operation robots to process streaming data in real-world operation scenarios. At the same time, it introduces prototype learning to transfer unified global knowledge to different modalities of local clients, greatly reducing the model bias caused by the non-independent and identically distributed problem of local data on different clients.

[0042] 2. Under the framework of federated learning, although the global prototype and global model obtained by the aggregation process do not contain sensitive data from a single client, this process still poses the risk of data leakage. Data thieves are likely to infer the user's private data through the leaked data. Therefore, the introduction of homomorphic encryption greatly protects privacy security.

[0043] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0045] Figure 1 This is a flow chart of a multi-terminal collaborative federated online cross-media matching method provided by an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of a multi-terminal collaborative federated online cross-media matching method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0048] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0049] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0050] Considering that in scenarios with multiple user robots, data is often not collected all at once, but is collected by the robots in the form of streaming data, the traditional methods of federated learning and cross-media matching for periodic batch processing may not be able to meet the dual requirements of real-time and efficient retrieval of large-scale data. New cross-media processing strategies, such as streaming data processing, are gradually becoming key tools to address this challenge. Due to the particularity of online data, it is necessary to meet the requirements of model learning new data while not losing generalization to old data. Therefore, the present invention combines federated learning and online cross-modal retrieval for the first time. While meeting the requirements of the local cross-media retrieval model of the robot to process the streaming information data in the room, it also realizes federated aggregation on the cloud server and protects the privacy of the data in the user's room. For the streaming data collected by multiple client robots, the Fedprox federated learning aggregation framework is used, combined with the local cross-media hashing online algorithm to collaboratively train a global model. In addition, prototype learning and homomorphic encryption algorithms are introduced to better alleviate the model bias of different robots and protect user privacy, so that multiple client robots can train a global cross-media matching model without sharing private data.

[0051] Example 1

[0052] like Figure 1 and Figure 2 As shown, this embodiment provides a multi-terminal collaborative federated online cross-media matching method, including the following steps:

[0053] Step 1: Obtain multiple media resources;

[0054] In this embodiment, media resources include image resources and audio resources; they are acquired by a working robot configured by the client.

[0055] Step 2: Extract features from the acquired media resources to obtain corresponding modal features, and map the modal features into a common subspace to obtain a common representation.

[0056] Step 3: Perform online clustering based on the common representation, combine the cluster centers obtained by online clustering into local prototypes, and perform homomorphic encryption on the local prototypes to obtain local encrypted prototypes;

[0057] Each client learns the local prototype through the Kmeans clustering algorithm locally ,in Indicates the k local prototypes for each customer, L Indicates the number of local prototypes;

[0058] When performing homomorphic encryption on the local prototype, select secure encryption parameters, including the modulus, the size of the polynomial degree and the noise parameter, and generate the public key pk and private key sk ; Use public key pk Local prototype Encrypt and calculate the ciphertext to get and upload the local encrypted prototype to the cloud;

[0059] Step 4: Perform homomorphic encryption on the local cross-modal retrieval model to obtain a local encrypted model;

[0060] When performing homomorphic encryption on the local cross-modal retrieval model, the BFV homomorphic encryption method is used to select secure encryption parameters, including the modulus, the size of the polynomial degree and the noise parameter, to generate the public key. pk and private key sk ; Use public key pk Weighting the local hash model Encrypt and calculate the ciphertext to get and upload the local encrypted model to the cloud;

[0061] Step 5: Aggregate the local encrypted model and the local encrypted prototype to obtain the global encrypted model and the global encrypted prototype. Then distribute the global encrypted model and the global encrypted prototype to each client. Decrypt the encrypted global model and use it as the local cross-modal retrieval model for this round. Decrypt the encrypted global prototype and use it to guide the generation of the hash code of the cross-modal retrieval model. After multiple rounds of training, the final local cross-modal retrieval model is obtained.

[0062] The specific steps include:

[0063] Step 501: Aggregate the local encryption model and the local encryption prototype on the server to obtain the global encryption model and the global encryption prototype, and then distribute the global encryption model and the global encryption prototype to each client;

[0064] Specifically include:

[0065] Assume there is k Clients participate in the aggregation, t The global model of the round time is , the local encryption model is .

[0066] t When new data arrives in round +1, the local encryption model is updated to , then the model update for each client is Therefore, when new data arrives at time t+1, the aggregation formula of the global encryption model is:

[0067] ,

[0068] in, Indicates the number of samples of newly arrived data for each client, is a positive number that controls the degree of deviation between the local model and the global model.

[0069] t At time +1, the operation of aggregating the local encrypted prototype of the new data to obtain the global prototype is as follows:

[0070] ,

[0071] in, for t+ The global encryption prototype for 1 round, is the local encryption prototype of the K-th client; K is the total number of clients.

[0072] To mitigate model bias caused by data distribution differences between different clients, the client performs prototype learning and uploads the local prototype and local model to the cloud for aggregation. Under the framework of federated learning, although the global prototype and global model obtained by the aggregation process do not contain sensitive data from a single client, this process still poses the risk of data leakage. Data thieves are likely to infer the user's private data through the leaked data. Therefore, the introduction of homomorphic encryption protects privacy security to a great extent.

[0073] Step 502: The global encrypted model is decrypted and used as the cross-modal retrieval model of the local client in this round. The global encrypted prototype is decrypted and used to guide the generation of the hash code of the local cross-modal retrieval model.

[0074] In each round of online training of the cross-modal retrieval model by the local client, a new online cross-modal hashing method called SEOCH (Semantic Embedding Online Cross-modal Hashing) is designed, whose objective function is defined as :

[0075] ,

[0076] ,

[0077] ,

[0078] ,

[0079] ,

[0080] in, is the set of variables to be optimized, is the modal balance parameter, Represents the basis vectors of different modes, is the feature-label association matrix, and is the hash code corresponding to round t-1 and round t, is a hash function, is the offset correction matrix, It's the Hadamard label. and is the modal feature representation corresponding to the t-1th round and the tth round, and are the new labels corresponding to rounds t-1 and t, represents the projection matrix, Represents the global prototype, Represents the prototype relationship of the samples, obtained from the local client Kmeans clustering process, Represents the client prototype, Represents the sample prototype relationship, and is the balance parameter, Represents the similarity relationship matrix between new and old data, Represents the similarity relationship matrix within the new data, represents the weight of the global model, Represents the mapping between new sample features and hash codes, represents the identity matrix, and represents the original one-hot labels corresponding to round t-1 and round t, Represents the total number of samples arriving at time t.

[0081] Data collected by robots from different clients typically arrives continuously in a streaming manner. Online cross-media hash learning ensures that only new data is used to update the hash model. It also generates hash codes for the new data and updates the hash codes for the old data. This avoids consuming large amounts of memory and greatly improves the efficiency of training streaming data.

[0082] Step 503: Repeat steps 501 and 502. After multiple rounds of training, the final cross-modal retrieval model is obtained.

[0083] In this embodiment, the overall aggregation objective function is:

[0084] ,

[0085] in, Indicates the k The number of samples of clients, Indicates the total number of client samples participating in the aggregation, represents the client-side local cross-modal objective function, Indicates the k local cross-modal hash model weights, Represents the global model weight aggregated from multiple client model weights. is a positive number that controls the degree of deviation between the local model and the global model.

[0086] Step 6: Generate resource matching results corresponding to the media resources to be matched based on the trained cross-modal model;

[0087] The matching can be implemented using existing methods, such as calculating the hash code distance in the Hamming space and matching the corresponding resources based on the distance.

[0088] Table 1 shows a simulation experiment of the proposed method. This experiment evaluates the MAP (mean average precision) of I2T (text retrieval from image) and T2I (text retrieval from image) at different bit positions on the NUS-WIDE and Mirflickr 25K datasets.

[0089] Table 1 Comparison of MAP performance of the present invention and other methods

[0090]

[0091] The data in Table 1 are MAP performance comparisons of the FedSEOCH algorithm of the present invention and other algorithms, with the best performance highlighted in bold. Table 1 shows that the performance of the method of the present invention is superior to other methods.

[0092] The present invention provides a multi-terminal collaborative federated online cross-media matching method, which introduces federated learning, prototype learning, homomorphic encryption and online cross-media hashing methods to train a federated online cross-media matching model for multi-user end operation robots in multiple scenarios. While protecting user privacy, it greatly improves the work efficiency of the operation robots.

[0093] Example 2

[0094] This embodiment provides a multi-terminal collaborative federated online cross-media matching system, including:

[0095] The model encryption module is used to extract features from the acquired multiple media resources to obtain corresponding features of each modality, map the features of each modality into a common subspace to obtain a common representation; perform online clustering based on the common representation, combine the cluster centers obtained by online clustering into a local prototype, encrypt the local prototype to obtain a local encrypted prototype; and encrypt the local cross-modal retrieval model to obtain a local encrypted model.

[0096] The encryption aggregation module is used to aggregate the local encryption model and the local encryption prototype to obtain the global encryption model and global encryption prototype. The encrypted global model is decrypted and used as the local cross-modal retrieval model for this round. The decrypted encrypted global prototype is used to guide the generation of the hash code of the local cross-modal retrieval model. After multiple rounds of training, the final cross-modal retrieval model is obtained.

[0097] Online cross-media matching is used to generate resource matching results for matching media resources based on the trained cross-modal model.

[0098] It should be noted that the specific implementation method of a multi-terminal collaborative federated online cross-media matching system in an embodiment of the present invention is similar to the specific implementation method of a multi-terminal collaborative federated online cross-media matching method in an embodiment of the present invention. Please refer to the description of the method part for details. In order to reduce redundancy, it will not be repeated here.

[0099] Example 3

[0100] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps in the multi-terminal collaborative federated online cross-media matching method described above are implemented.

[0101] Example 4

[0102] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the multi-terminal collaborative federated online cross-media matching method described above are implemented.

[0103] Example 5

[0104] This embodiment provides a program product, which is a computer program product and includes a computer program. When the computer program is executed by a processor, the steps in the multi-terminal collaborative federated online cross-media matching method described above are implemented.

[0105] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A multi-terminal collaborative federated online cross-media matching method, characterized in that: The steps include: Perform feature extraction on the acquired multiple media resources to obtain corresponding modal features, and then map each modal feature into a common subspace to obtain a common representation; Perform online clustering based on the common representation, combine the cluster centers obtained by online clustering into local prototypes, encrypt the local prototypes, and obtain local encrypted prototypes; Encrypt the local cross-modal retrieval model to obtain a local encrypted model; Aggregate the local encrypted model and the local encrypted prototype to obtain the global encrypted model and global encrypted prototype, distribute the global encrypted model and global encrypted prototype to each client, decrypt the global encrypted model and use it as the local cross-modal retrieval model for this round, and decrypt the global encrypted prototype to guide the generation of the hash code of the local cross-modal retrieval model. After multiple rounds of training, the final cross-modal retrieval model is obtained. The aggregation formula of the global encryption model is: ， in, express t The global encryption model of round time, express t The global encryption model at +1 round, is a positive number that controls the degree of deviation between the local model and the global model. , and express t +1 round time t Local encryption model for rounds, Indicates the number of samples of newly arrived data for each client; The aggregation formula of the global encryption prototype is: ， in, express t +1 round of global encryption prototype, is the local encryption prototype of the K-th client, where K is the total number of clients; The objective function for each round of local cross-modal retrieval model training is: ， ， ， ， ， in, is the set of variables to be optimized, is the modal balance parameter, Represents the basis vectors of different modes, is the feature-label association matrix, and is the hash code corresponding to round t-1 and round t, is a hash function, is the offset correction matrix, It's the Hadamard label. and is the modal feature representation corresponding to the t-1th round and the tth round, and are the new labels corresponding to rounds t-1 and t, represents the projection matrix, Represents the global prototype, Represents the prototype relationship of the samples, obtained from the local client Kmeans clustering process, Represents the client prototype, Represents the sample prototype relationship, and is the balance parameter, Represents the similarity relationship matrix between new and old data, Represents the similarity relationship matrix within the new data, represents the weight of the global model, Represents the mapping between new sample features and hash codes, represents the identity matrix, and represents the original one-hot labels corresponding to round t-1 and round t, Represents the total number of samples arriving at time t; Based on the trained cross-modal model, a resource matching result corresponding to the media resource to be matched is generated.

2. The multi-terminal collaborative federated online cross-media matching method according to claim 1, characterized in that: When performing homomorphic encryption on the local prototype and local cross-modal retrieval model, the BFV homomorphic encryption method is adopted.

3. The multi-terminal collaborative federated online cross-media matching method according to claim 1, characterized in that: When performing homomorphic encryption on the local prototype and local cross-modal retrieval model, select secure encryption parameters, including the modulus, the size of the polynomial degree, and the noise parameter, and generate the public key pk , using the public key pk Local prototype Encryption using public key pk Weighting the local hash model Encryption is performed.

4. The multi-terminal collaborative federated online cross-media matching method according to claim 1, characterized in that: The overall objective function of multiple rounds of training for the cross-modal retrieval model is: ， in, Indicates the k The number of samples of clients, Indicates the total number of client samples participating in the aggregation, represents the client-side local cross-modal objective function, Indicates the k local cross-modal hash model weights, Represents the global model weight aggregated from multiple client model weights, is a positive number that controls the degree of deviation between the local model and the global model.

5. A multi-terminal collaborative federated online cross-media matching system, characterized by: include: The model encryption module is used to extract features from the acquired multiple media resources to obtain corresponding modal features, map the modal features into a common subspace, and obtain a common representation; perform online clustering based on the common representation, combine the cluster centers obtained by online clustering into a local prototype, and encrypt the local prototype to obtain a local encrypted prototype; Encrypt the local cross-modal retrieval model to obtain a local encrypted model; The encryption aggregation module is used to aggregate the local encryption model and the local encryption prototype to obtain the global encryption model and the global encryption prototype. The global encryption model and the global encryption prototype are distributed to each client. The global encryption model is decrypted and used as the local cross-modal retrieval model for this round. The decrypted global encryption prototype is used to guide the generation of the hash code of the local cross-modal retrieval model. After multiple rounds of training, the final cross-modal retrieval model is obtained. The aggregation formula of the global encryption model is: ， in, express t The global encryption model of round time, express t The global encryption model at +1 round, is a positive number that controls the degree of deviation between the local model and the global model. , and express t +1 round time t Local encryption model for rounds, Indicates the number of samples of newly arrived data for each client; The aggregation formula of the global encryption prototype is: ， in, express t +1 round of global encryption prototype, is the local encryption prototype of the K-th client, where K is the total number of clients; The objective function for each round of local cross-modal retrieval model training is: ， ， ， ， ， in, is the set of variables to be optimized, is the modal balance parameter, Represents the basis vectors of different modes, is the feature-label association matrix, and is the hash code corresponding to round t-1 and round t, is a hash function, is the offset correction matrix, It's the Hadamard label. and is the modal feature representation corresponding to the t-1th round and the tth round, and are the new labels corresponding to rounds t-1 and t, represents the projection matrix, Represents the global prototype, Represents the prototype relationship of the samples, obtained from the local client Kmeans clustering process, Represents the client prototype, Represents the sample prototype relationship, and is the balance parameter, Represents the similarity relationship matrix between new and old data, Represents the similarity relationship matrix within the new data, represents the weight of the global model, Represents the mapping between new sample features and hash codes, represents the identity matrix, and represents the original one-hot labels corresponding to round t-1 and round t, Represents the total number of samples arriving at time t; Online cross-media matching is used to generate resource matching results corresponding to the media resources to be matched based on the trained cross-modal model.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of a multi-terminal collaborative federated online cross-media matching method as described in any one of claims 1 to 4 are implemented.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the multi-terminal collaborative federated online cross-media matching method according to any one of claims 1 to 4 are implemented.

8. A program product, wherein the program product is a computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the steps of the multi-terminal collaborative federated online cross-media matching method according to any one of claims 1 to 4 are implemented.

Images