Method and system for federated prompt tuning of black-box vision-language models

Abstract

The application is suitable for the field of artificial intelligence technology, and provides a federal prompt tuning method of a black box vision-language model, comprising: receiving a global prompt issued by a federal server; constructing a semantic base and performing prompt reconstruction on a text prompt according to the semantic base; calling a vision-language model interface to perform feature extraction on the reconstructed text prompt and a class label in local sample data; performing prompt reconstruction on a visual prompt based on a pre-constructed discrete cosine inverse transform base matrix; calling a model interface to perform feature extraction on the reconstructed visual prompt and a sample image in the local sample data; calculating a classification prediction probability according to semantic features and visual features, calculating a loss value according to the classification prediction probability and the class label, and performing iterative zero-order optimization on the text prompt and the visual prompt according to the loss value to obtain a local optimized prompt; and uploading the local optimized prompt to the federal server for aggregation. The application can improve the federal prompt tuning efficiency of the black box vision-language model.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a federated prompting optimization method and system for black-box visual-language models. Background Technology

[0002] The rapid development of large-scale visual-language models, such as Contrastive Language–Image Pre-training (CLIP) and A Large-scale Image and Noisy-text Embedding (ALIGN), has revolutionized multimodal labeling learning. These foundational models, pre-trained on network-scale image-text pairs, demonstrate exceptional zero-shot generalization capabilities and can serve as powerful backbones for various downstream tasks. To better adapt to user needs, local users often require task-specific adaptations, which typically rely on user-owned private data. However, some of this private data has confidentiality requirements and cannot be directly uploaded to a centralized server. Federated learning, because it supports collaborative modeling across distributed clients without sharing raw private data, has become a common paradigm for deploying visual-language models in privacy-sensitive applications.

[0003] However, in real-world deployment scenarios, driven by intellectual property protection and commercial considerations, visual-language models are increasingly being offered as services, a paradigm known as Model as a Service (MaaS). In this paradigm, users act as black-box clients, interacting with the model only through an inference application programming interface (API) without accessing the model's internal parameters or gradients.

[0004] To adapt to gradient-free optimization scenarios in black-box environments, zero-order optimization (or derivative-free optimization) schemes have been proposed. However, applying traditional zero-order optimization algorithms to federated vision-language models is not easy, mainly due to the "curse of dimensionality": the search space for continuous prompts typically reaches thousands of parameters. In such a high-dimensional space, gradient estimation becomes extremely noisy and inefficient, leading to slow convergence and high communication costs. Summary of the Invention

[0005] This application provides a federated cue tuning method and system for black-box visual-language models. The method introduces a structured subspace projection strategy of specific modalities to construct low-dimensional semantic and spectral sub-controls for text and visual cues respectively, thereby reducing the search space dimension of zero-order optimization and improving model tuning efficiency.

[0006] In a first aspect, embodiments of this application provide a federated feedback tuning method for a black-box visual-language model, the method being applied to a local client, the method comprising: Receive global prompts from the federated server, which include text prompts and visual prompts that need to be optimized; A semantic base is constructed based on the main semantic direction of the pre-trained lexical embedding matrix of CLIP, and the text prompt is reconstructed based on the semantic base to obtain the reconstructed text prompt. The visual-language model interface is invoked to extract features from the reconstructed text prompt and the category labels in the local sample data to obtain semantic features; The visual cues are reconstructed based on the pre-constructed discrete inverse cosine transform basis matrix to obtain the reconstructed visual cues; The visual-language model interface is invoked to extract features from the visual cues and sample images in the local sample data to obtain visual features. The classification prediction probability is calculated based on the semantic features and the visual features, and the loss value is calculated based on the classification prediction probability and the category label. The text prompt and the visual prompt are then iteratively optimized using the loss value to obtain the optimized local prompt. The local optimization suggestions are uploaded to the federated server for aggregation.

[0007] In one possible implementation of the first aspect, a semantic base is constructed based on the main semantic direction of the CLIP pre-trained lexical embedding matrix, and the text prompt is reconstructed based on the semantic base to obtain the reconstructed text prompt, including: Obtain the pre-trained lexical embedding matrix of CLIP, and center the pre-trained lexical embedding matrix to obtain a centered matrix; Singular value decomposition is performed on the centered matrix, and a semantic basis is constructed based on a predetermined number of right singular vectors sorted by the size of their singular values ​​from the right singular vectors obtained by the singular value decomposition. Obtain the initial text prompt, and calculate the reconstructed text prompt based on the initial text prompt, the semantic base, and the text prompt.

[0008] In this embodiment, the pre-trained lexical embedding matrix of CLIP is centered and subjected to singular value decomposition. Then, the right singular vectors with larger singular values ​​are extracted from the right singular vectors to construct a semantic base. Based on this semantic base, the original text prompts are mapped to the corresponding low-dimensional coordinates. Dimensionality reduction during the tuning process can then be further achieved by optimizing the text prompts corresponding to these low-dimensional coordinates. When calling the visual-language model API for feature generation, only the text prompts need to be reconstructed based on the semantic base to obtain reconstructed text prompts of standard dimensions.

[0009] In one possible implementation of the first aspect, the visual cue is reconstructed based on a pre-constructed discrete inverse cosine transform basis matrix to obtain a reconstructed visual cue, including: Obtain the pre-constructed discrete inverse cosine transform basis matrix, which includes a sequence basis matrix and an embedded basis matrix; The visual cue is reconstructed by performing a bilinear transformation on the visual cue based on the sequence basis matrix and the embedding basis matrix.

[0010] In this embodiment, by constructing a spectral subspace projector, the energy aggregation properties of the high-dimensional original visual cue are truncated into low-frequency coefficient visual cues through the discrete cosine transform matrix. This reduces the magnitude of the parameter search space for subsequent zero-order optimization, thereby improving tuning efficiency. When calling the vision-language API for feature generation, the inverse discrete cosine transform can be used to reconstruct the original visual cue from the visual cue.

[0011] For example, the step of calling the vision-language model interface to extract features from the visual cue and the sample images in the local sample data to obtain visual features includes: The sample images in the local sample data are divided into multiple image blocks, and the multiple image blocks are mapped to an embedding sequence; The input stream of the image encoder is constructed based on preset category terms, the embedded sequence, and the visual cues; Based on the vision-language model interface, the image encoder is called to extract features from the input stream to obtain visual features.

[0012] For example, the step of calculating the classification prediction probability based on the semantic features and the visual features, and calculating the loss value based on the classification prediction probability and the category label, includes: Calculate the cosine similarity between the semantic features and the visual features, and calculate the classification prediction probability based on the cosine similarity. The cross-entropy is calculated based on the predicted classification probability and the category label to obtain the loss value.

[0013] In one possible implementation of the first aspect, the step of iteratively optimizing the text prompt and the visual prompt based on the loss value to obtain the optimized local prompt includes: The text prompt and the visual prompt are combined to form a prompt to be optimized; Obtain the iteration rounds and determine the iteration cycle position based on the iteration rounds, wherein the iteration cycle position includes the global perception stage and the sparse acceleration stage; When in the global perception stage, the gradient is estimated based on the full-dimensional random perturbation of the prompt to be optimized and the gradient sensitivity is accumulated; When in the sparse acceleration stage, sensitive coordinates are selected from the prompts to be optimized based on the accumulated gradient sensitivity. The gradient is estimated based on the random perturbation of the sensitive coordinates and the prompts to be optimized are updated based on the estimated gradient values ​​to obtain local optimization prompts.

[0014] In the embodiments of this application, during the zero-order optimization stage, gradient-sensitive cue coordinates are identified through dynamic sensitivity perception. Then, these gradient-sensitive cue coordinates are randomly perturbed while the insensitive cue coordinates are frozen. This can further reduce the dimensionality of the optimization, thereby significantly reducing estimation noise.

[0015] In one possible implementation of the first aspect, when in the sparse acceleration phase, filtering sensitive coordinates from the optimization prompts based on the accumulated gradient sensitivity, estimating the gradient based on random perturbations of the sensitive coordinates, and updating the optimization prompts based on the estimated gradient values ​​to obtain local optimization prompts, includes: When in the sparse acceleration phase, sensitive coordinates are selected from the optimization prompts based on the accumulated gradient sensitivity. Calculate the effective proportion of the dimension of the sensitive coordinate in the total dimensions of the suggestions to be optimized; Based on the effective proportion, the number of perturbation samples required to estimate the gradient of the sensitive coordinates by random perturbation is adaptively reduced, and the learning rate used to update the cues to be optimized is adaptively increased. Based on the reduced number of perturbation samples and the increased learning rate, the gradient of the sensitive coordinates is estimated by random perturbation and the optimization prompt is updated to obtain the local optimization prompt.

[0016] In this embodiment of the application, estimation noise is further reduced and tuning efficiency is improved by adaptively reducing the number of perturbation samples and adaptively increasing the learning rate.

[0017] Secondly, embodiments of this application provide a federated feedback tuning method for a black-box visual-language model, the method comprising: At the start of each round of global communication, the federated server issues a global prompt; The local client extracts the text and visual cues to be optimized from the global prompts; The local client constructs a semantic base based on the main semantic direction of the CLIP pre-trained lexical embedding matrix, reconstructs the text prompt based on the semantic base, and calls the visual-language model interface to extract features from the reconstructed prompt and the category labels in the local sample data to obtain semantic features. The local client reconstructs the visual cues based on the pre-constructed discrete cosine inverse transform basis matrix, and calls the visual-language model interface to extract features from the reconstructed cues and sample images in the local sample data to obtain visual features. The local client calculates the classification prediction probability based on the semantic features and the visual features, calculates the loss value based on the classification prediction probability and the category label, and performs iterative zero-order optimization on the global prompt based on the loss value to obtain the optimized local prompt, and sends the optimized local prompt to the federated server. The federated server aggregates the local optimization tips received from multiple local clients and updates the global tips based on the aggregation results.

[0018] Thirdly, embodiments of this application provide an image classification method, the method comprising: Receive an image classification request, wherein the image classification request contains an image to be classified; Obtain global prompts, wherein the global prompts are global prompts optimized using the federated prompting optimization method of the black-box visual-language model according to any one of claims 1 to 8; Extract textual and visual cues from the global prompts; A semantic base is constructed based on the main semantic direction of the pre-trained lexical embedding matrix of CLIP, and the text prompt is reconstructed based on the semantic base to obtain the reconstructed text prompt. The visual-language model interface is invoked to extract features from the reconstructed text prompt and the category text corresponding to multiple candidate image categories, thereby obtaining the semantic features corresponding to the multiple candidate image categories; The visual cues are reconstructed based on the pre-constructed discrete inverse cosine transform basis matrix to obtain the reconstructed visual cues; The visual-language model interface is invoked to extract features from the visual cues and the image to be classified, thereby obtaining visual features; Calculate the similarity between the visual features and the semantic features corresponding to each candidate image category, and determine the target image category corresponding to the image to be classified among the multiple candidate image categories based on the similarity.

[0019] Fourthly, embodiments of this application provide a federated feedback tuning system for a black-box visual-language model, the system comprising: The federated server is used to send the current global prompt to the local client at the beginning of each round of communication, and to receive local update prompts uploaded by each local client and aggregate the local update prompts uploaded by each local client into a global prompt. Multiple local clients receive global prompts from the federated server, extracting text and visual prompts from them. They then reconstruct the text prompts in the text subspace and the visual prompts in the visual subspace to obtain reconstructed visual prompts. Next, they call a vision-language model interface to extract semantic features from the text input (composed of the reconstructed text prompts and category labels from local training samples) and image features from the image input (composed of the reconstructed visual prompts and sample images from local training samples). Based on the semantic and image features, they calculate the classification prediction probability, calculate the loss based on the prediction probability and category labels, and then iteratively optimize the global prompts using the loss value to obtain optimized local prompts. Finally, they send the optimized local prompts to the federated server.

[0020] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the method as described in any one of claims 1 to 7.

[0021] In a sixth aspect, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the federated prompting tuning method for the black-box visual-language model described in any of the first aspects above.

[0022] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0023] The beneficial effects of the embodiments of this application compared with the prior art are: This application embodiment constructs a semantic subspace and a spectral subspace, then reduces the original high-dimensional cue coordinates to low-dimensional semantic coordinates via the semantic subspace as text cue, and reduces the spectral coordinates to low-dimensional spectral coordinates via the spectral subspace as visual cue. Then, zero-order optimization is performed on the low-dimensional text cue and visual cue, which can greatly reduce the dimensionality of zero-order optimization, avoid the "curse of dimensionality" problem, and improve the federated optimization efficiency of the black-box visual-language model. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1This is a schematic diagram of a federated prompting tuning system for a black-box visual-language model provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a federated feedback tuning method for a black-box visual-language model provided in an embodiment of this application. Figure 3 This is a schematic flowchart of an image classification method provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of the federated prompting optimization device for the black-box visual-language model provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of an image classification device provided in an embodiment of this application. Detailed Implementation

[0026] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0027] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0028] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0029] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0030] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0032] The federated tuning method for black-box visual-language models provided in this application can be applied to terminal devices such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). This application does not impose any restrictions on the specific type of terminal device.

[0033] For example, the terminal device may be a station (STAION, ST) in a WLAN, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA) device, a handheld device with wireless communication capabilities, a computing device or other processing device connected to a wireless modem, an in-vehicle device, a vehicle networking terminal, a computer, a laptop computer, a handheld communication device, a handheld computing device, a satellite wireless device, a wireless modem card, a set-top box (STB), customer premises equipment (CPE), and / or other devices used for communication over a wireless system, as well as next-generation communication systems, such as mobile terminals in 5G networks or mobile terminals in future evolved Public Land Mobile Network (PLMN) networks, etc.

[0034] As an example and not a limitation, when the terminal device is a wearable device, the term "wearable device" can also refer to any device that utilizes wearable technology to intelligently design and develop everyday wearables, such as glasses, gloves, watches, clothing, and shoes. Wearable devices are portable devices worn directly on the body or integrated into a user's clothing or accessories. Wearable devices are not merely hardware devices; they achieve powerful functions through software support, data interaction, and cloud interaction. Broadly defined, wearable smart devices include those with comprehensive functions, large sizes, and the ability to perform complete or partial functions without relying on a smartphone, such as smartwatches or smart glasses, as well as those focused on a specific application function that require interaction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0035] Please see Figure 1 , Figure 1 This is a schematic diagram of a federated tuning system for a black-box visual-language model provided in an embodiment of this application. The system includes a federated server 110 and multiple local clients 120.

[0036] The federated server 110 is used to send the current global prompt to the local client at the beginning of each round of communication, and to receive the local update prompts uploaded by each local client, and to aggregate the local update prompts uploaded by each local client into a global prompt.

[0037] The local client 120 receives global prompts from the federated server and extracts text and visual prompts from them. It then reconstructs the text prompts in the text subspace and the visual prompts in the visual subspace to obtain reconstructed visual prompts. Next, it calls the vision-language model interface to extract semantic features from the text input (composed of the reconstructed text prompts and category labels from local training samples) and image features from the image input (composed of the reconstructed visual prompts and sample images from local training samples). Based on the semantic and image features, it calculates the classification prediction probability, calculates the loss based on the classification prediction probability and category labels, and then iteratively optimizes the global prompt using the loss value to obtain the optimized local prompt. Finally, it sends the optimized local prompt to the federated server.

[0038] In this system, the local data of the local client 120 does not need to be uploaded to the federated server 110, nor does it need to upload gradients to the federated server 110. Only after local optimization is completed, the local optimization prompts are uploaded to the federated server 110 for aggregation, which can avoid the leakage of local sensitive data and protect local data security.

[0039] Specifically, in the federated tuning system of the black-box visual-language model provided in this application, multiple local clients can adopt a set... This indicates that each local client Holding a private dataset ,in Indicates the input image. These are the corresponding tags. Indicates the number of samples.

[0040] Figure 2 A schematic flowchart of a federated hint tuning method for a black-box visual-language model provided in this application is shown. This is an example and not a limitation, and the method can be applied to the aforementioned local client 120.

[0041] S201 receives global prompts from the federal server.

[0042] It is understood that the federated prompting optimization method for the black-box visual-language model provided in this application embodiment is a multi-round communication optimization process, which terminates when the number of communication rounds reaches a preset number. In this application embodiment, S201 to S207 introduce a communication round, which can be the first communication round or an intermediate communication round, and is not limited here. When the communication round is the first communication round, the global prompt issued by the federated server is the initial global prompt; when the communication round is not the first communication round, the global prompt issued by the federated server is the global prompt obtained by globally aggregating the local optimization prompts uploaded by the local client in the previous round. In this application embodiment, the widely used pre-trained CLIP framework is used as the visual backbone model to provide a detailed description of the solution.

[0043] At the beginning of each communication round, the federated server 110 sends global prompts to multiple local terminals 120. The federated server 110 can send global prompts to all local clients 120, or it can send them to a subset of local clients 120. For example, if there are a total of 20 local clients, at the beginning of each communication round, 18 local clients can be randomly selected from a certain proportion (e.g., 90%) of all local clients to participate in the federated prompt optimization for that round, and then global prompts are sent to these randomly selected local clients.

[0044] In this embodiment, the optimization method for the black-box visual-language model is a cue-based optimization method. This method achieves efficient optimization without updating the complete model parameters. Specifically, it can employ... Indicates that there is a freeze parameter The pre-trained black-box visual-language model. However, related technologies use uniform prompting parameters to optimize the pre-trained black-box visual-language model, ignoring the essential differences in statistical regularities and semantic structures between the two modalities. Specifically, semantic features have semantic clustering structures, while visual features have spatial frequency structures. Therefore, the prompting parameters introduced in this application's embodiments include both visual and textual modalities, and can be specifically... ,in, Indicates visual cues, This indicates a text prompt. Thus, for a given image, the text is paired with a label. The probability is obtained through the image encoder. and text encoder Similarity calculation between them:

[0045] in, Represents cosine similarity. It is the temperature coefficient.

[0046] Thus, the global objective of federated learning is to minimize the weighted average of the empirical risks of all local clients:

[0047] in, This represents the sum of the sample sizes of all local clients. It is a loss function, such as cross-entropy.

[0048] Because the standard tipping tuning method optimizes the continuous context matrix ,in Here, is the context length, and D is the embedding dimension of the vision-language model. Thus, the total number of trainable parameters is... In black-box scenarios, estimating gradients for such high-dimensional parameters using zero-order optimization is extremely difficult because the query complexity of zero-order optimization methods is typically linearly proportional to the dimension M. To alleviate this problem, this application introduces subspace learning to optimize hints within a low-dimensional structured subspace.

[0049] Specifically, in the embodiments of this application, the pre-trained lexical embedding matrix of CLIP is used. Principal component analysis is applied to identify the main semantic directions, and then d main semantic directions are selected to form the semantic basis. This ensures that optimization occurs along the direction with the maximum semantic variance, rather than along arbitrary random directions in the semantic space, thus allowing the use of low-dimensional coordinate matrices. We can parameterize learnable text prompts, thereby reducing the dimensionality of gradient estimation for zero-order optimization while preserving the semantic integrity of the language prompts.

[0050] Adapting visual branches is also crucial for aligning multimodal labels. Related techniques typically employ pixel-level or lexical-level cues. Pixel-level cues superimpose learnable noise or frames onto the input image, but this method disrupts natural image statistics. Lexical-level cues suffer from the curse of dimensionality in derivative-free settings, and both methods lack explicit structural constraints, making them prone to overfitting high-frequency noise.

[0051] Therefore, in this embodiment, a spectrum subspace projector is proposed to parameterize visual cues in the frequency domain. The original visual cue sequence... It can be viewed as a 2D signal, and then the low-frequency coefficients can be extracted from the original visual cue using the energy convergence property of Discrete Cosine Transform (DCT). ,in This design essentially functions as a low-pass filter, significantly reducing the parameter search space for zero-order optimization.

[0052] That is, in the embodiments of this application, the optimization target can be It is determined as a concatenation of text subspace coordinates (text cues) and visual spectral coefficients (visual cues), i.e. Both text and visual cues are parameterized using low-dimensional structured search spaces customized based on the statistical regularities of their respective modalities, resulting in a significant reduction in parameter dimensionality.

[0053] Thus, in this embodiment, at the beginning of each communication round, the global prompt sent by the federated server to the local client is as follows: The global suggestions include text suggestions that need optimization. Visual cue C, which needs optimization.

[0054] S202: Construct a semantic base based on the main semantic direction of the pre-trained lexical embedding matrix of CLIP, and reconstruct the text prompt based on the semantic base to obtain the reconstructed text prompt.

[0055] In this embodiment of the application, when the local client receives the global prompt issued by the federated server, it can extract the text prompt from the global prompt and reconstruct the text prompt to obtain the original dimension text prompt.

[0056] In some embodiments, a semantic base is constructed based on the main semantic direction of the CLIP pre-trained lexical embedding matrix, and the text prompt is reconstructed based on the semantic base to obtain the reconstructed text prompt, including: Obtain the pre-trained word embedding matrix of CLIP, and center the pre-trained word embedding matrix to obtain the centered matrix; Perform singular value decomposition on the centered matrix, and construct a semantic basis based on a predetermined number of right singular vectors sorted by singular value from the right singular vectors obtained by the singular value decomposition. Obtain the initial text prompt, and calculate the reconstructed text prompt based on the initial text prompt, semantic base, and text prompt.

[0057] In this embodiment of the application, since the text prompt is obtained by mapping the semantic base determined by the main semantic direction based on the analysis of the pre-trained lexical embedding matrix of CLIP, the complete prompt embedding can be reconstructed based on the semantic base.

[0058] First, obtain the pre-trained lexical embedding matrix of CLIP. The matrix is ​​composed of The vocabulary consists of lexical units. Then, the pre-trained lexical embedding matrix is ​​centered to obtain the centered matrix. Furthermore, the centered matrix can be calculated. Singular Value Decomposition (SVD):

[0059] in, It contains orthogonal principal components (right singular vectors). In this embodiment, the top d-feature vector can be selected. To form a semantic base.

[0060] Then, further optimization of the text prompts based on semantic bases. Reconstruction:

[0061] in, For complete prompt embedding, To initialize text prompts (such as embedding human templates). It is the scaling factor.

[0062] S203, call the visual-language model interface to extract features from the reconstructed text prompts and category labels in the local sample data to obtain semantic features.

[0063] Once the text prompt is reconstructed to obtain a complete prompt embedding (reconstructed text prompt), the visual-language model API can be further called to extract features from the reconstructed text prompt and the category labels in the local sample data, thereby obtaining semantic features.

[0064] Specifically, the local sample data includes the aforementioned private dataset. The sample data in the image is, for example, (x, y), where x is the sample image and y is the class label.

[0065] The visual-language model API is called to extract features from the reconstructed text prompts and category labels in the local sample data. Specifically, the text encoder in the visual-language model is used to encode the features of the reconstructed text prompts and category labels to obtain semantic features.

[0066] S204. Based on the pre-constructed discrete inverse cosine transform basis matrix, visual cues are reconstructed to obtain reconstructed visual cues.

[0067] Similarly, after receiving a global suggestion for optimization from the federated server, the local client can select from the global suggestion... The visual cue C to be optimized is extracted, and then the visual cue C is reconstructed to obtain the original dimension cue sequence.

[0068] In some embodiments, visual cues are reconstructed based on a pre-constructed discrete inverse cosine transform basis matrix to obtain reconstructed visual cues, including: Obtain the pre-constructed discrete inverse cosine transform basis matrix, which includes the sequence basis matrix and the embedded basis matrix; The visual cues are reconstructed by performing a bilinear transformation on the visual cues based on the sequence basis matrix and the embedding basis matrix.

[0069] In this embodiment of the application, as described above, the visual cue C is derived from the original cue sequence based on the discrete cosine transform. The low-frequency coefficients are obtained by truncating the corresponding 2D signal. Therefore, when reconstructing visual cues, a pre-constructed discrete cosine inverse transform basis matrix can be obtained first, which may specifically include the sequence basis matrix. and embedding basis matrix .

[0070] Then, a bilinear transformation can be performed on the visual cue based on the sequence basis matrix and the embedding basis matrix to obtain the reconstructed visual cue:

[0071] Where the size is L The basis matrix of K Entries Defined as:

[0072] in and express .

[0073] S205, call the vision-language model interface to extract features from the reconstructed visual cues and sample images in the local sample data to obtain visual features.

[0074] Furthermore, after reconstructing the visual cues, the image encoder in the vision-language model can be called to encode the sample images in the local sample data and the reconstructed visual cues to obtain visual features.

[0075] In some embodiments, the vision-language model interface is invoked to extract features from the reconstructed visual cues and sample images in the local sample data to obtain visual features, including: The sample images in the local sample data are segmented into multiple image blocks, and these multiple image blocks are mapped to embedding sequences. The input stream of the image encoder is constructed based on preset category terms, embedded sequences, and reconstructed visual cues; The image encoder is called based on the vision-language model interface to extract features from the input stream and obtain visual features.

[0076] In this embodiment of the application, for sample images in the local sample data We can first divide it into image blocks, and then map the resulting image blocks into embedding sequences. .

[0077] Then, the input stream of the Transformer encoder is constructed as follows:

[0078] in, These are category-specific terms. Unlike pixel-level perturbations, the spectral terms in this application participate in the self-attention mechanism of all Transformer layers, thereby allowing for deep semantic modulation of visual features.

[0079] Then, the image encoder (i.e., the Transformer encoder) can be called based on the vision-language model API to extract features from the input stream and obtain visual features.

[0080] In the process of generating semantic and visual features, only the API of the black-box visual language model is called, without needing to access the model parameters or gradient values ​​of the black-box language.

[0081] S206. Calculate the classification prediction probability based on semantic and visual features, calculate the loss value based on the classification prediction probability and category label, and perform iterative zero-order optimization on the text prompt and visual prompt based on the loss value to obtain the optimized local prompt.

[0082] After generating semantic and image features, the classification prediction probability can be calculated based on the semantic and visual features. Then, the loss value is calculated based on the classification prediction probability and the category label. Furthermore, iterative zero-order optimization can be performed on the text prompt Z and the visual prompt C to be optimized based on the loss value, thereby obtaining the optimized local prompt.

[0083] In some embodiments, calculating a classification prediction probability based on the semantic features and the visual features, and calculating a loss value based on the classification prediction probability and the category label, includes: Calculate the cosine similarity between semantic features and visual features, and then calculate the classification prediction probability based on the cosine similarity. The cross-entropy is calculated based on the classification prediction probability and the category label to obtain the loss value.

[0084] In this embodiment, the classification prediction probability can be calculated by calculating the cosine similarity between semantic features and visual features, and then by calculating the ratio of the cosine similarity to the temperature coefficient. Finally, the cross-entropy is calculated based on the classification prediction probability and the category label to obtain the loss value.

[0085] In some embodiments, iterative zero-order optimization is performed on the text prompts and visual prompts based on the loss value to obtain optimized local prompts, including: Combine text and visual cues to create a cue to be optimized; Obtain the iteration round and determine the iteration cycle position based on the iteration round. The iteration cycle position includes the global perception stage and the sparse acceleration stage. When in the global perception stage, the gradient is estimated based on the full-dimensional random perturbation of the feedback to be optimized and the gradient sensitivity is accumulated. When in the sparse acceleration stage, sensitive coordinates are selected from the optimization prompts based on the cumulative gradient sensitivity. The gradient is estimated based on the random perturbation of the sensitive coordinates and the optimization prompts are updated based on the estimated gradient values ​​to obtain local optimization prompts.

[0086] In this application embodiment, the optimization objective is defined as a concatenation of textual cues (text subspace coordinates) and visual cues (visual spectral coefficients), i.e. Both Z and C are parameterized with low-dimensional structured search spaces customized according to the statistical laws of their respective modes. These structured subspaces achieve a significant reduction in parameter dimensionality. However, with a limited number of communication rounds, the remaining search space remains challenging for standard zero-order methods. Therefore, embodiments of this application further propose dynamic sensitivity-aware zero-order optimization, which employs a periodic strategy to alternately identify sensitive coordinates and then restricts the tuning to these sensitive coordinates to achieve further parameter dimensionality reduction.

[0087] Specifically, text cue Z and visual cue C can be concatenated to form the cue to be optimized. In other words, during the zero-order tuning phase, the modalities of the parameters are no longer distinguished; all parameters requiring tuning are subjected to a uniform random perturbation. Then, the tuning iteration rounds are obtained, and the tuning is performed based on the iteration rounds and a preset period. This determines the current optimization stage. For example, if the period is 50 and the result of modulo the number of iterations with the period is between 0 and 9, the current optimization stage is determined to be the global awareness stage; if the result of modulo the number of iterations with the period is between 10 and 49, the current optimization stage is determined to be the sparsity acceleration stage. For example, when the number of iterations is 65, 65 mod 50 = 15, so the current optimization stage is determined to be the sparsity acceleration stage.

[0088] In the global perception phase, the standard Simultaneous Perturbation Stochastic Approximation (SPSA) algorithm can be used to provide hints for optimization. Add perturbations to the complete parameter set to estimate the gradient Furthermore, in this embodiment, the history of gradient sensitivity is further preserved, specifically by tracking the exponential moving average of its squared norm:

[0089] in, Let be the sensitivity history vector at step t, which is used to accumulate the square of the gradient element by element to form the sensitivity history. As the attenuation factor, This represents element-wise multiplication. Wherein, express The cumulative gradient value of the i-th component indicates the area to be optimized. The sensitivity of the i-th parameter to the current optimization objective. When The larger the value, the more optimization is needed. The i-th parameter is highly sensitive to the current optimization objective.

[0090] During the sparse acceleration phase, based on accumulated sensitivity You can select one of them The k largest parameters are used as sensitive coordinates, and the other coordinates are frozen. Random perturbations are added only to the sensitive coordinates and the gradient is estimated. Then, the optimization hints are optimized based on the estimated gradients to obtain local optimization hints.

[0091] Specifically, you can choose middle The largest k parameters are used to generate a dynamic binary mask. ,in and It's the sparsity ratio. Then, random perturbation. Limited to these active dimensions:

[0092] By accumulating sensitivity to filter out sensitive parameters and then freezing insensitive parameters, the optimization dimensionality can be effectively reduced to k dimensions. According to the zeroth-order optimization complexity theory, the variance of the gradient estimator is proportional to the dimensionality; therefore, reducing the dimensionality can significantly reduce estimation noise. This variance reduction can be used to amplify the learning rate. This can be multiplied by a factor of 1 to accelerate convergence without affecting stability.

[0093] In some embodiments, during the sparse acceleration phase, sensitive coordinates are selected from the optimization prompts based on accumulated gradient sensitivity, the gradient is estimated based on random perturbations of the sensitive coordinates, and the optimization prompts are updated based on the estimated gradient values ​​to obtain local optimization prompts, including: When in the sparse acceleration phase, sensitive coordinates are selected from the optimization prompts based on the cumulative gradient sensitivity. Calculate the effective proportion of the dimension of the sensitive coordinate in the total dimensions of the suggestions to be optimized; Based on the effective proportion, the number of perturbation samples required for gradient estimation of sensitive coordinates is adaptively reduced, and the learning rate used to update the cues to be optimized is adaptively increased. Based on the reduced number of perturbation samples and the increased learning rate, the gradient of the sensitive coordinates is estimated by random perturbation and the optimization hints are updated to obtain local optimization hints.

[0094] In this embodiment, during the sparse acceleration stage, after filtering out sensitive coordinates from the prompts to be optimized based on cumulative gradient sensitivity, the effective proportion of the dimension of the sensitive coordinates in the total dimension of the prompts to be optimized can be further calculated. Then, based on the effective proportion, the number of perturbation samples required for estimating the gradient by random perturbation is adaptively reduced, and the learning rate used to update the prompts to be optimized is adaptively increased.

[0095] Then, based on the reduced number of perturbation samples and the increased learning rate, the gradient of the sensitive coordinates is estimated by random perturbation and the optimization hints are updated to obtain local optimization hints.

[0096] In the sparse acceleration stage, the gradient estimator is used with respect to the size deff = ρ The operation is performed on the active set of d, where d is a suggestion for optimization. The complete parameter dimensions. The estimated variance is then scaled to... When deff At time d, the original query count q is oversampled relative to the reduced noise baseline. In this embodiment, the number of perturbation samples used is adaptively reduced:

[0097] in, To reduce the number of perturbation samples. To represent the effective proportion, q represents the number of original perturbation samples. Its maintenance... - The geometric mean between the noise floor of the full space and the subspace. To compensate for the reduced average value, the learning rate is adaptively increased in this embodiment:

[0098] in, To increase the learning rate, This is the original learning rate. (And...) Compared to amplification, it is less invasive and, despite smaller sample sizes, still ensures stability. In this embodiment, the global perception stage maintains its original state. and constant.

[0099] S207 uploads local optimization suggestions to the federated server for aggregation.

[0100] Once the local optimization suggestions are obtained, multiple local clients upload them to the federated server. The federated server determines the weight of each local client's uploaded ontology optimization suggestion based on the proportion of the local sample in each local server to the total sample. Then, the federated server performs a weighted aggregation of the multiple local optimization suggestions uploaded by multiple local clients according to different weights to obtain the global suggestion after this round of communication optimization.

[0101] If the number of communication rounds has not reached the preset number of rounds, the federated server will further distribute the aggregated local optimization suggestions to each local client for tuning until the number of communication rounds reaches the preset number of rounds.

[0102] Please refer to Figure 3 This is a flowchart illustrating the image classification method provided in this application. As an example and not a limitation, this method can be applied to the aforementioned local client 120. The method includes: S301, Receive image classification request.

[0103] The federated prompting optimization method for black-box visual-language models provided in this case can be applied to multiple industry scenarios to perform federated prompting optimization on black-box visual-language models deployed in these scenarios. These include, but are not limited to, federated medical image diagnosis scenarios in multiple hospitals, product recognition scenarios in chain retail stores, industrial quality inspection and defect classification scenarios in multiple factories, and federated land cover classification scenarios based on remote sensing / satellite imagery.

[0104] In these scenarios, the black-box visual language model can be used for image recognition. For example, in a multi-factory industrial quality inspection and defect classification scenario, it can receive image classification requests, which contain images to be classified. These images can be images of industrial products that need to undergo appearance quality inspection.

[0105] S302, obtain global prompts and extract text prompts and visual prompts from global prompts.

[0106] After receiving an image classification request, the local terminal can obtain a global prompt issued by the federated server. The global prompt issued by the federated server can be the final global prompt obtained by using the federated prompt optimization method of the black-box visual language model provided in this application. That is, after a certain number of communication rounds, the federated server aggregates the uploaded local optimized prompts to obtain the global prompt.

[0107] Then, text and visual cues can be extracted from the global prompts.

[0108] S303: Construct a semantic base based on the main semantic direction of the pre-trained lexical embedding matrix of CLIP, and reconstruct the text prompt based on the semantic base to obtain the reconstructed text prompt.

[0109] After extracting the text prompts, a semantic base can be constructed, and the text prompts can be reconstructed based on the constructed semantic base to obtain the reconstructed text prompts. The specific execution process of this step is described in S202, and will not be repeated here.

[0110] S304, call the visual-language model interface to extract features from the reconstructed text prompt and the category text corresponding to multiple candidate image categories, and obtain the semantic features corresponding to multiple candidate image categories.

[0111] After obtaining the reconstructed text prompt, the visual-language model API can be further called to extract features from the combination of the reconstructed text prompt with the category text corresponding to multiple candidate image categories, thereby obtaining the semantic features corresponding to multiple candidate image categories.

[0112] Among them, multiple candidate image categories can include superior products, good products, defective products, and waste products.

[0113] S305, based on the pre-constructed discrete inverse cosine transform basis matrix, performs cues reconstruction on the reconstructed visual cues to obtain the reconstructed visual cues.

[0114] Similarly, after extracting visual cues from global cues, the visual cues can be further reconstructed based on a pre-constructed discrete inverse cosine transform basis matrix to obtain reconstructed visual cues. The specific execution process of this step is described in S204 and will not be repeated here.

[0115] S306, call the vision-language model interface to extract features from the reconstructed visual cues and the image to be classified, and obtain visual features.

[0116] Furthermore, the visual-language model API can be called to extract features from the reconstructed visual cues and the image to be classified, thereby obtaining visual features.

[0117] S307, calculate the similarity between visual features and semantic features corresponding to each candidate image category, and determine the target image category corresponding to the image to be classified among multiple candidate image categories based on the similarity.

[0118] Finally, the similarity between visual features and the semantic features corresponding to each candidate image category can be calculated. Then, based on the candidate image category corresponding to the highest similarity, the target image category corresponding to the image to be classified is determined, resulting in the image classification result. For example, the image might be classified as "superior quality".

[0119] Corresponding to the federated prompting optimization method for the black-box visual-language model described in the above embodiments, Figure 4 The diagram shows a structural block diagram of a federated cue tuning device 400 for a black-box visual-language model provided in an embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown.

[0120] Reference Figure 4 The device includes: The first receiving unit 410 is used to receive global prompts issued by the federated server. The global prompts include text prompts and visual prompts that need to be optimized. The first prompt reconstruction unit 420 is used to construct a semantic base based on the main semantic direction of the pre-trained lexical embedding matrix of CLIP, and to reconstruct the text prompt based on the semantic base to obtain the reconstructed text prompt; The first feature extraction unit 430 is used to call the visual-language model interface to extract features from the reconstructed text prompt and the category labels in the local sample data to obtain semantic features. The second cue reconstruction unit 440 reconstructs the visual cue based on the pre-constructed discrete cosine inverse transform basis matrix to obtain the reconstructed visual cue. The second feature extraction unit 450 calls the visual-language model interface to extract features from the reconstructed visual cues and sample images in the local sample data to obtain visual features. The prompt optimization unit 460 is used to calculate the classification prediction probability based on semantic features and visual features, calculate the loss value based on the classification prediction probability and category label, and perform iterative zero-order optimization on the text prompt and visual prompt based on the loss value to obtain the optimized local prompt. Upload unit 470 to upload local optimization suggestions to the federated server for aggregation.

[0121] Corresponding to the image classification method in the above embodiments, Figure 5 A structural block diagram of an image classification apparatus 500 provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0122] Reference Figure 5 The device includes: The second receiving unit 510 is used to receive an image classification request, which contains an image to be classified. The acquisition unit 520 is used to acquire global prompts and extract text prompts and visual prompts from the global prompts. The global prompts are global prompts optimized using the federated prompt tuning method of the black-box visual-language model of any one of claims 1 to 8. The third prompt reconstruction unit 530 is used to construct a semantic base based on the main semantic direction of the pre-trained lexical embedding matrix of CLIP, and to reconstruct the text prompt based on the semantic base to obtain the reconstructed text prompt. The third feature extraction unit 540 is used to call the visual-language model interface to extract features from the reconstructed text prompt and the category text corresponding to multiple candidate image categories, so as to obtain the semantic features corresponding to multiple candidate image categories. The fourth cue reconstruction unit 550 is used to reconstruct visual cues based on a pre-constructed discrete inverse cosine transform basis matrix to obtain reconstructed visual cues. The fourth feature extraction unit 560 is used to call the vision-language model interface to extract features from the reconstructed visual cues and the image to be classified, so as to obtain visual features; The classification unit 570 is used to calculate the similarity between visual features and semantic features corresponding to each candidate image category, and to determine the target image category corresponding to the image to be classified among multiple candidate image categories based on the similarity.

[0123] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0124] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0125] This application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0126] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0127] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0128] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0129] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0130] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0131] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0132] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0133] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A federated cueing optimization method for a black-box visual-language model, characterized in that, The method is applied to a local client, and the method includes: Receive global prompts from the federated server, which include text prompts and visual prompts that need to be optimized; A semantic base is constructed based on the main semantic direction of the pre-trained lexical embedding matrix of CLIP, and the text prompt is reconstructed based on the semantic base to obtain the reconstructed text prompt. The visual-language model interface is invoked to extract features from the reconstructed text prompt and the category labels in the local sample data to obtain semantic features; The visual cues are reconstructed based on the pre-constructed discrete inverse cosine transform basis matrix to obtain the reconstructed visual cues; The visual-language model interface is invoked to extract features from the reconstructed visual cue and the sample images in the local sample data to obtain visual features; The classification prediction probability is calculated based on the semantic features and the visual features, and the loss value is calculated based on the classification prediction probability and the category label. The text prompt and the visual prompt are then iteratively optimized using the loss value to obtain the optimized local prompt. The local optimization suggestions are uploaded to the federated server for aggregation.

2. The method according to claim 1, characterized in that, The main semantic direction of the CLIP-based pre-trained lexical embedding matrix is ​​used to construct a semantic base, and the text prompt is reconstructed based on the semantic base to obtain the reconstructed text prompt, including: Obtain the pre-trained lexical embedding matrix of CLIP, and center the pre-trained lexical embedding matrix to obtain a centered matrix; Singular value decomposition is performed on the centered matrix, and a semantic basis is constructed based on a predetermined number of right singular vectors sorted by the size of their singular values ​​from the right singular vectors obtained by the singular value decomposition. Obtain the initial text prompt, and calculate the reconstructed text prompt based on the initial text prompt, the semantic base, and the text prompt.

3. The method according to claim 1, characterized in that, The visual cue is reconstructed based on a pre-constructed discrete inverse cosine transform basis matrix to obtain the reconstructed visual cue, including: Obtain the pre-constructed discrete inverse cosine transform basis matrix, which includes a sequence basis matrix and an embedded basis matrix; The visual cue is reconstructed by performing a bilinear transformation on the visual cue based on the sequence basis matrix and the embedding basis matrix.

4. The method according to claim 3, characterized in that, The step of calling the vision-language model interface to extract features from the reconstructed visual cues and sample images in the local sample data to obtain visual features includes: The sample images in the local sample data are segmented into multiple image blocks, and the multiple image blocks are mapped into embedding sequences; The input stream of the image encoder is constructed based on preset category terms, the embedded sequence, and the reconstructed visual cues; Based on the vision-language model interface, the image encoder is called to extract features from the input stream to obtain visual features.

5. The method according to claim 1, characterized in that, The step of calculating the classification prediction probability based on the semantic features and the visual features, and calculating the loss value based on the classification prediction probability and the category label, includes: Calculate the cosine similarity between the semantic features and the visual features, and calculate the classification prediction probability based on the cosine similarity. The cross-entropy is calculated based on the predicted classification probability and the category label to obtain the loss value.

6. The method according to claim 1, characterized in that, The step of iteratively optimizing the text prompt and the visual prompt based on the loss value to obtain the optimized local prompt includes: The text prompt and the visual prompt are combined to form a prompt to be optimized; Obtain the iteration rounds and determine the iteration cycle position based on the iteration rounds, wherein the iteration cycle position includes the global perception stage and the sparse acceleration stage; When in the global perception stage, the gradient is estimated based on the full-dimensional random perturbation of the prompt to be optimized and the gradient sensitivity is accumulated; When in the sparse acceleration stage, sensitive coordinates are selected from the prompts to be optimized based on the accumulated gradient sensitivity. The gradient is estimated based on the random perturbation of the sensitive coordinates and the prompts to be optimized are updated based on the estimated gradient values ​​to obtain local optimization prompts.

7. The method according to claim 6, characterized in that, When in the sparse acceleration phase, sensitive coordinates are selected from the optimization prompts based on the accumulated gradient sensitivity. The gradient is estimated based on random perturbations of the sensitive coordinates, and the optimization prompts are updated based on the estimated gradient values ​​to obtain local optimization prompts, including: When in the sparse acceleration phase, sensitive coordinates are selected from the optimization prompts based on the accumulated gradient sensitivity. Calculate the effective proportion of the dimension of the sensitive coordinate in the total dimensions of the prompt to be optimized; Based on the effective proportion, the number of perturbation samples required to estimate the gradient of the sensitive coordinates by random perturbation is adaptively reduced, and the learning rate used to update the cues to be optimized is adaptively increased; Based on the reduced number of perturbation samples and the increased learning rate, the gradient of the sensitive coordinates is estimated by random perturbation and the optimization prompt is updated to obtain the local optimization prompt.

8. A federated cueing optimization method for a black-box visual-language model, characterized in that, The method includes: At the start of each round of global communication, the federated server issues a global prompt; The local client extracts the text and visual cues to be optimized from the global prompts; The local client constructs a semantic base based on the main semantic direction of the CLIP pre-trained lexical embedding matrix, reconstructs the text prompt based on the semantic base, and calls the visual-language model interface to extract features from the reconstructed prompt and the category labels in the local sample data to obtain semantic features. The local client reconstructs the visual cues based on the pre-constructed discrete cosine inverse transform basis matrix, and calls the visual-language model interface to extract features from the reconstructed cues and sample images in the local sample data to obtain visual features. The local client calculates the classification prediction probability based on the semantic features and the visual features, calculates the loss value based on the classification prediction probability and the category label, and performs iterative zero-order optimization on the global prompt based on the loss value to obtain the optimized local prompt, and sends the optimized local prompt to the federated server. The federated server aggregates the local optimization tips received from multiple local clients and updates the global tips based on the aggregation results.

9. An image classification method, characterized in that, The method includes: Receive an image classification request, wherein the image classification request contains an image to be classified; Obtain global prompts and extract text prompts and visual prompts from the global prompts, wherein the global prompts are global prompts optimized using the federated prompting optimization method of the black-box visual-language model according to any one of claims 1 to 8; A semantic base is constructed based on the main semantic direction of the pre-trained lexical embedding matrix of CLIP, and the text prompt is reconstructed based on the semantic base to obtain the reconstructed text prompt. The visual-language model interface is invoked to extract features from the reconstructed text prompt and the category text corresponding to multiple candidate image categories, thereby obtaining the semantic features corresponding to the multiple candidate image categories; The visual cues are reconstructed based on the pre-constructed discrete inverse cosine transform basis matrix to obtain the reconstructed visual cues; The visual-language model interface is invoked to extract features from the reconstructed visual cue and the image to be classified, thereby obtaining visual features; Calculate the similarity between the visual features and the semantic features corresponding to each candidate image category, and determine the target image category corresponding to the image to be classified among the multiple candidate image categories based on the similarity.

10. A federated cueing optimization system for a black-box visual-language model, characterized in that, The system includes: A federated server is used to send the current global prompt to local clients at the beginning of each round of communication, and to receive local update prompts uploaded by each local client and aggregate the update prompts into a global prompt; Multiple local clients are used to receive the global prompts issued by the federated server, extract text prompts and visual prompts from the global prompts; then, the text prompts are reconstructed in the text subspace to obtain reconstructed text prompts, and the visual prompts are reconstructed in the visual subspace to obtain reconstructed visual prompts; then, the vision-language model interface is called to extract semantic features from the text input composed of the reconstructed text prompts and category labels in the local training samples, and to extract image features from the image input constructed by the reconstructed visual prompts and sample images in the local training samples; and the classification prediction probability is calculated based on the semantic features and image features, the loss is calculated based on the classification prediction probability and category labels, and then the global prompts are iteratively optimized to zero order based on the loss value to obtain optimized local prompts, which are then sent to the federated server.

Images