Privacy-preserving multi-modal few-shot personalization method based on meta-learning and hypernetwork

Abstract

The application discloses a privacy multi-modal few-shot personalization method based on meta-learning and super network, and relates to the field of data processing. The method constructs a personalized generation framework of cloud large model and local small model cooperation, adopts a meta-learning initialization mechanism for visual encoder and language encoder, so that the model can complete rapid adaptation under the condition of a small number of historical examples; by limiting the gradient norm, parameter change amplitude and context field access range, the sensitive field is kept as much as possible in the local side processing; based on the personalized adapter generation mechanism of the super network, the LoRA parameters are generated according to the small sample examples, and the final output is guided in combination with the structure sketch without sensitive context or the collaborative reasoning result. The application can reduce the original user data transmission demand, improve the few-shot personalization adaptation efficiency, enhance the deployment feasibility in the multi-task scene, and improve the stability, migratability and resource overhead control ability of the personalization generation process.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a privacy-preserving multimodal few-shot personalization method based on meta-learning and hypernetworks. Background Technology

[0002] As the application of multimodal large models expands in personalized question answering, visual question answering, smart terminal assistants, and content generation scenarios, systems increasingly need to process multi-source information such as text commands, user profiles, interaction history, and images simultaneously to output results that match specific user needs. Existing collaborative generation frameworks typically employ a strategy of using a large cloud model for general inference while a local small model handles privacy context, thus mitigating the risk of directly uploading sensitive data to the cloud. For example, in context-aware instruction following tasks, without exposing the complete private context, a structured prior provided by a cloud model is used, followed by personalized completion by a local model, balancing generation quality and privacy constraints.

[0003] However, when the input is expanded to visual information, user screenshots, photos, and device usage images often contain stronger personal characteristics. If we still rely on a unified model for complete fine-tuning or direct cloud processing, we will face problems such as high cost, slow convergence, and increased privacy exposure risks. Summary of the Invention

[0004] Therefore, it is necessary to provide a privacy-preserving multimodal few-shot personalization method based on meta-learning and hypernetworks to address the aforementioned technical issues. This method can achieve fast, stable, and controllable personalized adaptation while ensuring privacy protection.

[0005] In a first aspect, this invention discloses a privacy-preserving multimodal few-shot personalization method based on meta-learning and hypernetworks, comprising:

[0006] Obtain the user's original image, text command, and several historical examples generated on the terminal side, and perform sensitive field filtering on user information to obtain the restricted context that allows participation in training and inference;

[0007] During the training phase, the images and text instructions preserved in the restricted context corresponding to the task are fused to obtain example prototypes, and combined with the historical examples preserved in the restricted context to form a support set and a query set;

[0008] Based on the support set and query set, meta-learning techniques are used to update the parameters of the visual encoder and the language encoder. Gradient clipping, gradient noise addition and parameter drift restriction operations are implemented during the parameter update process. Combined with the hypernetwork, shared adapter groups and private adapter groups are obtained.

[0009] During the inference phase, historical examples are input into the shared adapter to obtain the shared semantic stream, and the restricted context is input into the private adapter group to obtain the private semantic stream. The shared semantic stream and the private semantic stream are weighted and fused to obtain the fusion state. The fusion state is then combined to obtain the multimodal result.

[0010] In one embodiment, updating the parameters of the visual encoder and language encoder using meta-learning techniques based on the support set and query set includes:

[0011] For the current task, perform an inner update on the support set based on shared initialization parameters to obtain task-oriented temporary parameters;

[0012] The query set is used to evaluate the adaptation effect of the temporary parameters, and the shared initialization parameters are updated based on the evaluation results;

[0013] The next round of parameter updates begins based on the updated shared initialization parameters, and the iteration stops when the meta-objective function meets the conditions. The shared initialization parameters corresponding to the end of the iteration serve as the initialization parameters for the shared adapter group.

[0014] In one embodiment, when a new user joins, the method includes:

[0015] We obtain small sample examples of new users, and divide these small sample examples into support sets and query sets according to task type.

[0016] Based on the shared initialization parameters corresponding to the iteration stop and the support set corresponding to the new user, the temporary parameters for the new user are iteratively updated, and the iteration is evaluated based on the temporary parameters of the new user and the query set of the new user; wherein, the temporary parameters corresponding to the iteration stop are used as the initialization parameters of the private adapter group.

[0017] In one embodiment, the method further includes:

[0018] When iteratively updating temporary parameters for new users, an adaptive step size is used, generated based on the example prototype and task representation corresponding to the new user.

[0019] In one embodiment, after obtaining the initialization parameters of the shared adapter group's private adapter group, the method further includes:

[0020] Input a historical example into the hypernetwork to obtain the low-rank matrix of the shared adapter group;

[0021] Input the restricted context into the hypernetwork to obtain the low-rank matrix of the private adapter group;

[0022] The parameters of the shared adapter and private adapter group are updated using a low-rank matrix for use in the inference phase.

[0023] Secondly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the above-described method embodiments.

[0024] Thirdly, this application also provides a computer-readable storage medium. This computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps in the above-described method embodiments.

[0025] Fourthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps in the method embodiments described above.

[0026] The aforementioned privacy-preserving multimodal few-shot personalization method based on meta-learning and hypernetworks core idea is to introduce meta-learning initialization for visual and language encoders on a collaborative architecture between the cloud and the terminal. This allows the model to first learn shared parameter states across user and task distributions, and then quickly update personalization using a small number of samples when a specific user arrives. Simultaneously, this invention introduces a constrained parameter update mechanism during the adaptation phase, controlling the gradient norm, parameter changes, and noise injection process to prevent the model from over-memorizing sensitive visual information on a small number of samples, thereby improving privacy protection capabilities. Furthermore, this invention can be combined with lightweight low-rank adaptation or user-specific adapter generation driven by example embedding, concentrating personalization capabilities within a controlled, small-scale parameter space, balancing terminal computational efficiency and deployment flexibility. Therefore, this invention can reduce the need for external transmission of original user data while improving the efficiency of few-shot personalization adaptation, enhancing deployment feasibility in multi-task scenarios, and improving the stability, portability, and resource consumption control of the personalization generation process. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating a privacy-preserving multimodal few-shot personalization method based on meta-learning and hypernetworks in one embodiment. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0029] This application provides a privacy-preserving multimodal few-shot personalization method based on meta-learning and hypernetworks, such as... Figure 1 As shown, it includes:

[0030] S101, Receive images, text commands, and a small number of historical examples generated on the terminal side.

[0031] Here, the image refers to the observation of the task; the text instruction is the task description, such as "identify breed" for an image of a dog; and the historical examples are the actual annotations or expected outputs of the task, such as selecting or labeling the image.

[0032] S102. Perform field filtering on sensitive fields such as user profile, historical behavior and device usage information in user information, and retain only the restricted context that is allowed to participate in training and inference.

[0033] Based on images, text commands, and a very limited number of examples, the user terminal immediately performs a self-processing step to filter out content from fields such as user profiles, historical behavior, and device usage information that are not allowed to be shared externally, thus forming a restricted context. This restricted context can be accessed by the local model, while the cloud model can only receive text commands and general system prompts. General system prompts refer to general task constraints or system-level instructions that do not contain user privacy information, such as model roles, output formats, and general task rules. It should be noted that not all historical examples are inherently part of the restricted context; whether they can be sent to the cloud should be determined by privacy field filtering and access policies. This solution emphasizes that sensitive user information and restricted content are retained locally, while the cloud only receives non-sensitive task information that has been privacy-filtered and is permitted for use.

[0034] For terminal-side input, this invention first constructs a field mask vector. , used to represent fields Are participants allowed to participate in coding?

[0035] in, This indicates the representation of the fields after the mask. Represents the original field vector. Indicates the field access mask. This indicates element-wise multiplication.

[0036] Based on this, local prompts consist of task information and allowed fields, forming a restricted context input:

[0037]

[0038] in, Indicates user Local prompt sequences, i.e., only for user use The restricted context of the received data Indicates text instructions. , and These represent the masked user profile, historical behavior, and device usage information fields, respectively.

[0039] To ensure that cloud-based collaborative reasoning does not access sensitive fields, this invention constrains cloud input to contain only text commands with no sensitive prompts:

[0040]

[0041] in, This indicates input from the cloud. This indicates a general system prompt. This represents user text commands.

[0042] In one embodiment, field contributions that need further narrowing are filtered based on field leakage risk scores:

[0043]

[0044] in, Indicates user In the field The risk score of leakage on the screen This indicates the current representation of the field. This represents the reference representation of the field under the public distribution. This represents a smoothing constant to prevent the denominator from being zero.

[0045] when When the threshold is exceeded, the corresponding field will be further pruned or downsampled, and the restricted context will be updated, thereby reducing the probability that the local model will solidify rare privacy features into the parameters.

[0046] S103. Organize the restricted context into a support set and a query set.

[0047] Encode images into visual features using a visual encoder. Text instructions are encoded into language features using a language encoder. , This represents the current task. A history example is a triple containing an image, text instructions, and a label / feedback. It supports retrieving several history examples at a time, with the query group retrieving an unlabeled image and text instruction tuple.

[0048] S104. Form shared initialization parameters that can be quickly migrated across user task distributions, and provide initialization conditions for shared adaptation paths and private adaptation paths respectively.

[0049] Building cross-user task distribution Each task includes images, text instructions, and historical examples, allowing the model to learn relatively easy-to-fit initial parameter values ​​during training. For each task... In this invention, the visual encoder parameters and the language encoder parameters are jointly denoted as... And perform inner updates on the support set to obtain temporary parameters for this task. :

[0050]

[0051] in, Indicates shared initialization parameters. Indicates task Temporary adaptation parameters, Indicates the inner learning rate. Indicates task The loss function on the support set.

[0052] After obtaining the temporary parameters, this invention uses the query set to evaluate the adaptation effect and updates the shared initialization parameters accordingly, making them more sensitive to subsequent few-sample tasks.

[0053]

[0054] in, Indicates the outer learning rate. Indicates task The loss function on the query set, This represents the gradient with respect to the shared initialization parameters.

[0055] In this embodiment, to simultaneously consider both visual input and text context, the present invention outputs the visual encoder... With language encoder output The data is then fused to form a unified task representation used during meta-training:

[0056]

[0057] in, Indicates task The task representation, Represents the visual mapping matrix. Represents the language mapping matrix, Indicates visual characteristics, Indicates language features.

[0058] In one embodiment, the present invention sets task weights for different task types, enabling tasks such as visual question answering, image and text description, and instruction following to participate in meta-initialization learning without compromising convergence stability due to imbalanced sample size.

[0059]

[0060] in, Describe the meta-objective function. Indicates task The weighting coefficients, This indicates the error of the temporary parameter on the query set.

[0061] In this way, the local model does not need to learn visual preferences and language styles from scratch when facing new users. Instead, it can start from the parameter positions that are closer to the target solution to complete the subsequent few-sample personalization.

[0062] During the deployment phase, small sample examples provided by new users are received. These samples are divided into a support set and an adaptation target according to task type. A few personalized updates are then performed without altering the overall network structure. The adaptation target functions as a query set or adaptation evaluation set, used to assess the adaptation effect after the few-step update based on the support set and to determine whether to stop.

[0063] To ensure that different tasks have dedicated adaptation paths, this invention introduces task gating variables. Used to regulate the first The impact of task-like operations on shared parameters:

[0064]

[0065] in, Indicates user In the Temporary parameters during the next update Indicates the first Gating coefficients for different types of tasks, This indicates the user loss corresponding to this update. Initial temporary parameters. The value is the meta-objective function. The shared initialization parameters corresponding to the iteration stopping when the condition is met. .

[0066] To prevent overfitting to a single example in scenarios with few samples, this invention constructs a user prototype representation on the local side using multiple examples from a support set, and combines the prototype with the current sample for adaptation. Specifically, for the first... The visual features of the example images and text instructions were obtained by visual encoders and speech encoders, respectively. Text features Then, they are mapped to the same semantic space and fused into cross-modal embeddings. By aggregating the mean of multiple cross-modal embeddings of the same user, we obtain:

[0067]

[0068] in, Indicates user Example prototype, Indicates the number of user samples. Indicates the first Cross-modal embedding of an example, and These represent the mapping matrices for visual features and textual features, respectively.

[0069] For visual encoders and language encoders, this invention does not simply update all parameters synchronously, but generates an adaptive step size based on the example prototype and task representation:

[0070]

[0071] in, This indicates the user-level adaptive learning rate. Indicates the basic learning rate. This represents the Sigmoid function. and Indicates the step size mapping parameter. This represents the concatenated vector of the user prototype and the task representation.

[0072] After adaptation, this invention determines whether convergence has been achieved by measuring the deviation between the target output distribution and the reference output distribution:

[0073]

[0074] in, Indicates user Adaptation deviation, Indicates the number of evaluation samples. Indicates the Kullback-Leibler divergence. Indicates the target distribution. This indicates the current model output distribution.

[0075] like If the value is less than the preset threshold, it means that a few updates are sufficient to restore the user's personal preferences; if the value exceeds the threshold, the user will proceed to the subsequent privacy constraint update stage for further correction.

[0076] In one implementation, during the parameter update process for small-sample personalized adaptation of new users locally, this invention prunes the gradient norm to prevent excessive parameter jumps due to individual sensitive samples in each update. For any local gradient... The result of the cutting is as follows:

[0077]

[0078] in, This represents the gradient after clipping. Represents the original gradient. The L2 norm of the gradient, This represents the gradient clipping threshold.

[0079] To limit the deviation of personalized updates from the original initialization parameters, this invention adds an upper bound constraint to the distance of the updated parameters:

[0080]

[0081] in, This represents the parameter drift penalty term. This indicates the parameters after user adaptation. Indicates the initialization parameters. This indicates the maximum allowable drift amplitude.

[0082] Based on this, the present invention adds Gaussian noise to the pruning gradient during the local parameter update process for new users to satisfy differential privacy constraints:

[0083] in, This represents the gradient after noise injection. Indicates a Gaussian distribution. Represents the noise figure. Represents the identity matrix.

[0084] Ultimately, the optimization objective for local few-sample adaptation is expressed as:

[0085]

[0086] in, Indicates user Privacy constraint objective function, Indicates mission loss. and These represent the weights of the parameter drift penalty and the gradient stability penalty, respectively.

[0087] The privacy constraint objective function is used to constrain the local few-shot adaptation process for new users. The task loss ensures that the model completes the target task, the parameter drift penalty limits the personalized parameters from deviating too far from the original initial parameters, and the gradient stability penalty limits the update magnitude, thereby improving the stability of few-shot adaptation.

[0088] Through the above joint constraints, the personalization process is ensured to occur only within a lightweight and controllable parameter subspace, balancing output quality and sensitive information protection requirements.

[0089] S105, Generation of user-specific adapters based on hypernetworks.

[0090] Encode a small number of user samples locally and aggregate them into a cross-modal compressed prototype. Then input the prototype into the hypernet. The hypernetwork does not change the original weights of the base model, but instead outputs two sets of user-specific low-rank parameter increments. Here, "cross-modal" in the cross-modal compression prototype referred to in this paper refers to the model formed by fusing image features and text features. It is used to characterize the stable joint semantics of the same user on the current task. The "compressed prototype" refers to this fixed-length... Compared to the aggregated summary formed by multiple original images, text instructions, and context fields, compression only occurs during the local encoding and aggregation process; the hypernet and subsequent operations all use [compression techniques]. Instead of receiving or storing the original example.

[0091] The low-rank parameter increment of the shared adaptation path is used to adapt reusable visual and instruction semantics across users and tasks; the low-rank parameter increment of the private adaptation path is only used to express the current user's preferences in the local restricted context.

[0092] For any layer to be adapted The LoRA parameters of the two paths can be expressed as:

[0093]

[0094] in, and Representing users respectively In the The shared path low-rank matrix of the layers to be adapted. and These represent the low-rank matrices of the private paths, The parameter is Oriented towards the first Hypernetwork with layer output parameters This represents the aggregated representation of user examples. The shared path and the private path are two lightweight adaptation branches on the same base model, not two independent base models; the former only handles general task representations without sensitive contexts, while the latter handles user representations constrained by field masks locally.

[0095] The adapted linear mapping does not directly rewrite the original weights, but rather modifies the base weights. Superimposed low-rank increments:

[0096]

[0097] in, Indicates whether a path is shared or private in the first place. A user-specific mapping matrix for each layer to be adapted. This represents the basic mapping matrix of this layer. Indicates the scaling factor. This indicates a low-rank dimension.

[0098] In one embodiment, to ensure stable supernet output, the present invention maintains consistency constraints on adapters generated under different example sequences:

[0099]

[0100] in, This indicates the loss of consistency across the supernetwork. and This represents the aggregated representation obtained from different sample orders of the same user.

[0101] Meanwhile, this invention constrains the adapter size through storage efficiency metrics, avoiding excessive storage burden on end devices in multi-user scenarios:

[0102]

[0103] in, This represents the ratio of user-adapted parameters to the original data storage size. and Indicates the storage size of the adaptation matrix. This represents the storage size of the original user sample set. When When the threshold is exceeded, the LoRA rank can be reduced, low-contribution parameters can be pruned, or the adapter can be regenerated.

[0104] S106. During inference within the local adapter, the shared path is input to obtain the shared semantic flow using the task-insensitive representation. Private paths take local restricted context as input to obtain private semantic streams. The local fusion engine adjusts based on the current task type and adaptation bias. Generate non-negative sums Selection weight , And form a fused state for use by the local decoder:

[0105]

[0106] in, Indicates the first The fusion state at each decoding moment Indicates a shared semantic stream, Represents a private semantic stream; increases when the current task primarily depends on a general task structure. Increase when it is necessary to reflect local user preferences and field access masks allow. .

[0107] In the cloud-and-local collaborative reasoning stage, this invention preferably uses a large cloud model to first generate a structural sketch without sensitive context, and then a small local model combines the user's private context and an adapter generated by the hypernet to complete the refinement. The cloud sketch generation process is represented as follows:

[0108]

[0109] in, This represents a structural sketch generated in the cloud. This represents a large model in the cloud. This indicates that there is no sensitive cloud input.

[0110] After receiving local hints, sketches, and user adapter parameters, the local mini-model generates the final output:

[0111]

[0112] in, This indicates the final output. Indicates a local small model, Indicates local restricted context input, and This represents the user adapter parameters.

[0113] In another embodiment, the present invention may also employ a logits fusion strategy, enabling the privacy-related distribution of the local model and the general inference distribution of the cloud model to participate in decoding together:

[0114]

[0115] in, This represents the merged logits. This represents the local small model's logits. This represents the large cloud-based model's logits. This indicates the fusion weight.

[0116] If a geometric mean mechanism is further adopted, the probability of generating the next token is:

[0117]

[0118] in, Indicates the first The final conditional probability of each token. This represents the conditional probability of the local small model. This represents the conditional probability of a large cloud-based model. Represents the geometric fusion coefficient; symbol " "This means that the candidate token set needs to be divided by the normalization constant." So that the sum of the final probabilities of each candidate token is .

[0119] The geometric mean mechanism refers to, for a candidate token, weighting the same conditional probability given by the local small model and the cloud large model separately. and The tokens are multiplied by exponentiation and then normalized over all candidate tokens, so that tokens with higher probabilities in both methods receive higher final probabilities. Calculating the probability of the next token is used to select or sample the output token for the current decoding step. Repeating this process step by step allows for a balance between the general structural planning provided by the cloud model and the user preferences and visual details provided by the local model, without transmitting private context to the cloud.

[0120] Therefore, this invention utilizes the structural planning capabilities of the cloud model during the inference phase while retaining the local model's fine control over private contexts and visual preferences, thereby achieving high-quality multimodal collaborative generation under privacy protection.

[0121] In one embodiment, the user terminal submits an image, text commands, and a very small number of examples. The terminal immediately performs a self-processing step to filter out content from the profile, history, and device fields that is not allowed to be shared, thus forming a restricted context. Subsequently, the support set and query set are organized, and another self-processing step is performed to load the meta-learning initialization parameters, ensuring that the system does not start adapting from a random state. The trainer sends example prototype requests to the supernetwork, which returns lightweight parameter increments for shared and private paths. This process corresponds to the core step of few-shot personalized modeling. Afterward, the trainer only sends non-sensitive task descriptions to the cloud-based co-generator, which returns a structural sketch and high-level hints to provide a general inference framework without exposing local privacy details. The trainer then sends the fused state and adaptation parameters to the local decoder, which performs orthogonal constraint checks, local decoding, and consistency evaluation. If the candidate responses show significant conflicts between shared and private representations, the trainer will again request the supernetwork to shrink the conflicting components, keeping the parameter drift constraints within a controllable range. After the correction is completed, the local decoder outputs personalized multimodal results, and the local manager displays the results to the user and records feedback. New user feedback and supplementary examples flow back to the trainer, triggering the next round of rapid adaptation. This allows the invention to gradually improve individual matching accuracy through continuous interaction, while maintaining a balance between privacy boundaries, adaptation efficiency, and collaborative generation quality.

[0122] In one application embodiment of a multimodal consultation assistance terminal within a hospital, the terminal is deployed on the hospital's intranet. The terminal needs to receive patient images, textual complaints, and a small number of historical interaction examples, and generate personalized consultation suggestions without disclosing the patient's privacy data. This scenario is constrained by hospital privacy compliance requirements, terminal computing power limitations, and real-time response pressure. Therefore, it is necessary to utilize a collaborative framework to complete limited-sample visual adaptation locally, and achieve stable collaborative generation by providing only non-sensitive structural sketches in the cloud.

[0123] S201. The terminal receives the patient's current images, text complaints, and a small number of historical examples, organizes them into a task package to be processed, establishes a local session index according to the patient identifier, and outputs a controlled task description and a small sample cache, providing a unified entry point for the rapid loading of subsequent meta-learning initialization.

[0124] S202. Extract permission fields such as user profile fields, historical interaction fields, and device usage information fields, and output restricted context sequences and field masks based on the set gradient pruning threshold and rank parameter boundary pre-set adaptation constraints. This prevents sensitive fields from entering the outgoing link.

[0125] S203, Load meta-learning initialization parameters The system maps the current image task and text instructions to a cross-modal task representation, and then combines a few-shot cache to form a support set and a query set. Based on the high success rate performance under few-shot conditions, the system sets the default support set size to an appropriate size in this embodiment, and outputs the initial values ​​for quick adaptation and the task prototype. .

[0126] S204, Based on Task Prototype Perform a few-step update, and perform linked correction on the visual encoder and speech encoder to obtain temporary parameters for the current patient. And simultaneously generate adaptation deviation As a criterion for whether to continue the correction, the updated local representation is output.

[0127] S205. Prune the local gradient, limit parameter drift, and inject noise to control the parameter offset to not exceed the safety boundary; combined with the test verification results showing a significant decrease in the success rate of privacy attacks, output controlled incremental parameters and audit flags that meet privacy requirements.

[0128] S206. The cloud only receives the de-identified task description and returns a structural sketch. The local side sends the sketch and the restricted context into the hypernet. Based on its high personalization accuracy and good modal consistency, the system uses the sketch as a structural constraint rather than a direct answer and outputs the collaborative decoding conditions before fusion.

[0129] S207. The hypernetwork generates a user-specific low-rank adaptation matrix based on the few-sample prototype and selectively aggregates the semantic flows of shared paths and private paths to form a fused state. If the consistency check finds a conflict between shared semantics and individual preferences, the adaptation amplitude is rolled back and lightweight parameters are regenerated, outputting a decorable final state.

[0130] S208. The local decoder generates consultation assistance results based on the fusion status, and then writes the medical staff feedback and new examples back to the few-sample cache. Combining the overall performance and the setting of lightweight adaptation parameters, this embodiment outputs a personalized multimodal response that can run continuously on the hospital terminal and completes closed-loop updates.

[0131] This embodiment balances consultation accuracy, response efficiency, and data compliance within a closed-loop privacy environment within the hospital. The system exhibits excellent personalization accuracy and superior modal consistency, while significantly reducing the success rate of privacy attacks. Combined with a lightweight adaptation parameter strategy, this embodiment is suitable for long-term deployment on hospital intranet terminals and maintains stable few-sample visual adaptation capabilities.

[0132] In a personalized cold start scenario for new users of a home rehabilitation tablet, this embodiment is deployed on a patient's home rehabilitation tablet. The device needs to generate personalized graphic and textual guidance based on the user's uploaded motion images, rehabilitation questions, and a very small number of preference examples. The main constraints of this scenario are the extremely small sample size during the cold start of new users, significant fluctuations in home network bandwidth, and the device's limited computing power. Therefore, this implementation emphasizes the rapid transfer capability of meta-learning and a lightweight adapter storage mechanism to shorten the initial usability time and maintain privacy protection.

[0133] S301 The tablet terminal receives motion images, text questions, and a small number of personal preference examples taken by the user, writes the device status into the local task context, and outputs a small sample session packet for the new user for subsequent cold start adaptation.

[0134] S302. Locally perform field whitelist filtering on the session packet, retaining only rehabilitation preferences, recent interactions and device usage characteristics, and set the initialization loading strategy according to the predetermined meta-learning rounds and support set size, outputting restricted context and initial task labels to avoid unnecessary fields from participating in encoding.

[0135] S303. Read the shared initialization parameters and generate a fast-transferable starting point based on the current image task and historical metadata. Based on the excellent success rate under very few sample conditions, this embodiment prioritizes triggering a few-step fast adaptation when there are only a few examples, outputting the cold start initial value and the user prototype. .

[0136] S304, Utilizing User Prototypes Update key layers in the visual encoder and language encoder, perform only a limited number of gradient steps to reduce the load on home devices, and output temporary adaptation results for the current user's action preferences and expression style.

[0137] S305. Prune the gradient and limit parameter drift, while adding differential privacy noise to ensure that the updated parameters meet the controlled boundary; based on the high performance of the individual preference recovery rate, output a local parameter snapshot that balances individual recovery and privacy constraints.

[0138] S306. The supernet generates a lightweight low-rank adapter based on local parameter snapshots and a few-sample prototypes, and then solidifies the adapter into a reusable configuration. Combined with the low-rank configuration, this step outputs a personalized adapter set for new users, which is suitable for long-term retention on tablets without saving the original example.

[0139] S307. The cloud receives the de-identified task description and returns a structural sketch. The local fusion unit decodes the sketch together with the personalized adapter. Based on the improved privacy attack defense capabilities and the extremely low end-side adaptation latency, this step outputs the high-level collaboration conditions and the final local generated state that meet the bandwidth-constrained scenario.

[0140] S308: The tablet terminal generates rehabilitation guidance results with parallel text and images, and rewrites the user's corrective opinions and newly added images into the small sample cache; if the number of subsequent examples increases, it sends back to S303 to continue to fine-tune the initialization of the surrounding parameters, thereby outputting a personalized rehabilitation guidance closed loop that gradually enhances with interaction.

[0141] The beneficial effects of this invention are as follows:

[0142] 1. By leveraging the meta-learning initialization module, the visual encoder and language encoder can enter the effective working range with very few examples, significantly improving the availability of cold start with few samples;

[0143] 2. During the local adaptation process, field whitelists, gradient pruning, parameter drift limits, and differential privacy noise are introduced simultaneously, so that sensitive contexts do not need to leave the terminal device, significantly reducing the risk of privacy leakage.

[0144] 3. By using hypernetworks to directly generate user-specific lightweight adapters from a small number of examples, personalized features can be stably reproduced without saving the original samples;

[0145] 4. Instead of relying on full model retraining, it completes the generation through low-rank adapter and cloud-based structural sketch, which can reduce the deployment and maintenance costs on the edge.

[0146] 5. By using task prototyping, adaptive learning rate, and constrained parameter update strategies, ineffective iterations are reduced, enabling the system to reach a usable state faster with limited examples;

[0147] 6. By incorporating tasks such as visual question answering, image and text description, and instruction following into a unified meta-learning distribution, and then using task gating variables to achieve differentiated adaptation, the system can maintain stable output when switching tasks, thereby enhancing the system's robustness in transitioning between different tasks.

[0148] 7. This invention does not only pursue the improvement of a single indicator, but achieves a balance between small sample personalization, edge adaptation speed and privacy closed loop, and can take into account the generation quality, privacy boundary and engineering feasibility.

[0149] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0150] Based on the same inventive concept, this application also provides a privacy-preserving multimodal few-shot personalization device based on meta-learning and hypernetworks. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the privacy-preserving multimodal few-shot personalization device based on meta-learning and hypernetworks provided below can be found in the limitations of the privacy-preserving multimodal few-shot personalization method based on meta-learning and hypernetworks described above, and will not be repeated here.

[0151] In one embodiment, such as Figure 1 As shown, a privacy-preserving multimodal few-shot personalization device based on meta-learning and hypernetworks is provided, comprising:

[0152] The user terminal input unit is used to receive images, text commands, and a small number of historical examples generated on the terminal side.

[0153] The local privacy context management unit is used to perform field filtering on user profiles, historical behavior and device usage information, retaining only the restricted contexts that are allowed to participate in training and inference;

[0154] The small sample task building unit is used to organize the current task into a support sample set and a query set;

[0155] The meta-learning initialization unit is used to form shared initialization parameters that can be quickly transferred across user task distributions, and to provide initialization conditions for shared adaptation paths and private adaptation paths respectively.

[0156] Shared adapter groups are used to carry shared semantic streams that can be reused across users;

[0157] Private adapter groups are used to carry private semantic streams related to the current user's preferences;

[0158] The selective aggregation unit is used to selectively aggregate the two representations to form a fusion state for the reasoning stage. And send the non-sensitive collaborative conditions to the cloud-based co-generation unit;

[0159] The cloud-based collaborative generation unit does not directly access the complete privacy context, but only returns a structural sketch or high-level hints;

[0160] The fusion boundary and consistency verification unit is responsible for performing privacy constraints and decoupling verification. It controls the coupling degree between the shared space and the private space through orthogonal constraints, consistency comparisons, and parameter drift limits, so that the shared parameters maintain the task generalization ability while suppressing the leakage of private information.

[0161] Personalized multimodal response units are used to send user feedback back to small sample task construction units, forming a continuous update loop, so that subsequent adaptations can be locally modified based on the latest preferences.

[0162] The modules in the aforementioned privacy-preserving multimodal few-shot personalization device based on meta-learning and hypernetworks can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0163] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in all of the above method embodiments.

[0164] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in all of the above method embodiments.

[0165] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in all of the above method embodiments.

[0166] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0167] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0168] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0169] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A privacy-preserving multimodal few-shot personalization method based on meta-learning and hypernetworks, characterized in that, The method includes: Obtain the user's original image, text command, and several historical examples generated on the terminal side, and perform sensitive field filtering on user information to obtain the restricted context that allows participation in training and inference; During the training phase, the images and text instructions preserved in the restricted context corresponding to the task are fused to obtain example prototypes, and the historical examples preserved in the restricted context are combined to form a support set and a query set; Based on the support set and the query set, meta-learning techniques are used to update the parameters of the visual encoder and the language encoder. During the parameter update process, gradient clipping, gradient noise addition, and parameter drift restriction operations are implemented. Combined with the hypernetwork, shared adapter groups and private adapter groups are obtained. During the inference phase, the historical examples are input into the shared adapter to obtain the shared semantic stream, the restricted context is input into the private adapter group to obtain the private semantic stream, the shared semantic stream and the private semantic stream are weighted and fused to obtain the fusion state, and the fusion state is combined to obtain the multimodal result.

2. The method according to claim 1, characterized in that, The step of updating the parameters of the visual encoder and the language encoder using meta-learning techniques based on the support set and the query set includes: For the current task, perform an inner update on the support set based on the shared initialization parameters to obtain the task-oriented temporary parameters; The adaptation effect of the temporary parameters is evaluated using the query set, and the shared initialization parameters are updated based on the evaluation results; The next round of parameter updates will proceed based on the updated shared initialization parameters until the meta-objective function meets the conditions, at which point the iteration will stop; wherein, the shared initialization parameters corresponding to the iteration stop will serve as the initialization parameters of the shared adapter group.

3. The method according to claim 2, characterized in that, When a new user joins, the method includes: Obtain a small sample of new users, and divide the small sample into a support set and a query set according to the task type; Based on the shared initialization parameters corresponding to the iteration stop and the support set corresponding to the new user, the temporary parameters for the new user are iteratively updated, and the iteration is evaluated to determine whether to stop based on the temporary parameters of the new user and the query set of the new user; wherein, the temporary parameters corresponding to the iteration stop are used as the initialization parameters of the private adapter group.

4. The method according to claim 3, characterized in that, The method further includes: When iteratively updating the temporary parameters for new users, the updates are performed using an adaptive step size generated based on the example prototype and task representation corresponding to the new user.

5. The method according to claim 3, characterized in that, After obtaining the initialization parameters of the shared adapter group combined with the private adapter group, the method further includes: Input the historical example into the hypernetwork to obtain the low-rank matrix of the shared adapter group; Input the restricted context into the hypernetwork to obtain the low-rank matrix of the private adapter group; The parameters of the shared adapter and the private adapter group are updated using the low-rank matrix for use in the inference phase.

6. The method according to claim 1, characterized in that, The method further includes: Construct a privacy-constrained objective function based on task loss, parameter drift penalty, and gradient stability penalty.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images