Visual language model adaptation methods, devices, computer equipment, and storage media

Abstract

This invention relates to the field of artificial intelligence technology, applicable to scenarios such as finance and healthcare. It discloses a method, apparatus, computer device, and storage medium for adapting a visual language model. The method includes: acquiring task data from multiple scenarios; pre-training a visual language model using the task data; and extracting initial joint semantic features from the task data of multiple scenarios using the visual language model; injecting scenario-related parameters into each scenario using a lightweight adapter, and fusing them with the initial joint semantic features to obtain scenario features; predicting scenario weights using a dynamic routing mechanism, and fusing them based on scenario features and scenario weights to obtain fused features; mapping the fused features using cross-modal semantics, and decoding and outputting the fused features using a task decoder. This invention implements a lightweight adaptation structure based on task-scenario decoupling, which can effectively improve the transferability of visual language models in multi-scenario tasks.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer device, and storage medium for adapting visual language models. Background Technology

[0002] With the development of multimodal large models (such as CLIP, BLIP, GPT-4V, etc.), visual language models (VLMs) have shown broad application potential in tasks such as decision support, image-text alignment, and object recognition. However, the training data for these models mainly comes from general internet scenarios, possessing high abstraction and broad domain generalization capabilities, but lacking fine-grained adaptability to specific scenarios. In practical applications, such as indoor intelligent robots, assisted diagnosis and treatment systems, and intelligent logistics terminals, models need to understand language instructions in specific environments and complete object recognition tasks. Especially in situations where the scene changes significantly but the task type remains stable, such as medical institutions, home care, elderly care communities, and bank service halls, the performance of visual language models (VLMs) may significantly decrease when transferred from one environment to another after fine-tuning in one environment. For example, the instruction "get the red bottle" may refer to medicine in a hospital scenario, while it may refer to a condiment bottle in a home scenario, meaning that there are discrepancies between the semantics and visual form of the target. Such domain gaps cannot be solved by simple fine-tuning or image augmentation. Furthermore, existing methods often rely on full fine-tuning of the VLM model, resulting in high overall costs and a high risk of catastrophic forgetting. Therefore, how to improve the transfer and adaptation capabilities of visual language models is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0003] This invention provides a visual language model adaptation method, apparatus, computer device, and storage medium, aiming to enhance the adaptation effect of visual language models.

[0004] In a first aspect, embodiments of the present invention provide a visual language model adaptation method, comprising:

[0005] Acquire task data from multiple scenarios, use the task data to pre-train a visual language model, and extract initial joint semantic features from the task data of multiple scenarios using the visual language model.

[0006] Multiple lightweight adapters are used to inject scene-related parameters into each scene, and the scene features are obtained by fusing them with the initial joint semantic features.

[0007] A dynamic routing mechanism is used to predict scene weights, and fused features are obtained based on the scene features and scene weights.

[0008] Based on the scenario, the fused features are mapped using cross-modal semantics, and the mapped fused features are decoded and output through a task decoder to construct a target adaptation model.

[0009] The target adaptation model is used to perform transfer adaptation on the visual language model of the specified scene.

[0010] Secondly, embodiments of the present invention provide a visual language model adaptation device, comprising:

[0011] The pre-training unit is used to acquire task data from multiple scenarios, pre-train the visual language model using the task data, and extract initial joint semantic features from the task data of multiple scenarios using the visual language model.

[0012] The scene adaptation unit is used to inject scene-related parameters into each scene using multiple lightweight adapters, and to fuse the scene features with the initial joint semantic features.

[0013] The dynamic routing unit is used to predict scene weights using a dynamic routing mechanism, and to obtain fused features based on the scene features and scene weights.

[0014] The mapping and decoding unit is used to map the fused features based on the scenario using cross-modal semantics, and to decode and output the mapped fused features through the task decoder, thereby constructing a target adaptation model.

[0015] The transfer adaptation unit is used to perform transfer adaptation on the visual language model of the specified scene using the target adaptation model.

[0016] Thirdly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the visual language model adaptation method as described in the first aspect.

[0017] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the visual language model adaptation method as described in the first aspect.

[0018] This invention provides a visual language model adaptation method, apparatus, computer device, and storage medium. The method includes: acquiring task data from multiple scenarios; pre-training a visual language model using the task data; and extracting initial joint semantic features from the task data of multiple scenarios using the visual language model; injecting scenario-related parameters into each scenario using multiple lightweight adapters, and fusing them with the initial joint semantic features to obtain scenario features; predicting scenario weights using a dynamic routing mechanism, and fusing the scenario features and scenario weights to obtain fused features; mapping the fused features using cross-modal semantics based on the scenario, and decoding and outputting the mapped fused features using a task decoder to construct a target adaptation model; and using the target adaptation model to perform transfer adaptation on a visual language model for a specified scenario.

[0019] This invention pre-trains a visual language model on multiple scene task data to extract initial joint semantic features. Scene-related parameters are injected through a lightweight adapter, and scene features are fused. Scene weights are predicted using a dynamic routing mechanism, and fused features and weights are obtained. Finally, based on the scene, cross-modal semantics are used for mapping, and the output is decoded by a task decoder. Based on this, this invention implements a lightweight adaptable architecture based on task and scene decoupling. The design goal of this architecture is to significantly enhance the transferability of the visual language model (VLM) when facing tasks in various different scenarios. By adopting this lightweight adaptable structure, the model can more flexibly adapt to various task requirements, thereby achieving more efficient and accurate performance in different scenarios. This structure, by decoupling the dependency between tasks and scenes, allows the model to quickly adjust its parameters to adapt to new task requirements when handling tasks in new scenarios, without the need for large-scale retraining. This lightweight design not only improves the model's generalization ability but also reduces computational resource consumption, making the model more efficient and practical in real-world applications. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating a visual language model adaptation method provided in an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of a sub-process of a visual language model adaptation method provided in an embodiment of the present invention;

[0023] Figure 3 This is another flowchart illustrating a visual language model adaptation method provided in an embodiment of the present invention;

[0024] Figure 4 A schematic diagram illustrating the principle architecture of a visual language model adaptation method provided in an embodiment of the present invention;

[0025] Figure 5 A schematic block diagram of a visual language model adaptation device provided in an embodiment of the present invention;

[0026] Figure 6 A schematic block diagram of a visual language model adaptation device provided in an embodiment of the present invention;

[0027] Figure 7 Another schematic block diagram of a visual language model adaptation device provided in an embodiment of the present invention;

[0028] Figure 8 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0031] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0032] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0033] The visual language model adaptation method provided in this invention can be applied in an application environment where a client and a server interact, wherein the client communicates with the server via a network. The server acquires task data from multiple scenarios, pre-trains a visual language model using the task data, and extracts initial joint semantic features from the task data of multiple scenarios using the visual language model. Multiple lightweight adapters are used to inject scenario-related parameters into each scenario, and these are fused with the initial joint semantic features to obtain scenario features. A dynamic routing mechanism is used to predict scenario weights, and fused features are obtained based on the scenario features and scenario weights. Based on the scenario, cross-modal semantics is used to map the fused features, and a task decoder decodes and outputs the mapped fused features to construct a target adaptation model. The client sends a specified scenario, and the server uses the target adaptation model to perform transfer adaptation on the visual language model of the specified scenario. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0034] Please see below. Figure 1 The present invention provides a visual language model adaptation method, which specifically includes steps S101 to S105.

[0035] Step S101: Obtain task data from multiple scenarios, pre-train the visual language model using the task data, and extract initial joint semantic features from the task data of multiple scenarios using the visual language model.

[0036] Step S102: Use multiple lightweight adapters to inject scene-related parameters into each scene, and combine them with the initial joint semantic features to obtain scene features;

[0037] Step S103: Predict scene weights using a dynamic routing mechanism, and obtain fused features based on the scene features and scene weights;

[0038] Step S104: Based on the scenario, the fused features are mapped using cross-modal semantics, and the mapped fused features are decoded and output through a task decoder to construct a target adaptation model.

[0039] Step S105: Use the target adaptation model to perform transfer adaptation on the visual language model of the specified scene.

[0040] In this embodiment, combined with Figure 4First, initial joint semantic features are extracted by acquiring task data from multiple scenarios and pre-training a visual language model. Next, specific parameters are injected into each scenario, and these parameters are fused with the initial features using a lightweight adapter to obtain scene features. Then, scene weights are predicted using a dynamic router, and combined with the scene features to obtain the final fused features. Finally, cross-modal semantic mapping is performed on the fused features, and the final result is output through a task decoder.

[0041] This embodiment implements a lightweight adaptive architecture based on task and scene decoupling. The design goal of this architecture is to significantly enhance the transferability of the Visual Language Model (VLM) when facing tasks in various different scenarios. By adopting this lightweight adaptive structure, the model can more flexibly adapt to various task requirements, thereby achieving more efficient and accurate performance in different scenarios. This structure, by decoupling the dependency between tasks and scenarios, allows the model to quickly adjust its parameters to adapt to new task requirements when handling tasks in new scenarios, without requiring large-scale retraining. This lightweight design not only improves the model's generalization ability but also reduces computational resource consumption, making the model more efficient and practical in real-world applications.

[0042] In practical applications, the visual language model adaptation method provided in this embodiment can be applied to the medical field. For example, in an intelligent assisted diagnostic system, the visual language model can understand the doctor's verbal instructions, such as "show the abnormal areas in the patient's lung X-ray," and accurately mark the lesion sites in the image. Furthermore, this method can also be applied to the financial field. For instance, in an intelligent banking service system, the visual language model can recognize customer verbal instructions, such as "check my account balance," and quickly display the relevant information on the interface. Similarly, this method can be applied to the smart home field. For example, an intelligent voice assistant can understand the instructions of family members, such as "turn on the living room lights," and automatically execute the corresponding operations. Through the visual language model adaptation method provided in this embodiment, the model can be flexibly adjusted according to the task requirements in different scenarios, achieving efficient and accurate performance, thereby greatly improving the practicality and adaptability of the visual language model in practical applications.

[0043] In one embodiment, such as Figure 2 As shown, step S101 includes steps S201 to S203.

[0044] Step S201: Encode the image data into image features using a visual encoder;

[0045] Step S202: Encode the text data into language features using a language encoder;

[0046] Step S203: Fuse the image features and language features into the initial joint semantic features.

[0047] Specifically, fusing the image features and language features into the initial joint semantic features includes:

[0048] The initial joint semantic features are obtained by fusing according to the following formula:

[0049] f = VLM frozen (x,t)

[0050] Where f represents the initial joint semantic features, x represents image data, t represents text data, VLM represents a visual language model containing a visual encoder and a language encoder, and frozen represents parameter freezing.

[0051] This embodiment employs a frozen pre-training method to train the Visual Language Encoder (VLM), where the model parameters are frozen during the training phase. The VLM then extracts joint semantic features from the image and text, which are used as the base representation. Mainstream models can use architectures such as CLIP, BLIP, or Flamingo, and their main components include:

[0052] Visual encoder (ViT / ResNet): Input image x → Output image features f x ;

[0053] Language encoder (Transformer): Input text t → Output language features f t ;

[0054] Fusion module: For example, the BLIP architecture uses Cross-Attention to fuse text and images to obtain a cross-modal representation f, which is the initial joint semantic feature.

[0055] This approach ensures that the model retains key semantic connections between images and text when processing multimodal data, providing a solid foundation for subsequent scene adaptation. The advantage of frozen pre-training lies in its ability to flexibly adjust the model across different scenarios by introducing new task data and adaptation strategies without altering the pre-trained model parameters. This method not only improves the model's stability and reliability but also significantly reduces the time and cost of transferring the model between different tasks. Furthermore, using a Visual Language Model (VLM) as the basis for feature extraction fully leverages the model's powerful capabilities in visual and language understanding, further enhancing its performance in complex scenarios.

[0056] In one embodiment, step S102 includes:

[0057] The residual module is used to fuse the initial joint semantic features and scene-related parameters into scene features:

[0058] f i =f+Adapter i (f)

[0059] Among them, f i Let f represent the initial joint semantic features, and Adapter represent the scene features. i (f) represents scene-related parameters, and i represents the i-th scene.

[0060] After extracting the initial joint semantic features, a lightweight Adapter module is inserted into the image-text fusion layer, and each scene has its own independent Adapter to inject a small number of scene-related parameters for different task scenarios. This embodiment uses the classic residual BottleneckAdapter:

[0061] f i =f+Adapter i (f), i∈{family, community, institution...}

[0062] In practical applications, the initial joint semantic features can be first down-projected, then activated using GELU / ReLU, followed by up-projection, and finally the scene features are obtained by summing the residuals. In this way, the features for each scene can be effectively integrated with the initial joint semantic features and relevant parameters specific to the scene, enabling the model to better understand and handle task requirements in different scenarios. Through this approach, the lightweight adapter not only enhances the model's scene adaptability but also maintains its lightweight nature, avoiding increased computational burden caused by introducing too many parameters.

[0063] In one embodiment, step S103 includes:

[0064] Scene weights are predicted using a lightweight multilayer perceptron, and fused features are obtained according to the following formula:

[0065]

[0066] Among them, f * Represents the fusion feature, α i This represents the scene weight of the i-th scene.

[0067] This embodiment uses a dynamic routing mechanism to dynamically determine which weighted combination to use. The structure of the dynamic routing mechanism can employ a lightweight MLP to predict the weights α = [α1, α2, ..., α]. iThe model then fuses the predicted weights to obtain the fused features. In practical applications, a lightweight MLP can consist of multiple fully connected layers, with ReLU or GELU activation functions used between layers. Finally, the weights for each scene are obtained through Softmax normalization. This mechanism allows the model to dynamically adjust its focus in different scenes, thereby enhancing its adaptability to new scenes.

[0068] Furthermore, the introduction of the dynamic routing mechanism enhances model interpretability. Since the weights for each scene are predicted using a lightweight multilayer perceptron, it becomes intuitive to understand the model's emphasis on each scene when handling different tasks. This not only helps in better understanding the model's decision-making process but also facilitates subsequent model optimization and debugging.

[0069] In practical applications, data augmentation strategies can be employed to improve the generalization ability and robustness of models. Specifically, when acquiring task data from multiple scenarios, random transformations and perturbations can be applied to image and text data, such as image rotation, scaling, cropping, and text synonym replacement, to increase the diversity and complexity of the data. In this way, the model can be exposed to more data variations during the training phase, thereby learning more robust feature representations and improving its adaptability to new data.

[0070] In one embodiment, step S104 includes:

[0071] Obtain the task type, and select the corresponding task decoder according to the task type to decode and output the mapped fused features.

[0072] Different task types may correspond to different output formats and requirements. For example, in image recognition tasks, the task decoder may need to map fused features to specific category labels; in text generation tasks, the task decoder may need to convert fused features into natural language text. Therefore, selecting the appropriate task decoder based on the task type is crucial.

[0073] In practice, a set of task decoders can be predefined, each optimized for one or more specific task types. When a new task needs to be processed, the most suitable decoder can be selected from the predefined set based on the task type. This approach not only improves the model's flexibility and scalability but also ensures that the model achieves optimal performance when handling different tasks.

[0074] Furthermore, to further improve model performance, a multi-task learning strategy can be employed. Specifically, during the training phase, the model can be trained simultaneously using data from multiple tasks. This allows the model to learn how to handle individual tasks while also learning the correlations and commonalities between different tasks. In this way, the model can better utilize the complementary information between different tasks, thereby improving its performance across all tasks.

[0075] Specifically, such as Figure 3 As shown, the step of obtaining the task type and selecting the corresponding task decoder to decode and output the mapped fused features according to the task type includes steps S301 to S303.

[0076] Step S301: When the task type is a classification task, a multilayer perceptron is used as the task decoder;

[0077] Step S302: When the task type is a generation task, an LLM-style decoder is used as the task decoder;

[0078] Step S303: When the task type is an object detection task, the Transformer decoder is used as the task decoder.

[0079] In this embodiment, when mapping cross-modal semantics to task output, the appropriate task decoder can be selected based on the task type. For example, in classification tasks (such as scene selection), an MLP can be chosen as the task decoder. The decoding process can specifically include Transformer head output → Dropout → Linear → Softmax, that is, the fused feature f* is input into the MLP for classification prediction to obtain the classification result. In generation tasks (such as text generation), an LLM-style decoder can be chosen. The decoding process can perform autoregressive text generation based on the fused feature f*, with the input embedding from the Adapter output, and the decoding target being the answer or path. In object detection tasks (such as object detection), a Transformer decoder can be chosen. The decoding process can include position embedding → Transformer decoder → detection head output, thereby obtaining the position and label information of the target object. In this way, the visual language model adaptation method of this embodiment can flexibly adapt to different types of task output requirements, improving the model's versatility and practicality.

[0080] In summary, the visual language model adaptation method provided in this embodiment has the following advantages:

[0081] (1) Significantly improves adaptability to multiple scenarios. Unlike the traditional static fine-tuning method, this embodiment introduces a scenario adaptation mechanism to effectively improve the model's robustness in various environments. Experiments show that the same model can automatically adapt to the task without retraining in three environments: home, hospital, and elderly care community.

[0082] (2) Low training cost and fewer parameter updates. All lightweight parameters are inserted as Adapter modules, without affecting the original VLM structure, and can be flexibly loaded or unloaded. Compared with full fine-tuning, the amount of parameter updates is reduced by more than 90%.

[0083] (3) Strong structural compatibility, adaptable to a variety of large models. It is suitable for various visual language model structures such as BLIP, CLIP, MiniGPT, and LLaVA, and can be embedded in mobile deployment solutions.

[0084] (4) Deployment-friendly and inference-efficient. Scene judgment can rely solely on images, and only a portion of the Adapters are activated during the inference phase, eliminating the need for redundant computation of the entire model, making it suitable for deployment in edge scenarios.

[0085] Figure 5 This is a schematic block diagram of a visual language model adaptation device 500 provided in an embodiment of the present invention. The device 500 includes:

[0086] The pre-training unit 501 is used to acquire task data from multiple scenarios, pre-train a visual language model using the task data, and extract initial joint semantic features from the task data of multiple scenarios using the visual language model.

[0087] The scene adaptation unit 502 is used to inject scene-related parameters into each scene using multiple lightweight adapters, and to fuse the scene features with the initial joint semantic features.

[0088] The dynamic routing unit 503 is used to predict scene weights using a dynamic routing mechanism, and to obtain fused features based on the scene features and scene weights.

[0089] The mapping and decoding unit 504 is used to map the fused features based on the scenario using cross-modal semantics, and decode and output the mapped fused features through the task decoder, thereby constructing a target adaptation model.

[0090] The transfer adaptation unit 505 is used to perform transfer adaptation on the visual language model of the specified scene using the target adaptation model.

[0091] In this embodiment, combined with Figure 4First, initial joint semantic features are extracted by acquiring task data from multiple scenarios and pre-training a visual language model. Next, specific parameters are injected into each scenario, and these parameters are fused with the initial features using a lightweight adapter to obtain scene features. Then, scene weights are predicted using a dynamic router, and combined with the scene features to obtain the final fused features. Finally, cross-modal semantic mapping is performed on the fused features, and the final result is output through a task decoder.

[0092] This embodiment implements a lightweight adaptive architecture based on task and scene decoupling. The design goal of this architecture is to significantly enhance the transferability of the Visual Language Model (VLM) when facing tasks in various different scenarios. By adopting this lightweight adaptive structure, the model can more flexibly adapt to various task requirements, thereby achieving more efficient and accurate performance in different scenarios. This structure, by decoupling the dependency between tasks and scenarios, allows the model to quickly adjust its parameters to adapt to new task requirements when handling tasks in new scenarios, without requiring large-scale retraining. This lightweight design not only improves the model's generalization ability but also reduces computational resource consumption, making the model more efficient and practical in real-world applications.

[0093] In practical applications, the visual language model adaptation method provided in this embodiment can be applied to the medical field. For example, in an intelligent assisted diagnostic system, the visual language model can understand the doctor's verbal instructions, such as "show the abnormal areas in the patient's lung X-ray," and accurately mark the lesion sites in the image. Furthermore, this method can also be applied to the financial field. For instance, in an intelligent banking service system, the visual language model can recognize customer verbal instructions, such as "check my account balance," and quickly display the relevant information on the interface. Similarly, this method can be applied to the smart home field. For example, an intelligent voice assistant can understand the instructions of family members, such as "turn on the living room lights," and automatically execute the corresponding operations. Through the visual language model adaptation method provided in this embodiment, the model can be flexibly adjusted according to the task requirements in different scenarios, achieving efficient and accurate performance, thereby greatly improving the practicality and adaptability of the visual language model in practical applications.

[0094] In one embodiment, such as Figure 6 As shown, the pre-training unit 501 includes:

[0095] The visual encoding unit 601 is used to encode the image data into image features using a visual encoder.

[0096] The language encoding unit 602 is used to encode the text data into language features using a language encoder.

[0097] The first fusion unit 603 is used to fuse the image features and language features into the initial joint semantic features.

[0098] In one embodiment, the first fusion unit 603 includes:

[0099] The second fusion unit is used to fuse the initial joint semantic features according to the following formula:

[0100] f = VLM frozen (x,t)

[0101] Where f represents the initial joint semantic features, x represents image data, t represents text data, VLM represents a visual language model containing a visual encoder and a language encoder, and frozen represents parameter freezing.

[0102] This embodiment employs a frozen pre-training method to train the Visual Language Encoder (VLM), where the model parameters are frozen during the training phase. The VLM then extracts joint semantic features from the image and text, which are used as the base representation. Mainstream models can use architectures such as CLIP, BLIP, or Flamingo, and their main components include:

[0103] Visual encoder (ViT / ResNet): Input image x → Output image features;

[0104] Language encoder (Transformer): Input text t → Output language features;

[0105] Fusion module: For example, the BLIP architecture uses Cross-Attention to fuse text and images to obtain a cross-modal representation f, which is the initial joint semantic feature.

[0106] This approach ensures that the model retains key semantic connections between images and text when processing multimodal data, providing a solid foundation for subsequent scene adaptation. The advantage of frozen pre-training lies in its ability to flexibly adjust the model across different scenarios by introducing new task data and adaptation strategies without altering the pre-trained model parameters. This method not only improves the model's stability and reliability but also significantly reduces the time and cost of transferring the model between different tasks. Furthermore, using a Visual Language Model (VLM) as the basis for feature extraction fully leverages the model's powerful capabilities in visual and language understanding, further enhancing its performance in complex scenarios.

[0107] In one embodiment, the scene adaptation unit 502 includes:

[0108] The residual fusion unit is used to fuse the initial joint semantic features and scene-related parameters into scene features using a residual module.

[0109] f i =f+Adapter i (f)

[0110] Among them, f i Let f represent the initial joint semantic features, and Adapter represent the scene features. i (f) represents scene-related parameters, and i represents the i-th scene.

[0111] After extracting the initial joint semantic features, a lightweight Adapter module is inserted into the image-text fusion layer, and each scene has its own independent Adapter to inject a small number of scene-related parameters for different task scenarios. This embodiment uses the classic residual BottleneckAdapter:

[0112] f i =f+Adapter i (f), i∈{family, community, institution...}

[0113] In practical applications, the initial joint semantic features can be first down-projected, then activated using GELU / ReLU, followed by up-projection, and finally the scene features are obtained by summing the residuals. In this way, the features for each scene can be effectively integrated with the initial joint semantic features and relevant parameters specific to the scene, enabling the model to better understand and handle task requirements in different scenarios. Through this approach, the lightweight adapter not only enhances the model's scene adaptability but also maintains its lightweight nature, avoiding increased computational burden caused by introducing too many parameters.

[0114] In one embodiment, the dynamic routing unit 503 includes:

[0115] The weight prediction unit is used to predict scene weights using a lightweight multilayer perceptron and fuse them according to the following formula to obtain the fused features:

[0116]

[0117] Among them, f * Represents the fusion feature, α i This represents the scene weight of the i-th scene.

[0118] This embodiment uses a dynamic routing mechanism to dynamically determine which weighted combination to use. The structure of the dynamic routing mechanism can employ a lightweight MLP to predict the weights α = [α1, α2, ..., α].i The model then fuses the predicted weights to obtain the fused features. In practical applications, a lightweight MLP can consist of multiple fully connected layers, with ReLU or GELU activation functions used between layers. Finally, the weights for each scene are obtained through Softmax normalization. This mechanism allows the model to dynamically adjust its focus in different scenes, thereby enhancing its adaptability to new scenes.

[0119] Furthermore, the introduction of the dynamic routing mechanism enhances model interpretability. Since the weights for each scene are predicted using a lightweight multilayer perceptron, it becomes intuitive to understand the model's emphasis on each scene when handling different tasks. This not only helps in better understanding the model's decision-making process but also facilitates subsequent model optimization and debugging.

[0120] In practical applications, data augmentation strategies can be employed to improve the generalization ability and robustness of models. Specifically, when acquiring task data from multiple scenarios, random transformations and perturbations can be applied to image and text data, such as image rotation, scaling, cropping, and text synonym replacement, to increase the diversity and complexity of the data. In this way, the model can be exposed to more data variations during the training phase, thereby learning more robust feature representations and improving its adaptability to new data.

[0121] In one embodiment, the mapping decoding unit 504 includes:

[0122] The task decoding unit is used to obtain the task type and select the corresponding task decoder according to the task type to decode and output the mapped fused features.

[0123] Different task types may correspond to different output formats and requirements. For example, in image recognition tasks, the task decoder may need to map fused features to specific category labels; in text generation tasks, the task decoder may need to convert fused features into natural language text. Therefore, selecting the appropriate task decoder based on the task type is crucial.

[0124] In practice, a set of task decoders can be predefined, each optimized for one or more specific task types. When a new task needs to be processed, the most suitable decoder can be selected from the predefined set based on the task type. This approach not only improves the model's flexibility and scalability but also ensures that the model achieves optimal performance when handling different tasks.

[0125] Furthermore, to further improve model performance, a multi-task learning strategy can be employed. Specifically, during the training phase, the model can be trained simultaneously using data from multiple tasks. This allows the model to learn how to handle individual tasks while also learning the correlations and commonalities between different tasks. In this way, the model can better utilize the complementary information between different tasks, thereby improving its performance across all tasks.

[0126] In one embodiment, such as Figure 7 As shown, the task decoding unit includes:

[0127] The first setting unit 701 is used to employ a multilayer perceptron as the task decoder when the task type is a classification task.

[0128] The second setting unit 702 is used to use an LLM-style decoder as the task decoder when the task type is a generation task.

[0129] The third setting unit 703 is used to use the Transformer decoder as the task decoder when the task type is an object detection task.

[0130] In this embodiment, when mapping cross-modal semantics to task output, the appropriate task decoder can be selected based on the task type. For example, in classification tasks (such as scene selection), an MLP can be chosen as the task decoder. The decoding process can specifically include Transformer head output → Dropout → Linear → Softmax, that is, the fused feature f* is input into the MLP for classification prediction to obtain the classification result. In generation tasks (such as text generation), an LLM-style decoder can be chosen. The decoding process can perform autoregressive text generation based on the fused feature f*, with the input embedding from the Adapter output, and the decoding target being the answer or path. In object detection tasks (such as object detection), a Transformer decoder can be chosen. The decoding process can include position embedding → Transformer decoder → detection head output, thereby obtaining the position and label information of the target object. In this way, the visual language model adaptation method of this embodiment can flexibly adapt to different types of task output requirements, improving the model's versatility and practicality.

[0131] In summary, the visual language model adaptation method provided in this embodiment has the following advantages:

[0132] (1) Significantly improves adaptability to multiple scenarios. Unlike the traditional static fine-tuning method, this embodiment introduces a scenario adaptation mechanism to effectively improve the model's robustness in various environments. Experiments show that the same model can automatically adapt to the task without retraining in three environments: home, hospital, and elderly care community.

[0133] (2) Low training cost and fewer parameter updates. All lightweight parameters are inserted as Adapter modules, without affecting the original VLM structure, and can be flexibly loaded or unloaded. Compared with full fine-tuning, the amount of parameter updates is reduced by more than 90%.

[0134] (3) Strong structural compatibility, adaptable to a variety of large models. It is suitable for various visual language model structures such as BLIP, CLIP, MiniGPT, and LLaVA, and can be embedded in mobile deployment solutions.

[0135] (4) Deployment-friendly and inference-efficient. Scene judgment can rely solely on images, and only a portion of the Adapters are activated during the inference phase, eliminating the need for redundant computation of the entire model, making it suitable for deployment in edge scenarios.

[0136] Please see Figure 8 , Figure 8 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. The computer device is equipped with both wireless and wired communication capabilities.

[0137] The computer device includes a processor 802, a memory, and a network interface 805 connected via a system bus 801. The memory may include a non-volatile storage medium 803 and internal memory 804.

[0138] The non-volatile storage medium 803 may store an operating system 8031 ​​and a computer program 8032. When the computer program 8032 is executed, it causes the processor 802 to execute a visual language model adaptation method.

[0139] The processor 802 provides computing and control capabilities to support the operation of the entire computer device.

[0140] The internal memory 804 provides an environment for the execution of the computer program 8032 in the non-volatile storage medium 803. When the computer program 8032 is executed by the processor 802, the processor 802 can execute a visual language model adaptation method.

[0141] This network interface 805 is used for network communication with other devices. Those skilled in the art will understand that... Figure 8The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0142] The processor 802 is used to run a computer program 8032 stored in a memory to implement any embodiment of the above-described visual language model adaptation method.

[0143] It should be understood that, in this embodiment of the invention, the processor 802 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0144] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0145] Acquire task data from multiple scenarios, use the task data to pre-train a visual language model, and extract initial joint semantic features from the task data of multiple scenarios using the visual language model.

[0146] Multiple lightweight adapters are used to inject scene-related parameters into each scene, and the scene features are obtained by fusing them with the initial joint semantic features.

[0147] A dynamic routing mechanism is used to predict scene weights, and fused features are obtained based on the scene features and scene weights.

[0148] Based on the scenario, the fused features are mapped using cross-modal semantics, and the mapped fused features are decoded and output through a task decoder to construct a target adaptation model.

[0149] The target adaptation model is used to perform transfer adaptation on the visual language model of the specified scene.

[0150] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed, can perform the steps provided in the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0151] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0152] Acquire task data from multiple scenarios, use the task data to pre-train a visual language model, and extract initial joint semantic features from the task data of multiple scenarios using the visual language model.

[0153] Multiple lightweight adapters are used to inject scene-related parameters into each scene, and the scene features are obtained by fusing them with the initial joint semantic features.

[0154] A dynamic routing mechanism is used to predict scene weights, and fused features are obtained based on the scene features and scene weights.

[0155] Based on the scenario, the fused features are mapped using cross-modal semantics, and the mapped fused features are decoded and output through a task decoder to construct a target adaptation model.

[0156] The target adaptation model is used to perform transfer adaptation on the visual language model of the specified scene.

[0157] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0158] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0159] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0160] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A visual language model adaptation method, characterized in that, include: Acquire task data from multiple scenarios, use the task data to pre-train a visual language model, and extract initial joint semantic features from the task data of multiple scenarios using the visual language model. Multiple lightweight adapters are used to inject scene-related parameters into each scene, and the scene features are obtained by fusing them with the initial joint semantic features. A dynamic routing mechanism is used to predict scene weights, and fused features are obtained based on the scene features and scene weights. Based on the scenario, the fused features are mapped using cross-modal semantics, and the mapped fused features are decoded and output through a task decoder to construct a target adaptation model. The target adaptation model is used to perform transfer adaptation on the visual language model of the specified scene; The process involves using multiple lightweight adapters to inject scene-related parameters into each scene, and then fusing these parameters with the initial joint semantic features to obtain scene features, including: The residual module is used to fuse the initial joint semantic features and scene-related parameters into scene features: Among them, f i Let f represent the scene features, and let f represent the initial joint semantic features. This represents scene-related parameters, where i represents the i-th scene; The method of predicting scene weights using a dynamic routing mechanism and obtaining fused features based on the scene features and scene weights includes: Scene weights are predicted using a lightweight multilayer perceptron, and fused features are obtained according to the following formula: Among them, f * Represents the fusion feature, α i This represents the scene weight of the i-th scene.

2. The visual language model adaptation method according to claim 1, characterized in that, The task data includes image data and text data; the process of pre-training a visual language model using the task data and extracting initial joint semantic features from task data across multiple scenarios using the visual language model includes: The image data is encoded and output as image features using a visual encoder; The text data is encoded into language features using a language encoder. The image features and language features are fused into the initial joint semantic features.

3. The visual language model adaptation method according to claim 2, characterized in that, The process of fusing the image features and language features into the initial joint semantic features includes: The initial joint semantic features are obtained by fusing according to the following formula: Where f represents the initial joint semantic features, x represents image data, t represents text data, VLM represents a visual language model containing a visual encoder and a language encoder, and frozen represents parameter freezing.

4. The visual language model adaptation method according to claim 1, characterized in that, The process of mapping the fused features using cross-modal semantics based on the scenario, and decoding and outputting the mapped fused features through a task decoder, includes: Obtain the task type, and select the corresponding task decoder according to the task type to decode and output the mapped fused features.

5. The visual language model adaptation method according to claim 4, characterized in that, The step of obtaining the task type and selecting the corresponding task decoder based on the task type to decode and output the mapped fused features includes: When the task type is a classification task, a multilayer perceptron is used as the task decoder; When the task type is a generation task, an LLM-style decoder is used as the task decoder. When the task type is an object detection task, the Transformer decoder is used as the task decoder.

6. A visual language model adaptation device, characterized in that, include: The pre-training unit is used to acquire task data from multiple scenarios, pre-train the visual language model using the task data, and extract initial joint semantic features from the task data of multiple scenarios using the visual language model. The scene adaptation unit is used to inject scene-related parameters into each scene using multiple lightweight adapters, and to fuse the scene features with the initial joint semantic features. The dynamic routing unit is used to predict scene weights using a dynamic routing mechanism, and to obtain fused features based on the scene features and scene weights. The mapping and decoding unit is used to map the fused features based on the scenario using cross-modal semantics, and to decode and output the mapped fused features through the task decoder, thereby constructing a target adaptation model. The transfer adaptation unit is used to perform transfer adaptation on the visual language model of the specified scene using the target adaptation model. The scene adaptation unit 502 includes: The residual fusion unit is used to fuse the initial joint semantic features and scene-related parameters into scene features using a residual module. Among them, f i Let f represent the scene features, and let f represent the initial joint semantic features. This represents scene-related parameters, where i represents the i-th scene; The dynamic routing unit includes: The weight prediction unit is used to predict scene weights using a lightweight multilayer perceptron and fuse them according to the following formula to obtain the fused features: Among them, f * Represents the fusion feature, α i This represents the scene weight of the i-th scene.

7. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the visual language model adaptation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the visual language model adaptation method as described in any one of claims 1 to 5.

Images