Efficient fine tuning method and device for pre-training model, server and storage medium

Abstract

The invention discloses an efficient fine tuning method and device for a pre-training model, a server and a storage medium, and the method comprises the steps: obtaining a first text feature and an image prompt feature, the first text feature being generated by coding according to preset entity category text information and learnable text prompt information, the image prompt features are generated by encoding entity category image blocks corresponding to the first input image and preset learnable visual prompt parameters; performing mask processing on the image prompt feature to generate a first mask image feature, and processing the first mask image feature and the first text feature by using a preset multi-mode mask reconstruction network to generate a reconstructed image; and calculating loss values of the first input image and the reconstructed image, and finely adjusting parameters of the pre-training model according to the loss values. Through the method and the device, the problems of low accuracy and weak migration ability in downstream tasks of a parameter fine tuning method of a pre-training visual language model based on prompt in the related technology are solved.

Description

Technical Field

[0001] The present application relates to the technical field of pre-trained visual language models, and more specifically, to an efficient fine-tuning method, device, server, and storage medium for a pre-trained model. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the invention recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] With the development of big data technology, pre-trained visual language models for self-supervised learning have achieved remarkable performance in recognition tasks. Pre-trained visual language models use large-scale image-text pairs as input during the training process to achieve results comparable to fully supervised learning methods; at the same time, pre-trained visual language models have good transferability to new datasets and can be easily extended to other tasks, such as object detection, semantic segmentation, and 3D perception; pre-trained visual language models have excellent generalization capabilities, but when pre-trained visual language models are applied to downstream tasks with limited data, they are prone to catastrophic forgetting and lose the generalization capabilities obtained in the pre-training stage due to the need to support unbearable training resources.

[0004] In the related art, the parameter fine-tuning method based on the pre-trained visual language model can solve the problem of catastrophic forgetting of the above-mentioned visual language model. In the related art, the parameter fine-tuning method of the pre-trained visual language model includes an adapter-based method and a prompt-based method, among which the prompt-based method shows better performance in the knowledge transfer of downstream tasks. In the related art, the prompt-based method focuses on the design of single-mode prompt learning, ignores the mutual connection and difference between text and visual features, and reduces the fine-tuning accuracy of the pre-trained visual language model. In addition, the physical mechanism adopted by the prompt-based fine-tuning method cannot effectively improve the migration ability of the pre-trained visual language model in downstream tasks.

[0005] There is still a lack of better technical solutions to the problems of low accuracy and weak migration ability in downstream tasks in the parameter fine-tuning method of the prompt-based pre-trained visual language model in related technologies. Summary of the invention

[0006] The embodiments of the present application provide an efficient fine-tuning method, device, server and storage medium for a pre-trained model, so as to at least solve the problem of low accuracy and weak migration ability in downstream tasks in the parameter fine-tuning method of the prompt-based pre-trained visual language model in the related art.

[0007] According to one aspect of the present application, there is also provided an efficient fine-tuning method for a pre-trained model, the method comprising: obtaining a first text feature and an image prompt feature, wherein the first text feature is generated by encoding based on preset entity category text information and learnable text prompt information, and the image prompt feature is generated by encoding entity category image blocks corresponding to the first input image and preset learnable visual prompt parameters; performing mask processing on the image prompt feature to generate a first mask image feature, and using a preset multimodal mask reconstruction network to process the first mask image feature and the first text feature to generate a reconstructed image, wherein the multimodal mask reconstruction network uses a cross-attention mechanism to fuse the unimodal feature information processed by the self-attention mechanism; calculating the loss value of the first input image and the reconstructed image, and fine-tuning the parameters of the pre-trained model according to the loss value.

[0008] According to another aspect of the present application, there is also provided an efficient fine-tuning device for a pre-trained model, comprising:

[0009] An acquisition module, used to acquire a first text feature and an image prompt feature, wherein the first text feature is generated by encoding based on preset entity category text information and learnable text prompt information, and the image prompt feature is generated by encoding entity category image blocks corresponding to the first input image and preset learnable visual prompt parameters;

[0010] A processing module, configured to perform mask processing on the image prompt feature to generate a first mask image feature, and use a preset multimodal mask reconstruction network to process the first mask image feature and the first text feature to generate a reconstructed image, wherein the multimodal mask reconstruction network uses a cross-attention mechanism to fuse the unimodal feature information processed by the self-attention mechanism;

[0011] A fine-tuning module is used to calculate the loss value of the first input image and the reconstructed image, and fine-tune the parameters of the pre-trained model according to the loss value.

[0012] According to another aspect of the present application, a server is provided, comprising: a processor; and a memory storing a program, wherein the program comprises instructions, and when the instructions are executed by the processor, the processor executes the above method.

[0013] According to another aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to enable the computer to execute the steps of the above method.

[0014] According to another aspect of the present application, a computer program product is also provided. The computer program product includes a computer program. When the computer program is executed by a processor, the above method steps are implemented.

[0015] Compared with the related art, the efficient fine-tuning method, device, server and storage medium of the pre-trained model provided in the embodiment of the present application adopt the method of obtaining a first text feature and an image prompt feature, wherein the first text feature is generated by encoding according to preset entity category text information and learnable text prompt information, and the image prompt feature is generated by encoding the entity category image block corresponding to the first input image and the preset learnable visual prompt parameter; the image prompt feature is masked to generate a first mask image feature, and the first mask image feature and the first text feature are processed by a preset multimodal mask reconstruction network to generate a reconstructed image, and the multimodal mask reconstruction network adopts a cross-attention mechanism to utilize The unimodal feature information processed by the self-attention mechanism is fused; the loss value of the first input image and the reconstructed image is calculated, and the parameters of the pre-trained model are fine-tuned according to the loss value. Text prompts and attention-based visual prompts are adopted, and the interaction of text and visual prompts is implicitly learned using a multimodal mask signal reconstruction method to achieve multimodal prompt interaction, narrow the solution space, and effectively prompt fine-tuning the information of interest or specific information of the task of concern, thereby solving the problem of low accuracy and weak migration ability in downstream tasks of the parameter fine-tuning method of the prompt-based pre-trained visual language model in the related art, and achieving the beneficial effect of improving the fine-tuning effect of the pre-trained visual language model and reducing the amount of calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0017] Figure 1 A flowchart of an efficient fine-tuning method for a pre-trained model according to an embodiment of the present application is shown;

[0018] Figure 2 A framework diagram of multimodal prompt fine-tuning according to a preferred embodiment of the present application is shown;

[0019] Figure 3 A schematic diagram of a visual prompt module according to a preferred embodiment of the present application is shown;

[0020] Figure 4 A schematic diagram of an interaction module of visual language prompts according to a preferred embodiment of the present application is shown;

[0021] Figure 5 A structural block diagram of an efficient fine-tuning device for a pre-trained model according to an embodiment of the present application is shown;

[0022] Figure 6 A structural block diagram of an exemplary server that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0023] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0024] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0025] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0026] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0027] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0028] In this embodiment, an efficient fine-tuning method for a pre-trained model is provided. Figure 1 A flowchart of an efficient fine-tuning method for a pre-trained model according to an embodiment of the present application is shown, referring to Figure 1 The method flow includes:

[0029] Step S102, obtaining a first text feature and an image prompt feature, wherein the first text feature is generated by encoding based on preset entity category text information and learnable text prompt information, and the image prompt feature is generated by encoding entity category image blocks corresponding to the first input image and preset learnable visual prompt parameters.

[0030] The efficient fine-tuning method of the pre-trained model of the embodiment of the present application can be executed in a terminal, a server, a computer or a similar computing device, that is, the execution subject of the embodiment of the present application is a terminal, a server, a computer or a similar computing device; in this embodiment, the correspondence and difference between text and visual features are considered to modify the text or image input to the pre-trained model by multimodal (for example: text, vision) learnable parameters to learn task-specific knowledge to complete adaptation. Therefore, before fine-tuning the prompt of the pre-trained model, it is necessary to obtain the text prompt learnable parameters and learnable visual parameters corresponding to the input image, that is, to obtain the first text feature and the image prompt feature; in this embodiment, the first text feature is a target ( For example, a certain animal) corresponds to the entity category text (for example, white pet, white dog or white pet) and the learnable text prompt information (for example, Pomeranian, Samoyed, Great Pyrenees, cat); at the same time, in this embodiment, the image prompt feature includes the entity category image block representing the corresponding target as the set entity category in the first input image (for example, according to the mouth image block corresponding to the first input image, it is indicated that the target is a Great Pyrenees) and the set learnable visual prompt parameters (orange is used to prompt that the target corresponding to the first input image is a Great Pyrenees, and blue is used to prompt that the target corresponding to the first input image is a Samoyed); in this embodiment, for the first input image, in order to better learn the downstream task features, the entity category image block is fused with the learnable visual prompt parameters.

[0031] Step S104, masking the image prompt feature to generate a first mask image feature, and using a preset multimodal mask reconstruction network to process the first mask image feature and the first text feature to generate a reconstructed image, wherein the multimodal mask reconstruction network uses a cross-attention mechanism to fuse the unimodal feature information processed by the self-attention mechanism.

[0032] In this step, after obtaining the first text feature and the image prompt feature, in order to ensure that the first text feature is input unchanged into the multimodal mask reconstruction network, the image prompt feature is first masked to achieve implicit multimodal prompt interaction from a global perspective; at the same time, after the first mask image feature and the first text feature are fused and reconstructed to obtain the reconstructed image, only the multimodal mask signal is used in the pre-training part to reconstruct the part to achieve multimodal prompt interaction, thereby reducing inference time and memory usage.

[0033] Step S106, calculating the loss value of the first input image and the reconstructed image, and fine-tuning the parameters of the pre-trained model according to the loss value.

[0034] In this embodiment, the reconstructed image and the originally input first input image are subjected to loss calculation to optimize the multimodal prompt.

[0035] Through the above steps S102 to S106, a first text feature and an image prompt feature are obtained, wherein the first text feature is generated by encoding the preset entity category text information and the learnable text prompt information; the image prompt feature is generated by encoding the entity category image block corresponding to the first input image and the preset learnable visual prompt parameter; the image prompt feature is masked to generate a first mask image feature, and the first mask image feature and the first text feature are processed by a preset multimodal mask reconstruction network to generate a reconstructed image, wherein the multimodal mask reconstruction network uses a cross attention mechanism to process the single mask image feature and the first text feature after being processed by the self-attention mechanism. The invention adopts a method of multimodal mask signal reconstruction to implicitly learn the interaction between text and visual cues, realize multimodal prompt interaction, narrow the solution space, and effectively prompt fine-tune the information of interest or specific information of the task concerned, solves the problem of low accuracy and weak migration ability in downstream tasks of the parameter fine-tuning method of the prompt-based pre-trained visual language model in the related technology, and achieves the beneficial effect of improving the fine-tuning effect of the pre-trained visual language model and reducing the amount of calculation.

[0036] It should be noted that, in this embodiment, the interaction of multimodal information is completed by reconstructing the image, and the interaction of restoring the mask image based on the text information is implemented, so that the interaction of multimodal prompts can be implicitly performed from a global perspective; at the same time, in order to reduce the inference time and memory usage, only the multimodal mask signal reconstruction part is used in the pre-training part to realize the multimodal prompt interaction.

[0037] It should be noted that the efficient fine-tuning method of the pre-trained model of this embodiment can support a variety of existing and even future tasks, and has a wide range of applications; at the same time, the embodiment of the present application also reveals the potential physical mechanism of the prompt-based fine-tuning method of the pre-trained visual language model, and accordingly proposes a new migration framework, which adapts to multimodality by prompting from the perspective of attention, thereby narrowing the solution space, which helps to prompt fine-tuning to focus on the interesting / specific information of the task; furthermore, the embodiment of the present application only requires a small number of training parameters and does not require a large amount of training resources.

[0038] In some of the embodiments, the entity category text information includes object category text, and the learnable text prompt information includes learnable text prompts. Encoding is performed according to preset entity category text information and learnable text prompt information to generate a first text feature, which is achieved through the following steps: using a preset text encoder, encoding the object category text and the learnable text prompt to generate a first text feature.

[0039] In some embodiments, encoding the entity category image blocks corresponding to the first input image and the preset learnable visual cue parameters to generate image cue features is achieved by the following steps:

[0040] Step 21 : segment the first input image to obtain a plurality of original image blocks, wherein the original image blocks are used to represent at least one object category.

[0041] In this embodiment, after the first input image is segmented, a plurality of original image blocks are generated (refer to Figure 3 In the example, image blocks 1 to 9 represented by dotted dots are included, and each original image block represents an object category. For example, the categories represented by image blocks 1 and 2 represented by dotted dots are the legs of the target in the first input image. In this embodiment, Figure 3 The image block represented by the dotted dots in is a schematic representation, and the numbers in the image block do not limit or represent the image features actually corresponding to the first input image.

[0042] Step 22: Determine a plurality of visual cue image blocks in the learnable visual cue parameters, wherein the visual cue image blocks are used to represent a visual cue.

[0043] In this embodiment, the learnable visual cue parameters are image patches with visual cues (e.g., by color), for example: Figure 3 In the figure, the image blocks are represented by vertical lines, and the number of visual cues of the visual cue image blocks indicates the different object categories of the corresponding cues, for example: Figure 3 The image block 3 represented by the vertical line in the middle represents the visual cue that the target in the first input image is a “great white bear dog” (in practice, the corresponding visual cue color is set to orange).

[0044] Step 23, performing spatial information conversion processing on the original image block and the visual cue image block respectively to obtain a spatial feature map and a visual cue spatial feature map.

[0045] In this embodiment, in order to better restore the spatial information, the original image block (reference Figure 3 The dotted dots in the image block 1 to the image block 9) and the visual cue image block (reference Figure 3 The image blocks 1 to 4 represented by the vertical lines in the figure are respectively subjected to spatial information conversion processing to be converted into a spatial feature map of the spatial information structure (reference Figure 3 The spatial information structure under the "learnable parameters" in Figure 3 The spatial information structure under the “original image block” in Figure 1).

[0046] Step 24, using the visual cue module of the preset visual encoder, respectively detects class-related areas in the spatial feature map and the visual cue spatial feature map, and fuses the detected target spatial feature map and the target cue spatial feature map to generate an image cue feature, wherein the image cue feature represents an object category label of a visual cue. spatial information structure).

[0047] In some of the optional embodiments, the visual encoder is an encoder comprising N layers of Transformer layers, and the visual cue module is inserted into the i-th Transformer layer, i<N, i and N are positive integers.

[0048] By segmenting the first input image in the above steps, a plurality of original image blocks are obtained, wherein the original image blocks are used to represent at least one object category; in the learnable visual cue parameters, a plurality of visual cue image blocks are determined, wherein the visual cue image blocks are used to represent a visual cue; the original image blocks and the visual cue image blocks are respectively subjected to spatial information conversion processing to obtain a spatial feature map and a visual cue spatial feature map; using the visual cue module of the preset visual encoder, class-related areas are respectively detected in the spatial feature map and the visual cue spatial feature map, and the detected target spatial feature map and the target cue spatial feature map are fused to generate image cue features, wherein the image cue features represent an object category label of a visual cue, thereby achieving better learning of downstream task features for the first input image, and providing a data basis for multimodal cue interaction.

[0049] In order to better restore spatial information, in some embodiments, the original image blocks and the visual cue image blocks are respectively subjected to spatial information conversion processing, and the obtained spatial feature maps and visual cue spatial feature maps are realized by the following steps: raster data conversion is performed on the original image block sequence and the visual cue image sequence respectively to generate spatial feature maps and visual cue spatial feature maps, wherein the original image block sequence includes multiple original image blocks, and the visual cue image sequence includes multiple visual cue image blocks.

[0050] In this embodiment, the original image block (reference Figure 3 The dotted dots in the image block 1 to the image block 9) and the visual cue image block (reference Figure 3 The image blocks 1 to 4 represented by the vertical lines in the figure are respectively subjected to spatial information conversion processing, for example, the image block sequence or the visual cue image sequence is run-length encoded to convert it into a spatial feature map of the spatial information structure (refer to Figure 3 The spatial information structure under the "learnable parameters" in Figure 3 The spatial information structure under the “original image block” in Figure 1).

[0051] To ensure that the first text feature is input unchanged into the multimodal mask reconstruction network, in some embodiments, the image prompt feature is masked to generate a first mask image feature, which is achieved through the following steps: randomly masking the image prompt feature to generate a first mask image feature.

[0052] In some embodiments, processing the first mask image feature and the first text feature using a preset multimodal mask reconstruction network to generate a reconstructed image includes the following steps:

[0053] Step 31, using a self-attention mechanism to process the first mask image feature and the first text feature respectively to obtain a second mask image feature and a second text feature.

[0054] Step 32: Use a cross-attention mechanism to fuse the second mask image feature and the second text feature to generate an alternative fusion feature.

[0055] Step 33, input the candidate fusion features into the preset Transformer visual decoder for decoding to generate a reconstructed image.

[0056] In this embodiment, reference Figure 4 , use the self-attention mechanism to process the mask image feature F and the text feature G separately, then use the cross-attention mechanism to interact the information of the two modalities, and repeat the above process N f ; After that, the fused feature F CUse the visual decoder as input and process it, and repeat N times. d times, and finally obtain the reconstructed image.

[0057] Figure 2 A framework diagram of multimodal prompt fine-tuning according to a preferred embodiment of the present application is shown. Figure 3 shows a schematic diagram of a visual prompt module according to a preferred embodiment of the present application, Figure 4 A schematic diagram of an interactive module of visual language prompts according to a preferred embodiment of the present application is shown below. Figures 2 to 4 , the efficient fine-tuning method of the pre-trained model of the embodiment of the present application is further described as follows:

[0058] The efficient fine-tuning method of the pre-trained model of the embodiment of the present application designs a multimodal prompt from the perspective of reducing the solution space, including a traditional text prompt and an attention-based visual prompt module, and uses a multimodal mask signal reconstruction method to implicitly learn the interaction between text and visual prompts; Figure 2 As shown, it mainly includes two parts: a multimodal prompt module and a multimodal mask signal reconstruction module for interaction.

[0059] Multimodal prompt section

[0060] refer to Figure 2 ,The multimodal prompt part includes learnable parameters of text prompt and an ,attention-based visual prompt module for narrowing the solution space.

[0061] refer to Figure 2 For text prompts, learnable parameters are used to input object category text C i With learnable text prompts V i Encode using a text encoder.

[0062] refer to Figure 3 For the visual cue module, for the first input image, in order to better learn the downstream task features, before the image is input into the encoder, it is divided into multiple original image blocks and fused with some learnable visual parameters (corresponding to multiple visual cue image blocks); after inputting the visual encoder, a visual cue module that does not require training is used to narrow the solution space so that the visual encoder focuses on class-related areas; specifically, a visual cue module is inserted into the i-th layer encoder of the visual encoder. In order to better restore spatial information, the image features generated by the Transformer architecture are first converted back to the spatial information structure, and then processed by the visual cue module.

[0063] Multimodal information mask signal reconstruction section for interaction.

[0064] refer to Figure 4,In order to better complete the interaction of multimodal information, a multimodal mask signal reconstruction module is used, including:

[0065] The image features encoded by the image encoder are randomly masked, and the text features obtained from the text encoder are kept unchanged and input into the multimodal mask reconstruction module; in the fusion module of the multimodal mask reconstruction module, the self-attention mechanism is first used to process the mask image features F and the text features G separately, and then the cross-attention mechanism is used to interact with the information of the two modalities. Repeat the above process N f After that, the fused feature F C Use the visual decoder as input and process it, and repeat N times. d Finally, the reconstructed image is obtained, and then the loss of the reconstructed image and the original first input image is calculated to optimize the multimodal prompt.

[0066] This embodiment also provides an efficient fine-tuning device for a pre-trained model, which is used to implement the above embodiments and preferred implementations, and will not be repeated here. As used below, the terms "module", "unit", "subunit", etc. can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0067] Figure 5 The structure block diagram of the efficient fine-tuning device of the pre-training model according to the embodiment of the present application is shown as follows: Figure 5 As shown, the device includes: an acquisition module 51, a processing module 52 and a fine-tuning module 53, wherein:

[0068] An acquisition module 51 is used to acquire a first text feature and an image prompt feature, wherein the first text feature is generated by encoding the preset entity category text information and the learnable text prompt information, and the image prompt feature is generated by encoding the entity category image block corresponding to the first input image and the preset learnable visual prompt parameter;

[0069] The processing module 52 is coupled to the acquisition module 51 and is used to perform mask processing on the image prompt feature to generate a first mask image feature, and use a preset multimodal mask reconstruction network to process the first mask image feature and the first text feature to generate a reconstructed image, wherein the multimodal mask reconstruction network uses a cross-attention mechanism to fuse the unimodal feature information processed by the self-attention mechanism;

[0070] The fine-tuning module 53 is coupled to the processing module 52 and is used to calculate the loss value of the first input image and the reconstructed image, and fine-tune the parameters of the pre-trained model according to the loss value.

[0071] Through the efficient fine-tuning device of the pre-trained model of the embodiment of the present application, a first text feature and an image prompt feature are obtained, the first text feature is generated by encoding according to preset entity category text information and learnable text prompt information; the image prompt feature is generated by encoding the entity category image block corresponding to the first input image and the preset learnable visual prompt parameter; the image prompt feature is masked to generate a first mask image feature, and the first mask image feature and the first text feature are processed by a preset multimodal mask reconstruction network to generate a reconstructed image, and the multimodal mask reconstruction network uses a cross-attention mechanism to reconstruct the single mask image feature and the first text feature after being processed by the self-attention mechanism. The invention adopts a method of multimodal mask signal reconstruction to implicitly learn the interaction between text and visual cues, realize multimodal prompt interaction, narrow the solution space, and effectively prompt fine-tune the information of interest or specific information of the task concerned, solves the problem of low accuracy and weak migration ability in downstream tasks of the parameter fine-tuning method of the prompt-based pre-trained visual language model in the related technology, and achieves the beneficial effect of improving the fine-tuning effect of the pre-trained visual language model and reducing the amount of calculation.

[0072] The device is used to implement the functions of the method in the above-mentioned embodiment. Each module in the device corresponds to each step in the method. Those that have been explained in the method will not be repeated here.

[0073] Optionally, the entity category text information includes object category text, and the learnable text prompt information includes learnable text prompts. The acquisition module 51 is also used to use a preset text encoder to encode the object category text and the learnable text prompt to generate a first text feature.

[0074] Optionally, the acquisition module 51 further includes:

[0075] The segmentation unit is used to segment the first input image to obtain a plurality of original image blocks, wherein the original image blocks are used to represent at least one object category.

[0076] The determination unit is used to determine a plurality of visual cue image blocks in the learnable visual cue parameters, wherein the visual cue image blocks are used to represent a visual cue.

[0077] The conversion unit is used to perform spatial information conversion processing on the original image block and the visual cue image block respectively to obtain a spatial feature map and a visual cue spatial feature map.

[0078] The detection unit is used to use the visual cue module of the preset visual encoder to detect class-related areas in the spatial feature map and the visual cue spatial feature map respectively, and fuse the detected target spatial feature map and the target cue spatial feature map to generate an image cue feature, wherein the image cue feature represents an object category label of a visual cue.

[0079] Optionally, the visual encoder is an encoder comprising N layers of Transformer layers, and the visual cue module is inserted into the i-th Transformer layer, i<N, i and N are positive integers.

[0080] Optionally, the conversion unit is also used to perform raster data conversion on the original image block sequence and the visual cue image sequence respectively to generate a spatial feature map and a visual cue spatial feature map, wherein the original image block sequence includes multiple original image blocks and the visual cue image sequence includes multiple visual cue image blocks.

[0081] Optionally, the processing module 52 is further configured to randomly mask the image prompt feature to generate a first mask image feature.

[0082] Optionally, the processing module 52 further includes:

[0083] The processing unit is used to use a self-attention mechanism to process the first mask image feature and the first text feature respectively to obtain a second mask image feature and a second text feature.

[0084] The fusion unit is coupled to the processing unit and is used to fuse the second mask image feature and the second text feature using a cross attention mechanism to generate an alternative fusion feature.

[0085] The decoding unit is coupled to the fusion unit and is used to input the candidate fusion features into a preset Transformer visual decoder for decoding to generate a reconstructed image.

[0086] The exemplary embodiment of the present disclosure also provides a server, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and when the computer program is executed by the at least one processor, it is used to cause the electronic device to perform the method according to the embodiment of the present disclosure.

[0087] The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method according to the embodiments of the present disclosure.

[0088] The exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform the method according to the embodiments of the present disclosure.

[0089] refer to Figure 6 , a structural block diagram of a server 400 that can be used as the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0090] like Figure 6 As shown, the server 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0091] A plurality of components in the server 600 are connected to the I / O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 may be any type of device capable of inputting information to the server 600, and the input unit 606 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 607 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 608 may include, but is not limited to, a disk, an optical disk. The communication unit 609 allows the server 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0092] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above. For example, in some embodiments, the efficient fine-tuning method of the aforementioned pre-trained model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the server 600 via the ROM 602 and / or the communication unit 609. In some embodiments, the computing unit 601 may be configured to perform the efficient fine-tuning method of the pre-trained model by any other appropriate means (e.g., by means of firmware).

[0093] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0094] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0095] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0096] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or an LCD (liquid crystal display)) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0097] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0098] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0099] The above embodiments solve the problems of low accuracy and weak migration ability in downstream tasks in the parameter fine-tuning method of the prompt-based pre-trained visual language model, and achieve the beneficial effects of improving the fine-tuning effect of the pre-trained visual language model and reducing the amount of calculation.

[0100] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. An efficient fine-tuning method for a pre-trained model, characterized in that: include: Obtaining a first text feature and an image prompt feature, wherein the first text feature is generated by encoding according to preset entity category text information and learnable text prompt information, and the image prompt feature is generated by encoding entity category image blocks corresponding to the first input image and preset learnable visual prompt parameters; Masking the image prompt feature to generate a first mask image feature, and using a preset multimodal mask reconstruction network to process the first mask image feature and the first text feature to generate a reconstructed image, wherein the multimodal mask reconstruction network uses a cross-attention mechanism to fuse the unimodal feature information processed by the self-attention mechanism; Calculate the loss value of the first input image and the reconstructed image, and fine-tune the parameters of the pre-trained model according to the loss value.

2. The method according to claim 1, characterized in that The entity category text information includes object category text, and the learnable text prompt information includes learnable text prompts. Encoding is performed according to preset entity category text information and learnable text prompt information to generate the first text feature, including: using a preset text encoder to encode the object category text and the learnable text prompt to generate the first text feature.

3. The method according to claim 1, characterized in that Encoding the entity category image blocks corresponding to the first input image and the preset learnable visual cue parameters to generate the image cue features includes: Segmenting the first input image to obtain a plurality of original image blocks, wherein the original image blocks are used to represent at least one object category; Determining a plurality of visual cue image blocks in the learnable visual cue parameters, wherein the visual cue image blocks are used to represent a visual cue; The original image block and the visual cue image block are respectively subjected to spatial information conversion processing to obtain a spatial feature map and a visual cue spatial feature map; Using the visual cue module of the preset visual encoder, class-related areas are detected in the spatial feature map and the visual cue spatial feature map respectively, and the detected target spatial feature map and target cue spatial feature map are fused to generate the image cue feature, wherein the image cue feature represents an object category label of a visual cue.

4. The method according to claim 3, characterized in that The visual encoder is an encoder comprising N layers of Transformer layers, and the visual cue module is inserted into the i-th Transformer layer, i<N, i and N are positive integers.

5. The method according to claim 3, characterized in that: The original image block and the visual cue image block are respectively subjected to spatial information conversion processing to obtain a spatial feature map and a visual cue spatial feature map, including: raster data conversion is performed on the original image block sequence and the visual cue image sequence to generate the spatial feature map and the visual cue spatial feature map, wherein the original image block sequence includes a plurality of the original image blocks, and the visual cue image sequence includes a plurality of the visual cue image blocks.

6. The method according to claim 1, characterized in that Performing mask processing on the image prompt feature to generate a first mask image feature includes: performing random masking on the image prompt feature to generate the first mask image feature.

7. The method according to claim 1, characterized in that Processing the first mask image feature and the first text feature using a preset multimodal mask reconstruction network to generate a reconstructed image includes: Using a self-attention mechanism to process the first mask image feature and the first text feature respectively, to obtain a second mask image feature and a second text feature; Using a cross attention mechanism to fuse the second mask image feature and the second text feature to generate an alternative fused feature; The candidate fusion features are input into a preset Transformer visual decoder for decoding to generate the reconstructed image.

8. An efficient fine-tuning device for a pre-trained model, characterized in that: include: An acquisition module, used to acquire a first text feature and an image prompt feature, wherein the first text feature is generated by encoding based on preset entity category text information and learnable text prompt information, and the image prompt feature is generated by encoding entity category image blocks corresponding to the first input image and preset learnable visual prompt parameters; A processing module, configured to perform mask processing on the image prompt feature to generate a first mask image feature, and use a preset multimodal mask reconstruction network to process the first mask image feature and the first text feature to generate a reconstructed image, wherein the multimodal mask reconstruction network uses a cross-attention mechanism to fuse the unimodal feature information processed by the self-attention mechanism; A fine-tuning module is used to calculate the loss value of the first input image and the reconstructed image, and fine-tune the parameters of the pre-trained model according to the loss value.

9. A server, characterized in that: include: processor; and a memory for storing a program, wherein The program comprises instructions which, when executed by the processor, cause the processor to perform the method steps according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: in, The computer instructions are used to make the computer execute the method steps according to any one of claims 1-7.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method steps according to any one of claims 1 to 7 are implemented.

Images