Adaptive methods and electronic devices for visual language models during testing without training

Abstract

This application provides a method and electronic device for adaptive testing of visual language models without training. This embodiment introduces generalizable factors into the adaptive process of visual language models without training and innovatively uses prediction discrepancies to effectively quantify the influence of each generalizable factor. This allows the generalizable factors to effectively improve the performance of the visual language model in the adaptive process without training, thereby enhancing the performance of the visual language model in visual tasks such as image classification.

Description

Technical Field

[0001] This application relates to computer vision technology, and in particular to an adaptive method and electronic device for testing visual language models without training. Background Technology

[0002] The so-called training-free, test-time adaptive method for visual language models refers to applying a conventional or publicly available visual language model, such as the CLIP model, to a current visual task, such as image classification, without additional training or testing. This allows the model to adaptively improve its performance on the current visual task, improving its effectiveness in tasks like image classification. It's important to note that the "testing" in this training-free, test-time adaptive method for visual language models does not refer to testing the visual language model itself, but rather to its application on the visual task. Summary of the Invention

[0003] This application provides a training-free, test-time adaptive method and electronic device for visual language models, which introduces generalizable factors into the training-free, test-time adaptive process of visual language models to improve the performance of visual language models in visual tasks such as image classification.

[0004] This application provides a training-free, test-time adaptive method for visual language models, the method comprising:

[0005] For each generalizable factor matched by the current application, adjustments are made to the K enhanced images of the target image to adapt to the generalizable factor, resulting in K adjusted images corresponding to the generalizable factor. A first prediction set is obtained by predicting the K adjusted images using a visual language model. Based on the first prediction set and a second prediction set obtained by predicting the K enhanced images using the visual language model, a reference prediction difference corresponding to the generalizable factor is determined. The generalizable factor is the factor that affects the visual language model's performance of the visual task based on the target image under the current application.

[0006] Based on the reference prediction differences corresponding to each generalizable factor and the visual features of the K enhanced images, the target visual features of the target image are determined; the target visual features are used by the visual language model to perform the visual task based on the target image.

[0007] This application also provides an electronic device. The electronic device includes: a processor and a machine-readable storage medium;

[0008] The machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0009] The processor is used to execute machine-executable instructions to implement the steps of the disclosed method.

[0010] As can be seen from the above technical solutions, this embodiment introduces generalizable factors into the adaptive process of visual language model during training-free testing, and innovatively uses prediction differences to effectively quantify the influence of each generalizable factor, so that the generalizable factors can effectively affect the effect of the visual language model in the adaptive process during training-free testing, thereby improving the effect of visual language model in visual tasks such as image classification. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0012] Figure 1 A flowchart illustrating the method provided in this application embodiment;

[0013] Figure 2 Application flowcharts provided for embodiments of this application;

[0014] Figure 3 This is a structural diagram of the device provided in the embodiments of this application;

[0015] Figure 4 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0016] The adaptive method for testing a visual language model without training provided in this embodiment introduces a generalizable factor. A generalizable factor is a factor that influences the visual language model's performance of visual tasks based on the target image in the current application, such as shape factors and / or style factors. This embodiment introduces a generalizable factor into the adaptive method for testing a visual language model without training, which helps the visual language model adapt to scene shifts and improves the robustness of the visual language model in applications. To enable those skilled in the art to better understand the technical solutions provided in this application embodiment, and to make the above-mentioned objectives, features, and advantages of this application embodiment more apparent, the technical solutions in this application embodiment will be further described in detail below with reference to the accompanying drawings.

[0017] See Figure 1 , Figure 1 This is a flowchart illustrating a method provided in an embodiment of this application. This method is applicable to electronic devices such as host devices, but this embodiment is not specifically limited to it.

[0018] like Figure 1 As shown, the process may include the following steps:

[0019] Step 101: For each generalizable factor matched by the current application, perform adjustments to the K enhanced images of the target image to adapt to the generalizable factor, so as to obtain K adjusted images corresponding to the generalizable factor; use a visual language model to predict the K adjusted images to obtain a first prediction set; based on the first prediction set and the second prediction set obtained by predicting the K enhanced images using the visual language model, determine the reference prediction difference corresponding to the generalizable factor.

[0020] When a visual language model performs a visual task based on a target image in a current application, factors may affect its robustness and other performance. These factors can be referred to as generalizable factors matching the current application. For example, generalizable factors include shape factors and / or style factors. Different applications may introduce different generalizable factors, and even different visual tasks within different applications may involve different generalizable factors. This embodiment does not specifically limit these factors, but will determine matching generalizable factors based on the current application.

[0021] As an example, the target image can be enhanced using various conventional image enhancement methods, such as spatial domain methods, frequency domain methods, and AI algorithm enhancement, to obtain N enhanced images. Typically, N is greater than the number of enhanced images K required by the current application; therefore, this embodiment can select K enhanced images from these N enhanced images.

[0022] As an example, there are many ways to select K augmented images from the N augmented images. For instance, for each of the N augmented images, the visual encoder in the visual language model determines the visual features of the augmented image, and the text encoder in the visual language model determines the text features of the augmented image in C specified categories. Based on the visual features and the text features of the augmented image in the C specified categories, the prediction result of the visual language model for that augmented image is determined; the entropy corresponding to the prediction result of the visual language model for each augmented image is calculated; and based on the entropy corresponding to the prediction result of the visual language model for each augmented image, K augmented images are selected from the N augmented images. For example, K augmented images can be selected from the N augmented images in ascending order of entropy. The entropy corresponding to the prediction result of the visual language model for these K augmented images is less than the entropy corresponding to the prediction result of the remaining NK augmented images.

[0023] Here, based on the visual features of the enhanced image and the text features of the enhanced image in C specified categories, the prediction result of the visual language model for the enhanced image is determined. For example, based on the similarity between the text features of the enhanced image in each specified category and the visual features of the enhanced image, the prediction probability of the visual language model for the enhanced image in that specified category is determined. Based on the prediction probability of the visual language model for the enhanced image in C specified categories, the prediction result of the visual language model for the enhanced image is determined. For example, the prediction probability of the visual language model for the enhanced image in C specified categories is taken as the prediction result of the visual language model for the enhanced image.

[0024] In this embodiment, for each generalizable factor matched by the current application, the K enhanced images are adjusted to meet the requirements of the generalizable factor to obtain the K adjusted images corresponding to the generalizable factor. In this way, each enhanced image will have a corresponding adjustment object.

[0025] In this embodiment, the prediction of the K adjusted images using a visual language model is performed as follows: for each adjusted image, the visual encoder in the visual language model determines the visual features of the adjusted image, and the text encoder in the visual language model determines the text features of the adjusted image in C specified categories; based on the visual features and the text features of the adjusted image in the C specified categories, the prediction result of the visual language model for the adjusted image is obtained (the method of obtaining the prediction result is similar to the method of obtaining the prediction result of the enhanced image described above). Correspondingly, the first prediction set includes the prediction result of the visual language model for each of the K adjusted images.

[0026] In this embodiment, the second prediction set includes the prediction results of the visual language model for each of the K enhanced images; wherein, the prediction results of the visual language model for each enhanced image are as described above, and will not be repeated here.

[0027] In step 101, for each generalizable factor matched by the current application, adjustments are made to the K enhanced images of the target image to meet the requirements of the generalizable factor, resulting in K adjusted images corresponding to the generalizable factor. After predicting the K adjusted images using a visual language model to obtain a first prediction set, the prediction results of each enhanced image in the second prediction set are compared with the prediction results of the corresponding adjusted images in the first prediction set to obtain the prediction difference (denoted as the reference prediction difference) under the generalizable factor. Here, the reference prediction difference under the generalizable factor includes the difference between the prediction results of each enhanced image in the second prediction set and the prediction results of the corresponding adjusted images in the first prediction set.

[0028] Step 102: Based on the reference prediction differences corresponding to each generalizable factor and the visual features of the K enhanced images, determine the target visual features of the target image; the target visual features are used by the visual language model to perform the visual task based on the target image.

[0029] For example, in this embodiment, the target prediction difference can be determined based on the requirements of the visual language model for each generalizable factor when performing a visual task based on the target image, and the reference prediction difference corresponding to each generalizable factor.

[0030] Here, in the adaptive testing process for the visual language model without training, there are corresponding requirements for each generalizable factor matched by the current application. For example, the adaptive testing process for the visual language model without training is highly sensitive to at least one generalizable factor. Correspondingly, for the remaining generalizable factors matched by the current application, the adaptive testing process for the visual language model without training requires insensitivity to those remaining generalizable factors. In this embodiment, both being highly sensitive to at least one generalizable factor and being insensitive to the remaining generalizable factors in the adaptive testing process for the visual language model without training will affect the adaptive testing process for the visual language model without training.

[0031] Based on this, the requirements for any generalizable factor include: requiring sensitivity to the generalizable factor or requiring insensitivity to the generalizable factor.

[0032] For ease of description, the generalizable factors matched by the current application can be divided into a first set of factors and a second set of factors. Each generalizable factor in the first set of factors has the following requirement: it must be sensitive to that generalizable factor. Each generalizable factor in the second set of factors has the following requirement: it must be insensitive to that generalizable factor.

[0033] Under this premise, the above-mentioned requirements of the visual language model for each generalizable factor when performing a visual task based on the target image, and the reference prediction difference corresponding to each generalizable factor, determine the target prediction difference as follows: for each generalizable factor in the second factor set, the reference prediction difference corresponding to the visual language model when it is insensitive to the generalizable factor is determined based on the reference prediction difference corresponding to the generalizable factor; the target prediction difference is determined based on the reference prediction difference corresponding to each generalizable factor in the first factor set, and the reference prediction difference corresponding to the visual language model when it is insensitive to each generalizable factor in the second factor set.

[0034] Here, for each generalizable factor in the second factor set, the reference prediction difference corresponding to the visual language model's insensitivity to that generalizable factor is determined based on the reference prediction difference corresponding to that factor. For example, the negative value of the reference prediction difference corresponding to the generalizable factor is taken as the reference prediction difference corresponding to the visual language model's insensitivity to that generalizable factor. Taking style factors as an example, if the style factor has the following requirement: it must be insensitive to the style factor, then if the reference prediction difference corresponding to the style factor is... Then, when the visual language model is insensitive to style factors, the corresponding reference prediction difference is: .

[0035] Optionally, in this embodiment, the target prediction difference is determined based on the reference prediction difference corresponding to each generalizable factor in the first factor set and the reference prediction difference corresponding to the insensitivity of each generalizable factor in the second factor set. For example, the sum of the reference prediction difference corresponding to each generalizable factor in the first factor set and the reference prediction difference corresponding to the insensitivity of the visual language model to each generalizable factor in the second factor set is calculated, and the calculation result is determined as the target prediction difference.

[0036] After obtaining the target prediction difference, the target visual features of the target image can be determined by using the target prediction difference and the visual features of K enhanced images.

[0037] As one embodiment, this embodiment can perform a weighted calculation on the target prediction difference and the visual features of K enhanced images, and determine the target visual features of the target image based on the calculation result; the target prediction difference contains weight values ​​corresponding to the visual features of the K enhanced images. That is, this embodiment uses the target prediction difference as a set of weights to weight the visual features of the K enhanced images, rather than the conventional averaging, thus enhancing the high-perturbation prediction difference ( The characteristics of the Perturbed Prediction Difference (PERTED) index reduce low... The impact of indicator characteristics.

[0038] This concludes the process. Figure 1 The process is shown below.

[0039] pass Figure 1 As can be seen from the flowchart, this embodiment introduces generalizable factors into the adaptive process of visual language model during training-free testing, and innovatively uses prediction differences to effectively quantify the influence of each generalizable factor, so that the generalizable factors can effectively affect the effect of the adaptive process of visual language model during training-free testing, thereby improving the effect of visual language model in visual tasks such as image classification.

[0040] The following example illustrates... Figure 1The process is described as follows:

[0041] For an image (denoted as) Taking image classification as an example, if we consider the current application of visual language models... When performing image classification, both shape-sensitive (i.e., sensitive to shape) and style-insensitive (i.e., insensitive to style) factors need to be considered. Therefore, the generalizable factors for matching in the current application can be considered to include both shape and style factors. If... It is an image of a puppy.

[0042] First, this embodiment can be used for K enhanced images (denoted as K) The shape of the puppy in the image is perturbed to obtain the adjusted image corresponding to each enhanced image (the set of adjusted images is denoted as ). ).in, correspond And so on. The method for obtaining the K enhanced images is as described above, and will not be illustrated further here.

[0043] As an example, this embodiment can simply shuffle the image patches in each enhanced image to disrupt the shape of the puppy in the image while preserving local information.

[0044] Subsequently, this embodiment can utilize a visual language model to... The first prediction set (denoted as ) is obtained by predicting each adjusted image. The method for obtaining the first prediction set is, for example, for... For each adjusted image, the visual encoder in the visual language model determines its visual features, and the text encoder in the visual language model determines its text features in C specified categories (e.g., dog, cat, etc.). For each specified category, based on the similarity between the visual features and the text features of the adjusted image in that category, the visual language model determines its prediction probability for that adjusted image in that category. Based on the prediction probabilities of the adjusted image in the C specified categories, the visual language model determines its prediction result for that adjusted image. In other words, the prediction result of the visual language model for each adjusted image includes the prediction probabilities of the adjusted image in the C specified categories.

[0045] Then measure and The differences between them. Here, For visual language models The second prediction combination is obtained by predicting each enhanced image (the method for obtaining the second prediction combination is described above and will not be repeated here). Here, and The difference between them can be called the shape perturbation prediction difference. This also serves as a reference for predicting differences corresponding to shape factors. Similarly, The prediction result obtained by the visual language model for each augmented image will include the predicted probability of that augmented image in C specified categories.

[0046] As an example, The calculation may include: for each enhanced image, based on... In this enhanced image, the predicted probabilities of C specified categories are used to select a target category, for example, the category with the highest predicted probability. Then, the calculation is performed. The predicted probability of the enhanced image under this target category is... The difference between the predicted probability of the enhanced image and the adjusted image for that target category is denoted as the difference corresponding to the enhanced image. The differences corresponding to each enhanced image are denoted as... .Right now: This can be expressed by the following formula: ; where y represents the target category selected for each enhanced image.

[0047] Finally, the reference prediction difference corresponding to the shape factor was obtained. The greater the difference, the more the image was affected by shape factors during the test.

[0048] This embodiment can be used for K enhanced images (denoted as K) The styles are perturbed separately to obtain the adjusted images corresponding to each enhanced image (the set of adjusted images is denoted as ). ).in, correspond And so on.

[0049] As one embodiment, this embodiment can employ methods such as instance statistical permutation or Fourier-based style transfer to perturb the style of the image. Considering the efficiency of adaptation, this embodiment can choose efficient color transformation combined with tone adjustment to perturb the style of the image.

[0050] Subsequently, this embodiment can utilize a visual language model to... For each adjusted image, a prediction is performed to obtain a third prediction set (denoted as...). ).get The method is as described above and will not be repeated here.

[0051] Then measure and The differences between them. Here, As described above. Here, and The difference between them can be called the style perturbed prediction difference. This also serves as a reference for predicting differences corresponding to style factors. Similar to the above... The calculation method This can be expressed by the following formula: ; where y represents the target category selected for each enhanced image.

[0052] Finally, the reference prediction difference corresponding to the style factor was obtained. The greater the difference, the more the image was affected by style factors during the test.

[0053] In obtaining and Subsequently, this embodiment can be based on and Determine the target prediction difference (denoted as the perturbation prediction difference). )).

[0054] As described above, this embodiment is based on the current application of the visual language model. Image classification requires consideration of both shape-sensitive and style-insensitive factors. Based on this, the reference prediction differences corresponding to the aforementioned style factors are then used. In this embodiment, the reference prediction difference corresponding to style-insensitive factors can be obtained (i.e., That is, ultimately The calculation formula is:

[0055] .

[0056] get Subsequently, this embodiment can This is considered a generalizable factor deeply involved in the adaptive process of visual language models during training-free testing. Conventionally, in the context of... When performing image classification, the visual language model, such as the visual encoder of CLIP, is directly used to determine the image. The visual features of each of the K enhanced images are averaged, and the averaged visual features are used as the final visual features. The target visual features are used for image classification. In this embodiment, because a generalization factor is introduced, the target visual features can be used for image classification. As a set of weights, it comprises the visual features of each of the K enhanced images. Instead of the previous direct average, a weighted average is used, and the specific formula is as follows:

[0057] ; This indicates an enhanced image identifier. express The target visual features. This approach can enhance high-resolution visual features. The characteristics of the indicator reduce low The impact of indicator characteristics.

[0058] In obtaining After identifying the target visual features, a text encoder is used based on a visual language model such as CLIP. Definite Text features under C specified categories The process calculates the image-text similarity between the text features and the target visual features under each specified category, thus obtaining the classification probability under that specified category. Ultimately, the classification probability for each specified category is obtained, i.e. The classification probabilities under each specified category are also the visual language model's... The prediction results.

[0059] Meanwhile, visual language models such as CLIP maintain a dynamic visual cache sequence that stores high-quality visual features corresponding to C categories. and the corresponding category tags (Each category label has at least one visual feature). Initially, the visual cache sequence is empty.

[0060] For the above target visual features This embodiment can calculate the visual features of the target. Calculate another classification probability (denoted as ) based on the similarity between visual features in the visual cache sequence and each visual feature. As an example, This can be expressed by the following formula:

[0061] .

[0062] in Ultimately, a visual language model can be obtained that is tailored to... Classification probability Thus, a visual language model is obtained for... The final classification result (i.e.) (Target label).

[0063] As an example, this embodiment can target images, such as visual language models. The entropy corresponding to the prediction results is as described above. And the difference in the above-mentioned target predictions, namely This determines whether to update the visual cache sequence corresponding to the visual language model.

[0064] For example, compare the first difference with the reference difference corresponding to each reference visual feature under the reference label in the visual cache sequence; the reference label refers to the visual feature in the visual cache sequence that is related to the target image, for example... The target label is matched with sample labels such as the category labels mentioned above, and the target label is as described above; the first difference refers to the prediction result of the visual language model for the target image, such as the one described above. The difference between the corresponding entropy and the target prediction; the reference visual feature refers to the visual feature corresponding to the reference label in the visual cache sequence, such as the category label mentioned above, such as Q mentioned above.

[0065] If, based on the comparison results, it is determined that the target visual feature is superior to a reference visual feature under a reference label in the visual cache sequence, then the reference visual feature and its corresponding reference difference are deleted from the visual cache sequence, and the target visual feature and the first value are recorded in the visual cache sequence. This means that the dynamic visual caching process not only considers high confidence (entropy represents confidence) but also generalization factors, which further improves the model's performance.

[0066] For example, if the first difference is less than the reference difference corresponding to at least one reference visual feature under the reference label in the visual cache sequence, then the target visual feature is determined to be superior to a reference visual feature under the reference label in the visual cache sequence.

[0067] The method provided in the embodiments of this application has been described above. By applying the method provided in this embodiment, experiments have shown that the accuracy of ImageNet-A [9], ImageNet-V2

[10] , ImageNet-R

[11] and ImageNet-Sketch

[12] is significantly improved. Moreover, this method in this embodiment does not require heavy optimization and achieves better performance. Figure 2 An example diagram illustrating the application of the method provided in this embodiment is shown.

[0068] The apparatus provided in the embodiments of this application is described below:

[0069] See Figure 3 , Figure 3 A structural diagram of a device provided in an embodiment of this application. The device includes:

[0070] An adjustment unit is configured to perform adjustments on K enhanced images of the target image for each generalizable factor matched by the current application, adapting them to the requirements of that generalizable factor, to obtain K adjusted images corresponding to that generalizable factor; predict the K adjusted images using a visual language model to obtain a first prediction set; and determine a reference prediction difference corresponding to the generalizable factor based on the first prediction set and a second prediction set obtained by the visual language model predicting the K enhanced images; wherein the generalizable factor is a factor that affects the visual language model's performance of a visual task based on the target image under the current application;

[0071] The determining unit is used to determine the target visual features of the target image based on the reference prediction differences corresponding to each generalizable factor and the visual features of the K enhanced images; the target visual features are used by the visual language model to perform the visual task based on the target image.

[0072] Optionally, the generalizable factor is a shape factor; the target image and the K enhanced images all include the same target object;

[0073] The adjustment of the K enhanced images of the target image to be adapted to the generalizable factor includes: adjusting the shape of the target object in the K enhanced images of the target image respectively.

[0074] Optionally, the generalizable factor is a style factor;

[0075] The adjustment of the K enhanced images of the target image to be adapted to the generalizable factor includes: adjusting the style of the K enhanced images of the target image respectively.

[0076] Optionally, the step of using a visual language model to predict the K adjusted images includes:

[0077] For each adjusted image, the visual features of the adjusted image are determined using the visual encoder in the visual language model, and the text features of the adjusted image in C specified categories are determined using the text encoder in the visual language model; based on the visual features of the adjusted image and the text features of the adjusted image in C specified categories, the prediction result of the visual language model for the adjusted image is obtained.

[0078] The first prediction set contains the prediction results of the visual language model for each of the K adjusted images;

[0079] The second prediction set contains the prediction results of the visual language model for each of the K augmented images; wherein the prediction results of the visual language model for each augmented image are determined by the following steps: for each augmented image, the visual features of the augmented image are determined using the visual encoder in the visual language model, and the text features of the augmented image in C specified categories are determined using the text encoder in the visual language model; based on the visual features of the augmented image and the text features of the augmented image in C specified categories, the prediction results of the visual language model for the augmented image are obtained.

[0080] Optionally, determining the target visual features of the target image based on the reference prediction differences corresponding to each generalizable factor and the visual features of the K enhanced images includes:

[0081] Based on the requirements of the visual language model for each generalizable factor when performing a visual task on the target image, and the reference prediction difference corresponding to each generalizable factor, the target prediction difference is determined; wherein, the requirements for any generalizable factor include: requiring sensitivity to the generalizable factor or requiring insensitivity to the generalizable factor;

[0082] Based on the target prediction difference and the visual features of the K enhanced images, the target visual features of the target image are determined.

[0083] Optionally, determining the target prediction difference based on the requirements of the visual language model for each generalizable factor when performing a visual task based on the target image, and the reference prediction difference corresponding to each generalizable factor, includes:

[0084] For each generalizable factor in the second factor set, each generalizable factor in the second factor set has the following requirements: it is required that the generalizable factor is insensitive. Then, based on the reference prediction difference corresponding to the generalizable factor, the reference prediction difference when the generalizable factor is insensitive is determined.

[0085] The target prediction difference is determined based on the reference prediction difference corresponding to each generalizable factor in the first factor set and the reference prediction difference corresponding to the insensitivity of each generalizable factor in the second factor set; each generalizable factor in the first factor set has the following requirement: it is required to be sensitive to the generalizable factor.

[0086] Optionally, the generalizable factors include shape factors and style factors; wherein, when the visual language model performs a visual task based on the target image, the requirement for the shape factor is that it is sensitive to the shape factor, and the requirement for the style factor is that it is insensitive to the style factor.

[0087] The determination of the target prediction difference, based on the requirements of the visual language model for each generalizable factor when performing a visual task on the target image, and the reference prediction difference corresponding to each generalizable factor, includes:

[0088] Based on the reference prediction differences corresponding to style factors, determine the reference prediction differences when style factors are insensitive;

[0089] The target prediction difference is determined based on the reference prediction difference corresponding to the shape factor and the reference prediction difference when the style factor is insensitive.

[0090] Optionally, determining the target visual features of the target image using the target prediction difference and the visual features of the K enhanced images includes:

[0091] The target prediction difference and the visual features of the K enhanced images are weighted and calculated, and the target visual features of the target image are determined based on the calculation result; the target prediction difference contains weight values ​​corresponding to the visual features of the K enhanced images.

[0092] Optionally, the determining unit further includes:

[0093] Based on the target visual features and the text encoder in the visual language model, the text features of the target image in C specified categories are determined, and the prediction results of the visual language model for the target image are obtained.

[0094] Based on the entropy corresponding to the prediction result of the visual language model for the target image and the target prediction difference, it is determined whether to update the visual cache sequence corresponding to the visual language model.

[0095] Optionally, determining whether to update the visual cache sequence corresponding to the visual language model based on the entropy corresponding to the prediction result of the visual language model for the target image and the target prediction difference includes:

[0096] The first difference is compared with the reference difference corresponding to each reference visual feature under the reference label in the visual cache sequence. The reference label refers to the sample label in the visual cache sequence that matches the target label of the target image. The target label is determined based on the prediction result of the visual language model for the target image, the target visual features, the visual features under each sample label in the visual cache sequence, and each sample label. The first difference is the difference between the entropy corresponding to the prediction result of the visual language model for the target image and the difference between the target prediction. The reference visual feature refers to the visual feature corresponding to the reference label in the visual cache sequence.

[0097] If, based on the comparison results, it is determined that the target visual feature is superior to a reference visual feature under a reference label in the visual cache sequence, then the reference visual feature and its corresponding reference difference are deleted from the visual cache sequence, and the target visual feature and the first value are recorded in the visual cache sequence.

[0098] This application also provides embodiments that... Figure 3 The hardware structure of the device shown. See also Figure 4 , Figure 4 This is a structural diagram of an electronic device provided in an embodiment of this application. Figure 4 As shown, the hardware structure may include: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the method disclosed in the above example of this application.

[0099] Based on the same application concept as the above method, this application embodiment also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the method disclosed in the above examples of this application.

[0100] For example, the aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For instance, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0101] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A visual language model training-free test-time adaptation method, characterized in that, The method comprises: For each generalizable factor matched with the current application, performing adjustment required by the generalizable factor on K enhanced images of the target image respectively to obtain K adjustment images corresponding to the generalizable factor; obtaining a first prediction set by predicting the K adjustment images using a visual language model; determining a reference prediction difference corresponding to the generalizable factor based on the first prediction set and a second prediction set obtained by predicting the K enhanced images using the visual language model; the generalizable factor is a factor that affects the visual language model to perform a visual task based on the target image under the current application; Determining a target visual feature of the target image based on the reference prediction difference corresponding to each generalizable factor and visual features of the K enhanced images; the target visual feature is used for the visual language model to perform the visual task based on the target image.

2. The method of claim 1, wherein, The generalizable factor is a shape factor; the target image and the K enhanced images all include a same target object; The adjustment required by the generalizable factor on the K enhanced images of the target image respectively comprises adjusting shapes of the target object in the K enhanced images of the target image respectively.

3. The method of claim 1, wherein, The generalizable factor is a style factor; The adjustment required by the generalizable factor on the K enhanced images of the target image respectively comprises performing style adjustment on the K enhanced images of the target image respectively.

4. The method according to any one of claims 1 to 3, characterized in that, The prediction using the visual language model on the K adjustment images comprises: For each adjustment image, determining a visual feature of the adjustment image using a visual encoder in the visual language model, and determining text features of the adjustment image in C specified categories using a text encoder in the visual language model; obtaining a prediction result of the visual language model for the adjustment image based on the visual feature of the adjustment image and the text features of the adjustment image in the C specified categories; The first prediction set comprises the prediction result of the visual language model for each adjustment image in the K adjustment images; The second prediction set comprises the prediction result of the visual language model for each enhanced image in the K enhanced images; wherein the prediction result of the visual language model for each enhanced image is determined by the following steps: for each enhanced image, determining a visual feature of the enhanced image using a visual encoder in the visual language model, and determining text features of the enhanced image in C specified categories using a text encoder in the visual language model; obtaining a prediction result of the visual language model for the enhanced image based on the visual feature of the enhanced image and the text features of the enhanced image in the C specified categories.

5. The method of claim 1, wherein, The determination of the target visual feature of the target image based on the reference prediction difference corresponding to each generalizable factor and the visual features of the K enhanced images comprises: determine the target prediction difference according to requirements of each generalizable factor on the visual language model when the visual language model performs the visual task based on the target image and reference prediction differences corresponding to each generalizable factor, wherein the requirement of any generalizable factor comprises: requiring sensitivity to the generalizable factor or requiring insensitivity to the generalizable factor; determine the target visual feature of the target image based on the target prediction difference and visual features of the K enhanced images.

6. The method of claim 5, wherein, The determining the target prediction difference according to requirements of each generalizable factor on the visual language model when the visual language model performs the visual task based on the target image and reference prediction differences corresponding to each generalizable factor comprises: for each generalizable factor in the second factor set, if the generalizable factor is required to be insensitive, the reference prediction difference corresponding to the insensitivity of the visual language model to the generalizable factor is determined according to the reference prediction difference corresponding to the generalizable factor; determine the target prediction difference according to the reference prediction difference corresponding to each generalizable factor in the first factor set and the reference prediction difference corresponding to the insensitivity of the visual language model to each generalizable factor in the second factor set, wherein each generalizable factor in the first factor set is required to be sensitive.

7. The method according to claim 5 or 6, characterized in that, The generalizable factors comprise a shape factor and a style factor, wherein the requirement of the visual language model to the shape factor when the visual language model performs the visual task based on the target image is to be sensitive to the shape factor, and the requirement of the visual language model to the style factor when the visual language model performs the visual task based on the target image is to be insensitive to the style factor. The determining the target prediction difference according to requirements of each generalizable factor on the visual language model when the visual language model performs the visual task based on the target image and reference prediction differences corresponding to each generalizable factor comprises: determine the reference prediction difference corresponding to the insensitivity of the visual language model to the style factor based on the reference prediction difference corresponding to the style factor; determine the target prediction difference according to the reference prediction difference corresponding to the shape factor and the reference prediction difference corresponding to the insensitivity of the visual language model to the style factor.

8. The method of claim 5, wherein, The determining the target visual feature of the target image based on the target prediction difference and the visual features of the K enhanced images comprises: perform a weighted operation on the target prediction difference and the visual features of the K enhanced images, and determine the target visual feature of the target image according to an operation result, wherein the target prediction difference comprises a weight value corresponding to the visual features of the K enhanced images.

9. The method of claim 5, wherein, The method further comprises: determine text features of the target image in C specified categories based on the target visual feature and a text encoder in the visual language model, and obtain a prediction result of the visual language model for the target image. determine whether to update the visual cache sequence corresponding to the visual language model based on the entropy corresponding to the prediction result of the visual language model for the target image and the target prediction difference; wherein the visual cache sequence is used to store high-quality visual features of C categories and corresponding category labels, and a reference difference value corresponding to each visual feature.

10. The method of claim 9, wherein, The determination whether to update the visual cache sequence corresponding to the visual language model based on the entropy corresponding to the prediction result of the visual language model for the target image and the target prediction difference comprises: comparing the first difference value with a reference difference value corresponding to each reference visual feature under a reference label in the visual cache sequence, the reference label being a sample label in the visual cache sequence matching a target label of the target image, the target label being determined based on the prediction result of the visual language model for the target image, and based on the target visual feature, visual features under each sample label in the visual cache sequence, and each sample label; the first difference value being a difference value between the entropy corresponding to the prediction result of the visual language model for the target image and the target prediction difference; the reference visual feature being a visual feature corresponding to the reference label in the visual cache sequence; if it is determined based on the comparison result that the target visual feature is better than a reference visual feature under the reference label in the visual cache sequence, deleting the reference visual feature and the corresponding reference difference value from the visual cache sequence, and recording the target visual feature and the first difference value to the visual cache sequence.

11. An electronic device, comprising: The electronic device comprises a processor and a machine readable storage medium; The machine readable storage medium stores machine executable instructions capable of being executed by the processor; The processor is configured to execute the machine executable instructions to implement the method of any one of claims 1-10.

Images