Cross-domain small sample visual feature enhancement method based on language knowledge guidance

Abstract

The invention discloses a cross-domain small sample visual feature enhancement method, device and equipment based on language knowledge guidance. The method comprises the following steps: generating an independent task sample set containing a support set and a query set; respectively inputting an independent task sample set into a visual basic model and an image attribute generator by utilizing a cross-domain small sample visual feature enhancement algorithm, extracting visual features, image-level attribute language description and domain-level attribute language description, splicing the image-level attribute language description and the domain-level attribute language description, and inputting the spliced image-level attribute language description and domain-level attribute language description into an encoder; the obtained image attribute language description features are embedded into visual features, and the visual features are refocused by using a residual cross attention layer to obtain refined features; according to the method, effective adaptation of the pre-training visual model in a cross-domain small sample scene is realized, and the problem of insufficient generalization ability caused by significant inter-domain distribution difference and scarcity of labeled samples in a traditional method is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a cross-domain small sample visual feature enhancement method and device based on language knowledge guidance and equipment. BACKGROUND

[0002] In the field of artificial intelligence, few-shot learning (FSL) refers to a kind of learning method that enables the model to quickly adapt to a new task under the condition of only a small amount of labeled samples. This technology has wide application prospects in the field of computer vision, especially in image recognition, image classification and image segmentation tasks. With the development of deep learning technology, a large number of studies have been carried out around few-shot learning, aiming to improve the generalization ability of the model in the data scarce scene through pre-training model and transfer learning strategy. In the prior art, researchers have proposed a variety of solutions to the problem of cross-domain few-shot learning. For example, prior art one proposes a meta-learning method based on style adversarial training, which uses style disturbance to improve the robustness of the model to the style changes of the target domain; prior art two proposes a cross-domain segmentation method that combines the prompt mechanism and graph reasoning network, which realizes the transfer from natural image domain to extreme field by introducing task-level fine-tuning strategy. These methods generally rely on the general visual features extracted by the pre-training model of the source domain large-scale dataset, and fine-tune with a small amount of labeled samples in the target domain to improve the performance in the new task. However, due to the significant difference in visual feature distribution between the source domain and the target domain, from low-level color, texture, resolution difference, to high-level semantic composition, object structure, visual style and background environment changes, etc., the cross-domain distribution shift problem is caused, which further limits the generalization effect of the model in the target domain. Especially in the case of a large number of novel or complex visual structures in the target domain, the pre-training model often fails to effectively align the target domain features, resulting in a decrease in classification or segmentation accuracy. In addition, most of the existing methods focus on the adaptation of visual-level features, lack effective semantic transfer mechanism, and fail to fully exploit and utilize the potential semantic consistency information between the source domain and the target domain, thereby further weakening the adaptability of the model. SUMMARY

[0003] The main purpose of the present application is to provide a cross-domain small sample visual feature enhancement method, device and equipment based on language knowledge guidance, aiming to construct a mechanism that can effectively identify and transfer semantic information under the condition of significant visual distribution difference, so as to realize the efficient generalization of the model in the new field.

[0004] To achieve the above object, the application provides a cross-domain small sample visual feature enhancement method based on language knowledge guidance, comprising: using a meta-learning paradigm to perform task-level reorganization and sampling on training image data of a target domain to generate an independent task sample set containing a support set and a query set; using a pre-constructed cross-domain small sample visual feature enhancement algorithm based on language knowledge guidance, inputting the independent task sample set into a visual basic model and an image attribute generator respectively, extracting visual features, image-level attribute language descriptions and domain-level attribute language descriptions, inputting the image-level attribute language descriptions and the domain-level attribute language descriptions into an encoder after splicing to obtain image attribute language description features; embedding the image attribute language description features into the visual features, using a residual cross-attention layer to re-focus the visual features to obtain refined features; wherein the image-level attribute is the appearance feature of an object in an image, and the attribute language description includes features composed of global styles, background environments and semantics of a data set; using the support set to fine-tune the visual basic model, the residual cross-attention layer and the task-specific head, and evaluating the performance of the model in the cross-domain small sample visual task through the query set to obtain the trained cross-domain small sample visual feature enhancement algorithm based on language knowledge guidance; using the trained cross-domain small sample visual feature enhancement algorithm based on language knowledge guidance to process input image data and output target images with adaptive visual feature enhancement.

[0005] Optionally, the task-level reorganization and sampling of the training image data of the target domain based on the meta-learning paradigm to generate the independent task sample set containing the support set and the query set comprises: based on the meta-learning paradigm, constructing a plurality of independent tasks in each training batch; in each independent task, randomly sampling N categories from the target domain data set, and sampling K images of each category to form a candidate set; dividing the candidate set into a support set and a query set, wherein the support set contains S samples of each category, the query set contains Q samples of each category, and S+Q=K.

[0006] Optionally, the visual basic model is a source domain pre-training model for extracting visual features of an image, wherein the visual features are high-level visual feature representations; the image attribute generator comprises an image description model and a large language model, wherein the image description model is used to generate image-level attribute language descriptions, and the large language model is used to generate domain-level attribute language descriptions.

[0007] Optionally, the domain-level attribute language description is obtained by combining a large language model with an artificially designed prompt word to extract the overall attribute of the target domain in the independent task sample set; and the image-level attribute language description is obtained by extracting the attribute of a single image through an image description model.

[0008] Optionally, the encoder is a cross-modal pre-training encoder configured to encode the concatenated image-level attribute language description and the domain-level attribute language description into fixed-dimension image attribute language description features.

[0009] Optionally, the residual cross-attention layer maps the visual features and the image attribute language description features into a shared latent space through a learnable projection matrix, wherein the projection matrix comprises a query vector projection matrix for the visual features, a key vector projection matrix and a value vector projection matrix for the image attribute language description features.

[0010] Optionally, the refocusing of the visual features by the residual cross-attention layer comprises: mapping the visual features into query vectors through the query vector projection matrix, and mapping the image attribute language description features into key vectors and value vectors through the key vector projection matrix and the value vector projection matrix, so as to map the visual features and the image attribute language description features into the shared latent space; calculating scaled dot products of the query vectors and the key vectors to obtain attention scores, and applying a normalization exponential function to the attention scores to obtain attention weights; performing weighted summation on the value vectors by using the attention weights to obtain optimized visual features; mapping the original visual features into the shared latent space through a learnable matrix, and establishing a residual connection between the original visual features and the optimized visual features to obtain refined features after refocusing.

[0011] Optionally, the small sample fine-tuning of the visual base model, the residual cross-attention layer and the task-specific head by using the support set comprises: inputting the refined features into the task-specific head for prediction, and calculating a loss function value according to a prediction result and a true label of the sample; and performing end-to-end backpropagation optimization on parameters of the visual base model, the residual cross-attention layer and the task-specific head by using the loss function value; wherein the task-specific head comprises a classifier or a segmenter, wherein the classifier is used for a cross-domain small sample image classification task in a cross-domain small sample visual task, and the segmenter is used for a cross-domain small sample image segmentation task in the cross-domain small sample visual task.

[0012] To achieve the above object, the application further provides a cross-domain small sample visual feature enhancement device based on language knowledge guidance, comprising: a sample set generation module, configured to perform task-level reorganization and sampling on training image data of a target field by using a meta-learning paradigm, and generate an independent task sample set containing a support set and a query set; a model training module, configured to input the independent task sample set into a visual basic model and an image attribute generator by using a pre-constructed cross-domain small sample visual feature enhancement algorithm based on language knowledge guidance, extract visual features, image-level attribute language descriptions and domain-level attribute language descriptions, input the image-level attribute language descriptions and the domain-level attribute language descriptions after splicing into an encoder to obtain image attribute language description features, embed the image attribute language description features into the visual features, re-focus the visual features by using a residual cross-attention layer to obtain refined features; wherein the image-level attribute is an appearance feature of an object in an image, and the attribute language description comprises features composed of global styles, background environments and semantics according to a data set; the visual basic model, the residual cross-attention layer and a task-specific head are fine-tuned by using the support set, and the performance of the model in a cross-domain small sample visual task is evaluated by using the query set to obtain the trained cross-domain small sample visual feature enhancement algorithm based on language knowledge guidance; and an execution module, configured to process input image data by using the trained cross-domain small sample visual feature enhancement algorithm based on language knowledge guidance, and output a target image with adaptive visual feature enhancement.

[0013] To achieve the above object, the application further provides an electronic device, comprising: at least one processor, a memory and an input-output unit; wherein the memory is configured to store a computer program, and the processor is configured to call the computer program stored in the memory to execute the cross-domain small sample visual feature enhancement method based on language knowledge guidance provided by any of the preceding embodiments.

[0014] The embodiment of the application provides a cross-domain small sample visual feature enhancement method, device and equipment based on language knowledge guidance. The training image data of a target field is subjected to task-level reorganization and sampling by adopting a meta-learning paradigm, and an independent task sample set containing a support set and a query set is generated. The independent task sample set is input into a visual basic model and an image attribute generator by using a pre-constructed cross-domain small sample visual feature enhancement algorithm based on language knowledge guidance, visual features, image-level attribute language descriptions and domain-level attribute language descriptions are extracted, the image-level attribute language descriptions and the domain-level attribute language descriptions are spliced and input into an encoder, and image attribute language description features are obtained. The image attribute language description features are embedded in the visual features, the visual features are refocused by using a residual cross attention layer, and refined features are obtained. The application can fully utilize language prior knowledge to enhance the visual features, effectively improve the semantic alignment capability and generalization performance of a model in a cross-domain small sample visual task, and has superior migration effect and robustness in multiple tasks. The image-level attribute is the appearance feature of an object in an image, and the attribute language description includes features formed according to the global style, background environment and semantics of a data set. The visual basic model, the residual cross attention layer and the task-specific head are fine-tuned by using the support set, and the performance of the model in the cross-domain small sample visual task is evaluated by using the query set, so that the cross-domain small sample visual feature enhancement algorithm based on language knowledge guidance after training is obtained. The input image data is processed by using the cross-domain small sample visual feature enhancement algorithm based on language knowledge guidance after training, and a target image with adaptive visual feature enhancement is output, so that the cross-domain small sample visual feature enhancement method based on language knowledge guidance is solved, the effective adaptation of a pre-trained visual model in a cross-domain small sample scene is realized, and the problem of insufficient generalization capability caused by significant distribution difference between domains and lack of labeled samples in a traditional method is overcome. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 A flowchart is provided for an embodiment of the cross-domain small sample visual feature enhancement method based on language knowledge guidance of the application.

[0016] The implementation, functional features and advantages of the application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0017] It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application.

[0018] To overcome the shortcomings of the prior art, the present application provides a cross-domain small sample visual feature enhancement method based on language knowledge guidance. The method realizes adaptive feature adjustment of a specific target image on the basis of a pre-trained visual feature model by introducing language description of image attributes. The present application respectively uses a pre-trained image description model and a large language model (LLM) to extract image-level attributes (such as appearance features of objects in the image) and domain-level attributes (such as global semantic information such as the overall style of the data set and the background environment), thereby constructing a language representation system covering local and global attributes. On this basis, the present application designs a simple residual cross-attention mechanism for seamlessly embedding the above language attribute information into the visual feature representation to enhance the model's ability to migrate high-level semantic features and effectively compensate for the limitations of pure visual clues in cross-domain adaptation. In addition, the present application has good task-independent compatibility and can be directly integrated into the existing pre-trained visual feature model framework without substantial modification of the original model structure. The present application provides a new technical path for cross-modal small sample learning, and the core innovation point is the language prior guided visual feature enhancement strategy, which breaks through the semantic representation bottleneck faced by traditional methods in cross-domain scenarios and can significantly improve the generalization performance of the model in cross-modal small sample tasks in extreme fields.

[0019] Reference Figure 1 The cross-domain small sample visual feature enhancement method based on language knowledge guidance provided by the first embodiment of the present application can include: S10, using a meta-learning paradigm to perform task-level reorganization and sampling on training image data of a target domain to generate an independent task sample set containing a support set and a query set; In an embodiment of the present application, the process of using a meta-learning paradigm to perform task-level reorganization and sampling on training image data of a target domain to generate an independent task sample set containing a support set and a query set can include the following: S101, based on the meta-learning paradigm, constructing multiple independent tasks in each training batch; S102, in each independent task, randomly sampling N categories from the target domain data set, and sampling K images for each category to form a candidate set; S103, dividing the candidate set into a support set and a query set, wherein the support set contains S samples of each category, the query set contains Q samples of each category, and S+Q=K.

[0020] Specifically, for the training image data provided in the target domain, the processor employs a few-shot task construction strategy based on the meta-learning paradigm. This involves task-level reorganization and sampling of the data to generate an independent set of task samples that meets the requirements of few-shot learning. Formally, this is achieved by constructing [a set of tasks] in each training batch. An independent task Specifically, First, random samples are taken from the dataset. Categories Sampling for each category The images constitute the candidate set: (1) in, , indicating category The sample set is divided into a support set and a query set: (2) (3) in, , ,satisfy Supports collection Includes each category Zhang samples are used for rapid adaptive training of the model; query set Then it includes each category Ten samples are used to evaluate the model's generalization performance. Overlapping categories are allowed between different tasks during task construction.

[0021] S20. Using a pre-built cross-domain few-sample visual feature enhancement algorithm guided by language knowledge, the independent task sample set is input into the visual base model and the image attribute generator respectively to extract visual features, image-level attribute language descriptions, and domain-level attribute language descriptions. The image-level attribute language descriptions and domain-level attribute language descriptions are concatenated and input into the encoder to obtain image attribute language description features. The image attribute language description features are embedded into the visual features, and the visual features are refocused using a residual cross-attention layer to obtain refined features. Among them, image-level attributes are the appearance features of objects in the image, and the attribute language description includes features based on the global style, background environment and semantic composition of the dataset; Furthermore, the support set is used to fine-tune the visual base model, residual cross-attention layer and task-specific head with few samples, and the performance of the model in cross-domain few-sample visual tasks is evaluated by query set, resulting in a trained cross-domain few-sample visual feature enhancement algorithm based on language knowledge. In an embodiment of the present application, the visual base model is a source domain pre-training model, which is used to extract visual features of an image, wherein the visual features are high-level visual feature representations; Specifically, the processor inputs the image samples obtained by preprocessing in step 1 into the visual base model and the image attribute generator respectively, extracts corresponding visual feature representations and language description information of image attributes, and constructs a multi-modal semantic expression. Specifically, for the few-shot task one picture , the source domain pre-training model is used to extract visual features: (4) wherein, represents the image visual features, represents the parameters of the source domain pre-training model.

[0022] The image attribute generator includes an image description model and a large language model, wherein the image description model is used to generate image-level attribute language descriptions, and the large language model is used to generate domain-level attribute language descriptions.

[0023] In an embodiment of the present application, the domain-level attribute language description is obtained by combining a large language model with an artificially designed prompt word to extract the overall attribute of the target domain in the independent task sample set; In this embodiment, the domain-level attribute language description associated with the picture is obtained by means of the large language model LLM: (6) wherein, is an artificially designed prompt word, mainly involving relevant attributes known in the target domain, which is input into the large language model LLM to obtain the domain-level attribute language description .

[0024] The image-level attribute language description is obtained by extracting attributes of a single image by the image description model.

[0025] In this embodiment, the image-level attribute language description associated with the picture is obtained by means of the BLIP image-to-text model, and the expression of this process can be: (5) wherein, the encoder is a cross-modal pre-training encoder, which is used to encode the concatenated image-level attribute language description and domain-level attribute language description into fixed-dimensional image attribute language description features.

[0026] ​In an embodiment of the present application, the residual cross-attention layer maps the visual feature and the image attribute language description feature into a shared latent space through a learnable projection matrix, wherein the projection matrix includes a query vector projection matrix for the visual feature, a key vector projection matrix and a value vector projection matrix for the image attribute language description feature.

[0027] Specifically, the process of refocusing the visual feature through the residual cross-attention layer to obtain the refined feature can include the following: The visual feature is mapped into a query vector through the query vector projection matrix, and the image attribute language description feature is mapped into a key vector and a value vector through the key vector projection matrix and the value vector projection matrix, respectively, so as to map the visual feature and the image attribute language description feature into a shared latent space; The scaled dot product of the query vector and the key vector is calculated to obtain an attention score, and a normalized exponential function is applied to the attention score to obtain an attention weight; The value vector is weighted and summed using the attention weight to obtain an optimized visual feature, the original visual feature is mapped into the shared latent space through a learnable matrix, and a residual connection is established between the original visual feature and the optimized visual feature to obtain a refocused refined feature.

[0028] Specifically, the image attribute language description feature extracted in step S102 is embedded into the visual feature, and the residual cross-attention layer (Residual Cross-Attention Layer) is used to refocus the visual feature. Specifically, with the help of a learnable projection matrix , the image attribute language description feature and the visual feature are jointly mapped into a shared latent space with a dimension of , and the specific mapping process is as follows: (8) wherein, represents a query vector extracted from the visual feature, and and are a key vector and a value vector obtained from the image attribute language description feature .

[0029] Subsequently, the scaled dot product of the query vector and the key vector is calculated to obtain an attention score : (9) wherein, as a scaling factor, plays a role in stabilizing the gradient in the fine-tuning process. By applying a normalized exponential function to , the attention weight : (10) using the attention weight to weight-sum the value vector , so as to optimize the visual feature. In order to preserve the original visual information, the is mapped to the same latent space through a learnable matrix , and a residual connection is established: (11) wherein, the refined feature after refocusing is

[0030] S30, using the trained language knowledge guided cross-domain small sample visual feature enhancement algorithm to process the input image data, and outputting the target image with adaptive visual feature enhancement.

[0031] Specifically, the process of using the support set to fine-tune the visual base model, the residual cross-attention layer and the task-specific head can include the following: inputting the refined feature into the task-specific head for prediction, and calculating a loss function value according to the prediction result and the true label of the sample; using the loss function value to perform end-to-end back propagation optimization on the parameters of the visual base model, the residual cross-attention layer and the task-specific head; wherein, the task-specific head includes a classifier or a segmenter, wherein the classifier is used for the cross-domain small sample image classification task in the cross-domain small sample visual task, and the segmenter is used for the cross-domain small sample image segmentation task in the cross-domain small sample visual task.

[0032] In this embodiment, the present application uses the support set to fine-tune the small sample. For each support image , after steps 2 and 3, the refined feature is obtained, which is then input into the task-specific head (such as a classifier or a segmenter) for prediction, and finally the loss is calculated according to the prediction and the true label , and the calculation process is formalized as follows: (12) wherein, the task-specific head is represented by , and the learnable parameters are represented by ; and the task-specific loss function (such as cross-entropy loss, etc.) is represented by .

[0033] Finally, the present application uses the total loss to optimize the visual base model​ , residual cross-attention layer , task-specific head , end-to-end training, after the training is completed, the present application is in the order of step S10, step S20, step S30 for testing. Specifically, after step S10 to obtain the query image , after the completion of step S20 multi-modal feature extraction and step S30 feature re-aggregation, the corresponding refined features , input to the task-specific head , prediction, and finally by comparing the predicted score with the true label to evaluate the performance of the model on small sample tasks .

[0034] It should be noted that the present application shows significant effect in the small sample classification and segmentation tasks in the extreme field. In terms of small sample classification task, the test results based on the medical field dermoscopy image dataset ISIC, X-ray image dataset Chest, remote sensing field large scene image dataset EuroSAT, and crop disease image dataset CropDisease in the agricultural field show that: under the condition that only one labeled sample is provided for each class, the present application method can realize the average classification accuracy of 55.50%; under the condition that 5 labeled samples are provided for each class, the average classification accuracy is improved to 68.76%. Compared with the existing cross-domain small sample image classification method based on style adversarial training, the performance level is improved by 4.72% and 2.84%, respectively.

[0035] In terms of small sample segmentation task, the present application is evaluated on a plurality of representative cross-domain datasets, including the medical field dermoscopy image dataset ISIC, X-ray image dataset Chest, satellite remote sensing field image dataset DeepGlobe, and small sample segmentation dataset FSS-1000 containing 1000 categories. Test results show that: under the condition that only one labeled sample is provided for each class, the average segmentation accuracy of the present application method reaches 74.6%; under the condition that 5 labeled samples are provided for each class, the average segmentation accuracy is further improved to 77.7%. Compared with the existing method based on SAM prompt inference network, it is improved by 4.04% and 3.19%, which fully verifies the generalization ability and robustness of the present application in complex cross-domain environment.

[0036] In summary, the present application has the following beneficial effects: by constructing a cross-modal feature enhancement framework based on language guidance, the pre-trained visual model is effectively adapted in the cross-domain small sample scene, and the problem of insufficient generalization ability caused by significant distribution difference between domains and lack of labeled samples in traditional methods is overcome. The attribute generator and feature refocusing module proposed in the present application can fully utilize the language prior knowledge to enhance the visual features, effectively improve the semantic alignment ability and generalization performance of the model in the cross-domain small sample visual task, and show superior transfer effect and robustness in multiple tasks.

[0037] On the basis of the method embodiment, the present application further provides a cross-domain small sample visual feature enhancement device based on language knowledge guidance, which is used to solve the same technical problem as the method embodiment. The device can include a sample set generation module, a model training module and an execution module: the sample set generation module is used to perform task-level reorganization and sampling on the training image data of the target domain using the meta-learning paradigm, to generate an independent task sample set containing a support set and a query set; the model training module is used to input the independent task sample set into a visual base model and an image attribute generator respectively using a pre-constructed cross-domain small sample visual feature enhancement algorithm based on language knowledge guidance, to extract visual features, image-level attribute language descriptions and domain-level attribute language descriptions, to input the image-level attribute language descriptions and the domain-level attribute language descriptions after splicing into an encoder to obtain image attribute language description features; to embed the image attribute language description features into the visual features, and to use a residual cross-attention layer to refocus the visual features to obtain refined features; wherein the image-level attribute is the appearance feature of the object in the image, and the attribute language description includes features composed of global style, background environment and semantics according to the data set; the support set is used to fine-tune the visual base model, the residual cross-attention layer and the task-specific head, and the performance of the model in the cross-domain small sample visual task is evaluated through the query set to obtain the trained cross-domain small sample visual feature enhancement algorithm based on language knowledge guidance; the execution module is used to process the input image data using the trained cross-domain small sample visual feature enhancement algorithm based on language knowledge guidance, and to output the target image with adaptive visual feature enhancement.

[0038] On the basis of the above embodiments, the present application further provides an electronic device, which includes at least one processor, a memory and an input-output unit; wherein the memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the preceding language knowledge guided cross-domain small sample visual feature enhancement method.

[0039] The above merely preferred embodiments of the present application and are not intended to limit the patent scope of the present application, any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A cross-domain few-sample visual feature enhancement method guided by language knowledge, characterized in that, include: A meta-learning paradigm is used to perform task-level reorganization and sampling of training image data in the target domain, generating an independent set of task samples containing a support set and a query set. Using a pre-built, language-knowledge-guided, cross-domain few-sample visual feature enhancement algorithm, independent task sample sets are input into the visual base model and image attribute generator, respectively, to extract visual features, image-level attribute language descriptions, and domain-level attribute language descriptions. The image-level attribute language descriptions and domain-level attribute language descriptions are concatenated and input into the encoder to obtain image attribute language description features. The image attribute language description features are then embedded into the visual features, and the visual features are refocused using a residual cross-attention layer to obtain refined features. Among them, image-level attributes are the appearance features of objects in the image, and the attribute language description includes features based on the global style, background environment and semantic composition of the dataset; Furthermore, the visual base model, residual cross-attention layer, and task-specific head are fine-tuned using the support set with few samples, and the performance of the model in cross-domain few-sample visual tasks is evaluated by the query set, resulting in a trained cross-domain few-sample visual feature enhancement algorithm guided by language knowledge. The input image data is processed using a trained, language-knowledge-guided cross-domain few-shot visual feature enhancement algorithm, and the target image with adaptive visual feature enhancement is output.

2. The method for cross-domain few-sample visual feature enhancement based on language knowledge as described in claim 1, characterized in that, The method employs a meta-learning paradigm to perform task-level reorganization and sampling of training image data in the target domain, generating an independent set of task samples containing a support set and a query set, including: Based on the meta-learning paradigm, multiple independent tasks are constructed in each training batch; In each independent task, N categories are randomly sampled from the target domain dataset, and K images are sampled from each category to form a candidate set; The candidate set is divided into a support set and a query set, wherein the support set contains S samples for each category and the query set contains Q samples for each category, and S+Q=K.

3. The method for cross-domain few-sample visual feature enhancement based on language knowledge as described in claim 1, characterized in that, The visual foundation model is a source domain pre-trained model used to extract visual features from images, wherein the visual features are high-level visual feature representations. The image attribute generator includes an image description model and a large language model, wherein the image description model is used to generate image-level attribute language descriptions, and the large language model is used to generate domain-level attribute language descriptions.

4. The method for cross-domain few-sample visual feature enhancement based on language knowledge as described in claim 1, characterized in that, The domain-level attribute language description is obtained by extracting the overall attributes of the target domain from the independent task sample set through a large language model combined with manually designed prompt words; The image-level attribute language description is obtained by extracting attributes from a single image using an image description model.

5. The method for cross-domain few-sample visual feature enhancement based on language knowledge as described in claim 1, characterized in that, The encoder is a cross-modal pre-trained encoder, used to encode the concatenated image-level attribute language description and domain-level attribute language description into fixed-dimensional image attribute language description features.

6. The method for cross-domain few-sample visual feature enhancement based on language knowledge as described in claim 4, characterized in that, The residual cross-attention layer maps visual features and image attribute language description features to a shared latent space through a learnable projection matrix. The projection matrix includes a query vector projection matrix for visual features, a key vector projection matrix for image attribute language description features, and a value vector projection matrix.

7. The method for cross-domain few-sample visual feature enhancement based on language knowledge as described in claim 1, characterized in that, The method of refocusing visual features using a residual cross-attention layer to obtain refined features includes: Visual features are mapped to query vectors by query vector projection matrices, and image attribute language description features are mapped to key vectors and value vectors by key vector projection matrices and value vector projection matrices, respectively, so that the mapped values ​​of visual features and image attribute language description features share the latent space. The attention score is obtained by calculating the scaled dot product of the query vector and the key vector, and the attention weight is obtained by applying the normalized exponential function to the attention score. The value vector is weighted and summed using attention weights to obtain optimized visual features. The original visual features are then mapped to a shared latent space using a learnable matrix, and residual connections are established between the original and optimized visual features to obtain refined features after refocusing.

8. The method for cross-domain few-sample visual feature enhancement based on language knowledge as described in claim 1, characterized in that, The method of fine-tuning the visual base model, residual cross-attention layer, and task-specific head using the support set includes: The refined features are input into a task-specific header for prediction, and the loss function value is calculated based on the prediction results and the true labels of the samples. The parameters of the visual base model, residual cross-attention layer, and task-specific head are optimized through end-to-end backpropagation using the loss function value. The task-specific header includes a classifier or a segmenter. The classifier is used for cross-domain few-shot image classification in cross-domain few-shot vision tasks, and the segmenter is used for cross-domain few-shot image segmentation in cross-domain few-shot vision tasks.

9. A cross-domain few-sample visual feature enhancement device based on language knowledge guidance, characterized in that, include: The sample set generation module is used to perform task-level reorganization and sampling of training image data in the target domain using a meta-learning paradigm, generating an independent task sample set containing a support set and a query set. The model training module utilizes a pre-built, language-knowledge-guided, cross-domain few-sample visual feature enhancement algorithm. Independent task sample sets are input into the visual base model and image attribute generator, respectively, to extract visual features, image-level attribute language descriptions, and domain-level attribute language descriptions. The image-level and domain-level attribute language descriptions are concatenated and input into the encoder to obtain image attribute language description features. These features are then embedded into the visual features, and a residual cross-attention layer is used to refocus the visual features, resulting in refined features. Among them, image-level attributes are the appearance features of objects in the image, and the attribute language description includes features based on the global style, background environment and semantic composition of the dataset; We use the support set to fine-tune the visual base model, residual cross-attention layer, and task-specific head with few samples, and evaluate the performance of the model in cross-domain few-sample visual tasks through the query set, thus obtaining the trained cross-domain few-sample visual feature enhancement algorithm based on language knowledge guidance. The execution module is used to process the input image data using a trained, language-knowledge-guided, cross-domain few-shot visual feature enhancement algorithm, and output the target image with adaptive visual feature enhancement.

10. An electronic device, characterized in that, The electronic device includes: At least one processor, memory, and input / output unit; The memory is used to store computer programs, and the processor is used to call the computer programs stored in the memory to execute the cross-domain few-shot visual feature enhancement method based on language knowledge as described in any one of claims 1 to 8.

Images