Patent image few-sample classification method based on multi-modal representation fusion

Abstract

The invention relates to the technical field of patent information analysis, in particular to a patent image few-sample classification method based on multi-modal representation fusion. The method comprises the following steps: constructing a hybrid pre-training model by designing a language branch, a visual branch and a visual auxiliary branch; obtaining input corresponding patent image few samples, and carrying out image feature parallel extraction and text description feature generation to obtain patent image text prompt description features; a cross-modal fusion layer is designed by adopting an attention mechanism, and multi-modal representation fusion and feature fusion loss calculation are carried out, so that comparative learning loss and visual auxiliary branch loss between a language branch and a visual branch are obtained; and performing model fusion optimization on the mixed pre-training model to output fusion features, and constructing a patent image few-sample classification framework to execute corresponding patent image few-sample classification work. The feature representation corresponding to the patent image sample can be fully fused and learned to improve the classification effect of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of patent information analysis, and in particular to a few-shot classification method for patent images based on multi-modal representation fusion. Background Art

[0002] Patent data is an important scientific and technological resource, which can not only protect the rights and intellectual property of inventors, but also reflect the latest scientific research results and research directions. Through in-depth mining and analysis of patents, it can support application scenarios such as research topic selection, technical route evaluation, and patent value evaluation. The analysis and mining of patent data have important research significance.

[0003] Patent data contains not only text information, but also multi-modal information such as flowcharts, structure diagrams, chemical formulas, biological sequences, and formulas. The representation fusion of the above multi-modal information is a technical problem. Due to the asymmetry in quantity of the above multi-modal information itself and the strict professional requirements for standards, the few-shot patent classification technology has become a feasible technical approach for multi-modal patent information indexing.

[0004] In recent years, the few-shot classification (FSC) task has received extensive attention in the fields of computer vision and multi-modal information processing. The core goal of the FSC task is to effectively identify new categories with a very small amount of labeled data. Traditional few-shot classification methods mainly rely on metric learning and meta-learning. Among them, metric learning methods such as Matching Networks and Prototypical Networks construct effective feature distance metrics to make the features of samples in the same category as close as possible, while the features of samples in different categories are as far away as possible; meta-learning methods such as MAML (Model-Agnostic Meta-Learning) train an initial model parameter that can quickly adapt to new tasks to improve the generalization ability of the model in few-shot tasks.

[0005] With the development of deep learning, pre-trained Visual-Language Models (VLMs) have shown excellent performance in few-shot classification tasks. VLMs represented by CLIP (Contrastive Language-Image Pretraining) learn the cross-modal mapping relationship between images and texts through large-scale image-text contrastive learning, thus showing strong generalization ability in few-shot classification tasks. Few-shot classification methods based on VLMs, such as COOP and COCOOP, improve the feature expression ability and generalization performance of the model by optimizing text prompts. In addition, methods such as CLIP-Adapter and Tip-Adapter introduce adapter modules and only fine-tune a small number of parameters to reduce the risk of overfitting and improve the transfer ability of the model.

[0006] However, the existing few-shot classification methods based on VLMs still have limitations. When facing highly complex image content, these methods are difficult to fully learn effective feature representations, resulting in limited classification performance. In addition, due to the lack of prior information in few-shot data, the existing methods often lack effective feature fusion strategies in the few-shot learning process, making it difficult for the model to fully utilize the capabilities of the pre-trained model, thus affecting the classification effect. Summary of the Invention

[0007] Based on this, it is necessary to provide a few-shot classification method for patent images based on multi-modal representation fusion to solve at least one of the above technical problems.

[0008] To achieve the above object, a few-shot classification method for patent images based on multi-modal representation fusion includes the following steps:

[0009] Step S1: Construct a hybrid pre-training model by designing a language branch, a visual branch, and a visual auxiliary branch;

[0010] Step S2: Obtain the few-shot patent images corresponding to the input, and parallelly extract image features from the few-shot patent images based on the hybrid pre-training model to obtain patent image features corresponding to different types; generate text description features for each patent image in the few-shot patent images based on the text prompt generator corresponding to the large language model and in combination with the text encoder based on Transformer to obtain the patent image text prompt description features;

[0011] Step S3: Design a cross-modal fusion layer by using the attention mechanism among the model branches in the hybrid pre-training model, and use the cross-modal fusion layer to perform multi-modal representation fusion on the patent image features corresponding to different types and the patent image text prompt description features to generate the corresponding patent image multi-modal fusion process;

[0012] Step S4: Calculate the feature fusion loss for each model branch in the hybrid pre-trained model by designing a multi-level fusion loss function during the patent image multi-modal fusion process, so as to obtain the contrastive learning loss between the language branch and the visual branch and the visual auxiliary branch loss; optimize the model fusion of the hybrid pre-trained model corresponding to the patent image multi-modal fusion process based on the contrastive learning loss between the language branch and the visual branch and the visual auxiliary branch loss, output the corresponding multi-modal fusion features of the patent image samples, and construct a few-shot classification framework for patent images to perform the corresponding few-shot classification work of patent images.

[0013] Further, step S1 includes the following steps:

[0014] Step S11: Design and construct a language branch by using the text encoder corresponding to CLIP to convert the corresponding input image samples into text embeddings with deep semantic information;

[0015] Step S12: Design and construct a visual branch by consisting of a pre-trained visual feature extractor DINO, an auxiliary feature attention mechanism AFA, and an image encoder corresponding to CLIP, so as to extract visual features from the corresponding input image samples through the pre-trained visual feature extractor to generate visual representations for AFA, and optimize and output the corresponding image visual optimized representations through AFA combined with the image encoder;

[0016] Step S13: Design and construct a visual auxiliary branch by consisting of a pre-trained visual feature extraction model and a linear probe LP to balance the output results of the contrastive learning corresponding language branch and visual branch, select auxiliary features, and initialize the weights of LP with the feature means corresponding to each category of the output results to obtain the logit value corresponding to each input image sample;

[0017] Step S14: Fuse the language branch, the visual branch, and the visual auxiliary branch to construct a hybrid pre-trained model.

[0018] Further, the language branch described in step S11 is specifically:

[0019] Tokenize the corresponding input image samples: T = [t1, t2, …, t i , where T represents the text corresponding to the input image samples, and t i represents the i-th word in the text;

[0020] Perform word embedding processing on each word t i to obtain the text word embedding vector e i = W text [t i, where, W text represents the word embedding matrix;

[0021] Use the text encoder corresponding to CLIP to perform positional encoding processing on the text word embedding vectors to obtain the word vectors x corresponding to the positional encoding i = e i + p i , p i = PE(i), where, PE(·) represents the calculation of the positional encoding, p i represents the positional encoding;

[0022] Use the Transformer encoder to perform additional context-aware encoding on each word vector corresponding to the positional encoding to obtain the additional context-aware representation ct corresponding to each word vector L = Transformer L ({x1, x2, …, x n})), where, n represents the number of word vectors, L represents the number of layers of the Transformer encoder, Transformer L (·) represents the calculation of the last Transformer layer;

[0023] Based on the additional context-aware representation ct L Perform CLS-aware marking on each word vector corresponding to the positional encoding to obtain the text word vectors corresponding to the CLS-aware marking Perform text embedding processing on the word vectors corresponding to the CLS-aware marking to obtain the text embedding t corresponding to the input image sample e = Normalize(W text_proj t v ), where, W text_proj is the projection matrix, and Normalize(·) represents the calculation of performing L2 normalization on the text word vectors.

[0024] Furthermore, the visual branch described in step S12 is specifically:

[0025] Use the pre-trained visual feature extractor to extract the visual representation of the input image sample to obtain the visual representation matrix c = F(img) corresponding to the input image sample, where, img represents the input image sample, and F(·) represents the calculation of the visual feature extractor;

[0026] Perform normalized similarity calculation on the visual representation matrix c corresponding to the input image sample to obtain the normalized similarity matrix where, cc TDenotes the outer product between the visual representation matrix and its corresponding transposed visual representation matrix, which is used to represent the similarity between visual blocks. α represents the scaling parameter, λ represents the translation parameter, l represents the length of the median embedding in the CLIP image encoder, and ∑ a,b [cc T a,b Denotes the sum between each similarity element;

[0027] For each element S within the similarity matrix ab Perform a masking operation to obtain the corresponding element in the similarity masking matrix D Denotes that when the similarity score S ab is non - negative, the value at the corresponding position in the similarity masking matrix D is zero; when the similarity score S ab is negative, the value at the corresponding position in the similarity masking matrix D is negative infinity;

[0028] Perform the attention calculation corresponding to AFA on the normalized similarity matrix S and the similarity masking matrix D to obtain the attention matrix W AFA = SoftMax(S + D);

[0029] Output the corresponding visual representation z through the CLIP image encoder, and perform visual optimization processing on the visual representation z output by the CLIP image encoder based on the attention matrix to output the corresponding image visual optimization representation v = Proj(W AFA ·z), where Proj(·) represents the calculation of the projection layer.

[0030] Furthermore, the visual assistance branch described in step S13 is specifically:

[0031] Assume that the corresponding input image sample has M categories and K samples, and use the visual feature extraction model to extract the auxiliary visual features of the input image sample to obtain the input image auxiliary features where x jk represents the k - th image of the j - th category, represents the frozen weight parameter, f(·) represents the calculation of the visual feature extraction model, and h jk represents the auxiliary feature of the k - th image of the j - th category;

[0032] Calculate the mean of the input image auxiliary features to obtain the auxiliary feature mean corresponding to the j - th category

[0033] Use the linear probe LP to initialize the weights of the auxiliary feature means corresponding to each category to obtain the corresponding load feature weights And calculate the logit value corresponding to each input image sample based on the load feature weights ​

[0034] Further, the calculation of the visual feature extraction model is specifically as follows:

[0035]

[0036] In the formula, M is the number of categories corresponding to the input image samples, x jk represents the k-th image of the j-th category, is the frozen weight parameter.

[0037] Further, step S2 includes the following steps:

[0038] Step S21: Obtain few-shot patent images corresponding to the input;

[0039] Step S22: Parallelly extract image features from the few-shot patent images based on the hybrid pre-trained model to obtain patent image features corresponding to different types, including patent image text embeddings, patent image visual optimization representations, and patent image sample logit values;

[0040] Step S23: Assume that the set of patent image categories corresponding to the few-shot patent images is C = {C1, C2, …, C d , …, C N}, where N represents the number of image categories, and C d represents the name of the d-th category, and use the text prompt generator corresponding to the large language model combined with the Transformer-based text encoder to generate text description features for each patent image in the few-shot patent images to obtain patent image text prompt description features.

[0041] Further, the specific process of generating the text description features in step S23 is as follows:

[0042] Use the text prompt generator LLM corresponding to the large language model to construct a text prompt template for each category name in the set of patent image categories to generate a text prompt T d = P d (C d ), where P d represents the text description prompt template used by the LLM to generate the d-th category;

[0043] Receive the input corresponding text prompt T d through the LLM and convert it into a word vector E = Tokenize(T d ), where Tokenize(·) represents converting the text prompt into the input embedding calculation of the LLM;

[0044] The word vectors are input into a Transformer - based text encoder with m layers for encoding processing to output the context representation H at each layer position. (0) = E; H (m) = TransformerLayer (m) (H (m-1) ), where H (0) represents the initial input embedding of the LLM, H (m) represents the hidden state output by the m - th layer, and TransformerLayer (m) (·) represents the calculation of the m - th Transformer layer;

[0045] The text decoder is used to gradually generate the corresponding text description sequence Y = {y1, y2, …, y u} for each input text prompt embedding X corresponding to a patent image, where u is the length of the generated description, and the probability P(y h |y1, …, y h-1 , X) = Softmax(W y ·H (L) ) is calculated by quantization. Here, W y is the weight matrix of the output layer, H (L) is the output representation of the last layer of the Transformer, and Softmax(·) is the Softmax function, which is used to map the output to a probability distribution to output the corresponding text description, including text descriptions based on categories, text descriptions based on categories and similar categories, text descriptions based on category synonyms, and text descriptions based on category attributes, to obtain the patent image text prompt description features;

[0046] Among them, the generated patent image text prompt description features are specifically:

[0047] Y C = GPT(C, P C ); Y CS = GPT(C, P CS ); Y ST = GPT(C, P ST ); Y CA = GPT(C, P CA );

[0048] Among them, Y C , Y CS , Y ST and Y CA respectively represent text descriptions based on categories, text descriptions based on categories and similar categories, text descriptions based on category synonyms, and text descriptions based on category attributes, P C , PCS , P ST and P CA respectively represent the text prompt templates used to generate the above four types of text descriptions, and GPT(·) represents the Softmax probability distribution calculation corresponding to the LLM.

[0049] Furthermore, step S4 includes the following steps:

[0050] Step S41: Calculate the contrastive learning feature fusion loss for the contrastive learning between the language branch and the visual branch in the hybrid pre-trained model by designing a fusion loss function between the corresponding language branch and visual branch during the patent image multi-modal fusion process, so as to obtain the contrastive learning loss L between the language branch and the visual branch VL ;

[0051] Step S42: Calculate the auxiliary feature fusion loss for the visual auxiliary branch in the hybrid pre-trained model by designing a corresponding visual auxiliary loss function during the patent image multi-modal fusion process, so as to obtain the visual auxiliary branch loss L LP ;

[0052] Among them, the visual auxiliary loss function is specifically:

[0053]

[0054] Among them, G q represents the label of the q-th sample, represents the logit value corresponding to the q-th sample The class prediction probability obtained after passing through the softmax function, represents the logit value of the q-th sample on class n;

[0055] Step S43: Calculate the total loss based on the contrastive learning loss between the language branch and the visual branch and the visual auxiliary branch loss, and obtain the total loss L of the hybrid model total =βL VL +(1 - β)L LP , where β is a hyperparameter;

[0056] Step S44: Optimize the model fusion of the hybrid pre-trained model corresponding to the patent image multi-modal fusion process based on the total loss of the hybrid model, output the multi-modal fusion features of the corresponding patent image samples, and construct a few-shot classification framework for patent images to perform the corresponding few-shot classification work for patent images.

[0057] Furthermore, the fusion loss function between the language branch and the visual branch described in step S41 is specifically:

[0058]

[0059] where G represents the total number of samples, represents the logit value of the g-th sample output by the language branch and the visual branch, and τ represents the embedding coefficient.

[0060] Advantages of the present invention:

[0061] The few-shot classification method for patent images based on multi-modal representation fusion proposed by the present invention, compared with the prior art, the beneficial effects of the present application are as follows: a hybrid pre-training model is constructed by designing a language branch, a vision branch, and a vision assistance branch. Each branch is responsible for extracting different types of features to ensure that the model can effectively process multi-modal data. The language branch is responsible for processing text data and usually adopts an architecture based on Transformer, which can capture complex language semantic information and is suitable for patent text descriptions. The vision branch is responsible for processing patent images and extracts the visual features of the images through methods such as convolutional neural network (CNN) or vision Transformer (ViT). The vision assistance branch enhances the visual features, for example, by using auxiliary information of the images (such as image labels or category information) to optimize the quality of image feature extraction. In this way, not only can the visual features of patent images be processed, but also their text descriptions can be carefully understood, thus providing a basis for subsequent cross-modal fusion. The advantage of this design is that it can process data of different modalities separately and lay a solid foundation for the fusion of multi-modal data, improving the model's understanding ability of different types of patent data, so as to fully learn effective feature representations. Secondly, the input few-shot patent images are used to extract image features in parallel through the hybrid pre-training model. Here, the image features are extracted through the vision branch, while the language branch generates the text description features corresponding to the patent images based on the text prompt generator of the large language model using the Transformer text encoder. The parallel extraction of image features and text features helps to establish the association between images and texts and improve the model's processing ability for few-shot data. The few-shot feature extraction of patent images is very important because the samples of patent images are often few and there are class imbalance problems. By combining the visual features of the images and the text prompt description features, the model can obtain richer patent image information, thus effectively improving the performance of few-shot learning. Using the text prompt generator, corresponding text descriptions can be generated in the absence of labeled data, assisting in enhancing the model's understanding and analysis ability of image content and strengthening the model's multi-modal learning ability.Then, by designing a cross-modal fusion layer within the hybrid pre-trained model and using the attention mechanism to achieve effective fusion of different modal data, the cross-modal fusion layer can align and map visual features and text features in the same space, thereby ensuring that the model can generate rich multi-modal representations through interrelated visual and language features. This method can utilize the complementarity between vision and language to enhance the feature representation ability, improve the comprehensive understanding of patent images, and the benefit of designing the fusion layer by using the attention mechanism is that the attention mechanism can dynamically adjust the weights according to the relevance of the input features, making the model pay more attention to the features that contribute more to the classification of patent images. This cross-modal fusion design effectively enhances the model's ability to understand few-shot patent images. Especially in the context of scarce data, the model can execute effective feature fusion strategies through deep fusion of different modalities, generate more accurate image representations, and further improve the effect of patent image classification. Finally, a multi-level fusion loss function is designed to calculate the contrastive learning loss and the visual auxiliary branch loss between the language branch, the visual branch, and the visual auxiliary branch of the model. This loss function can optimize the multi-modal representation ability of the model by measuring the feature differences between different branches. By introducing the contrastive learning loss, the model can be prompted to perform more refined matching of image and text features during training, thereby improving the accuracy of few-shot patent image classification. In addition, the introduction of the visual auxiliary branch loss further optimizes the process of image feature extraction and fusion, helping the model better process different types of image data. Through the calculation of these loss functions, the optimization of the model enables the hybrid pre-trained model to accurately and efficiently perform feature fusion when processing multi-modal data of patent images, thereby improving the performance of the few-shot patent image classification framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Other features, objects, and advantages of the present invention will become more apparent by reading the detailed description of the non-limiting embodiments with reference to the following drawings:

[0063] Figure 1 It is a schematic flowchart of the steps of the method for few-shot classification of patent images based on multi-modal representation fusion of the present invention;

[0064] Figure 2 is Figure 1 a detailed schematic flowchart of step S1 in DETAILED DESCRIPTION OF THE EMBODIMENTS

[0065] The technical method of the present invention will be clearly and completely described below with reference to the drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0066] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus repeated descriptions thereof will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor methods and / or microcontroller methods.

[0067] It should be understood that although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.

[0068] To achieve the above object, please refer to Figures 1 to 2 , the present invention provides a few-shot classification method for patent images based on multi-modal representation fusion, and the method includes the following steps:

[0069] Step S1: Construct a hybrid pre-training model by designing a language branch, a vision branch, and a vision assistance branch;

[0070] Step S2: Obtain the few-shot patent images corresponding to the input, and parallelly extract the image features of the few-shot patent images based on the hybrid pre-training model to obtain the patent image features corresponding to different types; generate the text description features of each patent image in the few-shot patent images based on the text prompt generator corresponding to the large language model and in combination with the text encoder based on Transformer to obtain the patent image text prompt description features;

[0071] Step S3: Design a cross-modal fusion layer by adopting an attention mechanism among the model branches in the hybrid pre-training model, and use the cross-modal fusion layer to perform multi-modal representation fusion on the patent image features corresponding to different types and the patent image text prompt description features to generate the corresponding patent image multi-modal fusion process;

[0072] Step S4: Calculate the feature fusion loss for each model branch in the hybrid pre-trained model by designing a multi-level fusion loss function during the patent image multi-modal fusion process, so as to obtain the contrastive learning loss between the language branch and the visual branch and the visual auxiliary branch loss; Based on the contrastive learning loss between the language branch and the visual branch and the visual auxiliary branch loss, optimize the model fusion of the hybrid pre-trained model corresponding to the patent image multi-modal fusion process, output the corresponding multi-modal fusion features of the patent image samples, and construct a patent image few-shot classification framework to perform the corresponding patent image few-shot classification work.

[0073] In the embodiment of the present invention, please refer to Figure 1 As shown, it is a schematic diagram of the step flow of the patent image few-shot classification method based on multi-modal representation fusion of the present invention. In this example, the patent image few-shot classification method based on multi-modal representation fusion includes the following steps:

[0074] Step S1: Construct a hybrid pre-trained model by designing a language branch, a visual branch, and a visual auxiliary branch;

[0075] In the embodiment of the present invention, when constructing the hybrid pre-trained model, first, a language branch, a visual branch, and a visual auxiliary branch are designed. The visual branch mainly extracts image features from patent images through a vision feature extractor DINO pre-trained by a convolutional neural network (CNN). These features can effectively represent the visual information in patent images, such as shape, texture, and color, etc. The language branch uses a large-scale pre-trained language model (such as BERT or GPT series) to extract text features using the patent text description information, aiming to capture the technical details and semantic content of the patent. In order to enhance the semantic understanding of the image, a visual auxiliary branch is designed to combine the image features with the meta-information of the image, such as metadata like patent number, inventor, etc., to further enrich the model's visual understanding ability. All three branches are fused into a whole through a unified framework to achieve collaborative learning and optimization of image and text information in subsequent tasks, and finally, a hybrid pre-trained model is constructed.

[0076] Step S2: Obtain the corresponding patent image few-shot samples, and perform parallel extraction of image features on the patent image few-shot samples based on the hybrid pre-trained model to obtain patent image features corresponding to different types; Generate text description features for each patent image in the patent image few-shot samples based on the text prompt generator corresponding to the large language model and in combination with the Transformer-based text encoder to obtain the patent image text prompt description features;

[0077] In the embodiment of the present invention, after obtaining a small sample of the input patent image, based on the hybrid pre-trained model, image feature parallel extraction is first performed. The visual branch uses classical deep learning networks such as ResNet or EfficientNet to extract the image features of the patent image. These features represent the low-level (such as edges, colors) and high-level (such as object recognition) information of the image. At the same time, a text prompt generator based on a large language model generates a corresponding text description for each patent image. This process first generates prompt words through the basic information of the image, and then combines the text encoder of the Transformer structure to generate the text description features of the patent image. This text description feature can reflect the technical content, innovation points, and functional description of the patent image. This process ensures the parallel extraction of image and text information, and the image and text features have a high degree of matching, which is convenient for subsequent multimodal fusion. Finally, the patent image text prompt description features are obtained, including text descriptions based on categories, text descriptions based on categories and similar categories, text descriptions based on category synonyms, and text descriptions based on category attributes.

[0078] Step S3: Design a cross-modal fusion layer by adopting an attention mechanism between each model branch in the hybrid pre-trained model, and use the cross-modal fusion layer to perform multimodal feature fusion on the patent image features corresponding to different types and the patent image text prompt description features to generate the corresponding patent image multimodal fusion process;

[0079] In the embodiment of the present invention, within the hybrid pre-trained model, by designing a cross-modal fusion layer driven by an attention mechanism, the patent image features extracted by the visual branch are fused with the text description features. This cross-modal fusion layer adopts the self-attention mechanism based on Transformer, which can calculate the correlation between the image features and the text features, ensuring that when each image feature is aligned with the text feature, it can fully reflect the semantic relationship between the two. Specifically, first, the image features and the text features are mapped to the same dimensional space through the embedding layer, and then the mutual influence between the image and text features is calculated through the multi-head self-attention mechanism. The multimodal feature fusion features output by the cross-modal fusion layer are the fusion representation of the image and text information in the same feature space, which can effectively capture the deep connection between vision and text in the patent image. Through the processing of this layer, the model can obtain richer semantic information and finally generate the corresponding patent image multimodal fusion process.

[0080] Step S4: Calculate the feature fusion loss for each model branch in the hybrid pre-trained model by designing a multi-level fusion loss function during the patent image multi-modal fusion process, so as to obtain the contrastive learning loss between the language branch and the visual branch and the visual auxiliary branch loss; optimize the model fusion of the hybrid pre-trained model corresponding to the patent image multi-modal fusion process based on the contrastive learning loss between the language branch and the visual branch and the visual auxiliary branch loss, output the corresponding multi-modal fusion features of the patent image samples, and construct a few-shot classification framework for patent images to perform the corresponding few-shot classification work of patent images.

[0081] In the embodiment of the present invention, during the patent image multi-modal fusion process, by designing a multi-level fusion loss function to calculate the feature fusion loss, it is ensured that the contrastive learning loss between the language branch and the visual branch and the loss of the visual auxiliary branch can be effectively optimized. First, calculate the feature similarity between the language branch and the visual branch through the contrastive learning loss, and use the strategy of maximizing mutual information to ensure that the image and text feature embeddings of the same category are close, and the image and text features of different categories are far apart. In addition, a loss function for the visual auxiliary branch is designed to focus on optimizing the image features using visual information and ensure the independence and effectiveness of visual information in multi-modal fusion. Then, combine the contrastive learning loss between the language branch and the visual branch and the visual auxiliary branch loss to calculate the total loss of the model. Based on the total loss, use the gradient descent method to optimize the hybrid pre-trained model, so that the language branch, the visual branch and the visual auxiliary branch can jointly improve the classification performance. During the optimization process, gradually adjust the model parameters through backpropagation and multiple rounds of iteration to output the final multi-modal fusion features of the patent image. These fusion features are used to construct a few-shot classification framework for patent images, enabling the model to efficiently complete the patent image classification task with the support of a small amount of labeled data, significantly improving the accuracy of few-shot classification, and finally performing the corresponding few-shot classification work of patent images.

[0082] Further, step S1 includes the following steps:

[0083] Step S11: Design and construct a language branch by using the text encoder corresponding to CLIP to convert the corresponding input image sample into a text embedding with deep semantic information.

[0084] Step S12: Design and construct a visual branch by consisting of a pre-trained visual feature extractor DINO, an auxiliary feature attention mechanism AFA, and an image encoder corresponding to CLIP, so as to extract visual features from the corresponding input image sample through the pre-trained visual feature extractor to generate a visual representation for AFA, and optimize and output the corresponding image visual optimized representation through AFA combined with the image encoder.

[0085] Step S13: Design and construct a visual assistance branch by combining a pre-trained visual feature extraction model and a linear probe LP, which is used to balance the output results of the corresponding language branch and visual branch in contrastive learning, select auxiliary features, and initialize the weights of LP using the feature means corresponding to each category of the output results to obtain the logit value corresponding to each input image sample;

[0086] Step S14: Fuse the language branch, visual branch, and visual assistance branch to construct a hybrid pre-trained model.

[0087] As an embodiment of the present invention, refer to Figure 2 shown in Figure 1 is a detailed step flow schematic diagram of step S1 in

[0088] Step S11: Design and construct a language branch by using the text encoder corresponding to CLIP to convert the corresponding input image sample into a text embedding with deep semantic information;

[0089] In the embodiment of the present invention, the core of the design of this language branch is to convert the input image sample into a text embedding with deep semantic information through the text encoder in CLIP (Contrastive Language-Image Pretraining). First, use the text encoder of the CLIP model to process the descriptive text corresponding to the given image to obtain the embedding representation of the text. This text encoder uses pre-trained language models (such as GPT, BERT, etc.) to process the text information describing the image content and generates a text embedding vector with a fixed dimension. This vector contains the semantic information of the image description. For each input image sample, first provide a semantic framework through the manual annotation or text description of the image, and then this framework will be mapped to a text embedding by the text encoder of CLIP. These embeddings will be used for contrastive learning operations in subsequent multimodal tasks to ensure that the semantic information of the image can be accurately expressed and used for classification tasks, and finally obtain the corresponding text embedding.

[0090] Step S12: Design and construct a visual branch by combining a pre-trained visual feature extractor DINO, an auxiliary feature attention mechanism AFA, and an image encoder corresponding to CLIP, so as to extract visual features from the corresponding input image sample through the pre-trained visual feature extractor to generate visual representations for AFA, and optimize the output corresponding image visual optimized representation through AFA combined with the image encoder;

[0091] In the embodiment of the present invention, the construction of the visual branch adopts a pre-trained visual feature extractor (DINO), an auxiliary feature attention mechanism (AFA), and the image encoder in CLIP. First, the input image is processed by the DINO network to extract its visual features. The DINO network can perform self-supervised learning on the high-level features in the image to obtain the latent feature representation of the image. Then, through the AFA module, these features will be further optimized. The AFA module can adaptively assign an attention weight to each feature in the image, thereby highlighting the important parts in the image and strengthening the attention to key features in the visual representation. The image encoder of CLIP further fuses and encodes the visual features optimized by AFA with the image content to generate an embedding vector representing the semantic information of the image. Through this step, it can be ensured that the input image is deeply understood at the visual level and transformed into a high-quality feature representation, and finally the corresponding visual optimization representation of the image is output after optimization.

[0092] Step S13: Design and construct a visual auxiliary branch by using a pre-trained visual feature extraction model and a linear probe LP to balance the output results of the contrastive learning corresponding language branch and visual branch, select auxiliary features, and initialize the weights of LP with the feature means corresponding to each category of the output results to obtain the logit value corresponding to each input image sample.

[0093] In the embodiment of the present invention, the design of the visual auxiliary branch integrates a pre-trained visual feature extraction model and a linear probe (LP). The visual branch and the language branch are jointly trained through contrastive learning, and the role of the visual auxiliary branch is to establish a balance between the two. First, the pre-trained visual feature extractor is used to extract features from the input image samples to obtain their preliminary visual representations. Then, the linear probe LP takes over this visual feature and, through the contrastive learning algorithm, makes the visual representation of each input image sample match the corresponding text embedding. At this time, the linear probe will adjust the weights and parameters according to the contrast relationship between the visual representation and the language representation, thereby optimizing the extraction effect of the visual features. To improve the training efficiency, the weights of LP are initialized according to the feature means of each category, enabling the model to converge more quickly and obtain higher accuracy in the contrastive learning process. Through the visual auxiliary branch, the fusion of visual and language information can be better balanced, improving the performance of the overall model in few-shot classification tasks, and finally obtaining the logit value corresponding to each input image sample.

[0094] Step S14: Fuse the language branch, the visual branch, and the visual auxiliary branch to construct a hybrid pre-trained model.

[0095] In the embodiments of the present invention, a multi-modal hybrid pre-training model is constructed by fusing the language branch, visual branch, and visual assistance branch obtained from the previous design. First, the text embeddings and image embeddings obtained in the previous steps are fused through contrastive learning. This fusion process deeply integrates language and visual information through a common feature space, enabling the information of these two modalities to complement each other and optimize synergistically. Specifically, the model calculates the similarity between language and visual features through a contrastive loss function, thereby optimizing their respective embedding spaces, making the same image and its description text close in the embedding space, while keeping the embeddings of different images and texts at a large distance. The output of the entire multi-modal fusion model can be used for few-shot classification tasks. Through this multi-modal representation, the model can effectively overcome the few-shot problem and achieve more accurate classification results. When dealing with the few-shot problem in fields such as patent images, the fused hybrid pre-training model can give full play to the advantages of language and vision, improve the classification accuracy and robustness, and finally obtain the hybrid pre-training model.

[0096] Further, the language branch described in step S11 is specifically as follows:

[0097] Tokenize the corresponding input image sample: T = [t1, t2, …, t i , where t represents the text corresponding to the input image sample, and t i represents the i-th word in the text;

[0098] In the embodiments of the present invention, through the tokenization process of the input image sample, first, the corresponding text information of the input image sample is extracted to form a word sequence T = [t1, t2, …, t i , where t i represents the i-th word in the text. The purpose of this step is to split the text description in the image sample (such as the text content in the patent image) into individual words for subsequent processing. The tokenization of the text can be performed through standard natural language processing tools. For example, the Chinese text can be segmented using a tokenization tool such as jieba, or the English text can be segmented using other language processing tools such as spaCy to ensure that the text can be effectively and accurately decomposed into independent words.

[0099] Preferably, each word t i is subjected to word embedding processing to obtain the text word embedding vector e i = W text [t i , where W text represents the word embedding matrix;

[0100] In the embodiments of the present invention, through the word embedding processing of each word obtained from the previous tokenization, the text word embedding vector ei , the process of word embedding is achieved by looking up a pre-trained word embedding matrix W text , where W text is a high-dimensional matrix that represents the correspondence between each word and its semantic space. Each word t i will be mapped to a high-dimensional vector e i . Usually, the dimensions of these word vectors are relatively high, between 100 and 1000, specifically according to the design of the pre-trained word vector model (such as Word2Vec, GloVe, etc.). The representation after word embedding can effectively capture the semantic relationship between words, laying a foundation for subsequent context understanding, and finally obtaining the text word embedding vector.

[0101] Preferably, the text encoder corresponding to CLIP is used to perform position encoding processing on the text word embedding vector to obtain the word vector x i = e i + p i , p i = PE(i), where PE(·) represents the calculation of position encoding, and p i represents the position encoding;

[0102] In the embodiments of the present invention, the text encoder of the CLIP (Contrastive Language-Image Pre-training) model is used to perform position encoding processing on the word embedding vector. The position encoding is used to represent the relative or absolute position of a word in the text, which can help the model understand the order information between words. In a specific implementation, the position encoding is calculated through a function PE(i). For each word t i the corresponding word embedding e i will be added to the position encoding p i to obtain the word vector x i containing position information. The PE(i) function is usually calculated based on sine and cosine functions. It can generate a unique encoding for each word according to the order of words, ensuring that the model can consider the order relationship of words when processing text, and finally obtaining the word vector corresponding to the position encoding.

[0103] Preferably, the Transformer encoder is used to perform additional context-aware encoding on each word vector corresponding to the position encoding to obtain the additional context-aware representation ct L = Transformer L ({x1,x2,…,x n}), where n represents the number of word vectors, L represents the number of layers of the Transformer encoder, and Transformer L(·) represents the calculation of the last Transformer layer;

[0104] In the embodiments of the present invention, by using a Transformer encoder to further process the word vectors containing positional encoding, the context awareness ability between vocabularies is enhanced. The Transformer encoder is generally composed of multiple layers of self-attention mechanisms and feed-forward neural networks. Each layer can capture semantic information at different levels. In this step, each input word vector x1, x2, …, x n (i.e., the word embedding vector containing positional encoding) will pass through the Transformer L (·) for encoding. L represents the number of layers of the Transformer encoder. The output of each layer will take into account all other words in the context, thereby generating a richer context representation ct L , and these context-aware representations can better capture the meaning of each word in the context, and finally obtain the additional context-aware representation corresponding to each word vector.

[0105] Preferably, based on the additional context-aware representation ct L perform CLS-aware marking on each word vector corresponding to the positional encoding to obtain the text word vector corresponding to the CLS-aware marking perform text embedding processing on the word vector corresponding to the CLS-aware marking to obtain the text embedding t corresponding to the input image sample e = Normalize(W text_proj t v ), where W text_proj is the projection matrix, and Normalize(·) represents the calculation of L2 normalization on the text word vector.

[0106] In the embodiments of the present invention, through the output ct of the Transformer L generate the word vector t containing the CLS-aware marking v . The CLS marking is usually achieved by inserting a special CLS marking ([CLS]) at the beginning of the text sequence. This marking can aggregate the overall semantic information of the text. After the context-aware processing of the Transformer encoder, select the vector at the corresponding CLS position as the final text vector t v . Then, use the projection matrix W text_proj to project t v into the embedding space of the text to obtain the final embedding representation t of the text e , and the projection matrix W text_projis a linear transformation matrix that can map the output vector of the CLS token into the embedding space of the text, further enhancing the expressive power of the text vector. Using the L2 normalization technique for t v to perform normalization processing to ensure that the text vector t e has a norm of 1 in the embedding space, which helps to perform effective similarity calculation and fusion in multimodal tasks, and finally obtain the text embedding corresponding to the input image sample.

[0107] Furthermore, the visual branch described in step S12 is specifically:

[0108] Use a pre-trained visual feature extractor to extract the visual representation of the input image sample to obtain the visual representation matrix c = F(img) corresponding to the input image sample, where img represents the input image sample and F(·) represents the calculation of the visual feature extractor;

[0109] In the embodiment of the present invention, by using a pre-trained visual feature extractor to process the input image sample img, the image is processed by the visual feature extractor F(·) to obtain a visual feature matrix c representing the image. The visual feature extractor can be a deep learning-based model, such as ResNet, VGG, or more advanced models such as the CLIP (Contrastive Language-Image Pretraining) model, etc. These models are trained on large-scale datasets and can extract high-dimensional and abstract information from images. In specific implementation, F(·) will perform convolution operations, pooling operations, and other non-linear transformations on the input image img to generate a feature matrix c, which represents different visual information in the image in each dimension, and finally obtain the visual representation matrix corresponding to the input image sample.

[0110] Preferably, perform a normalized similarity calculation on the visual representation matrix c corresponding to the input image sample to obtain a normalized similarity matrix where cc T represents the outer product between the visual representation matrix and its corresponding transposed visual representation matrix, used to represent the similarity between visual blocks, α represents a scaling parameter, λ represents a translation parameter, l represents the length of the value embedding in the CLIP image encoder, and ∑ a,b [cc T a,b represents the sum of each similarity element;

[0111] In the embodiment of the present invention, the visual representation matrix c of the input image needs to undergo a similarity calculation to generate a similarity matrix S. In specific implementation, first through cc T ​The outer product between the calculated visual representation matrix c and its transpose matrix is obtained. This outer product represents the similarity between each visual block (such as different regions or features in an image). To enhance the effectiveness of the calculation, a scaling parameter α and a translation parameter λ are introduced. The calculation result of the outer product is adjusted and added with a correction term, which considers the sum ∑ a,b [cc T a,b ). The role of this term is to normalize the similarity matrix so that the influence of each element is neither too large nor too small. The obtained matrix S is represented as This similarity matrix S contains the correlation information between each visual block in the image.

[0112] Preferably, a masking operation is performed on each element S ab in the similarity matrix to obtain the corresponding element in the similarity masking matrix D, which means that when the similarity score S ab is non - negative, the value of the corresponding position in the similarity masking matrix D is zero; when the similarity score S ab is negative, the value of the corresponding position in the similarity masking matrix D is negative infinity;

[0113] In the embodiment of the present invention, by performing a masking operation on the normalized similarity matrix S, the similarity masking matrix D is obtained. Specifically, for each element S ab in the matrix, if the similarity score is greater than or equal to zero, the corresponding element D ab is zero; if the similarity score is less than zero, D ab is set to negative infinity. The purpose of this masking operation is to emphasize the positive similarity between features in the image and suppress the negative similarity, so that in subsequent calculations, the model can pay more attention to visually relevant blocks with positive correlation and avoid being interfered by negative information. The core purpose of this operation is to improve the model's ability to recognize important regions in the image, and finally obtain the corresponding element

[0114] Preferably, the attention calculation corresponding to AFA is performed on the normalized similarity matrix S and the similarity masking matrix D to obtain the attention matrix W AFA = SoftMax(S + D);

[0115] In the embodiment of the present invention, by applying the AFA (Attention Feature Aggregation) model, combining the normalized similarity matrix S and the similarity masking matrix D, the attention matrix W AFA ​, the attention matrix is calculated by adding S and D and then processed by the SoftMax function. The SoftMax function normalizes the input values into a probability distribution, resulting in the final attention matrix W AFA reflects the attention levels of different regions in the image. The purpose of this process is to emphasize the more important visual feature regions through the attention mechanism so that in subsequent visual optimization steps, information integration can be carried out more effectively. The attention matrix W AFA After the SoftMax operation, it can reflect the influence of different regions in the image on the final decision, and finally obtain the attention matrix W AFA = SoftMax(S + D).

[0116] Preferably, the corresponding visual representation z is output by the CLIP image encoder, and the corresponding visual representation z output by the CLIP image encoder is visually optimized based on the attention matrix to output the corresponding image visual optimization representation v = Proj(W AFA ·z), where Proj(·) represents the calculation of the projection layer.

[0117] In the embodiment of the present invention, based on the visual representation z output by the CLIP image encoder, further visual optimization processing is performed through the attention matrix W AFA to obtain the optimized visual representation v. In the specific implementation, first, the original visual representation z of the image is extracted by the CLIP model, and then the attention matrix W AFA is multiplied by this visual representation to generate a weighted visual feature representation. Next, the weighted visual feature is further transformed by the projection layer Proj(·) to obtain the final visual optimization representation. This optimized representation can not only better retain the key information in the image but also meet the requirements of downstream tasks (such as few-shot classification). The optimized visual representation will be used as input and passed to subsequent classification models or other task modules to improve the final classification performance and accuracy, and finally obtain the corresponding image visual optimization representation.

[0118] Furthermore, the visual assistance branch described in step S13 is specifically:[[]]

[0119] By assuming that there are M categories and K samples for the corresponding input image samples, and using the visual feature extraction model to extract the auxiliary visual features of the input image samples to obtain the input image auxiliary features where x jk represents the k-th image of the j-th category, represents the frozen weight parameter, f(·) represents the calculation of the visual feature extraction model, and h jk represents the auxiliary feature of the k-th image of the j-th category;

[0120] In an embodiment of the present invention, by assuming that the input image samples contain M categories, and there are K samples in each category. First, a visual feature extraction model is used to extract auxiliary visual features for each input image sample. Specifically, for each input image sample x jk , the k-th image sample of the j-th category, by inputting it into a pre-trained visual feature extraction model wherein, the visual feature extraction model is specifically In the formula, M is the number of categories corresponding to the input image samples, x jk represents the k-th image of the j-th category, is the frozen weight parameter to obtain the auxiliary feature h of this image sample jk , where represents the frozen weight parameter of the visual feature extraction model, that is, the fixed parameter obtained during the model training stage, ensuring that the extracted features are stable and reliable. Through the operation of this model, the mapping from the original image to the high-dimensional feature space is performed to generate a vector representing the image content. This auxiliary feature h jk captures the key visual information of the image, such as shape, color, texture, etc., and finally obtains the auxiliary feature of the input image.

[0121] Preferably, the mean value of the auxiliary features of the input image is calculated to obtain the mean value of the auxiliary features corresponding to the j-th category

[0122] In an embodiment of the present invention, by calculating the mean value of the auxiliary features of all samples in each category, assuming that for category j, there are K image samples, and their corresponding auxiliary features are h j1 , h j2 ,..., h jK , by calculating the mean value of these features, the mean value of the auxiliary features corresponding to the j-th category h j is obtained. Specifically, the mean value of the auxiliary features of category j can be expressed as: This operation is to average the features of all image samples in the category, aiming to generate an overall visual feature representing the category at the category level. This mean feature can effectively fuse the visual information of each image sample, reduce noise and improve the expression ability of the category features. The calculated mean feature h j can reflect the main visual features of this category, and finally obtain the mean value of the auxiliary features corresponding to the j-th category.

[0123] Preferably, a linear probe LP is used to initialize the weights of the mean values of the auxiliary features corresponding to each category to obtain the corresponding load feature weights And calculate the logit value corresponding to each input image sample based on the load feature weight

[0124] In the embodiment of the present invention, by introducing a Linear Probe (LP), the weight of the auxiliary feature mean corresponding to each category is initialized. Specifically, first, the auxiliary feature mean h of each category j is used as the input, and a linear probe is used for mapping to initialize the category feature weight This weight matrix includes the mean features of all categories, which is expressed as: This step maps the mean features to a new space through a linear transformation to generate the initial load feature weight for each category. Then, using these load feature weights, the auxiliary features of each image sample are weighted and summed to calculate the logit value corresponding to each image sample The calculation process is as follows: Among them, represents the logit value obtained after mapping the auxiliary features of category j through a linear probe. This logit value is used as the basis for classification decisions and can reflect the matching degree of the image in each category.

[0125] Furthermore, step S2 includes the following steps:

[0126] Step S21: Obtain the few-shot patent images corresponding to the input;

[0127] In the embodiment of the present invention, through a specific patent image acquisition module, the images from the patent database or patent storage system are screened and extracted. The construction process of the few-shot patent image set needs to be filtered according to preset conditions, such as the source, category, validity of the patent image, and image quality requirements. The input images should go through an image preprocessing module, including operations such as denoising, standardizing the size, grayscale conversion, or color space conversion, to ensure that each patent image has a unified input format. When collecting images, the number of images is strictly limited to ensure a few-shot data set is obtained, enabling subsequent feature extraction and model training to be carried out with few samples. At the same time, the patent images in the few-shot set should cover multiple categories, and the number of patent images in each category should be as balanced as possible to improve the generalization ability of the classification model, and finally obtain the corresponding few-shot patent images.

[0128] Step S22: Parallelly extract the image features of the few-shot patent images based on the hybrid pre-trained model to obtain the patent image features corresponding to different types, including patent image text embedding, patent image visual optimization representation, and patent image sample logit value;

[0129] In the embodiments of the present invention, by using a hybrid pre-trained model to extract features from few-shot patent images, the hybrid model combines multiple pre-trained networks, such as convolutional neural networks (CNNs) and vision transformers (ViTs), to utilize their advantages for parallel feature extraction. Each patent image undergoes multi-layer convolutional networks, attention mechanisms, and chunking of image regions to extract low-level and mid-level visual features of the image. On this basis, the Transformer structure is further used for high-level visual optimization representation of the image, combining the spatial information and context association of the image, thereby improving the accuracy and stability of the image representation. The process of extracting image features includes multi-stage extraction of image features, including extraction of patent image text embeddings, visual optimization representation of patent images, and calculation of logit values for patent image samples. The process of extracting patent image text embeddings is to perform OCR recognition on the text contained in the image (such as the drawing description in the patent document), and combine a specific text embedding model, such as the BERT or GPT series models, to represent the extracted text information. In this way, the text features of the patent image can be obtained. The visual optimization representation further optimizes the visual features of the image through convolutional operations of high-level neural network layers, making the details and category information of the image more accurate and improving the accuracy of image classification. The logit value of a patent image refers to the predicted value of each category obtained through model calculation, which reflects the confidence level of the model for each category. Finally, patent image features corresponding to different types are obtained, including patent image text embeddings, visual optimization representation of patent images, and logit values of patent image samples.

[0130] Step S23: Assume that the set of patent image categories corresponding to few-shot patent images is C = {C1, C2, …, C d , …, C N}, where N represents the number of image categories, and C d represents the name of the d-th category, and use the text prompt generator corresponding to the large language model combined with the Transformer-based text encoder to generate text description features for each patent image in the few-shot patent images, obtaining patent image text prompt description features.

[0131] In the embodiments of the present invention, by defining the set of categories C of few-shot patent images as C = {C1, C2, …, C d , …, C N}, where N represents the total number of image categories, and C dDenote the name of category d. For each patent image, first, combine its category attribute with the text prompt generated by the large language model to generate text information describing the features of the patent image. The text prompt generator of the large language model will automatically generate an accurate text description according to the category and content of the patent image. Each text description not only includes the basic information of the image (such as image content, patent type, etc.), but also includes the important features and uses of the image in the patent category. The generated text prompt can be encoded by a fine-tuned Transformer text encoder to obtain the text embedding features of each image. This process includes two main operations: text prompt generation and text encoding. Text prompt generation uses the large language model to generate descriptive text that matches the image based on the category information and image content on the given few-shot patent images. Next, the Transformer text encoder encodes the generated text prompt to obtain text embedding features, forming complementary text description information for each image. Finally, obtain the patent image text prompt description features.

[0132] Further, the specific process of generating the text description features in step S23 is as follows:

[0133] Use the text prompt generator LLM corresponding to the large language model to construct a text prompt template for each category name in the patent image category set to generate the text prompt T corresponding to the d-th category d = P d (C d ), where P d represents the text description prompt template used by the LLM to generate the d-th category;

[0134] In the embodiment of the present invention, by using the large language model (LLM) to construct a text prompt template for each category name in the patent image category set, the specific operation is, first, according to the name of the patent image category, design an appropriate text prompt template for each category, and generate the corresponding text description prompt T for each category name through the LLM model d , the template form is P d (C d ), where P d represents a text prompt template design function for a corresponding category, C d is the category name. The design of the text prompt template needs to be accurately matched with the features of the category name. For example, for the "optical image processing" category, the template is a language structure containing "describe a technical solution for optical image processing". When designing the template, the characteristics of the category need to be fully considered to avoid vague and unclear descriptions to ensure that the generated text prompt can effectively guide the subsequent model training and inference processes, and finally generate the text prompt corresponding to the d-th category.

[0135] Preferably, receive the text prompt T corresponding to the input through the LLM d and convert it into a word vector E = Tokenize(T d ), where Tokenize(·) represents converting the text prompt into the input embedding calculation of the LLM;

[0136] In the embodiment of the present invention, receive the text prompt T through the LLM d and convert it into a word vector E. Specifically, use the Tokenize function of the LLM model to convert the text prompt T d into a word vector E that can be understood by the model. The Tokenize function maps each word in T d to the corresponding word vector according to the vocabulary of the pre-trained model. The generated word vector E is used as the input embedding of the LLM, providing a basic representation for the subsequent encoding and generation processes. In this process, the input text prompt T d is converted into vector form word by word, ensuring that the LLM can perform effective semantic parsing on each input.

[0137] Preferably, input the word vector into the text encoder based on Transformer with m layers for encoding processing to output the context representation H at each layer position (0) = E; H (m) = TransformerLayer (m) (H (m-1) ), where H (0) represents the initial input embedding of the LLM, H (m) represents the hidden state output by the m-th layer, and TransformerLayer (m) (·) represents the calculation of the m-th Transformer layer;

[0138] In the embodiment of the present invention, by inputting the word vector into the text encoder based on Transformer for multi-level encoding processing, the input word vector E passes through multiple Transformer encoding layers and is calculated layer by layer to output the context representation at each layer position. At initialization, the word vector E directly serves as the input embedding H through the 0th layer (0) , and in each layer of Transformer, the hidden state will be adjusted and updated according to the self-attention mechanism. In the formula, H (m) represents the hidden state output by the m-th layer, where m represents the number of layers. In each layer of Transformer, the current hidden state H (m-1) will be calculated, and through the action of the multi-head self-attention mechanism and the feed-forward neural network, a updated representation H (m) will be generated until the output H (L), which contains deep context information and can reflect the semantic information of the input text prompt T d and its interrelationships under different categories.

[0139] Preferably, the text decoder is used to gradually generate the corresponding text description sequence Y = {y1, y2, …, y u} for each text prompt embedding X corresponding to the input patent image, where u is the length of the generated description, and the probability P(y h |y1, …, y h-1 , X) = Softmax(W y ·H (L) ) is calculated by quantization, where W y is the weight matrix of the output layer, H (L) is the output representation of the last layer of the Transformer, and Softmax(·) is the Softmax function, which is used to map the output to a probability distribution to output the corresponding text description, including category-based text descriptions, category- and similar-category-based text descriptions, category-synonym-based text descriptions, and category-attribute-based text descriptions, to obtain the patent image text prompt description features;

[0140] In the embodiments of the present invention, the text decoder gradually generates the text description sequence Y = {y1, y2, …, y u} corresponding to each patent image according to the input text prompt embedding x. In the specific implementation process, the final output H (L) after being processed by the Transformer encoding layer is input into the text decoder. The decoder generates a text description according to the current word vector embedding X, and uses the Softmax function to calculate the generation probability P(y h |y1, …, y h-1 , X) = Softmax(W y ·H (L) ), where W y represents the weight matrix of the output layer, H (L) is the output representation of the last layer. H (L) is mapped to a probability distribution through Softmax to predict the next word. This process continues until the complete text description sequence is generated. This sequence includes different types of texts such as category-based descriptions, category- and similar-category descriptions, category-synonym-based descriptions, and category-attribute-based descriptions. The generation process completes the description by gradually generating each word, and the output of each step is predicted based on the previously generated words and context information to ensure that the generated text has logical coherence and accuracy. The generated text description features can be specifically divided into four categories: category-based text description YC 1. Text description Y based on categories and similar categories CS 2. Text description Y based on category synonyms ST 3. Text description Y based on category attributes CA Among these four types of descriptions, the generation of each type depends on different text prompt templates P C P CS P ST and P CA . The LLM parses different text prompt templates to generate different patent image text descriptions, which can effectively represent the patent images from different dimensions and levels, improve the performance of subsequent classification and retrieval, and finally obtain the patent image text prompt description features

[0141] Among them, the generated patent image text prompt description features are specifically:

[0142] Y C = GPT(C, P C ); Y CS = GPT(C, P CS ); Y ST = GPT(C, P ST ); Y CA = GPT(C, P CA );

[0143] Among them, Y C , Y CS , Y ST and Y CA respectively represent the text description based on categories, the text description based on categories and similar categories, the text description based on category synonyms, and the text description based on category attributes. P C , P CS , P ST and P CA respectively represent the text prompt templates for generating the above four types of text descriptions. GPT(·) represents the Softmax probability distribution calculation corresponding to the LLM

[0144] Furthermore, step S4 includes the following steps:

[0145] Step S41: Calculate the contrast learning feature fusion loss for the contrast learning between the language branch and the visual branch in the hybrid pre-trained model by designing a fusion loss function between the corresponding language branch and the visual branch during the multi-modal fusion of patent images, so as to obtain the contrast learning loss L VL ;

[0146] In the embodiments of the present invention, in the process of multi-modal fusion of patent images, a fusion loss function between the language branch and the visual branch is first designed. Specifically, the fusion loss function between the language branch and the visual branch is where G represents the total number of samples, represents the logit value of the g-th sample output by the language branch and the visual branch, and τ represents the embedding coefficient, which is used to calculate the contrastive learning loss L between the language branch and the visual branch in the hybrid pre-trained model VL . Specifically, using the contrastive learning strategy, by embedding the visual information of the patent image and the corresponding text description (language information), a pre-trained model with a dual-branch structure is constructed. In this process, the visual branch first extracts image features, and the language branch extracts language features from the text description. To optimize the cooperation between the two, a fusion loss function is designed. This loss function is optimized according to the similarity of the two modalities, so that the embedding distance between the image and text pairs of the same category is as close as possible, and the distance between different categories is as far as possible. Through this way, contrastive learning helps the model capture the deep semantic connection between vision and language, so as to achieve effective feature fusion and accurate classification prediction. In addition, the specific calculation process of the logit value of the g-th sample corresponding to the language branch and the visual branch is t C =T(Y C ); t CS =T(Y CS ); t SY =T(Y SY ); t CA =T(Y CA ); where T(·) represents the calculation of the model's language branch, and t C , t CS , t ST and t CA represent the text embeddings of four types of text descriptions respectively, and a concatenation operation is performed on them t = [t1, t2,..., t i ,..., t N , where t i represents the embedding after concatenating the embeddings of four types of text descriptions of the i-th image category; t represents the text embeddings of all image categories; represents the embedding of the k-th type of text description of the i-th image category; represents the embedding obtained from the j-th piece of text of the k-th type of text description of the i-th image category; O k,i represents the number of the k-th type of text description of the i-th image category, Among them, v represents the image representation obtained by the visual branch; represents the cosine similarity calculated from the embedding of the j-th text description of the i-th image category in the k-th text description; represents the maximum cosine similarity calculated from the text embedding of the i-th image category in the k-th text description. Finally, the contrastive learning loss L between the language branch and the visual branch is calculated VL .

[0147] Step S42: Calculate the auxiliary feature fusion loss of the visual auxiliary branch in the hybrid pre-trained model by designing a corresponding visual auxiliary loss function during the patent image multi-modal fusion process to obtain the visual auxiliary branch loss L LP ;

[0148] Among them, the visual auxiliary loss function is specifically:

[0149]

[0150] Among them, G q represents the label of the q-th sample, represents the logit value corresponding to the q-th sample The class prediction probability obtained after passing through the softmax function, represents the logit value of the q-th sample on class n;

[0151] In the embodiment of the present invention, by designing a visual auxiliary loss function during the multi-modal fusion process, the learning ability of the visual auxiliary branch is further enhanced. The visual auxiliary branch processes the visual information of the patent image to assist in optimizing the accuracy of image classification. Specifically, the loss function is defined as:

[0152]

[0153] Among them, G q is the label of the q-th sample, is the logit value of this sample The corresponding class probability is obtained after passing through the Softmax transformation. The Softmax function converts the original logit value into a probability distribution, enabling the model to effectively distinguish between different classes. During the training process, for each sample, the logit value on each class is calculated, and the class prediction probability distribution is obtained through the Softmax function, thereby calculating the loss value for each class. This loss function is designed to optimize the learning of the visual branch for patent image features, enabling the model to accurately predict the class of the image based on visual features, and finally obtaining the visual auxiliary branch loss L LP .

[0154] Step S43: Calculate the total loss based on the contrastive learning loss between the language branch and the visual branch and the visual auxiliary branch loss to obtain the total loss L of the hybrid model total = βL VL + (1 - β)L LP , where β is a hyperparameter;

[0155] In the embodiments of the present invention, based on steps S41 and S42, by combining the contrastive learning loss between the language branch and the visual branch and the loss of the visual auxiliary branch to calculate the total loss, the total loss of the hybrid model is obtained. Specifically, the total loss of the hybrid model is: L total = βL VL + (1 - β)L LP , where β is a hyperparameter used to control the weight ratio between the language branch and the visual auxiliary branch. By adjusting β, according to the requirements of the task, the key training direction of the model can be adjusted. If the language information of the patent image is more critical, β can be increased to strengthen the influence of the language branch; conversely, β can be decreased to make the visual information play a greater role in classification. The key to this step is to comprehensively consider the fusion of visual features and language features through the weighted combination of loss functions, optimize the overall performance of the model, improve the effect of few-shot classification of patent images, and finally obtain the total loss of the hybrid model.

[0156] Step S44: Based on the total loss of the hybrid model, perform model fusion optimization on the hybrid pre-training model corresponding to the multi-modal fusion process of the patent image, output the corresponding multi-modal fusion features of the patent image samples, and construct a few-shot classification framework for the patent image to perform the corresponding few-shot classification work of the patent image.

[0157] In the embodiments of the present invention, based on the calculated total loss, the hybrid pre-training model is optimized to enable it to better process the multi-modal features of the patent image. Specifically, the gradient of the loss function with respect to the model parameters is calculated through the backpropagation algorithm, and then the weights of the language branch, visual branch, and visual auxiliary branch in the hybrid model are updated. The optimized model will output the multi-modal fusion features of the patent image samples, which integrate visual and language information and help improve the accuracy in the few-shot classification task. During this process, through multiple rounds of training and validation, the generalization ability of the model for few-shot data is gradually improved, thereby constructing a few-shot classification framework for the patent image. During the classification process, a small number of labeled samples are used for training, and the classification accuracy of the model in the few-shot scenario is effectively improved through methods such as reinforcement learning or transfer learning. The model learns the semantic relationships between different modalities to achieve efficient patent image classification. Especially in the few-shot scenario, it can still show good performance and finally perform the corresponding few-shot classification work of the patent image.

[0158] Therefore, from any perspective, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, it is intended to encompass all changes that fall within the meaning and scope of the equivalent elements of the application documents within the present invention.

[0159] The above are only specific embodiments of the present invention, enabling those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but rather to the broadest scope consistent with the principles and novel features invented herein.

Claims

1. A few-shot classification method for patent images based on multi-modal representation fusion, characterized in that, Including the following steps: Step S1: Construct a hybrid pre-training model by designing a language branch, a visual branch, and a visual assistance branch; Step S2: Obtain few-shot patent images corresponding to the input, and parallelly extract image features from the few-shot patent images based on the hybrid pre-training model to obtain patent image features corresponding to different types; Generate text description features for each patent image in the few-shot patent images based on the text prompt generator corresponding to the large language model and in combination with the Transformer-based text encoder to obtain patent image text prompt description features; Step S3: Design a cross-modal fusion layer by adopting an attention mechanism among the model branches in the hybrid pre-training model, and use the cross-modal fusion layer to perform multi-modal representation fusion on the patent image features corresponding to different types and the patent image text prompt description features to generate a corresponding patent image multi-modal fusion process; Step S4: Calculate the feature fusion loss of each model branch in the hybrid pre-training model by designing a multi-level fusion loss function in the patent image multi-modal fusion process to obtain the contrast learning loss between the language branch and the visual branch and the visual assistance branch loss; Optimize the model fusion of the hybrid pre-training model corresponding to the patent image multi-modal fusion process based on the contrast learning loss between the language branch and the visual branch and the visual assistance branch loss, output the corresponding multi-modal fusion features of the patent image samples, and construct a few-shot patent image classification framework to perform the corresponding few-shot patent image classification work.

2. The method for few-shot classification of patent images based on multimodal representation fusion according to claim 1, wherein, Step S1 includes the following steps: Step S11: Design and construct a language branch by adopting the text encoder corresponding to CLIP to convert the corresponding input image samples into text embeddings with deep semantic information; Step S12: Design and construct a visual branch by consisting of a pre-trained visual feature extractor DINO, an auxiliary feature attention mechanism AFA, and an image encoder corresponding to CLIP, so as to extract visual features from the corresponding input image samples through the pre-trained visual feature extractor to generate visual representations for AFA, and optimize and output the corresponding image visual optimized representations through AFA in combination with the image encoder; Step S13: Design and construct a visual assistance branch by consisting of a pre-trained visual feature extraction model and a linear probe LP to balance the output results of the contrast learning corresponding language branch and visual branch, select auxiliary features, and initialize the weights of LP using the feature means corresponding to each category of the output results to obtain the logit value corresponding to each input image sample; Step S14: Fuse the language branch, the visual branch, and the visual assistance branch to construct a hybrid pre-training model.

3. The method for few-shot classification of patent images based on multi-modal representation fusion according to claim 2, wherein The language branch described in Step S11 is specifically: Tokenize the corresponding input image sample: T = [t1, t2, …, t i , where T represents the text corresponding to the input image sample, and t i represents the i-th word in the text; Embed each word t i to obtain the text word embedding vector e i = W text [t i , where W text represents the word embedding matrix; The position encoding process is performed on the text word embedding vector using the text encoder corresponding to CLIP to obtain the word vector x corresponding to the position encoding i = e i + p i , p i = PE(i), where PE(·) represents the calculation of the position encoding, and p i represents the position encoding; Use a Transformer encoder to perform additional context-aware encoding on each word vector corresponding to the positional encoding to obtain the additional context-aware representation ct corresponding to each word vector L = Transformer L ({x1, x2, …, x n}), where n represents the number of word vectors, L represents the number of layers of the Transformer encoder, and Transformer L (·) represents the calculation of the last Transformer layer; Based on the additional context-aware representation ct L Perform CLS-aware marking on each word vector corresponding to the positional encoding to obtain the text word vector corresponding to the CLS-aware marking Perform text embedding processing on the word vector corresponding to the CLS-aware marking to obtain the text embedding t corresponding to the input image sample e = Normalize(W text_proj t v ), where W text_proj is the projection matrix, and Normalize(·) represents the calculation of L2 normalization on the text word vector 4. The method for few-shot classification of patent images based on multimodal representation fusion according to claim 2, wherein, The visual branch described in Step S12 is specifically: The visual representation of the input image sample is extracted by using a pre-trained visual feature extractor to obtain the visual representation matrix c = F(img) corresponding to the input image sample, where img represents the input image sample and F(·) represents the calculation of the visual feature extractor; Perform normalized similarity calculation on the visual representation matrix c corresponding to the input image sample to obtain the normalized similarity matrix Among them, cc T represents the outer product between the visual representation matrix and the corresponding transposed visual representation matrix, which is used to represent the similarity between visual blocks. α represents the scaling parameter, λ represents the translation parameter, l represents the length of the median embedding in the CLIP image encoder, and ∑ a,b [cc T a,b represents the sum of each similarity element;​ For each element S in the similarity matrix ab perform a masking operation to obtain the corresponding element in the similarity masking matrix D It is expressed that when the similarity score S ab is non - negative, the value at the corresponding position in the similarity masking matrix D is zero; when the similarity score S ab is negative, the value at the corresponding position in the similarity masking matrix D is negative infinity; Perform the attention calculation corresponding to AFA on the normalized similarity matrix S and the similarity masking matrix D to obtain the attention matrix W AFA = SoftMax(S + D); Output the corresponding visual representation z through the CLIP image encoder, and perform visual optimization processing on the visual representation z output by the CLIP image encoder based on the attention matrix to output the corresponding image visual optimization representation v = Proj(W AFA ·z), where Proj(·) represents the calculation of the projection layer.

5. The method for few-shot classification of patent images based on multi-modal representation fusion according to claim 2, wherein, Specifically, the visual assistance branch described in step S13 is as follows: By assuming that there are M categories and K samples of the corresponding input image samples, and using the visual feature extraction model to extract the auxiliary visual features of the input image samples, the auxiliary features of the input image are obtained where x jk represents the k-th image of the j-th category, represents the frozen weight parameters, f(·) represents the calculation of the visual feature extraction model, and h jk represents the auxiliary feature of the k-th image of the j-th category; Calculate the mean of the auxiliary features of the input image to obtain the mean of the auxiliary features corresponding to the j-th category Initialize the weights of the auxiliary feature means corresponding to each category using the linear probe LP to obtain the corresponding load feature weights And calculate the logit value corresponding to each input image sample based on the load feature weights 6. The method for few-shot classification of patent images based on multimodal representation fusion according to claim 5, wherein Specifically, the calculation of the visual feature extraction model is as follows: where M is the number of classes corresponding to the input image samples, and x jk represents the k-th image of the j-th class, is the frozen weight parameter.

7. The method for few-shot classification of patent images based on multimodal representation fusion according to claim 1, characterized in that Step S2 includes the following steps: Step S21: Obtain the few-shot patent images corresponding to the input; Step S22: Parallelly extract the image features of the few-shot patent images based on the hybrid pre-trained model to obtain the patent image features corresponding to different types, including patent image text embeddings, visually optimized representations of patent images, and patent image sample logits; Step S23: The set of patent image categories corresponding to the few-shot assumption of patent images is \(C = \{C_1, C_2, \ldots, C\) d , \ldots, C\) N \}, where \(N\) represents the number of image categories, and \(C\) d represents the name of the \(d\)-th category. Then, use the text prompt generator corresponding to the large language model and combine it with the Transformer-based text encoder to generate text description features for each patent image in the few-shot patent images, obtaining the text prompt description features of the patent images.

8. The method for few-shot classification of patent images based on multimodal representation fusion according to claim 7, wherein Specifically, the process of generating the text description features in step S23 is as follows: Use the text prompt generator LLM corresponding to the large language model to construct a text prompt template for each category name in the patent image category set, so as to generate the text prompt T corresponding to the d-th category d = P d (C d ), where P d represents the text description prompt template used by the LLM to generate the d-th category; Receive the corresponding text prompt T for the input through the LLM d and convert it into a word vector E = Tokenize(T d ), where Tokenize(·) represents converting the text prompt into the input embedding calculation of the LLM; The word vectors are input into a Transformer-based text encoder with m layers for encoding processing to output the context representation H at each layer position (0) = E; H (m) = TransformerLayer (m) (H (m-1) ), where H (0) represents the initial input embedding of the LLM, and H (m) represents the hidden state output by the m-th layer, and TransformerLayer (m) (·) represents the calculation of the m-th Transformer layer; Using a text decoder, gradually generate the corresponding text description sequence Y = {y1, y2, …, y u} for each text prompt embedding X corresponding to the input patent image, where u is the length of the generated description, and the probability P(y h |y1, …, y h-1 , X) = Softmax(W y ·H (L) ) is calculated by quantization, where W y is the weight matrix of the output layer, H (L) is the output representation of the last layer of the Transformer, and Softmax(·) is the Softmax function, which is used to map the output to a probability distribution to output the corresponding text description, including the text description based on categories, the text description based on categories and similar categories, the text description based on category synonyms, and the text description based on category attributes, so as to obtain the text prompt description features of the patent image; Specifically, the generated patent image text prompt description features are as follows: Y C = GPT(C, P C ); Y CS = GPT(C, P CS ); Y ST = GPT(C, P ST ); Y CA = GPT(C, P CA ); Among them, Y C , Y CS , Y ST and Y CA respectively represent text descriptions based on categories, text descriptions based on categories and similar categories, text descriptions based on category synonyms, and text descriptions based on category attributes. P C , P CS , P ST and P CA respectively represent text prompt templates for generating the above four types of text descriptions. GPT(·) represents the Softmax probability distribution calculation corresponding to the LLM.

9. The method for few-shot classification of patent images based on multimodal representation fusion according to claim 1, wherein Step S4 includes the following steps: Step S41: Calculate the contrastive learning feature fusion loss for the contrastive learning between the language branch and the visual branch in the hybrid pre-trained model by designing a fusion loss function between the corresponding language branch and the visual branch during the patent image multi-modal fusion process, so as to obtain the contrastive learning loss L between the language branch and the visual branch VL ; Step S42: Calculate the auxiliary feature fusion loss of the visual auxiliary branch in the hybrid pre-training model by designing a corresponding visual auxiliary loss function during the multi-modal fusion of patent images, so as to obtain the visual auxiliary branch loss L LP ; Specifically, the visual assistance loss function is as follows: Among them, G q represents the label of the q-th sample, represents the logit value corresponding to the q-th sample which is the class prediction probability obtained after passing through the softmax function, represents the logit value of the q-th sample on class n; Step S43: Calculate the total loss according to the contrastive learning loss between the language branch and the visual branch and the visual auxiliary branch loss to obtain the total loss \(L\) of the hybrid model total =\(\beta L\) VL +(1 - \(\beta\))\(L\) LP , where \(\beta\) is a hyperparameter; Step S44: Optimize the model fusion of the hybrid pre-trained model corresponding to the multi-modal fusion process of the patent images based on the total loss of the hybrid model, output the corresponding multi-modal fusion features of the patent image samples, and construct a few-shot patent image classification framework to perform the corresponding few-shot patent image classification work.

10. The method for few-shot classification of patent images based on multi-modal representation fusion according to claim 9, wherein Specifically, the fusion loss function between the language branch and the visual branch in step S41 is as follows: Among them, G represents the total number of samples, represents the logit value of the g-th sample output by the language branch and the visual branch, and τ represents the embedding coefficient.

Images