Image classification method and system based on collaboration of small model and multi-modal large image recognition model

Abstract

The invention discloses an image classification method and system based on cooperation of a small model and a multi-modal large image recognition model, and the method comprises the steps: inputting a to-be-classified image into a small image recognition model, and outputting a rough image classification result; matching a preset prompt lexicon according to a rough classification result; if the matching fails, outputting that the picture category is other, and if the matching succeeds, calling a multi-modal large-recognition picture model for processing; inputting the to-be-classified image and a preset prompt word bank matched with the to-be-classified image into the multi-modal large image recognition model to obtain a fine image classification result; the multi-modal large-recognition graph model comprises four composition modules: a visual encoder, a language model, a visual language fusion device and a result output module. And fusing the image rough classification result and the image fine classification result to obtain a final fine-grained classification result of the image.

Description

Technical Field

[0001] The present invention relates to the field of image detection technology classification, and specifically to an image classification method and system based on the collaboration of a small model and a multimodal large image recognition model. Background Art

[0002] In many cases of image processing, a single algorithm cannot effectively process all image types. Therefore, faced with a wide range of image types, the optimal approach is to first classify the images and then apply algorithms tailored to the specific image type. Existing technologies often rely on single models (such as CNNs and SVMs) for end-to-end classification, requiring a predefined closed set of categories and struggling to meet the fine-grained classification requirements of open-world scenarios. Existing technologies often rely on single models (such as CNNs and SVMs) for end-to-end classification, requiring a predefined closed set of categories and struggling to meet the fine-grained classification requirements of open-world scenarios. The multimodal large-scale image recognition model Qwen2.5-VL leverages its large model understanding capabilities to automatically extract semantic meaning, making it more robust to ambiguous descriptions. Furthermore, similarity calculations based on models like clip can overlook complex semantic relationships between text and image features. Furthermore, adding new categories to the clip model requires redesigning and encoding all text prompts, making real-time adaptation difficult. Qwen-VL, on the other hand, flexibly supports new categories by simply adjusting the input prompt. Multimodal large-scale image recognition models such as Qwen2.5-vl support open-ended question-answering classification. However, their direct processing of fine-grained tasks is inefficient, consumes large amounts of computational resources, and lacks domain-specific optimization strategies, resulting in insufficient classification accuracy. Effective image recognition cannot distinguish fine-grained categories based solely on image features. For example, it is difficult to distinguish between contract documents and legal documents in document images based solely on image features. Summary of the Invention

[0003] In order to overcome the shortcomings of the above-mentioned existing technologies of low efficiency and high computing resource consumption of multimodal large image recognition models in directly processing fine-grained image classification tasks, the present invention provides an image classification method and system based on the collaboration of small models and multimodal large image recognition models. Pre-screening by small models can avoid frequent calls to large models, and the design of precise prompt words can improve the convergence effect of large models and reduce computing costs. By simply adding prompt words, it is possible to flexibly respond to new categories in the open world and improve adaptability to open scenes.

[0004] According to one aspect of the present invention, a method for image classification based on collaboration between a small model and a multimodal large image recognition model is provided, comprising:

[0005] Input the image to be classified into the small image recognition model and output the rough classification result of the image; the model main network of the small image recognition model adopts MobileNet, and the output layer includes a global average pooling layer, a fully connected layer and an activation function layer;

[0006] Match the preset prompt word library based on the coarse classification results: If the match fails, the image category is output as "other". If the match succeeds, the multimodal large image recognition model is called for processing;

[0007] The image to be classified and its matching preset prompt word library are input into the multimodal large image recognition model to obtain the image sub-classification result; the multimodal large image recognition model comprises four components: a visual encoder for extracting image visual features, a language model for extracting text features from the preset prompt word library, a visual language fusion module for fusing visual features and text features, and a result output module for outputting the image sub-classification result based on the fused features;

[0008] The coarse classification results of the image are combined with the fine classification results of the image to obtain the final fine-grained classification results of the image.

[0009] As a further implementation plan, the steps for the small image recognition model to process the image to be classified are:

[0010] The image to be classified is input into MobileNet to extract high-dimensional features of the image. Global average pooling is used to flatten the channel dimension of the high-dimensional features of the image into a vector. The fully connected layer sets the number of output feature channels corresponding to the number of categories of the predicted coarse classification label of the image and obtains the channel score. Finally, the activation function layer is input to normalize the channel score and output the coarse classification label of the image in the form of probability. The coarse classification result of the image is output according to the preset highest-scoring coarse classification label probability threshold.

[0011] As a further implementation plan, MobileNet consists of seven stages. The first stage uses a convolutional layer to extract preliminary features of the image. The second to seventh stages are stacked inverted residual modules. By adjusting the number of channels and resolution, features are extracted stage by stage, and finally high-dimensional features of the image are output.

[0012] As a further implementation scheme, a preset prompt word library is set corresponding to the document coarse classification label, wherein the picture coarse classification labels include but are not limited to: books, test papers, other documents, posters and backgrounds. Except for the coarse classification label of background, each document coarse classification label is set with a preset prompt word library.

[0013] As a further implementation scheme, the setting process of the preset prompt word library is as follows:

[0014] For each coarse classification label of a document, a preset number of fine classification labels are determined. Prompt words corresponding to the fine classification labels are designed from four dimensions: design purpose, visual style, core content, and communication scenario. The prompt words of all fine classification labels are integrated to obtain a preset prompt word library corresponding to the coarse classification label of the document.

[0015] As a further implementation scheme, the visual encoder is a ViT model, including CNN feature extraction and visual feature encoder, and the working steps include:

[0016] First, the image to be classified is preprocessed, and then the block size of non-overlapping blocks is set to divide the original image into image blocks;

[0017] Each image block is linearly projected into the embedding space through CNN feature extraction to obtain visual labels;

[0018] The input visual sequence is constructed based on the visual markers, and the input visual sequence passes through the visual feature encoder of the ViT architecture to output visual features.

[0019] As a further implementation plan, the input of the language model is a preset prompt vocabulary. Tags are added to the prompt words in the preset prompt vocabulary to construct a dialogue structure. Then, a word segmenter is used to process the prompt words to form text tags. The text tags are mapped to the embedding space, and the obtained word vectors are used as text features.

[0020] As a further implementation plan, the visual language fusion device uses two multi-layer perceptrons to reduce the dimensionality of visual features until they are aligned with the dimensions of text features, processes visual and text features through a multi-head attention mechanism, and outputs the fused features.

[0021] According to another aspect of the present specification, there is provided an image classification system based on the collaboration of a small model and a multimodal large image recognition model, characterized by comprising:

[0022] The image coarse classification module is used to input the image to be classified into the small image recognition model and output the image coarse classification result; the model main network of the small image recognition model adopts MobileNet, and the output layer includes a global average pooling layer, a fully connected layer, and an activation function layer;

[0023] The preset prompt word library matching module is used to match the preset prompt word library according to the rough classification results: if the match fails, the image category is output as "other"; if the match succeeds, the multimodal large image recognition model is called for processing;

[0024] The image sub-classification module inputs the image to be classified and its matching preset prompt word library into the multimodal large image recognition model to obtain the image sub-classification results. The multimodal large image recognition model includes four components: a visual encoder for extracting image visual features, a language model for extracting text features from the preset prompt word library, a visual language fusion module for fusing visual features and text features, and a result output module that outputs the image sub-classification results based on the fused features.

[0025] The result output module is used to fuse the coarse classification results of the image with the fine classification results of the image to obtain the final fine-grained classification results of the image.

[0026] According to another aspect of the present specification, an electronic device is provided, comprising a memory and a processor, wherein the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute an image classification method based on the collaboration of a small model and a multimodal large image recognition model.

[0027] Compared with the existing technology, the beneficial effects of the present invention are: the present invention performs image classification based on the collaboration of large and small models, pre-screening with the small model can avoid frequent calls to the large model, and the design of precise prompt words improves the convergence effect of the large model and reduces computing costs. Only by adding prompt words, it can flexibly respond to new categories in the open world and improve adaptability to open scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction will be given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 A flowchart of an image classification method based on the collaboration of a small model and a multimodal large image recognition model provided by an embodiment of the present invention;

[0030] Figure 2 This is a flowchart of an example of an image classification method based on the collaboration of a small model and a multimodal large image recognition model according to an embodiment of the present invention;

[0031] Figure 3 Schematic diagram of the structure of an image classification system based on the collaboration of a small model and a multimodal large image recognition model in an embodiment of the present invention;

[0032] Figure 4 A schematic diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] It should be noted that:

[0034] The terms "including" and "having" and any variations thereof in the description and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions, for example, a process, method, system, product or apparatus that includes a series of steps or units is not necessarily limited to the steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0035] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily need to be executed in the order described. For example, some operations / steps may be further decomposed, while others may be combined or partially combined, so the actual execution order may vary depending on the actual situation.

[0036] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a new technical solution. This combination is not restricted by the sequence of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0037] like Figure 1 As shown, Figure 1 This is a flowchart of an image classification method based on the collaboration of a small model and a multimodal large image recognition model in an embodiment of the present invention, the method comprising:

[0038] Provides an image classification method based on the collaboration of a small model and a multimodal large image recognition model, including:

[0039] Input the image to be classified into the small image recognition model and output the rough classification result of the image; the model main network of the small image recognition model adopts MobileNet, and the output layer includes a global average pooling layer, a fully connected layer and an activation function layer;

[0040] Match the preset prompt word library based on the coarse classification results: If the match fails, the image category is output as "other". If the match succeeds, the multimodal large image recognition model is called for processing;

[0041] The image to be classified and its matching preset prompt word library are input into the multimodal large image recognition model to obtain the image sub-classification result; the multimodal large image recognition model comprises four components: a visual encoder for extracting image visual features, a language model for extracting text features from the preset prompt word library, a visual language fusion module for fusing visual features and text features, and a result output module for outputting the image sub-classification result based on the fused features;

[0042] The coarse classification results of the image are combined with the fine classification results of the image to obtain the final fine-grained classification results of the image.

[0043] Furthermore, the steps for the small image recognition model to process the image to be classified are:

[0044] The image to be classified is input into MobileNet to extract high-dimensional features of the image. Global average pooling is used to flatten the channel dimension of the high-dimensional features of the image into a vector. The fully connected layer sets the number of output feature channels corresponding to the number of categories of the predicted coarse classification label of the image and obtains the channel score. Finally, the activation function layer is input to normalize the channel score and output the coarse classification label of the image in the form of probability. The coarse classification result of the image is output according to the preset highest-scoring coarse classification label probability threshold.

[0045] Furthermore, MobileNet consists of seven stages. The first stage uses a convolutional layer to extract preliminary features of the image. The second to seventh stages are stacked inverted residual modules. By adjusting the number of channels and resolution, features are extracted stage by stage, and finally high-dimensional features of the image are output.

[0046] Furthermore, the preset prompt word library corresponds to the document coarse classification label setting, wherein the image coarse classification labels include but are not limited to: books, test papers, other documents, posters and backgrounds. Except for the coarse classification label of background, each document coarse classification label is set with a preset prompt word library.

[0047] Furthermore, the setting process of the preset prompt word library is as follows:

[0048] For each coarse classification label of a document, a preset number of fine classification labels are determined. Prompt words corresponding to the fine classification labels are designed from four dimensions: design purpose, visual style, core content, and communication scenario. The prompt words of all fine classification labels are integrated to obtain a preset prompt word library corresponding to the coarse classification label of the document.

[0049] Furthermore, the visual encoder is a ViT model, which includes CNN feature extraction and visual feature encoder. The working steps include:

[0050] First, the image to be classified is preprocessed, and then the block size of non-overlapping blocks is set to divide the original image into image blocks;

[0051] Each image block is linearly projected into the embedding space through CNN feature extraction to obtain visual labels;

[0052] The input visual sequence is constructed based on the visual markers, and the input visual sequence passes through the visual feature encoder of the ViT architecture to output visual features.

[0053] Furthermore, the input of the language model is a preset prompt word library. Tags are added to the prompt words in the preset prompt word library to build a dialogue structure, and then the word segmenter is used to process them to form text tags. The text tags are mapped to the embedding space, and the obtained word vectors are used as text features.

[0054] Furthermore, the visual language fusion device uses two multi-layer perceptrons to reduce the dimensionality of visual features until they are aligned with the dimensions of text features, processes visual and text features through a multi-head attention mechanism, and outputs the fused features.

[0055] Specifically, the result output module uses the fused features as input and gradually generates text through an autoregressive transformer. Each step calculates the output based on the current context and visual features. The final output is the output of the large language model, which includes the confidence level of the current category and an explanation of the judgment. The autoregressive mechanism is that the model generates a probability distribution for the current word based on previously generated results and selects candidate words through search.

[0056] Specifically, the logic for obtaining the final image classification result from the coarse and fine classification results is as follows: The multimodal large image recognition model is invoked to identify the fine-grained labels only when the coarse label confidence (coarse classification label probability) output by the small image recognition model exceeds a certain threshold. For small model label confidences less than or equal to the threshold, the sub-label confidence is set to 0. Finally, all small model confidences are multiplied by their sub-label confidences. The maximum value is the current image category.

[0057] As a preferred implementation, if the confidence of the "other" label is greater than the threshold and the other labels are less than the threshold, the current image label is considered to be "other". If the confidence of the "other" label is greater than the threshold and the other labels are also greater than the threshold, then when performing the threshold size comparison, p5 needs to be multiplied by a weight, which is set to 0.9 in this article.

[0058] This paper proposes an image classification method based on the collaboration of a small model and a multimodal large image recognition model. This approach employs a hierarchical collaborative architecture of small and large models. The small model is responsible for coarse-grained classification (e.g., "document," "certificate," "portrait," and "animal"), while the large model focuses on fine-grained recognition (e.g., "contract," "legal document," and "rules and regulations" within "document"). Different prompt words are then used to answer questions based on different fine-grained labels. For example, a document-related prompt word reads: "To determine whether this image is a contract, the following characteristics must be met: 1) It contains signature fields for both parties; 2) It has a clause number; and 3) The title contains the word 'contract.' Please answer 'yes' or 'no.'" Finally, the outputs of the small and large models are weightedly fused, and low-confidence results are filtered using a threshold.

[0059] like Figure 2 The figure shows an example of an image classification method based on the collaboration of a small model and a multimodal large image recognition model. The system architecture of this method includes image input, a small image recognition model (MobileNetV2), a preset prompt word library, a multimodal large image recognition model (qwen2.5-vl), and a result fusion module. The large model performs semantic analysis based on the prompt word and returns a binary classification result (yes / no) and a confidence score. The result fusion module combines the small model confidence with the large model result to generate the final classification label. The entire framework flow chart is shown below. It involves six steps:

[0060] Step 1: Determine five coarse label categories: books, test papers, other documents, posters, and others, where "others" is the background category.

[0061] Step 2: For each coarse category, determine three fine-grained labels, specifically:

[0062] Books (textbooks, novels, professional books);

[0063] Examination papers (Chinese, English, and mathematics);

[0064] Other documents (contracts, government documents, regulations);

[0065] Posters (commercial advertising posters, movie / entertainment posters, public service posters);

[0066] Others (no fine-grained labels, image content extraction possible).

[0067] Step 3: Training the Small Image Recognition Model: First, a large number of relevant images are collected, then the image data is annotated to construct a corresponding training set. The model uses a single-label recognition framework. Images are first scaled to the MobileNetV2 input size (e.g., 224x224) and converted to RGB three-channel format. Pixel values ​​are then normalized (e.g., mean [0.485, 0.456, 0.406], standard deviation [0.229, 0.224, 0.225]). The main network of the model uses a lightweight small MobileNetV2 network. The output layer is a global average pooling followed by a fully connected layer. MobileNetV2 consists of a series of stacked Bottleneck Residual Blocks and is divided into seven stages. Each stage progressively extracts features by adjusting the number of channels and resolution. The entire network is end-to-end. The network structure is shown below. The model inputs a 224x224x3 image and uses MobileNetV2 to obtain image features F, with feature dimensions of 1x1280. The model result R is then output through a fully connected layer, with dimensions of 1xm. The fully connected layer has 1280 input feature channels and m output feature channels. m here is set based on the specific model's predicted category. In this paper, the predicted category is 5, so m is set to 5. Finally, R is normalized using the sigmoid function, resulting in a value in the range [0, 1]. Five probability values ​​representing the corresponding labels are obtained: p1, p2, p3, p4, and p5. Each value represents the confidence level of the five current labels: book, test paper, other document, poster, and other. During training, the loss function uses binary cross entropy (BCE). During the inference phase, the predicted image type is the label with the highest confidence level, and this value must be greater than or equal to a threshold. In this paper, the threshold is 0.6.

[0068] Step 4: Preset prompt words. Taking the fine-grained classification of posters as an example, we first need to know the characteristics of commercial advertisements, movie posters, and public welfare posters, as shown in Table 1. Prompt words are designed based on these four dimensions, and finally a preset prompt word library is formed.

[0069] Table 1. Dimensions of prompt word design Dimensions Commercial advertising posters Movie / Entertainment Posters Public welfare posters Design purpose Promote products / services and boost consumption Promote film and television works and attract audiences Spread the concept of public welfare and call for social attention Visual style Bright colors, strong sense of design, focus on attractiveness Strong visual impact, focusing on narrative and atmosphere Concise and clear, focusing on information transmission and emotional resonance Core Content Product / service information, promotional information, brand logo The protagonist's image, the title of the work, the release date, and the slogan Public welfare themed text, call to action slogans, and relevant data Communication scenarios Shopping malls, supermarkets, e-commerce platforms, social media Cinemas, ticketing websites, social media Schools, communities, charity events, and social media

[0070] Take the poster-fine-grained classification prompt word example:

[0071] The specific process of determining whether it is a commercial advertising poster prompt is as follows:

[0072] [Context] You are an experienced poster designer, skilled at creating and analyzing various types of poster images. Now, you need help determining whether an image is a commercial advertising poster image.

[0073] [Task Objective] Analysis of the characteristics of commercial advertising poster images:

[0074] Product display: Is the product clearly displayed in the image? Is the product in a prominent position? Is there any promotional information (such as price, discount, purchase link, etc.)

[0075] Background design: Is the background simple? Does it have a brand logo, slogan, or other promotional elements (such as a QR code, contact information).

[0076] Composition and angle: Whether the product is displayed at an eye-catching angle and whether the composition highlights the characteristics of the product.

[0077] Color and style: whether the colors are bright, whether there is a clear main brand color, and whether it conforms to the visual impact characteristics of the advertising design.

[0078] Other elements: whether there is promotional copy (such as "limited time offer" or "buy one get one free"), brand logo, contact information, etc.

[0079] Judgment Basis: Carefully analyze the image content, including the product display, background, composition, color, and copy, to determine whether it meets the design characteristics of a commercial advertising poster. This analysis should be based on professional poster design standards and provide a convincing judgment.

[0080] [Tone] Maintain professionalism and rigor in your analysis, avoid subjective assumptions, and ensure that the basis for your judgment is clear and organized.

[0081] [Output Format] The returned output should be in the following JSON format, containing two fields:

[0082] score: The credibility of whether the image is a commercial poster image, ranging from [0, 1], where 1 means it is completely consistent and 0 means it is completely inconsistent.

[0083] Reason: The reason for the judgment based on the image analysis, and the specific basis for compliance or non-compliance must be explained in detail.

[0084] [Example]

[0085] Input image: A poster showing a mobile phone with a solid color background, the phone occupying the center, the text "Limited Time Discount" and the brand logo.

[0086] Output: {"score":0.9,"reason":"The image shows a product (a mobile phone) in a prominent position, against a simple background, with promotional text ('Limited Time Discount') and a brand logo, meeting the design requirements of a commercial poster."}

[0087] Input image: A landscape photograph with a natural background and no product display or promotional information.

[0088] Output: {"score":0.1,"reason":"The image does not display any products, promotional information, or brand logos. The background is a natural landscape, which is more consistent with the characteristics of artistic photography or landscape posters rather than commercial advertising posters."}.

[0089] The specific process of determining whether it is a movie / entertainment poster prompt word is as follows:

[0090] [Context] You are an experienced poster designer, skilled at creating and analyzing various types of poster images. Now, you need help determining whether an image is a movie / entertainment poster image.

[0091] [Task Objective] Analysis of the characteristics of movie / entertainment poster images:

[0092] Visual Focus: Does the image have a clear visual focal point (e.g., the film's protagonist, key scenes, iconic props, etc.) and does it highlight the film's core theme or atmosphere?

[0093] Background design: Does the background fit the narrative style of the film (such as the futuristic scenes of science fiction films, the magical atmosphere of fantasy films, the tense scenes of action films, etc.).

[0094] Composition and style: Is the composition dramatic or narrative? Does it create a strong visual impact or emotional atmosphere through light, shadow, color, and other techniques?

[0095] Text information: Does it include the film's title, subtitle, release date, and promotional slogans (e.g., "Coming Soon," "Don't Miss It"). Other elements: Does it include iconic elements related to the film (e.g., director, lead actor's name, film logo, iconic symbols, etc.).

[0096] Judgment Basis: Carefully analyze the image's content, including focal point, background, composition, text, and iconic elements, to determine if it meets the design requirements for film / entertainment posters. This analysis should be based on professional poster design standards and incorporate the film's narrative and emotional expression to provide a convincing assessment.

[0097] [Tone] Maintain professionalism and rigor in your analysis, avoid subjective assumptions, and ensure that the basis for your judgment is clear and organized.

[0098] [Output Format] The returned output should be in the following JSON format, containing two fields:

[0099] score: The confidence level of whether the image is a movie / entertainment poster image, ranging from [0, 1], where 1 indicates complete compliance and 0 indicates complete disconfirmation.

[0100] Reason: The reason for the judgment based on image analysis, and the specific basis for compliance or non-compliance must be explained in detail.

[0101] [Example] Input image: A poster showing the protagonist of a movie in an explosion scene, with a futuristic city in the background, the movie title "Future Wars" and the release date "December 2024".

[0102] Output: {"score":0.9,"reason":"The image has a clear visual focal point (the protagonist is in the explosion scene), and the background is a futuristic city, which is consistent with the narrative style of a science fiction film. It also has the film title and release date, and the overall composition has a strong visual impact, which is consistent with the design characteristics of movie / entertainment posters."}.

[0103] Input image: A poster showing a natural landscape with mountains and a lake in the background, without any movie-related text or iconic elements.

[0104] Output: {"score":0.1,"reason":"The image has no clear focal point or movie-related elements, and the background is a natural landscape, making it more suitable for a landscape poster than a movie / entertainment poster."}.

[0105] The specific process of determining whether it is a public service poster prompt is as follows:

[0106] "[Context] You are an experienced poster designer, skilled at creating and analyzing various types of poster images. Now, you need help determining whether an image is a public service poster image.

[0107] [Task Objective] Analysis of the characteristics of public welfare poster images:

[0108] Theme expression: Does the image have a clear public welfare theme (such as environmental protection, caring for children, healthy living, social equity, etc.), and does it convey a sense of social responsibility or call for action?

[0109] Visual Focus: Does the image have a clear visual focus (such as people, animals, natural landscapes, etc.) and does it highlight the core emotions or information of the public welfare theme?

[0110] Background design: Is the background related to the public welfare theme (such as the natural environment, urban problem scenes, life scenes of vulnerable groups, etc.), and does it contain symbolic elements (such as the earth, love, gestures, etc.).

[0111] Text information: Whether there are public welfare-related slogans (such as "Protect the Earth", "Care for Children", "Reject Waste"), calls to action (such as "Take Action Now", "Join Us") and other text information.

[0112] Color and style: whether the color conveys emotions (such as green represents environmental protection, red represents urgency, and blue represents hope); whether the style is concise and clear, focusing on information transmission and emotional resonance.

[0113] Other elements: Whether there is a public welfare organization logo, relevant data or facts (such as "reducing 1 million tons of plastic pollution each year"), symbolic patterns (such as recycling symbols, peace doves, etc.).

[0114] Judgment Basis: Carefully analyze the image's content, including thematic expression, visual focus, background design, textual information, color and style, to determine whether it meets the design characteristics of a public service poster. This analysis should be based on professional poster design standards, taking into account the public service poster's emotional expression and sense of social responsibility, and provide a convincing judgment.

[0115] [Tone] Maintain professionalism and rigor in your analysis, avoid subjective assumptions, and ensure that the basis for your judgment is clear and organized.

[0116] [Output Format] The returned output should be in the following JSON format, containing two fields:

[0117] Score: The credibility of whether the image is a public service poster. The range is [0, 1], where 1 indicates complete compliance and 0 indicates complete noncompliance. Reason: The reason for the judgment based on the image analysis, which requires a detailed description of the specific basis for compliance or noncompliance.

[0118] [Example] Input image: A poster showing marine life trapped by plastic waste, with a blue ocean background and the slogan "Say No to Plastic, Save the Ocean."

[0119] Output: {"score":0.9,"reason":"The image has a clear visual focus (marine life trapped by plastic waste), a blue ocean background, and is related to environmental protection. It also features the slogan 'Reject Plastic, Protect the Ocean.' The overall style is concise and clear, meeting the design requirements of a public service poster."}

[0120] Input image: A poster showing fashion clothing with a city street background, a brand logo, and a promotional message "50% off everything."

[0121] Output: Copy {"score":0.1,"reason":"The image lacks a clear public welfare theme or emotional expression. The background is a city street, making it more like a commercial advertisement than a public welfare poster."}.

[0122] The same applies to other prompt words.

[0123] Step 5: The multimodal large-scale image recognition model outputs the image classification results. This method uses the open-source multimodal large-scale image recognition model Qwen2.5-VL. Based on the Qwen2.5 language model, it expands visual capabilities, deeply integrates text and image understanding, and supports complex cross-modal tasks such as visual question answering, image captioning, and multimodal reasoning. Qwen2.5-VL employs a joint vision encoder-language model architecture consisting of three components: a vision encoder, a language model, and a multi-layer perceptron-based vision-language fusion module. The vision encoder includes CNN feature extraction and a visual feature encoder, while the vision-language fusion module includes feature compression and projection, as well as cross-modal interaction. The entire model inference process is shown in the figure below.

[0124] Step 1: First, preprocess the input image, including scaling the image width and height to multiples of 28, and then divide the image into image patches with a step size of 14×14. Each patch is linearly projected to generate a visual token.

[0125] Step 2: Use the Tokenizer (word segmentation tool) to segment the text, generate a word vector sequence, and add tags (such as <|im_start|>, <|im_end|>) to build a dialogue structure and form text tokens.

[0126] Step 3: Perform visual feature encoding through the Vision Transformer (ViT) architecture, input the visual token from the first step, and output visual features.

[0127] Step 4: Reduce the dimensionality of the image features to align with the features of the text token through feature compression and projection. The specific operation is to pass the image features through two multi-layer perceptrons to obtain the visual features with reduced dimensionality.

[0128] Step 5: Align visual and textual tokens in a unified semantic space through cross-modal interaction. This involves dynamically associating image regions with textual vocabulary through a cross-attention mechanism. Specifically, textual tokens are used as queries, and visual tokens are used as keys and values. A multi-head attention mechanism is used to interact with visual and textual features, forming fused features.

[0129] Step 6: Finally, based on the Qwen2.5 LLM architecture, the fused features from step 5 are used as input, and text is generated step by step through an autoregressive transformer. Each step calculates the output based on the current context and visual features. The final output is the large language model output, which includes the confidence level and judgment explanation for the current category. The autoregressive mechanism is that the model generates a probability distribution for the current word based on the previously generated results and selects candidate words through search.

[0130] The main parameters involved in the Qwen2.5-vl calling interface include temperature (temperature value), top_p (kernel sampling probability), and max_tokens (maximum number of tokens). Temperature controls the randomness of the model's generated results. This method prefers deterministic results, so a smaller value, such as 0.2, is used. Top_p controls the diversity of the model's generated results, truncating low-probability candidates based on the probability distribution. For classification tasks like this one, it is set to 0.9, retaining only candidates with a cumulative probability of 90%. max_tokens limits the maximum number of tokens generated by the model. For classification tasks, generating long text is generally not necessary, so a smaller value, such as 64, can be used. The multimodal large image recognition model is called as follows: an image and a prompt word are input, and the corresponding category confidence scores q1, q2, and q3 are output.

[0131] Step 6: Fusion of results based on prior information. Steps 3 and 5 yield relevant label information. The logic is as follows: When the label confidence of a small model exceeds a certain threshold, the multimodal large image recognition model is invoked to identify its fine-grained label. For small model label confidences less than or equal to the threshold, the sub-label confidence is set to 0. Finally, all small model confidences are multiplied by their sub-label confidences. The maximum value represents the current image category.

[0132] Note that if the "other" label, that is, p5 is greater than the threshold and other labels are less than the threshold, the current image label is considered to be "other". If p5 is greater than the threshold and other labels are also greater than the threshold, then p5 needs to be multiplied by a weight when performing the threshold size comparison. This article sets it to 0.9.

[0133] The implementation basis of each embodiment of the present invention is achieved through programmed processing by a system with processor functions. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this reality, on the basis of the above embodiments, an embodiment of the present invention provides an image classification system based on the collaboration of a small model and a multimodal large image recognition model, which is used to execute an image classification method based on the collaboration of a small model and a multimodal large image recognition model in the above method embodiment.

[0134] See also Figure 3 , the system comprises:

[0135] The image coarse classification module is used to input the image to be classified into the small image recognition model and output the image coarse classification result; the model main network of the small image recognition model adopts MobileNet, and the output layer includes a global average pooling layer, a fully connected layer, and an activation function layer;

[0136] The preset prompt word library matching module is used to match the preset prompt word library according to the rough classification results: if the match fails, the image category is output as "other"; if the match succeeds, the multimodal large image recognition model is called for processing;

[0137] The image sub-classification module inputs the image to be classified and its matching preset prompt word library into the multimodal large image recognition model to obtain the image sub-classification results. The multimodal large image recognition model includes four components: a visual encoder for extracting image visual features, a language model for extracting text features from the preset prompt word library, a visual language fusion module for fusing visual features and text features, and a result output module that outputs the image sub-classification results based on the fused features.

[0138] The result output module is used to fuse the coarse classification results of the image with the fine classification results of the image to obtain the final fine-grained classification results of the image.

[0139] It should be noted that the system embodiments provided by the present invention are not only used to implement the methods in the above-mentioned method embodiments, but also used to implement the methods in other method embodiments provided by the present invention. The only difference lies in the setting of corresponding functional modules, and its principles are basically the same as the principles of the above-mentioned system embodiments provided by the present invention. As long as those skilled in the art refer to the specific technical solutions in other method embodiments on the basis of the above-mentioned system embodiments, obtain corresponding technical means and technical solutions composed of these technical means by combining technical features, and on the premise of ensuring the practicality of the technical solutions, they will improve the system in the above-mentioned system embodiments to obtain corresponding system class embodiments for implementing the methods in other method class embodiments.

[0140] The method of the embodiment of the present invention is implemented by electronic devices, so it is necessary to introduce the relevant electronic devices. Based on this purpose, the embodiment of the present invention provides an electronic device, such as Figure 4As shown, the electronic device includes: at least one processor, a communications interface, at least one memory, and a communications bus, wherein the at least one processor, the communications interface, and the at least one memory communicate with each other via the communications bus. The at least one processor invokes logic instructions in the at least one memory to execute all or part of the steps of the methods provided in the aforementioned method embodiments.

[0141] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, located in one place or distributed across multiple network units. Depending on practical needs, some or all of these modules may be selected to achieve the objectives of this embodiment. Persons of ordinary skill in the art will understand and implement these embodiments without inventive effort.

[0142] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0144] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0146] In summary of the above embodiments, the purpose of the invention of this article is to propose an image classification method based on the collaboration of a small model and a multimodal large image recognition model, to reduce the annotation of fine-grained classification and save the computational cost of the large model. The first step is to first use a small model to identify large categories, reduce the computational load of the large model, and improve the adaptability of open scenes. The second step is to use the semantic understanding ability of the multimodal large image recognition model to directly complete the category judgment of the fine-grained categories under the large categories in the form of questions and answers, and solve the problems of predefined category restrictions and repeated training. Improve the classification accuracy and efficiency, through a hierarchical collaborative mechanism, combine the fast reasoning of the small model with the semantic analysis advantages of the large model, and optimize the classification effect in complex scenarios.

[0147] The key point of this invention is to propose an image classification method based on the collaboration of a small model and a multimodal large image recognition model. This method adopts a hierarchical collaborative architecture of small and large models. The small model is responsible for coarse-grained classification (such as "document," "certificate," "portrait," and "animal"), while the large model focuses on fine-grained recognition (such as "contract," "legal document," and "rules and regulations" under "document"). Different prompt words are then used to answer questions based on different fine-grained labels. For example, a document-class prompt word reads: "To determine whether this image is a contract, the following characteristics must be met: 1) Contains signature fields for both parties; 2) Has a clause number; 3) The title contains the word 'contract.' Please answer 'yes' or 'no.'" Finally, the outputs of the small and large models are weightedly fused, and low-confidence results are filtered using a threshold.

[0148] The present invention achieves beneficial effects by enabling collaborative image classification based on large and small models. Pre-screening with the small model avoids frequent calls to the large model, reducing computational costs and improving adaptability to open scenarios. Furthermore, fine-grained classification labels are directly identified using a multimodal large image recognition model, avoiding the need for fine-grained annotation of the entire dataset. This reduces data annotation costs, model training complexity, and optimizes resource utilization.

[0149] Furthermore, fine-grained classification tasks place high demands on model capabilities. Multimodal large-scale image recognition models offer superior accuracy compared to traditional convolutional models. Furthermore, pre-classification by a small model can provide the large model with more precise prompts, further enhancing its effectiveness. Furthermore, simply by adding prompts, this method can flexibly adapt to new categories in the open world (such as new contract types and document styles), avoiding the retraining challenges inherent in traditional methods due to category expansion. Therefore, this invention offers significant benefits, both theoretically and practically.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. An image classification method based on the collaboration of a small model and a multimodal large image recognition model, characterized in that: include: Input the image to be classified into the small image recognition model and output the rough classification result of the image; The main network of the small image recognition model adopts MobileNet, and the output layer includes a global average pooling layer, a fully connected layer, and an activation function layer; Match the preset prompt word library based on the coarse classification results: if the match fails, the image category is output as "other"; if the match succeeds, the multimodal large image recognition model is called for processing; The image to be classified and its matching preset prompt word library are input into the multimodal large image recognition model to obtain the image sub-classification result; the multimodal large image recognition model comprises four components: a visual encoder for extracting image visual features, a language model for extracting text features from the preset prompt word library, a visual language fusion module for fusing visual features and text features, and a result output module for outputting the image sub-classification result based on the fused features; The coarse classification results of the image are combined with the fine classification results of the image to obtain the final fine-grained classification results of the image.

2. The image classification method based on the collaboration of a small model and a multimodal large image recognition model as described in claim 1 is characterized in that: The steps of processing the image to be classified by the small image recognition model are as follows: The image to be classified is input into MobileNet to extract high-dimensional features of the image. Global average pooling is used to flatten the channel dimension of the high-dimensional features of the image into a vector. The fully connected layer sets the number of output feature channels corresponding to the number of categories of the predicted coarse classification label of the image and obtains the channel score. Finally, the activation function layer is input to normalize the channel score and output the coarse classification label of the image in the form of probability. The coarse classification result of the image is output according to the preset highest-scoring coarse classification label probability threshold.

3. The image classification method based on the collaboration of a small model and a multimodal large image recognition model as described in claim 2 is characterized in that: The MobileNet consists of seven stages. The first stage uses a convolutional layer to extract preliminary features of the image. The second to seventh stages are stacked inverted residual modules. By adjusting the number of channels and resolution, features are extracted stage by stage, and finally high-dimensional features of the image are output.

4. The image classification method based on the collaboration of a small model and a multimodal large image recognition model according to claim 1, characterized in that: The preset prompt word library corresponds to the document coarse classification label setting, wherein the image coarse classification label includes but is not limited to: books, test papers, other documents, posters and backgrounds. Except for the coarse classification label of background, each document coarse classification label is set with a preset prompt word library.

5. The image classification method based on the collaboration of a small model and a multimodal large image recognition model as described in claim 4 is characterized in that: The process of setting the preset prompt word library is as follows: For each coarse classification label of a document, a preset number of fine classification labels are determined. Prompt words corresponding to the fine classification labels are designed from four dimensions: design purpose, visual style, core content, and communication scenario. The prompt words of all fine classification labels are integrated to obtain a preset prompt word library corresponding to the coarse classification label of the document.

6. The image classification method based on the collaboration of a small model and a multimodal large image recognition model as claimed in claim 1, characterized in that: The visual encoder is a ViT model, which includes CNN feature extraction and visual feature encoder. The working steps include: First, the image to be classified is preprocessed, and then the block size of non-overlapping blocks is set to divide the original image into image blocks; Each image block is linearly projected into the embedding space through CNN feature extraction to obtain visual labels; The input visual sequence is constructed based on the visual markers, and the input visual sequence passes through the visual feature encoder of the ViT architecture to output visual features.

7. The image classification method based on the collaboration of a small model and a multimodal large image recognition model as claimed in claim 1, characterized in that: The input of the language model is a preset prompt word library. Tags are added to the prompt words in the preset prompt word library to build a dialogue structure. Then, a word segmenter is used to process the prompt words to form text tags. The text tags are mapped to the embedding space, and the obtained word vectors are used as text features.

8. The image classification method based on the collaboration of a small model and a multimodal large image recognition model as claimed in claim 1, characterized in that: The visual-language fusion device uses two multi-layer perceptrons to reduce the dimensionality of visual features until they are aligned with the dimensions of text features, processes visual and text features through a multi-head attention mechanism, and outputs fused features.

9. An image classification system based on the collaboration of a small model and a multimodal large image recognition model, characterized in that: include: The image coarse classification module is used to input the image to be classified into the small image recognition model and output the image coarse classification result; The main network of the small image recognition model adopts MobileNet, and the output layer includes a global average pooling layer, a fully connected layer, and an activation function layer; The preset prompt word library matching module is used to match the preset prompt word library according to the rough classification results: if the match fails, the image category is output as "other"; if the match succeeds, the multimodal large image recognition model is called for processing; The image sub-classification module inputs the image to be classified and its matching preset prompt word library into the multimodal large image recognition model to obtain the image sub-classification results. The multimodal large image recognition model includes four components: a visual encoder for extracting image visual features, a language model for extracting text features from the preset prompt word library, a visual language fusion module for fusing visual features and text features, and a result output module that outputs the image sub-classification results based on the fused features. The result output module is used to fuse the coarse classification results of the image with the fine classification results of the image to obtain the final fine-grained classification results of the image.

10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the method according to any one of claims 1 to 8.

Images