Image classification model training method and device, image classification method and device, equipment and medium

Abstract

The invention discloses an image classification model training method and device, an image classification method and device, equipment and a medium. The method comprises the steps that an original image is acquired, an image description text corresponding to the original image, N attribute description texts and corresponding saliency areas are determined, and N is larger than 2; performing mask processing on each saliency region, and determining N region mask images; feature coding fusion is carried out on the original image, the N area mask images, the image description text and the N attribute description texts, and fusion coding features are determined; and performing model training based on the fused coding features, and determining a target classification model. According to the method, cross-modal feature code fusion is carried out on the whole image and a saliency region to obtain a fusion feature code capable of accurately representing the image, accurate training of a model is realized through the fusion feature code, and a target classification model with relatively high image category recognition accuracy is obtained. Therefore, image classification with high precision can be carried out according to the target classification model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an image classification model training method, image classification method, apparatus, device and medium. Background Technology

[0002] With the continuous development of artificial intelligence technology, image recognition and classification are widely used in various professional scenarios. For example, in industrial quality inspection, defect identification and classification of acquired workpiece images are required to facilitate repair based on the corresponding defect categories. Similarly, in biological classification, biological images need to be categorized to determine the biological type within them. Furthermore, in vehicle brand and model classification, the brand and model of vehicles in images need to be identified. Current technologies typically employ large-scale visual language models for visual understanding and image classification. However, existing large-scale visual language models can only perform a global understanding of images and cannot analyze image details, resulting in low accuracy in image classification. Therefore, improving the accuracy of image classification is a pressing technical problem that needs to be solved. Summary of the Invention

[0003] This invention provides an image classification model training method, an image classification method, an apparatus, a device, and a medium to address the technical problem of how to improve the accuracy of image classification.

[0004] An image classification model training method, comprising: Obtain the original image, and determine the image description text, N attribute description texts and their corresponding salient regions corresponding to the original image, where N > 2; Each of the aforementioned salient regions is masked to determine N region mask images; The original image, N region mask images, the image description text, and N attribute description text are fused using feature encoding to determine the fused encoding features; The target classification model is determined by training the model based on the fused encoding features.

[0005] An image classification method, comprising: Obtain the image to be classified, and determine the image description text corresponding to the image to be classified, as well as the region mask image and attribute description text corresponding to N salient regions in the image to be classified, where N > 2; A target classification model is used to perform classification prediction on the image to be classified, the image description text, N region mask images, and N attribute description texts to determine the target category corresponding to the image to be classified. The target classification model was trained using the image classification model training method described above.

[0006] An image classification model training device, comprising: The image and text acquisition module is used to acquire the original image, determine the image description text, N attribute description texts and their corresponding salient regions corresponding to the original image, where N>2; The region mask image determination module is used to perform masking processing on each of the aforementioned salient regions to determine N region mask images; The fusion coding feature determination module is used to perform feature coding fusion on the original image, N region mask images, the image description text, and N attribute description text to determine the fusion coding features; The target classification model determination module trains the model based on the fused encoded features to determine the target classification model.

[0007] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the image classification model training method described above, or the processor implements the image classification method described above when executing the computer program.

[0008] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the image classification model training method described above, or, when executed by a processor, implements the image classification method described above.

[0009] The aforementioned image classification model training method, image classification method, apparatus, device, and medium determine N salient regions in the original object, enabling the extraction of highly identifiable image details from the original image. This facilitates masking of the N salient regions, resulting in N region mask images that focus on image details and prevent the details of the original image from being obscured by redundant information. Feature fusion encoding is performed on the original image, the region mask images corresponding to the N salient regions, the image description text corresponding to the original image, and the attribute description text corresponding to the N salient regions. The fusion feature encoding is determined, and cross-modal deep fusion is performed on the image features and text features representing image attributes. This achieves strong correlation between the image features and text features of the original image, facilitating model training using the fusion feature encoding of deep cross-modal features. The result is a target classification model for accurately identifying the categories of objects contained in the image.

[0010] This method fuses cross-modal feature codes for a region mask image that strongly identifies image details, an original image that represents the overall image condition, image description text that represents the overall image attributes, and attribute description text that strongly identifies image details. This results in a fused feature code that accurately represents image attributes. Through this precise fused feature code, the model can be accurately trained, resulting in a target classification model with high accuracy in image category recognition. This model facilitates high-accuracy image classification and recognition, and has broad application prospects and high application value. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of an image classification model training method according to an embodiment of the present invention; Figure 2 This is another flowchart of an image classification model training method in one embodiment of the present invention; Figure 3 This is another flowchart of an image classification model training method in one embodiment of the present invention; Figure 4 This is another flowchart of an image classification model training method in one embodiment of the present invention; Figure 5 This is another flowchart of an image classification model training method in one embodiment of the present invention; Figure 6 This is another flowchart of an image classification model training method in one embodiment of the present invention; Figure 7 This is a flowchart of an image classification method according to an embodiment of the present invention; Figure 8 This is a schematic diagram of an image classification model training device in one embodiment of the present invention; Figure 9 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] The image classification model training method provided in this invention involves acquiring an original image, determining the image description text, N attribute description texts, and their corresponding salient regions (N > 2) corresponding to the original image, performing masking processing on each salient region to determine N region mask images, fusing feature encodings of the original image, the N region mask images, the image description text, and the N attribute description texts to determine fused encoding features, and training the model based on the fused encoding features to determine the target classification model. This facilitates the training of a target classification model capable of accurately classifying images, thereby improving the accuracy of image classification.

[0015] In one embodiment, such as Figure 1 As shown, an image classification model training method is provided, and this image classification model training method is applied to... Figure 9 Taking a computer device as an example, the following steps are included: S101: Obtain the original image, determine the image description text corresponding to the original image, N attribute description texts and their corresponding salient regions, N>2; S102: Perform masking processing on each salient region to determine N region mask images; S103: Perform feature encoding fusion on the original image, N region mask images, image description text, and N attribute description text to determine the fused encoding features; S104: Train the model based on fused coding features to determine the target classification model.

[0016] In this context, the original image refers to the image used for model training. A salient region is a region within the image that reflects its attributes. Image description text describes the overall attributes of the image. Attribute description text describes the attributes of the salient regions.

[0017] As an example, in step S101, the computer device acquires multiple original images in an image classification scenario, performs image attribute description on each original image, determines the corresponding image description text for each original image, extracts region attribute descriptors from the corresponding image description text for each original image, determines N attribute description texts corresponding to each original image, and determines N regions in each original image that correspond to the N attribute description texts. These N regions are then identified as the N salient regions corresponding to the original image. Where N > 2. In this example, when the number of determined salient regions is less than N, the computer device upsamples the original image to increase the number of salient regions, resulting in N salient regions. When the number of determined salient regions is not less than N, the computer device downsamples the original image to randomly delete salient regions, reducing the number of salient regions to obtain N salient regions.

[0018] For example, in a biological classification scenario, if the creature in the original image is a dog, the image description text is determined as follows: The head is typically apple-shaped, with a round forehead, a short nose, a wide face, large, erect ears, an upward-curving tail, short limbs, and a narrow chest. When N=4, the image description text is analyzed by extracting and analyzing the regional attribute descriptors for four regions: head, tail, limbs, and chest. The attribute description text for the head is determined as: The head is typically apple-shaped, with a round forehead, a short nose, a wide face, and large, erect ears. The attribute description text for the tail is: An upward-curving tail. The attribute description text for the limbs is: Short limbs. The attribute description text for the chest is: Narrow chest. The regions corresponding to the head, tail, limbs, and chest in the original image are then identified as salient regions.

[0019] In this example, text descriptions are performed on the original image in the image recognition scenario to determine the image description text and attribute description text, and salient regions in the original image are obtained. This facilitates feature extraction from the image description text, attribute description text, and original image during subsequent model training, and the fusion of image features and text features enables accurate model training.

[0020] Among them, a region mask image refers to a grayscale image obtained by masking an image in a salient region.

[0021] As an example, in step S102, the computer device uses a preset adaptive masking strategy to mask the N salient regions of each original image, determining the region mask image corresponding to each salient region of each original image, thus obtaining N region mask images corresponding to each original image. This facilitates subsequent image feature encoding of the salient regions using the region mask images. For example, for an original image in a biological classification scenario, if the organism in the original image is a dog, the computer device performs masking on four salient regions in the original image: head, tail, limbs, and chest, determining the region mask images corresponding to the head, tail, limbs, and chest, respectively.

[0022] Among them, fusion coding features refer to features obtained by fusing multiple features.

[0023] As an example, in step S103, the computer device uses the encoder in the model to be trained, which is used for encoding images, to perform image feature encoding processing on the original image and the N region mask images respectively, determining the image features corresponding to the original image and the image features corresponding to the N region mask images respectively. Then, the computer device uses the language encoder in the model to be trained to perform text feature encoding on the image description text and the N attribute description text respectively, determining the text features corresponding to the image description text and the text features corresponding to the N attribute description text respectively. The computer device fuses the image features output by the encoder used for image encoding and the text features output by the language encoder to determine the fused encoding features corresponding to the original image. The computer device determines the fused encoding features corresponding to each original image using the above method. Here, the model to be trained refers to the image classification model that needs to be trained, including the parallel encoder used for image encoding and the language encoder used for text encoding. In this example, the encoder used for image encoding can be a VisionTransformer encoding network.

[0024] For example, in a biological classification scenario, if the organism in the original image is a dog, the computer device uses the encoder in the model to encode the image to encode the original image, the masked region images corresponding to the head, tail, limbs, and chest, respectively, determining the image features corresponding to the original image, the encoded features corresponding to the head, the image features corresponding to the tail, the image features corresponding to the limbs, and the image features corresponding to the chest. Then, the language encoder in the model is used to encode the image description text and the attribute description text corresponding to the four salient regions (head, tail, limbs, and chest), respectively, determining the text features corresponding to the image description text and the text features corresponding to the four salient regions. The computer device then fuses these image and text features to determine the fused encoded features corresponding to the original image in the biological classification scenario.

[0025] Among them, the target classification model refers to the trained model used to classify images.

[0026] As an example, in step S104, the computer device inputs the fused encoded features corresponding to the original image in the image classification scenario into the category prediction model in the model to be trained, outputs the category prediction result, and determines the training loss function value based on the category prediction result, for example, the cross-entropy loss function value. When the loss function value does not meet the preset convergence condition, the computer device updates the model parameters of the model to be trained; when the loss function value meets the preset convergence condition, the model to be trained after updating the model parameters is determined as the target classification model in the image classification scenario. In this example, steps S101 to S104 are applied to a biological classification scenario to obtain a target classification model capable of classifying and recognizing images containing organisms. Steps S101 to S104 are applied to an industrial quality inspection scenario to obtain a target classification model capable of classifying and recognizing defects in images containing workpieces. Steps S101 to S104 are applied to a scenario where vehicles are classified by brand and model to obtain a target classification model capable of classifying and recognizing vehicles by brand and model in images.

[0027] In this embodiment, N salient regions are identified in the original object to extract highly identifiable image details from the original image. This facilitates masking of the N salient regions, resulting in N region mask images that focus on the image details and prevent the details of the original image from being obscured by redundant information. Feature fusion encoding is performed on the original image, the region mask images corresponding to the N salient regions in the original image, the image description text corresponding to the original image, and the attribute description text corresponding to the N salient regions. The fusion feature encoding is determined, and cross-modal deep fusion is performed on the image features and the text features used to represent image attributes. This achieves strong correlation between the image features and text features of the original image, facilitating model training using the fusion feature encoding of deep fusion cross-modal features to obtain a target classification model for accurately identifying the categories of objects contained in the image.

[0028] This method fuses cross-modal feature codes for a region mask image that strongly identifies image details, an original image that represents the overall image condition, image description text that represents the overall image attributes, and attribute description text that strongly identifies image details. This results in a fused feature code that accurately represents image features. Through this precise fused feature code, the model can be accurately trained, resulting in a target classification model with high accuracy in image category recognition. This model facilitates high-accuracy image classification and recognition, and has broad application prospects and high application value.

[0029] In one embodiment, such as Figure 2 As shown, step S101, namely acquiring the original image and determining the image description text, N attribute description texts, and their corresponding salient regions corresponding to the original image, includes: S201: Describe the attributes of the original image and determine the image description text corresponding to the original image; S202: Extract attribute text from the image description text corresponding to the original image and determine N attribute description texts; S203: Annotate the original image based on N attribute description texts to determine the N salient regions corresponding to the original image.

[0030] As an example, in step S201, the computer device describes the attributes of the original image using a preset attribute description method to determine the image description text corresponding to the original image. In this example, the computer device can use a pre-trained language model to describe the attributes of the original image and determine the image description text corresponding to the original image. For example, the question "Please describe the visual attributes of the original image" and the original image are input into GPT4v, and GPT4v outputs the image description text corresponding to the original image. In this example, the image description text corresponding to the original image can be determined in advance using a manual description method, and the computer device receives the image description text corresponding to the original image input by the user.

[0031] For example, in the original image for biological classification, the image description text is: The head shape is usually apple-shaped, with a round forehead, a short nose, a wide face, large and erect ears, an upward-curving tail, short limbs, and a narrow chest.

[0032] As an example, in step S202, the computer device recognizes the image description text, determines the description text related to N detailed attributes of the original image, and identifies the description text for each detailed attribute as an attribute description text, resulting in N attribute description texts. For example, for the image description text corresponding to the original image in the biological classification scenario in step S201, when N=4, the image description text extracts the detailed attributes of four regions: head, tail, limbs, and chest, determining four attribute description texts: The attribute description text for the head is: The head shape is usually apple-shaped, with a round forehead, a short nose, a wide face, and large, erect ears. The attribute description text for the tail is: An upward-curving tail. The attribute description text for the limbs is: Short limbs. The attribute description text for the chest is: A narrow chest.

[0033] As an example, in step S203, the computer device searches for regions in the original image that correspond to the N attribute description texts and performs region annotation, which can accurately obtain the N annotated salient regions corresponding to the original image. For example, for the original image in the biological classification scenario determined in step S202 and the determined four attribute description texts (head, tail, limbs, and chest), the computer device searches for regions in the original image in the biological classification scenario that correspond to the four attribute description texts (head, tail, limbs, and chest) and performs region annotation, obtaining the salient regions corresponding to the head, tail, limbs, and chest. Understandably, in the process of training a target classification model using the original image, in order to improve the accuracy of model training, the model needs to have a strong recognition and classification ability for the original image during training. When the model can recognize more detailed attributes in the original image, its recognition ability will be improved. Therefore, annotating the salient regions used to represent the detailed attributes of the image in the original image is necessary to improve the accuracy of subsequent model training.

[0034] In this embodiment, the original image is described by attributes to obtain image description text that describes the entire image. The image description text is then used to extract text corresponding to the image detail attributes to obtain N attribute description texts that characterize the image detail attributes. The original image is then accurately labeled based on the N attribute description texts to determine N salient regions. This method can adaptively label the original image based on the attribute description texts and reasonably determine the size of the salient regions so that accurate adaptive image masking can be performed subsequently based on the N salient regions.

[0035] In one embodiment, such as Figure 3 As shown, step S103 involves fusing feature encoding of the original image, N region mask images, image description text, and N attribute description text to determine the fused encoding features, including: S301: Perform image feature encoding processing on the original image and N region mask images to determine the overall image features corresponding to the original image and the region image features corresponding to the N region mask images; S302: Perform text feature encoding processing on the image description text and N attribute description texts to determine the first text feature corresponding to the image description text and the second text feature corresponding to the N attribute description texts; S303: Perform alternating image feature-text feature concatenation on the overall image features, the first text features, the N regional image features, and the N second text features to determine the fused coding features.

[0036] Among them, overall image features are used to represent the overall features of the original image. Region image features are used to represent the features of the salient regions corresponding to the region mask image.

[0037] As an example, in step S301, the computer device performs image encoding processing on the original image and the N region mask images respectively, determines the image features corresponding to the original image and the image features corresponding to each region mask image in the N region mask images, determines the image features corresponding to the original image as the overall image features, and determines the image features corresponding to each region mask image as the region image features corresponding to each region mask image, thereby obtaining the N region image features.

[0038] The first text feature refers to the text feature corresponding to the image description text. The second text feature refers to the text feature corresponding to the attribute description text.

[0039] As an example, in step S302, the computer device performs text feature encoding processing on the image description text and N attribute description texts respectively, determines the text features corresponding to the image description text and the text features corresponding to each attribute description text, determines the text features corresponding to the image description text as the first text feature, and determines the text features corresponding to each attribute description text as the second text feature, thereby obtaining N second text features.

[0040] As an example, in step S303, the computer device performs concatenation and fusion of the overall image features, the first text features, N regional image features, and N second text features according to a preset image feature-text feature alternating concatenation method, to obtain the fused coding features corresponding to the original image. The computer device determines the fused coding features corresponding to multiple original images used for model training in accordance with steps S301 to S303.

[0041] Understandably, during model training, to improve the classification accuracy of the trained model, it is necessary to ensure strong interpretability of the model's recognition. This means the model needs a basis for image recognition, such as determining the image category based on specific parts or components. Therefore, in this embodiment, feature encoding is performed on the image description text and N attribute description text corresponding to the original image to obtain features corresponding to the overall image and salient regions, resulting in more comprehensive text features. This allows for the subsequent training of a highly interpretable target classification model based on these comprehensive text features, improving the accuracy of model training and pattern recognition. Furthermore, in this embodiment, feature encoding is performed on the image description text and N attribute description text to obtain multiple text features. This facilitates cross-modal concatenation and fusion of text features and image features, establishing a strong correlation between the image and text description, further improving the accuracy of model training and pattern recognition.

[0042] In one embodiment, such as Figure 4As shown, step S301, which involves performing image feature encoding processing on the original image and the N region mask images to determine the overall image features corresponding to the original image and the region image features corresponding to the N region mask images, includes: S401: Input the original image into the convolutional layer of the visual encoder and output the first image feature corresponding to the original image; S402: Input the first image features into the general self-attention coding layer of the visual encoder to determine the second image features corresponding to the original image; S403: Input the second image features and N region mask images into the region self-attention coding layer to determine the third image features and N fourth image features corresponding to the original image; S404: Input the third image feature and N fourth image features into the modal connector for feature dimension adjustment processing to determine the overall image features corresponding to the original image and the regional image features corresponding to the N regional mask images.

[0043] In this context, a visual encoder refers to an encoder used for feature encoding of an image. In this embodiment, the visual encoder is part of a training model used for image classification and recognition, and is used for feature encoding of the image. The visual encoder includes concatenated convolutional layers, multiple general self-attention encoding layers, and region self-attention encoding layers. The convolutional layers are used to encode the original image in blocks. Both the general self-attention encoding layers and the region self-attention encoding layers are used for self-attention encoding.

[0044] The first image feature refers to the image feature corresponding to the original image output by the convolutional layer.

[0045] As an example, in step S401, the computer device inputs the original image into the convolutional layer of the visual encoder, performs local feature encoding on the original image using the convolutional layer, and outputs the first image feature patch token corresponding to the original image.

[0046] The second image feature refers to the image feature after the first image feature has been self-attention encoded using a general self-attention coding layer.

[0047] As an example, in step S402, the computer device inputs the first image features into the general self-attention coding layer of the visual encoder. The general self-attention coding layer uses a preset visual token embedding to perform self-attention coding on the first image features to obtain the second image features corresponding to the original image.

[0048] The third image feature refers to the image feature after the second image feature has been self-attention encoded by a region self-attention encoding layer. The fourth image feature refers to the image feature after the region mask image has been self-attention encoded by a region self-attention encoding layer.

[0049] As an example, in step S403, the computer device inputs the second image features and N region mask images into the region self-attention coding layer. The region self-attention coding layer encodes the third image features and the N fourth image features using an attention formula, outputting the third image features corresponding to the original image and one fourth image feature corresponding to each region mask image. The attention formula is as follows: in, Here, Q is the activation function, K is the key, and V is the value. It can be a third image feature or an arbitrary region mask image. This can be the dimension corresponding to the third image feature or the dimension corresponding to any region mask image.

[0050] As an example, in step S404, the computer device inputs the third image feature and N fourth image features to the modal connector for feature dimension adjustment processing, adjusting the feature dimensions corresponding to the third image feature and N fourth image features to the feature dimensions required by the language model in the image to be trained, thereby obtaining the overall image features corresponding to the original image and the region image features corresponding to the N region mask images.

[0051] In this embodiment, the original image is sequentially encoded using a convolutional layer, a general self-attention coding layer, and a region self-attention coding layer in the visual encoder to obtain accurately encoded third image features. Then, the region self-attention coding layer in the visual encoder is used to accurately encode N region mask images to obtain N fourth image features. A modal connector is used to adjust the feature dimensions of the third image features and the N fourth image features to obtain overall image features and N region image features with reasonable dimensions, making it feasible to train the model based on the overall image features and the N region image features.

[0052] In one embodiment, such as Figure 5 As shown, step S303 involves alternating image feature-text feature concatenation of the overall image features, the first text features, the N region image features, and the N second text features to determine the fused coding features, including: S501: Perform alternating image feature-text feature concatenation on the overall image features, the first text features, the N regional image features, and the N second text features to determine the initial fusion features; S502: Perform major category classification label and minor category classification label splicing on the initial fusion features to determine the fusion coding features.

[0053] The initial fusion feature refers to the feature obtained by alternately splicing image features and text features.

[0054] As an example, in step S501, the computer device uses the concatenation layer in the model to be trained, which is connected in series with the visual encoder and the language encoder, to perform alternating image feature-text feature concatenation on the overall image features and N region image features output by the visual encoder, and the first text features and N second text features output by the language encoder, to obtain the concatenated and fused features, which are then determined as the initial fused features. In this example, the computer device inputs the overall image features, the first text features, the N region image features, and the N second text features into the concatenation layer in the model to be trained. The concatenation layer sequentially concatenates and fuses the overall image features, the first text features, and the region image features and second text features corresponding to the same region mask image to determine the initial fused features.

[0055] For example, in the original image of a biological classification scenario, there are region image features and second text features corresponding to the region mask images of four salient regions: head, tail, limbs, and chest. Therefore, the stitching layer stitches and fuses the overall image features, first text features, region image features corresponding to the head, second text features corresponding to the head, region image features corresponding to the tail, second text features corresponding to the tail, region image features corresponding to the limbs, second text features corresponding to the limbs, region image features corresponding to the chest, and second text features corresponding to the chest, to achieve alternating stitching and fusion of image-text pairs and obtain the initial fused features.

[0056] In this example, by alternately concatenating image features and text features, it is helpful to perform cross-modal fusion of the corresponding features of the image and to fuse features that are strongly correlated with the image category, so as to improve the targeting of the detailed features of the image.

[0057] In this context, the broad category label refers to the label used to prompt the model to identify the broad category of an image. The sub-category label refers to the label used to prompt the model to identify the specific category of an image. For example, if an image contains a dog, "dog" is the broad category; if the dog in the image is specifically a Chihuahua, then "Chihuahua" is the sub-category.

[0058] As an example, in step S502, the computer device sequentially concatenates a major category identifier "major category CLS token" and a minor category identifier "minor category CLS token" at the end position of the initial fused features to obtain the concatenated fused encoded features. Understandably, the major category identifier is used to capture global cross-modal features when encoding fused features corresponding to an image, guiding the model to predict the major category of the object to be classified in the image; for example, in a biological classification scenario, the major category is identified as "dog". The minor category identifier is used to capture cross-modal features corresponding to salient regions when encoding fused features corresponding to an image, guiding the model to predict the minor category of the object to be classified in the image; for example, in a biological classification scenario, the minor category is identified as "Chihuahua".

[0059] In this embodiment, by alternately concatenating image features and text features from the overall image features, the first text features, N regional image features, and N second text features, an initial fusion feature capable of deep fusion of cross-modal features is obtained, achieving cross-modal feature complementarity. Furthermore, this initial fusion feature can accurately fuse global and regional features of the image, achieving strong correlations between different regions of the image. The initial fusion feature is also processed by concatenating major and minor classification labels to guide the model in classifying objects in the image according to their major and minor categories. This facilitates the subsequent training of the target classification model based on the cross-modal complementarity and the strong correlation between the overall and local fusion features, enabling accurate classification of objects in the image into their major and minor categories.

[0060] In one embodiment, such as Figure 6 As shown, step S104, which involves training the model based on fused encoded features to determine the target classification model, includes: S601: Use preset image-text pairs to pre-train the modal connectors in the model to be trained, and determine the pre-trained model; S602: Using fused coding features, fine-tuning training is performed on the visual encoder, language encoder, modal connector, and language model in the pre-trained model to determine the target classification model.

[0061] The preset image-text pairs refer to the preset data used for model pre-training. The pre-trained model refers to the model after pre-training the model to be trained.

[0062] As an example, in step S601, the computer device inputs the image from the preset image-text pair into the visual encoder of the model to be trained, inputs the image features output by the visual encoder into the modal connector of the model to be trained, and outputs the image features after the modal connector adapts the feature dimensions of the image features output by the visual encoder. The text from the preset image-text pair is input into the language encoder of the model to be trained, and the text features are output. The image features and text features adapted to the feature dimensions are input into the language model of the model to be trained, and the language model outputs the classification and recognition result. A loss function value is calculated based on the classification and recognition result. If the loss function value does not meet the convergence condition, the parameters of the modal connector in the model to be trained are adjusted to obtain an updated model to be trained. If the loss function value meets the convergence condition, the updated model to be trained is determined as the pre-trained model, thus achieving the purpose of pre-training the classification model using the preset image-text pair. In this example, the language model can be a large-scale visual-language model.

[0063] As an example, in step S602, the computer device inputs the fused encoding features obtained in steps S101 to S103 into the language model in the pre-trained model, outputs an initial classification result, and fine-tunes the visual encoder, language encoder, modal connector, and language model in the pre-trained model based on this initial classification result to obtain the fine-tuned target classification model. Here, the initial classification result refers to the classification result corresponding to the object to be identified in the original image during the model training process.

[0064] In this example, the computer device calculates the fine-tuning loss function value based on the initial classification result. When the fine-tuning loss function value does not meet the preset convergence condition, the parameters of the visual encoder, language encoder, modal connector, and language model in the pre-trained model are adjusted, and the label parameters corresponding to the major category label and minor category label used to form the fusion coding features are also adjusted to obtain the fine-tuned pre-trained model. When the fine-tuning loss function value meets the preset convergence condition, the fine-tuned pre-trained model is determined as the target classification model. In this example, the fine-tuning loss function value is determined by the major category classification loss function value, the minor category classification loss function value, and the autoregressive loss function value. The major category loss function value is determined based on the difference between the major category recognition result output during the classification model training process and the preset major category label. The minor category classification loss function value is determined based on the difference between the minor category recognition result output during the classification model training process and the preset minor category label. The autoregressive loss function value is determined based on the autoregressive loss function during model training, which includes a loss function determined based on the region image features corresponding to N region mask images and the second text features corresponding to N attribute description texts.

[0065] In this embodiment, the model training of the above steps is performed in different image classification scenarios to obtain target classification models corresponding to different image classification scenarios. It does not require high training costs and has strong generalization performance and low cost advantages.

[0066] In this embodiment, the modal connector in the model to be trained is first pre-trained to determine the pre-trained model. This ensures that the modal connector can output image features that are dimensionally compatible with the language model in the model to be trained. Then, fusion coding features are used to fine-tune the visual encoder, language encoder, modal connector, and language model in the pre-trained model, resulting in a target classification model capable of classifying and recognizing images. Since the fusion coding features are determined not only by fusing the overall image features corresponding to the whole image and the regional image features corresponding to salient regions, but also by complementary fusion of cross-modal features (text features and image features), this method, when fine-tuning training based on fusion coding features, can fully consider the overall image features corresponding to the whole image and the regional image features corresponding to salient regions. This avoids the regional image features corresponding to salient regions being overwhelmed by redundant information in the overall image. The fusion of cross-modal features achieves cross-modal feature complementarity, enabling model training with high classification accuracy. This allows the fine-tuned target classification model to accurately classify and recognize objects in images that require classification.

[0067] The image classification model training method in this embodiment can achieve image classification tasks in image classification and recognition scenarios without extensive annotation of the original images used for model training. It can train the corresponding model for different classification scenarios using original images of different classification scenarios, resulting in low training costs and low transfer costs for model training scenarios. It has broad application prospects and high application value.

[0068] In another embodiment, such as Figure 7 As shown, an image classification method is provided, which is then applied to... Figure 9 Taking a computer device as an example, the following steps are included: S701: Obtain the image to be classified, and determine the image description text corresponding to the image to be classified, as well as the region mask image and attribute description text corresponding to N salient regions in the image to be classified, where N>2; S702: Using a target classification model, classify and predict the image to be classified, image description text, N region mask images and N attribute description text to determine the target category corresponding to the image to be classified. The target classification model is trained using the image classification model training method described in any of the above embodiments.

[0069] Here, the image to be classified refers to the image in which the objects contained need to be classified and identified.

[0070] As an example, in step S701, the computer device acquires the image to be classified in the image classification scenario, performs image attribute description on the image to be classified, determines the image description text corresponding to the image to be classified, and extracts region attribute descriptive words from the image description text corresponding to the image to be classified, determining N attribute description texts corresponding to the image to be classified, and determining N salient regions in the image to be classified that correspond to the N attribute description texts. The computer device performs adaptive masking processing on the image corresponding to each salient region, determining a region mask image corresponding to each salient region, thereby obtaining N region mask images corresponding to the N salient regions.

[0071] For example, for an image to be identified in a biological classification scenario, the computer device uses the processing method of step S701 to determine the image description text corresponding to the image to be classified, as well as the N region mask images and N attribute description texts corresponding to the N salient regions in the image to be classified.

[0072] The target category refers to the category of the object in the image to be recognized.

[0073] As an example, in step S702, the computer device inputs the image to be classified and N region mask images into the visual encoder of the target classification model to obtain the image features output by the visual encoder. The image features output by the visual encoder are then input into the modality connector of the target classification model to output image features adapted to the language model dimension of the target classification model. The computer device inputs image description text and N attribute description texts into the language encoder of the target classification model to output text features corresponding to the image to be classified. The image features adapted to the language model dimension of the target classification model, along with the text features corresponding to the image to be classified, are input into the stitching layer for alternating image feature-text feature stitching and major category and minor category stitching to obtain the fused encoded features corresponding to the image to be classified. These fused encoded features are then input into the language model of the target classification model for image category recognition, determining the target category of the image to be classified and completing the category recognition of the objects contained in the image to be classified.

[0074] For example, for an image to be identified in a biological classification scenario, the computer device uses the processing methods in steps S701 to S702 to perform classification prediction on the image to be classified, image description text, N region mask images, and N attribute description texts to determine the target category corresponding to the image to be classified. For example, the image to be identified in the biological classification scenario is identified as having a major category of tiger and a subcategory of Siberian tiger.

[0075] In this embodiment, a precisely trained target classification model is used to classify and predict the target category of the image to be classified, along with image description text, N region mask images, and N attribute description texts. This method uses a precisely trained target classification model to process cross-modal data, as well as image and text data with strong correlations between the whole and regions, to accurately determine the target category of the image to be classified.

[0076] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0077] In one embodiment, an image classification model training device is provided, which corresponds one-to-one with the image classification model training method described in the above embodiments. For example... Figure 8 As shown, the image classification model training device includes an image and text acquisition module 801, a region mask image determination module 802, a fusion coding feature determination module 803, and a target classification model determination module 804. Detailed descriptions of each functional module are as follows: Image and text acquisition module 801 is used to acquire the original image, determine the image description text corresponding to the original image, N attribute description texts and their corresponding salient regions, where N>2; The region mask image determination module 802 is used to perform masking processing on each salient region to determine N region mask images; The fusion coding feature determination module 803 is used to perform feature coding fusion on the original image, N region mask images, image description text and N attribute description text to determine the fusion coding features; The target classification model determination module 804 trains the model based on fused encoding features to determine the target classification model.

[0078] In one embodiment, the image and text acquisition module 801 includes: The image description text determination submodule is used to perform attribute descriptions on the original image and determine the image description text corresponding to the original image. The attribute description text determination submodule is used to extract attribute text from the image description text corresponding to the original image and determine N attribute description texts. The salient region determination submodule annotates the original image based on N attribute description texts to determine the N salient regions corresponding to the original image.

[0079] In one embodiment, the fusion coding feature determination module 803 includes: The image feature determination submodule is used to perform image feature encoding processing on the original image and N region mask images to determine the overall image features corresponding to the original image and the regional image features corresponding to the N region mask images. The text feature determination submodule is used to perform text feature encoding processing on the image description text and N attribute description texts to determine the first text feature corresponding to the image description text and the second text feature corresponding to the N attribute description texts; The fusion coding feature determination submodule is used to perform alternating image feature-text feature concatenation on the overall image features, the first text features, the N regional image features, and the N second text features to determine the fusion coding features.

[0080] In one embodiment, the image feature determination submodule includes: The first encoding unit is used to input the original image into the convolutional layer of the visual encoder and output the first image feature corresponding to the original image. The second encoding unit is used to input the first image features into the general self-attention encoding layer of the visual encoder to determine the second image features corresponding to the original image; The third coding unit is used to input the second image features and N region mask images into the region self-attention coding layer to determine the third image features and N fourth image features corresponding to the original image; The feature dimension adjustment unit is used to input the third image feature and N fourth image features into the modal connector for feature dimension adjustment processing, and to determine the overall image feature corresponding to the original image and the regional image features corresponding to the N regional mask images.

[0081] In one embodiment, the fusion coding feature determination submodule includes: The initial fusion feature unit is used to perform alternating image feature-text feature concatenation on the overall image features, the first text features, the N regional image features, and the N second text features to determine the initial fusion features; The fusion coding feature determination unit is used to perform major category classification labeling and minor category classification labeling concatenation processing on the initial fusion features to determine the fusion coding features.

[0082] In one embodiment, the target classification model determination module 804 includes: The pre-training submodule is used to pre-train the modal connectors in the model to be trained using preset image-text pairs, and to determine the pre-trained model. The fine-tuning submodule is used to fine-tune the visual encoder, language encoder, modal connector, and language model in the pre-trained model by using fused encoding features to determine the target classification model.

[0083] In another embodiment, an image classification device is provided, which corresponds one-to-one with the image classification methods in the above embodiments. The image classification device includes an image processing module and a target category determination module. Detailed descriptions of each functional module are as follows: The image processing module is used to acquire the image to be classified, determine the image description text corresponding to the image to be classified, and the region mask image and attribute description text corresponding to N salient regions in the image to be classified, where N>2; The target category determination module is used to perform classification prediction on the image to be classified, image description text, N region mask images and N attribute description text using a target classification model, and determine the target category corresponding to the image to be classified. The target classification model is trained using the image classification model training method described in any of the above embodiments.

[0084] Specific limitations regarding the image classification model training device can be found in the limitations regarding the image classification model training method above, and specific limitations regarding the image classification device can be found in the limitations regarding the image classification method above; these will not be repeated here. The various modules in the aforementioned image classification model training device and image classification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0085] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data used or generated during the execution of image classification model training methods and image classification methods. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements an image classification model training method, or, when executed by the processor, implements an image classification method.

[0086] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the image classification model training method described in the above embodiment, for example... Figure 1 As shown in S101-S104, or Figures 2 to 6 As shown, to avoid repetition, it will not be described again here. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in this embodiment of the image classification model training device, for example... Figure 8 The functions of the image and text acquisition module 801, the region mask image determination module 802, the fusion coding feature determination module 803, and the target classification model determination module 804 shown are not described again here to avoid repetition.

[0087] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the image classification method described in the above embodiment, for example... Figure 7 S701-S702, as shown, will not be described again here to avoid repetition. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in this embodiment of the image classification device, such as the functions of the image processing module and the target category determination module. To avoid repetition, these will not be described again here.

[0088] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program implements the image classification model training method described in the above embodiment, for example... Figure 1 As shown in S101-S104, or Figures 2 to 6 As shown, to avoid repetition, it will not be described again here. Alternatively, when the computer program is executed by the processor, it implements the functions of each module / unit in this embodiment of the image classification model training device, for example... Figure 8 The functions of the image and text acquisition module 801, the region mask image determination module 802, the fusion coding feature determination module 803, and the target classification model determination module 804 shown are not described again here to avoid repetition. The computer-readable storage medium can be non-volatile or volatile.

[0089] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program implements the image classification method described in the above embodiment, for example... Figure 7 S701-S702, as shown, will not be described again here to avoid repetition. Alternatively, when the computer program is executed by the processor, it implements the functions of each module / unit in this embodiment of the image classification device, such as the functions of the image processing module and the target category determination module. To avoid repetition, these will not be described again here. The computer-readable storage medium can be non-volatile or volatile.

[0090] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0091] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0092] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for training an image classification model, characterized in that, include: Obtain the original image, and determine the image description text, N attribute description texts and their corresponding salient regions corresponding to the original image, where N > 2; Each of the aforementioned salient regions is masked to determine N region mask images; The original image, N region mask images, the image description text, and N attribute description text are fused using feature encoding to determine the fused encoding features; The target classification model is determined by training the model based on the fused encoding features.

2. The image classification model training method according to claim 1, characterized in that, The process of acquiring the original image and determining the image description text, N attribute description texts, and their corresponding salient regions for the original image includes: The original image is described by attributes, and the corresponding image description text is determined. Attribute text extraction is performed on the image description text corresponding to the original image to determine N attribute description texts; The original image is labeled based on N attribute description texts to determine N salient regions corresponding to the original image.

3. The image classification model training method according to claim 1, characterized in that, The step of performing feature encoding fusion on the original image, N region mask images, the image description text, and N attribute description texts to determine the fused encoding features includes: Image feature encoding processing is performed on the original image and the N region mask images to determine the overall image features corresponding to the original image and the region image features corresponding to the N region mask images; Text feature encoding is performed on the image description text and the N attribute description texts to determine the first text feature corresponding to the image description text and the second text feature corresponding to the N attribute description texts; The overall image features, the first text features, the N regional image features, and the N second text features are alternately spliced ​​together to determine the fused coding features.

4. The image classification model training method according to claim 3, characterized in that, The step of performing image feature encoding processing on the original image and the N region mask images to determine the overall image features corresponding to the original image and the region image features corresponding to the N region mask images includes: The original image is input into the convolutional layer of the visual encoder, and the first image feature corresponding to the original image is output. The first image features are input into the general self-attention encoding layer of the visual encoder to determine the second image features corresponding to the original image; The second image feature and N region mask images are input into the region self-attention coding layer to determine the third image feature and N fourth image features corresponding to the original image; The third image feature and N fourth image features are input to the modal connector for feature dimension adjustment processing to determine the overall image features corresponding to the original image and the regional image features corresponding to the N regional mask images.

5. The image classification model training method according to claim 3, characterized in that, The step of alternatingly concatenating image features and text features from the overall image features, the first text features, N regional image features, and N second text features to determine the fused coding features includes: The overall image features, the first text features, the N regional image features, and the N second text features are alternately spliced ​​together using image features and text features to determine the initial fusion features; The initial fusion features are processed by concatenating the major category identifier and the minor category identifier to determine the fusion coding features.

6. The image classification model training method according to claim 1, characterized in that, The step of training the model based on the fused encoded features to determine the target classification model includes: Pre-training is performed on the modal connectors in the model to be trained using pre-set image-text pairs to determine the pre-trained model. Using the fused encoding features, the visual encoder, language encoder, modal connector, and language model in the pre-trained model are fine-tuned to determine the target classification model.

7. An image classification method, characterized in that, include: Obtain the image to be classified, and determine the image description text corresponding to the image to be classified, as well as the region mask image and attribute description text corresponding to N salient regions in the image to be classified, where N > 2; A target classification model is used to perform classification prediction on the image to be classified, the image description text, N region mask images, and N attribute description texts to determine the target category corresponding to the image to be classified. The target classification model is trained using the image classification model training method described in any one of claims 1 to 6.

8. An image classification model training device, characterized in that, include: The image and text acquisition module is used to acquire the original image, determine the image description text, N attribute description texts and their corresponding salient regions corresponding to the original image, where N>2; The region mask image determination module is used to perform masking processing on each of the aforementioned salient regions to determine N region mask images; The fusion coding feature determination module is used to perform feature coding fusion on the original image, N region mask images, the image description text, and N attribute description text to determine the fusion coding features; The target classification model determination module trains the model based on the fused encoded features to determine the target classification model.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image classification model training method as described in any one of claims 1 to 6, or, when the processor executes the computer program, it implements the image classification method as described in claim 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the image classification model training method as described in any one of claims 1 to 6, or, when the computer program is executed by the processor, it implements the image classification method as described in claim 7.

Images