Training method of image classification model, image classification method and related device thereof

Abstract

The present disclosure relates to a training method of an image classification model, an image classification method and related equipment thereof, the training method comprising: obtaining a scene category attribute of an image classification scene to which a training image set belongs, configuring a preset image classification head of a multi-network cascade according to the scene category attribute to obtain an exclusive image classification head of the image classification scene, splicing the exclusive image classification head to the end of a base model, and jointly fine-tuning the base model and the exclusive image classification head according to the training image set to obtain an image classification model adapted to the image classification scene, wherein the preset image classification head comprises a plurality of feature extraction modules cascaded, and the configuration content comprises at least one of adjustment of model parameters, reduction of the feature extraction modules and addition of a feature projection layer, which can not only meet the accurate classification requirement of the image classification scene, but also avoid repeated development of the network architecture, so as to quickly respond to diversified scene classification requirements.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a training method for an image classification model, an image classification method, and related equipment. Background Technology

[0002] With the rapid development of artificial intelligence in the field of image processing, image classification models have been widely used in many scenarios such as medical, industrial, and daily vision. This has also brought higher technical challenges and requirements to them in terms of classification performance and cross-scenario transfer.

[0003] The current mainstream model training scheme is as follows: first, a base model is used to complete the pre-training process, and then an image classification head is spliced ​​to the end of the obtained base model. Using a labeled dataset for a specific scene, the base model and the image classification head are jointly fine-tuned. The configuration type of the image classification head is either general or scene-specific.

[0004] However, general-purpose image classification heads are poorly adapted to high-precision image classification scenarios and cannot meet the accurate classification requirements of the scenarios. Scenario-specific image classification heads, on the other hand, require redesigning the network architecture for different image classification scenarios, increasing model development costs and model training complexity, thus making it difficult to quickly respond to diverse scenario classification needs. Summary of the Invention

[0005] This disclosure provides a training method for an image classification model, an image classification method, and related equipment to solve the problem of poor adaptability of fixed image classification heads across scenes, and to achieve the goal of fast and accurate cross-scene transfer of image classification models.

[0006] One aspect of this disclosure provides a method for training an image classification model, comprising: Obtain a preset image classification head with multiple cascaded networks. The preset image classification head includes an input adaptation network, a multi-scale feature network, and a classification output network. The multi-scale feature network includes multiple cascaded feature extraction modules. Obtain the scene category attribute of the image classification scene to which the training image set belongs, and configure the preset image classification head according to the scene category attribute to obtain the exclusive image classification head for the image classification scene; The dedicated image classification head is spliced ​​to the end of the base model, and the base model and the dedicated image classification head are jointly fine-tuned according to the training image set to obtain an image classification model adapted to the image classification scenario. The scene category attribute represents the feature set used to distinguish different image classification scenes. The configuration includes adjusting the model parameters of the preset image classification head, deleting the feature extraction module in the multi-scale feature network, and adding at least one of the following to the multi-scale feature network and the classification output network: a feature projection layer that matches the scene category attribute.

[0007] Another aspect of this disclosure provides an image classification method, comprising: A trained image classification model that matches the image classification scene to which the image to be tested belongs is obtained, wherein the image classification model is obtained using the training method of the image classification model described in any embodiment of this disclosure; The image classification model is deployed to obtain the target classification model; The image to be tested is input into the target classification model to obtain the target classification result corresponding to the image to be tested.

[0008] Another aspect of this disclosure provides a training apparatus for an image classification model, comprising: A preset image classification head acquisition module is used to acquire a preset image classification head of multiple cascaded networks. The preset image classification head includes an input adaptation network, a multi-scale feature network, and a classification output network. The multi-scale feature network includes multiple cascaded feature extraction modules. A dedicated image classification head determination module is used to obtain the scene category attribute of the image classification scene to which the training image set belongs, and configure the preset image classification head according to the scene category attribute to obtain the dedicated image classification head of the image classification scene; The image classification model training module is used to attach the dedicated image classification head to the end of the base model, and to jointly fine-tune the base model and the dedicated image classification head according to the training image set to obtain an image classification model adapted to the image classification scenario. The scene category attribute represents the feature set used to distinguish different image classification scenes. The configuration includes adjusting the model parameters of the preset image classification head, deleting the feature extraction module in the multi-scale feature network, and adding at least one of the following to the multi-scale feature network and the classification output network: a feature projection layer that matches the scene category attribute.

[0009] Another aspect of this disclosure provides an image classification method, comprising: The image classification model acquisition module is used to acquire a trained image classification model that matches the image classification scene to which the image to be tested belongs. The image classification model is obtained by the training method of the image classification model described in any embodiment of this disclosure. An image classification model deployment module is used to deploy the image classification model to obtain a target classification model; The target classification result determination module is used to input the image to be tested into the target classification model to obtain the target classification result corresponding to the image to be tested.

[0010] Another aspect of this disclosure provides an electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the training method of the image classification model according to any embodiment of the present disclosure and / or the image classification method according to any embodiment of the present disclosure.

[0011] Another aspect of this disclosure provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the training method of the image classification model described in any embodiment of this disclosure and / or the image classification method described in any embodiment of this disclosure.

[0012] Another aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the training method of the image classification model described in any embodiment of this disclosure and / or the image classification method described in any embodiment of this disclosure.

[0013] The technical solution of this disclosure provides a preset image classification head composed of a cascaded input adaptation network, a multi-scale feature network, and a classification output network. The multi-scale feature network includes multiple cascaded feature extraction modules. Based on the scene category attributes of the image classification scene to which the training image set belongs, the preset image classification head is configured to obtain a dedicated image classification head for the image classification scene. The configuration includes at least one of the following: adjusting model parameters, deleting feature extraction modules, and adding feature projection layers. This solves the problem of poor cross-scene adaptability of fixed image classification heads. The flexible configuration of the preset image classification head can meet the accurate classification requirements of image classification scenes. At the same time, the cross-scene reuse of the preset image classification head avoids the repeated development of networks in different scenes, thereby enabling rapid response to diverse scene classification requirements and balancing cross-scene classification performance and training cost.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating a training method for an image classification model provided in one embodiment of this disclosure; Figure 2 A flowchart illustrating a training method for another image classification model provided in one embodiment of this disclosure; Figure 3 A flowchart illustrating an image classification method provided in one embodiment of this disclosure; Figure 4 A flowchart illustrating a specific example of an image classification method provided in an embodiment of this disclosure; Figure 5 This is a schematic diagram of the structure of a training device for an image classification model provided in one embodiment of the present disclosure; Figure 6 This is a schematic diagram of the structure of an image classification device provided in one embodiment of the present disclosure; Figure 7 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present disclosure. Detailed Implementation

[0017] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.

[0018] It should be noted that the terms "preset," "exclusive," "first," "second," "predicted," and "target," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] Figure 1 This is a flowchart illustrating a training method for an image classification model according to an embodiment of this disclosure. This embodiment is applicable to training image classification models using a training paradigm that employs a pedestal model and an image classification head. The method can be executed by an image classification model training device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes: S110. Obtain the preset image classification head of the multi-network cascade.

[0020] Specifically, the preset image classification head is a multi-network cascade structure pre-built to adapt to different image classification scenarios for multiple image classification scenarios.

[0021] In this embodiment, the preset image classification head includes an input adaptation network, a multi-scale feature network, and a classification output network. The multi-scale feature network includes multiple cascaded feature extraction modules. Specifically, the input adaptation network aligns with the output feature dimensions of the base model, smoothly mapping the general visual features output by the base model to the input feature dimensions required by the input adaptation network, thereby avoiding gradient vanishing caused by abrupt changes in feature dimensions. The multi-scale feature network performs multi-scale feature extraction, strengthening key features relevant to the image classification scenario and suppressing irrelevant noise features. The classification output network performs image category mapping in the image classification scenario.

[0022] In one optional embodiment, the input adaptation network includes a linear mapping layer for linearly mapping the general visual features output by the pedestal model to obtain scene adaptation features. Exemplarily, the linear mapping layer can be a fully connected layer, an affine mapping layer, or a Transformer module, but is not limited to the given examples.

[0023] The feature extraction module aims to extract key feature information from the raw data. Specifically, the network architecture used by each feature extraction module in the multi-scale feature network can be the same or different.

[0024] For example, the feature extraction module can consist of one or more convolutional modules, pooling layers, and activation functions, and the classification output network can be a fully connected layer, support vector machine, decision tree, or nearest neighbor classifier, etc. The specific implementation can be flexibly adjusted and expanded according to the scene coverage requirements, data characteristics, and performance goals, and is not limited to the example given above.

[0025] S120. Obtain the scene category attribute of the image classification scene to which the training image set belongs, and configure the preset image classification head according to the scene category attribute to obtain the exclusive image classification head of the image classification scene.

[0026] Specifically, the training image set contains at least one training image acquired in its respective image classification scenario.

[0027] For example, according to the application field, the scene set can include medical imaging scene, industrial inspection scene and natural image scene, etc.; according to the imaging method, the scene set can include visible light scene, infrared light scene, multispectral scene and tomography scene, etc. The specific division method of image classification scene is not limited here.

[0028] It is understandable that there can be one or more classification criteria. Based on different classification criteria, the resulting sets of scenarios may have overlap or hierarchical nesting relationships. However, each set of scenarios has its own unique distinguishing features. These distinguishing features determine the differentiated design of the dedicated image classification head in terms of structural complexity, parameter scale, and regularization strategy. For example, medical imaging scenarios tend to introduce attention mechanisms to enhance the representation ability of regions of interest, while industrial inspection scenarios focus on lightweight deployment and real-time optimization, and natural image scenarios pay more attention to the model's generalization ability and cross-domain robustness.

[0029] In this embodiment, the scene category attribute represents the set of features used to distinguish different image classification scenes, describes the constraints of scene adaptability modeling, and can be customized according to the different scene requirements or data distribution characteristics of different image classification scenes. This embodiment does not exhaustively limit it.

[0030] In one optional embodiment, the scene category attribute includes at least one of the following: classification accuracy, number of category labels, and category image size corresponding to the image classification scene. Here, classification accuracy represents the minimum threshold of classification accuracy required by the image classification model for the image classification scene; the number of category labels reflects the classification complexity of the image classification scene; and the category image size is the order of magnitude of the smallest training images under the category labels covered by the image classification scene, reflecting the distributional balance of the training image set.

[0031] In this embodiment, the configuration includes adjusting the model parameters of the preset image classification head, deleting the feature extraction module in the multi-scale feature network, and adding at least one of the following to the multi-scale feature network and the classification output network: adding a feature projection layer that matches the scene category attribute.

[0032] The model parameters include the hyperparameters and / or trainable parameters of the preset image classification head. The hyperparameters represent the structured settings that control the model capacity and convergence behavior, such as the learning rate, weight decay coefficient, convolution kernel size, batch size, loss function weights, and gradient clipping threshold. The trainable parameters represent the weights and biases in the model that can be updated through backpropagation.

[0033] Specifically, the multi-scale feature network in the preset image classification head has sufficient redundant feature channels and layer depth to match the needs of various scenarios. Its redundant design reserves ample space for subsequent on-demand reduction. The feature projection layer is used to dynamically adjust the output feature dimension of the multi-scale feature network according to the scene category attribute, avoiding information loss or redundant calculation caused by feature dimension mismatch, thereby avoiding underfitting or overfitting in the joint fine-tuning process.

[0034] In an optional embodiment, configuring the preset image classification head according to the scene category attribute to obtain a dedicated image classification head for the image classification scene includes: adjusting the number of neurons in the classification output network of the preset image classification head to the number of category labels in response to the scene category attribute including the number of category labels; and using the cascaded input adaptation network, multi-scale feature network and adjusted classification output network as the dedicated image classification head for the image classification scene.

[0035] In this embodiment, the classification output network includes a linear mapping layer and a Softmax activation function. The linear mapping layer contains multiple neurons, each corresponding to a category label. By adjusting the number of neurons, the output feature dimension of the dedicated image classification head is made strictly consistent with the number of category labels in the image classification scene. This ensures that the output space of the image classification model is precisely aligned with the semantic space of the image classification scene, avoiding classification ambiguity or redundant computation caused by dimension mismatch. The Softmax activation function transforms the real-valued vector output by the linear mapping layer into a probability distribution, and the sum of the predicted probabilities for each category is always 1, satisfying the probability normalization constraint.

[0036] In another optional embodiment, configuring the preset image classification head according to the scene category attribute to obtain a dedicated image classification head for the image classification scene includes: in response to the scene category attribute including classification accuracy, determining the number of cascaded modules matching the classification accuracy, retaining the number of feature extraction modules in the input adaptation network, classification output network, and multi-scale feature network of the preset image classification head, to obtain a dedicated image classification head for the image classification scene.

[0037] In this embodiment, the classification accuracy is positively correlated with the number of cascaded modules. In multi-scale feature networks, the semantic representation ability of cascaded feature extraction modules increases progressively with the increase of the number of cascades. High-precision image classification scenarios require more cascaded levels to capture deeper discriminative features, while low-precision image classification scenarios can balance inference speed and save computing power by reducing the number of cascaded modules.

[0038] In another optional embodiment, configuring the preset image classification head according to the scene category attribute to obtain a dedicated image classification head for the image classification scene includes: in response to the scene category attribute including classification accuracy, retaining the feature extraction module in the input adaptation network, classification output network, and multi-scale feature network of the preset image classification head that matches the classification accuracy, to obtain a dedicated image classification head for the image classification scene.

[0039] In this embodiment, different feature extraction modules in the multi-scale feature network correspond to different semantic representation capabilities. In high-precision scenarios, feature extraction modules with high semantic representation capabilities, such as attention enhancement modules, are retained to ensure that the capture of fine-grained features meets the high-precision classification requirements. In low-precision scenarios, feature extraction modules with low semantic representation capabilities, such as lightweight convolution modules, can be enabled to balance the classification efficiency of image classification scenarios.

[0040] It is understood that the multi-scale feature network in this embodiment also possesses the progressive enhancement characteristic of semantic representation capabilities, and its hierarchical structure naturally supports progressive abstraction from low-level texture to high-level semantics. In a specific embodiment, while retaining the feature extraction module that matches the classification accuracy, one or more feature extraction modules from adjacent levels can also be retained to further improve the classification accuracy of the dedicated image classification head and its adaptability to the high-precision requirements of image classification scenarios.

[0041] In another optional embodiment, configuring the preset image classification head according to the scene category attribute to obtain a specific image classification head for the image classification scene includes: in response to the scene category attribute including the category image size, when the category image size is less than a first size threshold, configuring the preset image classification head with at least one of the following configurations to obtain a specific image classification head for the image classification scene: 1) Freeze the input adaptation network in the preset image classification head; 2) Adjust the number of trainable parameters of the multi-scale feature network according to the image size of the category; 3) Add a feature projection layer to the multi-scale feature network and the classification output network to reduce the dimensionality of the output features of the multi-scale feature network; 4) Adjust the dropout rate of the Dropout layer in the multi-scale feature network according to the size of the category image, wherein the size of the category image is positively correlated with the dropout rate.

[0042] Specifically, the first scale threshold is used to distinguish between small sample classification scenarios and regular classification scenarios. When the size of the category images is smaller than the first scale threshold, the model is prone to overfitting due to data scarcity.

[0043] In an optional embodiment, the configuration condition for freezing the input adaptation network in the preset image classification head is that the size of the category image is less than a second size threshold. The second size threshold is lower than the first size threshold, used to further distinguish extremely small sample sub-scenes in small sample classification scenarios. In this case, freezing the input adaptation network forces the dedicated image classification head to reuse the underlying feature extraction capabilities of the base model, avoiding the introduction of additional noise due to fine-tuning of the input adaptation network, thereby ensuring the stability and generalization of feature representation when data is extremely scarce.

[0044] Specifically, when the size of the category image is between the second size threshold and the first size threshold, a partial freezing strategy is adopted for the input adaptation network. For example, only the normalization layer parameters in the input adaptation network that are sensitive to data distribution are frozen, while the update capability of the remaining learnable modules is retained. This achieves a dynamic balance between feature alignment accuracy and model robustness under limited samples.

[0045] In this embodiment, the size of the category image is positively correlated with the number of trainable parameters. The trainable parameters represent the network parameters that the multi-scale feature network in the dedicated image classification head can learn and adjust during the joint fine-tuning stage with the base model. These parameters are adjusted accordingly as the size of the category image changes, which can avoid the risk of overfitting caused by parameter redundancy in the dedicated image classification head.

[0046] Specifically, the dimensionality reduction of the feature projection layer can be positively correlated with the size of the class image. That is, the smaller the size of the class image, the lower the dimensionality reduction of the feature projection layer, so as to compress feature redundancy in small sample classification scenarios and reduce the risk of overfitting in the joint fine-tuning stage.

[0047] In this embodiment, the size of the category image is positively correlated with the dropout rate. Specifically, the Dropout layer is used to enhance the generalization ability of the model. Its dropout rate decreases as the size of the category image decreases, thereby reducing the regularization strength of the dedicated image classification head and preventing underlearning of features due to the random deactivation of too many neurons in small sample classification scenarios.

[0048] S130. The dedicated image classification head is spliced ​​to the end of the base model, and the base model and the dedicated image classification head are jointly fine-tuned according to the training image set to obtain an image classification model adapted to the image classification scenario.

[0049] Specifically, a base model refers to a general visual backbone network that has completed supervised pre-training, whose network parameters have fully converged and possess strong generalization feature representation capabilities. For example, the network architecture used in a base model can be a ResNet-50 architecture, a Visual Transformer architecture (ViT-Base), or a ConvNeXt-Tiny architecture, etc.

[0050] In an optional embodiment, the method further includes: preprocessing the original image set to obtain a pre-trained image set, and performing self-supervised pre-training on the base model based on the pre-trained image set to obtain a base model.

[0051] Image preprocessing includes image enhancement and / or image normalization. For example, image enhancement includes, but is not limited to, random cropping, color dithering, normalization, and Gaussian noise injection. Normalization can map image pixel values ​​to the [-1,1] range to improve the robustness of image classification. Image normalization includes, but is not limited to, artifact removal, grayscale normalization, and contrast normalization, which aim to eliminate quality deviations caused by differences in imaging devices.

[0052] Specifically, the base model refers to the original model structure that has not been trained or whose weights have only been randomly initialized. Self-supervised pre-training refers to training the base model by constructing supervision signals using the data itself without the need for manual data labeling. The base model is able to autonomously learn the ability to extract general visual features of images.

[0053] In one optional embodiment, the training tasks of self-supervised pre-training include image masking and / or image reconstruction. The image masking task involves randomly masking local regions of an image and then having the base model predict the pixel values ​​of the masked regions, enabling the base model to gain a deeper understanding of the global semantic structure and local texture relationships of the pre-trained image. The image reconstruction task requires the base model to completely reconstruct the pre-trained image from its reduced-dimensional representation, enhancing its ability to encode and decode detailed image features.

[0054] Taking the ViT-Base network architecture of the base model as an example, the trainable parameters of the first three Transformer encoders in the base model cascade are unfrozen, the general feature extraction capability of the base model's underlying layer is retained, and only the network structure and dedicated image classification head of the base model that are unfrozen and related to the image classification scenario are jointly fine-tuned.

[0055] For example, the loss function used in the joint fine-tuning stage can be the cross-entropy loss function, the optimizer can be the AdamW optimizer, the learning rate can be 0.00001, or a cosine annealing learning rate scheduling strategy can be used to dynamically adjust the learning rate in the joint fine-tuning stage to improve convergence speed and generalization. The weight decay coefficient can be 0.001, and the number of joint fine-tuning rounds can be 20-50, etc., but it is not limited to the example given above.

[0056] In one optional embodiment, a label smoothing technique is introduced during the joint fine-tuning stage to mitigate overfitting during the joint fine-tuning process. For example, the label smoothing coefficient can be 0.1.

[0057] Based on the above embodiments, optionally, according to the training image set, the base model and the dedicated image classification head are jointly fine-tuned to obtain an image classification model adapted to the image classification scenario, including: dividing the training image set into a training set and a validation set according to a preset ratio; jointly fine-tuning the base model and the dedicated image classification head according to the training set to obtain an initial classification model; using the validation set to verify the classification accuracy of the initial classification model; and if the classification accuracy exceeds a preset accuracy threshold, using the initial classification model as the image classification model adapted to the image classification scenario.

[0058] For example, the preset ratio of training set to validation set can be 8:2 or 7:3, and the preset accuracy threshold can be 93%, but it is not limited to the example given above.

[0059] In one alternative embodiment, during backpropagation in the joint fine-tuning phase, a gradient clipping operation is performed at the stitching layer between the base model and the dedicated image classification head. For example, the gradient clipping parameter can be 1.0.

[0060] The advantage of this setup is that it prevents the gradient explosion phenomenon generated by the dedicated image classification head from being transmitted to the base model during backpropagation, thus avoiding the destruction of the general feature extraction capability formed by the base model during the pre-training stage and ensuring the stability of the base model.

[0061] The technical solution of this embodiment obtains a preset image classification head composed of a cascaded input adaptation network, a multi-scale feature network, and a classification output network. The multi-scale feature network includes multiple cascaded feature extraction modules. Based on the scene category attributes of the image classification scene to which the training image set belongs, the preset image classification head is configured to obtain a dedicated image classification head for the image classification scene. The configuration includes at least one of the following: adjusting model parameters, deleting or adding feature projection layers. This solves the problem of poor cross-scene adaptability of fixed image classification heads. The flexible configuration characteristics of the preset image classification head can meet the accurate classification requirements of image classification scenes. At the same time, the cross-scene reuse characteristics of the preset image classification head avoid the repeated development of networks in different scenes, thereby enabling rapid response to diverse scene classification requirements and balancing cross-scene classification performance and training cost.

[0062] Figure 2 This is a flowchart illustrating another training method for an image classification model provided in one embodiment of this disclosure. This embodiment further refines the training method of the base model in the above embodiment. For example... Figure 2 As shown, the method includes: S210. Obtain a pre-training sample set containing at least one pre-training sample.

[0063] In this embodiment, the pre-training samples include pre-training images, text feature vectors matched with the pre-training images, and acquisition parameter features, wherein the acquisition parameter features are obtained by encoding a structured sequence of acquisition parameters from the pre-training images.

[0064] In an optional embodiment, the method further includes: collecting an original image set and corresponding text data and parameter sequence data based on multiple image classification scenarios; performing image preprocessing on the original image set, vectorizing the text data, and encoding the parameter sequence data to obtain a pre-training sample set.

[0065] The original image set contains the original images for each image classification scenario, covering original images with different shooting content and conditions for each scenario. For example, image preprocessing includes image enhancement and / or image normalization. Image normalization includes unifying the resolution of all training images, such as 224×224 or 384×384.

[0066] Specifically, the text data includes descriptive text corresponding one-to-one with the original images in the original image set. The descriptive text describes the semantic content of the original image, such as "red car, frontal view" or "lung CT image, suspected nodule in the upper lobe of the right lung," etc. Specifically, each descriptive text undergoes stop word removal, word segmentation, and word vector mapping to generate a corresponding text feature vector. For example, word vector mapping can be performed using a pre-trained BERT or CLIP text encoder, but is not limited to the given examples.

[0067] Specifically, the parameter sequence data contains a sequence of acquisition parameters that corresponds one-to-one with the original images in the original image set. The acquisition parameters in the sequence describe the structured metadata at the time of acquisition of the original images. Acquisition parameter sequences in different image classification scenarios may contain different acquisition parameters. For example, acquisition parameter sequences in medical imaging scenarios may include slice thickness, device type and resolution, etc. Acquisition parameter sequences in industrial inspection scenarios may include exposure time, lens focal length and light source intensity, etc. Acquisition parameter sequences in natural image scenarios may include camera model, shutter speed, brightness and contrast, etc.

[0068] For example, the encoding method for the acquired parameter sequence can be one-hot encoding, embedding encoding, or numerical encoding, but it is not limited to the given example.

[0069] S220. For each pre-training sample, the text feature vector and the acquisition parameter features in the pre-training sample are concatenated to obtain the dual-modal fusion feature, and the pre-training image in the pre-training sample is input into the base model to obtain the predictive general feature of the pre-training image.

[0070] Specifically, the feature dimension of the bimodal fusion features is consistent with the output feature dimension of the base model. Taking the network architecture of the base model as ViT-Base as an example, the ViT-Base architecture adopts a multi-head attention mechanism with 12 Transformer encoder layers and 12 attention heads. After dividing the input pre-training image into blocks (such as image blocks of size 16×16 and sequence length 196), it extracts 768-dimensional predicted image features. Therefore, the feature dimension of the bimodal fusion features obtained by feature concatenation or feature concatenation mapping is also 768-dimensional.

[0071] S230. Based on the dual-modal fusion features and the general prediction features, the pedestal model is pre-trained under self-supervised supervision to obtain the pedestal model.

[0072] In one alternative embodiment, the training task of the self-supervised pre-training includes at least one of a cross-modal alignment task, a mask modeling task, and a class consistency task.

[0073] The cross-modal alignment task aims to enable the base model to learn the semantic alignment relationship between pre-trained images and their corresponding text descriptions and acquired parameter sequences. Specifically, the pre-trained sample set is divided into positive and negative sample pairs. Features are extracted from the positive and negative sample pairs using the base model and mapped to feature spaces respectively. A contrastive loss function is used to maximize the similarity of the feature spaces corresponding to the positive sample pairs and minimize the similarity of the feature spaces corresponding to the negative sample pairs. The network parameters of the base model are iteratively updated to obtain the base model.

[0074] In this dataset, the positive sample pairs in the positive sample pair set are pre-trained samples, while the negative sample pairs in the negative sample pair set consist of pre-trained images and text descriptions and / or sequences of acquisition parameters that do not match the pre-trained images. For example, the contrastive loss function can be the InfoNCE loss function.

[0075] The masking modeling task includes text masking and / or image masking. The text masking task randomly masks a portion of tokens in pre-trained text, requiring the base model to predict the masked tokens based on the context. The image masking task randomly occludes a portion of a pre-trained image, requiring the base model to reconstruct the pixel values ​​of the occluded region based on the visible region image of the pre-trained model. By reconstructing the masked portion, the masking modeling task enhances the base model's ability to jointly model local details and global semantics. For example, the masking ratio can be 15%, and the modeling loss can be a mean squared error function.

[0076] The category consistency task aims to enable the base model to learn feature information related to category labels in pre-trained images. Specifically, classification is performed on the multimodal fusion features and the predicted general features output by the base model to obtain a first classification result and a second classification result for the pre-trained image. A consistency loss function is used to determine the first classification loss value between the first classification result and the category label of the pre-trained image, and a second classification loss value between the second classification result and the category label of the pre-trained image. The consistency loss value is obtained by weighted summation of the classification loss values, and backpropagation is used to optimize the trainable parameters of the base model to obtain the base model. For example, the classification loss value can use the cross-entropy loss function.

[0077] In one specific embodiment, the training tasks of self-supervised pre-training include at least two of the following: cross-modal alignment task, mask modeling task, and class consistency task. Specifically, the loss values ​​corresponding to each training task are weighted and summed to obtain the pre-training loss value, and the trainable parameters of the base model are updated by backpropagation based on the pre-training loss value.

[0078] For example, during the pre-training phase, the batch size can be 64, the learning rate can be 0.00005, the number of pre-training rounds can be 100, and the weight decay coefficient can be 0.01, but it is not limited to the given example.

[0079] In one optional embodiment, the pre-training sample set is divided into a training set and a validation set, and an early stopping strategy is used to supervise the alignment accuracy of the cross-modal alignment task on the validation set. When the alignment accuracy fails to improve for several consecutive rounds, the self-supervised pre-training process ends, and the base model of this round is saved as the pedestal model to ensure that the generalization ability of the pedestal model on the cross-modal alignment task does not decrease due to overfitting. For example, the early stopping rounds can be 10.

[0080] S240. Obtain the preset image classification head of the multi-network cascade.

[0081] In another optional embodiment, the input adaptation layer includes a linear mapping layer, which is used to linearly map the general visual features output by the base model to obtain linear mapping features, and to perform layer normalization processing on the linear mapping features to obtain scene adaptation features.

[0082] For example, the number of neurons in the linear mapping layer can be 512. When the network architecture of the base model is ViT-Base, the input feature dimension of the linear mapping layer is 768-dimensional and the output feature dimension is 512-dimensional. The normalization parameter used for the layer normalization process can be 0.00001.

[0083] By not configuring an activation function after the linear mapping layer, the original information of general visual features can be preserved, avoiding feature distortion introduced by nonlinear transformation. Layer normalization can avoid training instability caused by feature distribution shift, and improve the convergence speed and generalization performance of the downstream joint fine-tuning stage.

[0084] In another optional embodiment, the input adaptation network includes a linear mapping layer, a linear dimensionality reduction layer, and a feature fusion layer; wherein, the linear mapping layer is used to linearly map the general visual features output by the base model to obtain linear mapping features; the linear dimensionality reduction layer is used to linearly reduce the general visual features to obtain linear dimensionality reduction features; the feature fusion layer is used to element-wise add the linear mapping features and the linear dimensionality reduction features to obtain image residual features, and to perform layer normalization processing on the image residual features to obtain scene adaptation features.

[0085] Among them, the linear dimensionality reduction layer can be a fully connected layer or a convolutional dimensionality reduction module. The linear mapping layer focuses on maintaining the semantic integrity of the visual general features, while the linear dimensionality reduction layer focuses on compressing the feature dimension of the visual general features to reduce computational redundancy. They complement each other in feature expression, so that the output scene-adaptive features have both high-order abstraction capabilities and contain fine-grained spatial structure information.

[0086] Taking the network architecture of the base model as ViT-Base as an example, the feature dimensions of both the linear mapping feature and the linear dimensionality reduction feature are 512-dimensional. The feature dimension of the scene adaptation feature obtained by element-wise addition of the two is still 512-dimensional.

[0087] This embodiment, by setting a residual structure, can alleviate the gradient vanishing problem caused by multi-layer network cascading or multi-layer module cascading, thereby ensuring the stability and effectiveness of gradients during backpropagation, and can significantly improve the convergence efficiency of dedicated image classification heads with long links.

[0088] In an optional embodiment, the feature extraction module includes a first nonlinear layer, a second nonlinear layer, and a stitching layer; wherein, the first nonlinear layer is used to perform nonlinear processing on the input image features to obtain a first nonlinear feature; the stitching layer is used to stitch the image features and the first nonlinear feature to obtain a stitched feature; and the second nonlinear layer is used to perform nonlinear processing on the stitched feature to obtain the module output feature of the feature extraction module.

[0089] Taking the ViT-Base network architecture of the base model as an example, with the input image features having a feature dimension of 512, the first nonlinear layer has 512 neurons and uses the GELU function as the activation function to achieve nonlinear mapping and enhance feature expression capabilities. The second nonlinear layer has 256 neurons and also uses the GELU function as the activation function to extract higher-order semantic abstract features by compressing the feature dimension.

[0090] The stitching layer stitches the input image features and the first nonlinear feature along the channel dimension to form a 1024-dimensional stitched feature. After dimensionality reduction by 1×1 convolution of the second nonlinear layer, it outputs a 256-dimensional module output feature, realizing cross-level multi-scale feature fusion.

[0091] In one optional embodiment, the feature extraction module includes a first fully connected layer, a second fully connected layer, and a channel fusion layer; wherein, the first fully connected layer is used to perform linear dimensionality reduction on the input image features to obtain a first channel weight, and determine a nonlinear channel weight based on the first channel weight and a first activation function; the second fully connected layer is used to perform linear dimensionality increase on the nonlinear channel weight to obtain a second channel weight, and determine a normalized channel weight based on the second channel weight and a second activation function; the channel fusion layer is used to perform element-wise multiplication of the image features channel by channel based on the normalized channel weight to obtain the module output features of the feature extraction module.

[0092] In this embodiment, the feature extraction module performs channel-level weight allocation on the input image features, enabling the module's output features to focus on semantic channels highly relevant to the image classification scenario, while mitigating interference from irrelevant redundant channels. Specifically, the input feature dimension of the first fully connected layer is consistent with the output feature dimension of the second fully connected layer.

[0093] Taking the ViT-Base network architecture of the base model as an example, with the input image features having a feature dimension of 256, the first fully connected layer has an input feature dimension of 256 and an output feature dimension of 64. The first activation function is the ReLU function, which realizes the initial activation of the nonlinear channel weights. The second fully connected layer has an input feature dimension of 64 and an output feature dimension of 256. It works with the second activation function to normalize the channel weights. The second activation function can be the Softmax function, which ensures that the normalized channel weights are in the range of [0,1] and that the sum is 1.

[0094] The channel fusion layer performs channel-by-channel weighting on the input image features based on normalized channel weights, thereby achieving precise enhancement of key semantic channels and adaptive suppression of redundant channels.

[0095] In the feature extraction module of the above embodiment, the image feature is the scene adaptation feature output by the input adaptation network based on the general visual features of the base model, or the image feature is the module output feature of the previous feature extraction module in the multi-scale feature network, and the feature dimension of the image feature is higher than the feature dimension of the module output feature.

[0096] In one alternative embodiment, the multi-scale feature network reduces the feature dimension and strengthens semantic abstraction features by cascading multiple identical feature extraction modules.

[0097] In another optional embodiment, the multi-scale feature network includes a cascaded first feature extraction module and a second feature extraction module. The first feature extraction module includes a first nonlinear layer, a second nonlinear layer, and a concatenation layer. The second feature extraction module includes a first fully connected layer, a second fully connected layer, and a channel fusion layer. The first feature extraction module is used to perform preliminary adaptation of the scene-adaptive features for image classification, while the second feature extraction module is used to perform deep semantic focusing on the module output features of the first feature extraction module.

[0098] Based on the above embodiments, optionally, the multi-scale feature network employs a regularization mechanism. For example, a Dropout layer and an L2 regularization layer are added to the multi-scale feature network to doubly suppress the risk of overfitting during the joint fine-tuning process and improve the generalization ability of the image classification model.

[0099] S250. Obtain the scene category attribute of the image classification scene to which the training image set belongs, and configure the preset image classification head according to the scene category attribute to obtain the exclusive image classification head of the image classification scene.

[0100] S250 in this embodiment is the same as in the above embodiment. Figure 1 The S120 shown is the same or similar, and will not be described again in this embodiment.

[0101] In one specific embodiment, the multi-scale feature network in the preset image classification head includes a first feature extraction module and a second feature extraction module. Specifically, when the image classification scenario is a medical imaging scenario or an industrial inspection scenario, the dedicated image classification head consists of a cascaded input adaptation network, a first feature extraction module, a second feature extraction module, and a classification output network to meet the high sensitivity requirements of fine-grained texture in medical imaging or industrial inspection scenarios, achieving the goal of accurately identifying subtle structural defects. When the image classification scenario is a natural image scenario, the dedicated image classification head consists of a cascaded input adaptation network, a first feature extraction module, and a classification output network. Since natural image scenarios have lower sensitivity to texture details and focus more on the generalization ability and inference efficiency of the image classification model, the computational complexity of the dedicated image classification head is reduced by omitting the second feature extraction module, thus balancing the requirements of coarse-grained classification and real-time response.

[0102] S260. The dedicated image classification head is spliced ​​to the end of the base model, and the base model and the dedicated image classification head are jointly fine-tuned according to the training image set to obtain an image classification model adapted to the image classification scenario.

[0103] S260 in this embodiment is the same as in the above embodiment. Figure 1The S130 shown is the same or similar, and will not be described again in this embodiment.

[0104] In one optional embodiment, attaching the dedicated image classification head to the end of the base model includes: initializing the dedicated image classification head and then attaching it to the end of the base model.

[0105] When the dedicated image classification head includes a linear mapping layer or a fully connected layer, the weight parameters of the linear mapping layer or the fully connected layer are initialized using Xavier to ensure that the weight parameters follow a uniform distribution, thus guaranteeing the consistency and stability of variance when transferring features between layers. When the dedicated image classification head includes a layer normalization or batch normalization layer, its trainable parameters are initialized to a unit scaling factor and a zero bias. When the dedicated image classification head includes a second feature extraction module, the He initialization strategy is used to initialize the weight parameters of the two fully connected layers in the second feature extraction module, and the bias of the Sigmoid function is initialized to zero to improve the learning efficiency and convergence stability of the channel weights.

[0106] This embodiment overcomes the convergence difficulties and feature collapse problems in the joint fine-tuning stage caused by random initialization by setting different initialization strategies for the network structures in the dedicated image classification head, accurately matching the nonlinear characteristics and gradient propagation laws of each network structure.

[0107] The general visual features output by the base model obtained from single-modal pre-training have inherent limitations in understanding image semantics. They are difficult to accurately distinguish visually similar but semantically different images, such as medical images with similar shapes but different lesion levels, or workpiece images with similar appearances but different defect types. This problem directly leads to poor generalization ability of image classification models for different data distributions and classification scenarios, resulting in a significant decrease in classification accuracy. Especially in small sample classification scenarios, the effective semantic information that the model can learn is limited, and it is very easy to use general visual features as the main discrimination criteria, which further aggravates the misclassification rate of image classification models and makes it difficult to meet the practical application requirements of high accuracy and high robustness.

[0108] This embodiment breaks through the limitations of single-modal semantic understanding by employing multimodal features that integrate descriptive text, collected parameter sequences, and images. This enables the base model to represent visual details, semantic information, and scene context. Furthermore, through self-supervised pre-training based on multimodal features and cross-modal training tasks, the base model can learn more discriminative general visual features. This overcomes the technical shortcomings of the aforementioned single-modal pre-training methods and improves the generalization performance and robustness of the image classification model in cross-scene, small-sample, or long-tailed distribution scenarios.

[0109] Figure 3This is a flowchart illustrating an image classification method according to an embodiment of the present disclosure. This embodiment is applicable to classifying images acquired in an image classification scenario. The method can be executed by an image classification device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 3 As shown, the method includes: S310. Obtain a trained image classification model that matches the image classification scene to which the image to be tested belongs.

[0110] The image classification model in this embodiment is obtained using the training method of the image classification model described in any embodiment of this disclosure. The training method of the image classification model will not be described in detail in this embodiment.

[0111] S320. Deploy the image classification model to obtain the target classification model.

[0112] In an optional embodiment, deploying the image classification model to obtain a target classification model includes: obtaining deployment environment parameters of the image classification model; if the deployment environment parameters do not meet the conditions for full deployment, quantizing and compressing the dedicated image classification head in the image classification model, and deploying the compressed image classification model to obtain the target classification image.

[0113] Among them, the deployment environment parameters characterize the set of parameters of hardware, software and performance constraints required for model deployment, and the full deployment conditions represent the conditions under which the image classification model can run stably and efficiently in the current deployment environment without any compression, cropping or other optimization processing.

[0114] For example, deployment environment parameters include deployment device type, hardware resource constraints, inference latency requirements, and energy efficiency ratio. Among these, the deployment device type, whether it is an edge device or a cloud device, is often limited by hardware resources such as limited computing power and insufficient storage capacity. Quantization compression can effectively reduce the model's storage footprint and computational complexity, reduce the computing power consumption during model inference, and thus improve the compatibility between the target classification model and edge devices.

[0115] For example, quantization compression methods may include INT8 fixed-point quantization, FP16 half-precision quantization, pruned quantization, and combinations thereof, but are not limited to the given examples.

[0116] In this embodiment, the step of deploying the image classification model to obtain the target classification model further includes: fully deploying the image classification model to obtain the target classification model when the deployment environment parameters meet the conditions for full deployment.

[0117] In another optional embodiment, deploying the image classification model to obtain the target classification model includes: obtaining deployment environment parameters of the image classification model and obtaining the number of modules corresponding to the feature extraction modules of the dedicated image classification head in the image classification model; if the deployment environment parameters do not meet the full deployment condition and the number of modules is multiple, deleting the earlier feature extraction modules in the dedicated image classification head according to the cascading order of the feature extraction modules in the dedicated image classification head to obtain a compressed image classification model; obtaining the accuracy loss value and model parameter size corresponding to the compressed image classification model; and deploying the compressed image classification model to obtain the target classification model if the accuracy loss value is less than a preset loss threshold and the number of model parameters is less than a preset parameter size threshold.

[0118] In this context, the cascaded order refers to the sequential arrangement of feature extraction modules in the dedicated image classification head from front to back, with the abstract representation capabilities increasing progressively. The earlier feature extraction modules typically undertake basic feature extraction tasks, have a lower level of abstraction, and higher feature redundancy, so prioritizing their removal can balance classification accuracy and lightweight objectives. Conversely, the later feature extraction modules focus on high-level semantic modeling, so prioritizing their removal can easily lead to a significant decrease in classification accuracy.

[0119] Specifically, the accuracy loss value refers to the difference in accuracy between the compressed image classification model and the uncompressed full image classification model on the same test set. The model parameter size refers to the total number of parameters in the dedicated image classification head of the compressed image classification model, and its specific value can be obtained directly by counting the number of trainable parameters in the dedicated image classification head. For example, the preset loss threshold can be 3%, and the preset parameter size threshold can be 500,000, which can be customized and adjusted according to the specific requirements of the image classification scene to which the image to be tested belongs.

[0120] In one alternative embodiment, the compressed image classification model for deployment can be selected by traversing all possible compressed image classification models and selecting the compressed image classification model with the smallest accuracy loss value (less than a preset loss threshold) and the smallest model parameter size (less than a preset parameter threshold); or by using a greedy strategy, sequentially eliminating the earlier feature extraction modules in a cascaded order, evaluating the accuracy loss value and model parameter size after each elimination, until the double threshold constraint is met for the first time, at which point the elimination stops, and the compressed image classification model obtained in this round of elimination is selected for deployment.

[0121] Based on the above embodiments, optionally, the step of deploying the image classification model to obtain the target classification model further includes: when the deployment environment parameters meet the conditions for full deployment, performing full deployment of the image classification model to obtain the target classification model; when the deployment environment parameters do not meet the conditions for full deployment and the number of modules is one, quantizing and compressing the dedicated image classification head in the image classification model, and deploying the resulting compressed image classification model to obtain the target classification model.

[0122] S330. Input the image to be tested into the target classification model to obtain the target classification result corresponding to the image to be tested.

[0123] For example, the target classification result includes the target probability value corresponding to each category label in the image classification scenario and the category label corresponding to the maximum target probability value, wherein the target probability value represents the confidence level that the image to be tested belongs to the corresponding category label.

[0124] Based on the above embodiments, the method may optionally further include: comparing the target classification result with the true category label of the image to be tested, and adding the image to be tested with inconsistent category labels as a misclassified image to the fine-tuning image set; and performing a fine-tuning training on the target classification model based on the fine-tuning image dataset in response to meeting the preset fine-tuning conditions.

[0125] For example, preset fine-tuning conditions may include accumulating the number of images in the fine-tuning image set to a preset number, such as 100 images, or the time interval between the current moment and the previous fine-tuning training reaching a preset period, such as 7 days. Fine-tuning training may only update the parameters of the image classification head in the target classification model.

[0126] The technical solution of this embodiment solves the problem that a fixed image classification model cannot be generalized to adapt to multiple scenarios by adapting the trained image classification model to the image classification scene to which the image to be tested belongs. This achieves accurate classification in diverse image classification scenarios. At the same time, relying on the cross-scenario reuse characteristics of the preset image classification head, it avoids the repeated deployment of the same network structure, thus taking into account both classification performance and response efficiency across scenarios.

[0127] Figure 4This is a flowchart illustrating a specific example of an image classification method provided in one embodiment of this disclosure. Specifically, multimodal raw data is constructed, including original images, descriptive text matching the original images, and a sequence of acquisition parameters. The original images are enhanced to obtain one or more pre-training images. The descriptive text is embedded to obtain text feature vectors that are uniformly matched to each pre-training image. The acquisition parameter sequence is encoded to obtain acquisition parameter features that are uniformly matched to each pre-training image. A pre-training sample set is constructed based on each pre-training image, text feature vector, and acquisition parameter features. Using the pre-training sample set, a base model using the ViT-Base architecture is self-supervised pre-trained to obtain a base model. The training tasks include cross-modal alignment, mask modeling, and class consistency tasks.

[0128] Based on the image classification scene of the training image set, a dedicated image classification head is adapted and configured. The top-level parameters of the base model are unfrozen, and the dedicated image classification head is concatenated at the end. Joint fine-tuning training is then performed based on the training sample set, and the trained image classification model is saved. The image classification model is loaded and deployed according to the image classification scene of the image to be tested, resulting in the target classification model. For each image to be tested, it is determined whether the image classification scene has changed. If so, the training image set corresponding to the image classification scene of the image to be tested is obtained, and the dedicated image classification head is adapted and configured according to the image classification scene of the training image set. If no scene change has occurred, the target classification model is directly called for inference classification. During the inference classification process, it is determined whether the cumulative number of misclassified images in the target classification model is greater than or equal to 100. If so, the fine-tuning training mechanism is triggered, and the image classification head in the target classification model is incrementally updated. If the number is less than 100, inference classification continues for the next image to be tested.

[0129] The following are embodiments of the image classification model training apparatus provided in this disclosure. This apparatus and the image classification model training method described above belong to the same inventive concept. For details not described in detail in the embodiments of the image classification model training apparatus, please refer to the content of the image classification model training method described in the above embodiments.

[0130] Figure 5 This is a schematic diagram of the structure of a training device for an image classification model provided in one embodiment of this disclosure. Figure 5 As shown, the device includes: a preset image classification head acquisition module 410, a dedicated image classification head determination module 420, and an image classification model training module 430.

[0131] The preset image classification head acquisition module 410 is used to acquire a preset image classification head of multiple cascaded networks. The preset image classification head includes an input adaptation network, a multi-scale feature network, and a classification output network. The multi-scale feature network includes multiple cascaded feature extraction modules. The dedicated image classification head determination module 420 is used to obtain the scene category attribute of the image classification scene to which the training image set belongs, and configure the preset image classification head according to the scene category attribute to obtain the dedicated image classification head of the image classification scene. The image classification model training module 430 is used to attach the dedicated image classification head to the end of the base model, and to jointly fine-tune the base model and the dedicated image classification head according to the training image set to obtain an image classification model adapted to the image classification scenario. The scene category attribute represents the feature set used to distinguish different image classification scenes. The configuration includes adjusting the model parameters of the preset image classification head, deleting the feature extraction module in the multi-scale feature network, and adding at least one of the following to the multi-scale feature network and the classification output network: a feature projection layer that matches the scene category attribute.

[0132] The technical solution of this embodiment solves the problem of poor cross-scene adaptability of fixed image classification heads. The flexible configuration of the preset image classification head can meet the accurate classification requirements of image classification scenarios. At the same time, the cross-scene reuse feature of the preset image classification head avoids the repeated development of networks in different scenarios, thereby enabling rapid response to diverse scenario classification requirements and balancing cross-scene classification performance and training cost.

[0133] In one optional embodiment, the scene category attribute includes at least one of the following: classification accuracy, number of category labels, and category image size corresponding to the image classification scene.

[0134] In an optional embodiment, the dedicated image classification head determination module 420 includes: The first dedicated image classification head determination unit is used to, in response to the scene category attribute including classification accuracy, retain the feature extraction module in the input adaptation network, classification output network and multi-scale feature network of the preset image classification head that matches the classification accuracy, to obtain the dedicated image classification head for the image classification scene; or, The number of cascaded modules matching the classification accuracy is determined, and the number of feature extraction modules in the input adaptation network, classification output network, and multi-scale feature network of the preset image classification head is retained to obtain the exclusive image classification head for the image classification scene. The classification accuracy is positively correlated with the number of cascaded modules.

[0135] In an optional embodiment, the dedicated image classification head determination module 420 includes: The second dedicated image classification head determination unit is configured, in response to the scene category attribute including the category image size, to obtain a dedicated image classification head for the image classification scene by configuring the preset image classification head with at least one of the following settings when the category image size is less than a first size threshold: 1) Freeze the input adaptation network in the preset image classification head; 2) Adjust the number of trainable parameters of the multi-scale feature network according to the size of the category image, wherein the size of the category image is positively correlated with the number of trainable parameters; 3) Add a feature projection layer to the multi-scale feature network and the classification output network to reduce the dimensionality of the output features of the multi-scale feature network; 4) Adjust the dropout rate of the Dropout layer in the multi-scale feature network according to the size of the category image, wherein the size of the category image is positively correlated with the dropout rate.

[0136] In an optional embodiment, the dedicated image classification head determination module 420 includes: The third dedicated image classification head determination unit is used to adjust the number of neurons in the classification output network of the preset image classification head to the number of category labels in response to the scene category attribute including the number of category labels; The cascaded input adaptation network, multi-scale feature network, and adjusted classification output network are used as the dedicated image classification head for the image classification scenario.

[0137] In one optional embodiment, the feature extraction module includes a first nonlinear layer, a second nonlinear layer, and a splicing layer; The first nonlinear layer is used to perform nonlinear processing on the input image features to obtain the first nonlinear feature; the stitching layer is used to stitch the image features and the first nonlinear feature to obtain the stitched feature; the second nonlinear layer is used to perform nonlinear processing on the stitched feature to obtain the module output feature of the feature extraction module. Wherein, the image feature is the scene adaptation feature output by the input adaptation network based on the general visual features of the base model, or the image feature is the module output feature of the previous feature extraction module in the multi-scale feature network, and the feature dimension of the image feature is higher than the feature dimension of the module output feature.

[0138] In one optional embodiment, the feature extraction module includes a first fully connected layer, a second fully connected layer, and a channel fusion layer; The first fully connected layer is used to perform linear dimensionality reduction on the input image features to obtain the first channel weights, and determine the nonlinear channel weights based on the first channel weights and the first activation function; the second fully connected layer is used to perform linear dimensionality increase on the nonlinear channel weights to obtain the second channel weights, and determine the normalized channel weights based on the second channel weights and the second activation function; the channel fusion layer is used to perform element-wise multiplication of the image features channel by channel based on the normalized channel weights to obtain the module output features of the feature extraction module. Wherein, the image features are scene adaptation features output by the input adaptation network based on the general visual features input by the base model, or the image features are module output features of the previous feature extraction module in the multi-scale feature network.

[0139] In one optional embodiment, the input adaptation network includes a linear mapping layer, a linear dimensionality reduction layer, and a feature fusion layer; The linear mapping layer is used to linearly map the general visual features output by the base model to obtain linear mapping features; the linear dimensionality reduction layer is used to linearly reduce the general visual features to obtain linear dimensionality reduction features; the feature fusion layer is used to element-wise add the linear mapping features and the linear dimensionality reduction features to obtain image residual features, and to perform layer normalization processing on the image residual features to obtain scene adaptation features.

[0140] In an optional embodiment, the device further includes: The base model pre-training module is used to obtain a pre-training sample set containing at least one pre-training sample. The pre-training sample includes a pre-training image, a text feature vector matched with the pre-training image, and acquisition parameter features. The acquisition parameter features are obtained by encoding a structured acquisition parameter sequence of the pre-training image. For each pre-training sample, the text feature vector and the acquisition parameter features in the pre-training sample are concatenated to obtain the dual-modal fusion feature, and the pre-training image in the pre-training sample is input into the base model to obtain the predictive general feature of the pre-training image; Based on the dual-modal fusion features and the general prediction features, the base model is pre-trained in a self-supervised manner to obtain the base model. The training task of the self-supervised pre-training includes at least one of the following: cross-modal alignment task, mask modeling task, and category consistency task.

[0141] The image classification model training apparatus provided in this disclosure can execute the image classification model training method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0142] The following are embodiments of the image classification device provided in this disclosure. This device and the image classification method in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the image classification device, please refer to the content about the image classification method in the above embodiments.

[0143] Figure 6 This is a schematic diagram of the structure of an image classification device provided in one embodiment of this disclosure. Figure 6 As shown, the device includes: an image classification model acquisition module 510, an image classification model deployment module 520, and a target classification result determination module 530.

[0144] The image classification model acquisition module 510 is used to acquire a trained image classification model that matches the image classification scene to which the image to be tested belongs. The image classification model is obtained by the training method of the image classification model described in any embodiment of this disclosure. The image classification model deployment module 520 is used to deploy the image classification model to obtain the target classification model; The target classification result determination module 530 is used to input the image to be tested into the target classification model to obtain the target classification result corresponding to the image to be tested.

[0145] The technical solution of this embodiment solves the problem that a fixed image classification model cannot be generalized to adapt to multiple scenarios by adapting the trained image classification model to the image classification scene to which the image to be tested belongs. This achieves accurate classification in diverse image classification scenarios. At the same time, relying on the cross-scenario reuse characteristics of the preset image classification head, it avoids the repeated deployment of the same network structure, thus taking into account both classification performance and response efficiency across scenarios.

[0146] In one optional embodiment, the image classification model deployment module 520 is specifically used for: Obtain the deployment environment parameters of the image classification model, and obtain the number of modules corresponding to the feature extraction module of the dedicated image classification head in the image classification model; If the deployment environment parameters do not meet the conditions for full deployment and there are multiple modules, the feature extraction modules at the beginning of the dedicated image classification head are deleted according to the cascading order of the feature extraction modules in the dedicated image classification head to obtain a compressed image classification model. Obtain the accuracy loss value and model parameter scale corresponding to the compressed image classification model; When the accuracy loss value is less than a preset loss threshold and the number of model parameters is less than a preset parameter threshold, the compressed image classification model is deployed to obtain the target classification model.

[0147] The image classification apparatus provided in this disclosure can execute the image classification method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0148] Figure 7 This is a schematic diagram of an electronic device provided according to one embodiment of the present disclosure. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0149] like Figure 7 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor 11. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0150] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information or data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0151] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the image classification method provided in the above embodiments.

[0152] In some embodiments, the image classification method provided in the above embodiments can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the image classification method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the image classification method by any other suitable means (e.g., by means of firmware).

[0153] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of embodiments of this disclosure.

[0154] Various embodiments of the systems and techniques described above can be implemented in the following systems or combinations thereof: digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), system-on-chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0155] Computer programs for implementing the image classification methods of this disclosure can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0156] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable storage medium. Examples of machine-readable storage media include, based on an electrical connection of at least one wire, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0157] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a cathode-ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0158] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0159] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0160] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0161] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A training method for an image classification model, characterized in that, include: Obtain a preset image classification head with multiple cascaded networks. The preset image classification head includes an input adaptation network, a multi-scale feature network, and a classification output network. The multi-scale feature network includes multiple cascaded feature extraction modules. Obtain the scene category attribute of the image classification scene to which the training image set belongs, and configure the preset image classification head according to the scene category attribute to obtain the exclusive image classification head for the image classification scene; The dedicated image classification head is spliced ​​to the end of the base model, and the base model and the dedicated image classification head are jointly fine-tuned according to the training image set to obtain an image classification model adapted to the image classification scenario. The scene category attribute represents the feature set used to distinguish different image classification scenes. The configuration includes adjusting the model parameters of the preset image classification head, deleting the feature extraction module in the multi-scale feature network, and adding at least one of the following to the multi-scale feature network and the classification output network: a feature projection layer that matches the scene category attribute.

2. The training method according to claim 1, characterized in that, The scene category attribute includes at least one of the following: classification accuracy, number of category labels, and category image size corresponding to the image classification scene.

3. The training method according to claim 2, characterized in that, The step of configuring the preset image classification head according to the scene category attribute to obtain a dedicated image classification head for the image classification scene includes: In response to the scene category attribute including classification accuracy, the feature extraction modules in the input adaptation network, classification output network, and multi-scale feature network of the preset image classification head that match the classification accuracy are retained to obtain a dedicated image classification head for the image classification scene. or, The number of cascaded modules matching the classification accuracy is determined, and the number of feature extraction modules in the input adaptation network, classification output network, and multi-scale feature network of the preset image classification head is retained to obtain the exclusive image classification head for the image classification scene. The classification accuracy is positively correlated with the number of cascaded modules.

4. The training method according to claim 2, characterized in that, The step of configuring the preset image classification head according to the scene category attribute to obtain a dedicated image classification head for the image classification scene includes: In response to the scene category attribute including category image size, if the category image size is less than a first size threshold, the preset image classification head is configured with at least one of the following to obtain a dedicated image classification head for the image classification scene: 1) Freeze the input adaptation network in the preset image classification head; 2) Adjust the number of trainable parameters of the multi-scale feature network according to the size of the category image, wherein the size of the category image is positively correlated with the number of trainable parameters; 3) Add a feature projection layer to the multi-scale feature network and the classification output network to reduce the dimensionality of the output features of the multi-scale feature network; 4) Adjust the dropout rate of the Dropout layer in the multi-scale feature network according to the size of the category image, wherein the size of the category image is positively correlated with the dropout rate.

5. The training method according to claim 2, characterized in that, The step of configuring the preset image classification head according to the scene category attribute to obtain a dedicated image classification head for the image classification scene includes: In response to the scene category attribute including the number of category labels, the number of neurons in the classification output network of the preset image classification head is adjusted to the number of category labels; The cascaded input adaptation network, multi-scale feature network, and adjusted classification output network are used as the dedicated image classification head for the image classification scenario.

6. The training method according to claim 1, characterized in that, The feature extraction module includes a first nonlinear layer, a second nonlinear layer, and a splicing layer; The first nonlinear layer is used to perform nonlinear processing on the input image features to obtain the first nonlinear feature; The stitching layer is used to stitch the image features and the first nonlinear feature to obtain stitched features; the second nonlinear layer is used to perform nonlinear processing on the stitched features to obtain the module output features of the feature extraction module. Wherein, the image feature is the scene adaptation feature output by the input adaptation network based on the general visual features of the base model, or the image feature is the module output feature of the previous feature extraction module in the multi-scale feature network, and the feature dimension of the image feature is higher than the feature dimension of the module output feature.

7. The training method according to claim 1, characterized in that, The feature extraction module includes a first fully connected layer, a second fully connected layer, and a channel fusion layer; The first fully connected layer is used to perform linear dimensionality reduction on the input image features to obtain the first channel weights, and to determine the nonlinear channel weights based on the first channel weights and the first activation function. The second fully connected layer is used to linearly increase the dimensionality of the nonlinear channel weights to obtain the second channel weights, and to determine the normalized channel weights based on the second channel weights and the second activation function; the channel fusion layer is used to perform element-wise multiplication of the image features channel by channel based on the normalized channel weights to obtain the module output features of the feature extraction module. Wherein, the image features are scene adaptation features output by the input adaptation network based on the general visual features input by the base model, or the image features are module output features of the previous feature extraction module in the multi-scale feature network.

8. The training method according to claim 1, characterized in that, The input adaptation network includes a linear mapping layer, a linear dimensionality reduction layer, and a feature fusion layer; The linear mapping layer is used to linearly map the general visual features output by the base model to obtain linear mapping features; the linear dimensionality reduction layer is used to linearly reduce the general visual features to obtain linear dimensionality reduction features; the feature fusion layer is used to element-wise add the linear mapping features and the linear dimensionality reduction features to obtain image residual features, and to perform layer normalization processing on the image residual features to obtain scene adaptation features.

9. The training method according to any one of claims 1-8, characterized in that, Also includes: Obtain a pre-training sample set containing at least one pre-training sample. The pre-training sample includes a pre-training image, a text feature vector matched with the pre-training image, and acquisition parameter features. The acquisition parameter features are obtained by encoding a structured sequence of acquisition parameters of the pre-training image. For each pre-training sample, the text feature vector and the acquisition parameter features in the pre-training sample are concatenated to obtain the dual-modal fusion feature, and the pre-training image in the pre-training sample is input into the base model to obtain the predictive general feature of the pre-training image; Based on the dual-modal fusion features and the general prediction features, the base model is pre-trained in a self-supervised manner to obtain the base model. The training task of the self-supervised pre-training includes at least one of the following: cross-modal alignment task, mask modeling task, and category consistency task.

10. An image classification method, characterized in that, include: Obtain a trained image classification model that matches the image classification scene to which the image to be tested belongs, wherein the image classification model is obtained using the training method of the image classification model as described in any one of claims 1-9; The image classification model is deployed to obtain the target classification model; The image to be tested is input into the target classification model to obtain the target classification result corresponding to the image to be tested.

11. The image classification method according to claim 10, characterized in that, The process of deploying the image classification model to obtain the target classification model includes: Obtain the deployment environment parameters of the image classification model, and obtain the number of modules corresponding to the feature extraction module of the dedicated image classification head in the image classification model; If the deployment environment parameters do not meet the conditions for full deployment and there are multiple modules, the feature extraction modules at the beginning of the dedicated image classification head are deleted according to the cascading order of the feature extraction modules in the dedicated image classification head to obtain a compressed image classification model. Obtain the accuracy loss value and model parameter scale corresponding to the compressed image classification model; When the accuracy loss value is less than a preset loss threshold and the number of model parameters is less than a preset parameter threshold, the compressed image classification model is deployed to obtain the target classification model.

12. A training device for an image classification model, characterized in that, include: A preset image classification head acquisition module is used to acquire a preset image classification head of multiple cascaded networks. The preset image classification head includes an input adaptation network, a multi-scale feature network, and a classification output network. The multi-scale feature network includes multiple cascaded feature extraction modules. A dedicated image classification head determination module is used to obtain the scene category attribute of the image classification scene to which the training image set belongs, and configure the preset image classification head according to the scene category attribute to obtain the dedicated image classification head of the image classification scene; The image classification model training module is used to attach the dedicated image classification head to the end of the base model, and to jointly fine-tune the base model and the dedicated image classification head according to the training image set to obtain an image classification model adapted to the image classification scenario. The scene category attribute represents the feature set used to distinguish different image classification scenes. The configuration includes adjusting the model parameters of the preset image classification head, deleting the feature extraction module in the multi-scale feature network, and adding at least one of the following to the multi-scale feature network and the classification output network: a feature projection layer that matches the scene category attribute.

13. An image classification device, characterized in that, include: The image classification model acquisition module is used to acquire a trained image classification model that matches the image classification scene to which the image to be tested belongs. The image classification model is obtained by the training method of the image classification model as described in any one of claims 1-9. An image classification model deployment module is used to deploy the image classification model to obtain a target classification model; The target classification result determination module is used to input the image to be tested into the target classification model to obtain the target classification result corresponding to the image to be tested.

14. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the training method of the image classification model according to any one of claims 1-9 and / or the image classification method according to any one of claims 10-11.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute and implement the training method of the image classification model according to any one of claims 1-9 and / or the image classification method according to any one of claims 10-11.

16. A computer program product comprising a computer program that, when executed by a processor, implements a training method for an image classification model according to any one of claims 1-9 and / or an image classification method according to any one of claims 10-11.

Images