Image classification method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide an image classification method and device, electronic equipment and storage medium. The method comprises: obtaining an image to be classified; inputting the image to be classified into a trained image classification model to obtain a classification result of the image to be classified, wherein a model loss in a training process of the image classification model is obtained by loss calculation based on a first target feature, a second target feature and a classification label of a sample image, the first target feature is obtained from a plurality of first image features obtained by encoding processing of the sample image by an initial image classification model, and the second target feature is obtained from a plurality of second image features obtained by decoding processing of a target image feature by the initial image classification model. Since the model loss is calculated based on the plurality of image features obtained in the encoding and decoding processes, the model loss calculated has high accuracy, so that the classification result obtained by classifying the image to be classified based on the trained image classification model is more accurate.

Description

Technical Field

[0001] This application relates to the fields of image processing and machine learning technology, and more specifically, to an image classification method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of computer technology, image processing technology has an increasingly wide range of applications. Through image processing technology, images can be classified based on their content and image classification models.

[0003] Currently, when training image classification models, the image label information is usually used directly for training. However, the accuracy of the image classification model trained in this way is poor, and consequently, the classification results obtained by using the trained image classification model are also inaccurate. Summary of the Invention

[0004] In view of this, embodiments of this application propose an image classification method, apparatus, electronic device, and storage medium, which can classify images using a more accurate image classification model, thereby improving the accuracy of image classification.

[0005] In a first aspect, embodiments of this application provide an image classification method, the method comprising: acquiring an image to be classified; inputting the image to be classified into a trained image classification model to obtain a classification result of the image to be classified, wherein the model loss during the training process of the image classification model is calculated based on a first target feature, a second target feature, and a classification identifier of a sample image, wherein the first target feature is obtained by encoding a sample image using an initial image classification model to obtain multiple first image features, each first image feature having a different size of its corresponding feature map, and the second target feature is obtained by decoding a target image feature among the multiple first image features using an initial image classification model to obtain multiple second image features, each second image feature having a different size of its corresponding feature map.

[0006] Secondly, embodiments of this application provide an image classification apparatus, comprising: an image acquisition module and a classification module. The image acquisition module is used to acquire an image to be classified; the classification module is used to input the image to be classified into a trained image classification model to obtain a classification result for the image to be classified. The model loss during the training process of the image classification model is calculated based on a first target feature, a second target feature, and a classification identifier of the sample image. The first target feature is obtained by encoding multiple first image features of the sample image using an initial image classification model, and the feature maps corresponding to each first image feature have different sizes. The second target feature is obtained by decoding multiple target image features among the multiple first image features using an initial image classification model, and the feature maps corresponding to each second image feature have different sizes.

[0007] In one possible implementation, the device further includes a sample acquisition module, a sampling module, an encoding module, a processing module, a loss calculation module, and a training module. The sample acquisition module acquires sample images, which have classification identifiers. The encoding module encodes the sample images using an image classification model to obtain multiple first image features, each first image feature corresponding to a feature map of different sizes. The decoding module decodes a target image feature among the multiple first image features using the image classification model to obtain multiple second image features, each second image feature corresponding to a feature map of different sizes. The processing module processes the multiple first image features to obtain a first target feature and processes the multiple second image features to obtain a second target feature. The loss calculation module calculates a model loss based on the first target feature, the second target feature, and the classification identifier of the sample image. The training module trains the image classification model based on the model loss.

[0008] In one possible implementation, the image classification model includes an encoding network comprising multiple cascaded encoding layers. The encoding module includes an encoding submodule, a feature output submodule, and a first feature acquisition submodule. The encoding submodule is used to downsample and encode the sample image using the first encoding layer of the cascaded encoding layers to obtain image features. The feature output submodule is used to output the image features encoded by the first encoding layer to subsequent cascaded encoding layers, so that downsampling and encoding of the encoded image features can continue in subsequent cascaded encoding layers until the last encoding layer. The first feature acquisition submodule is used to acquire the image features output by at least two encoding layers in the cascaded encoding layers to obtain at least two first image features, wherein the at least two first image features include the image features output by the last encoding layer.

[0009] In one possible implementation, the image classification model further includes a decoding network comprising multiple cascaded decoding layers, each decoding layer being connected to a corresponding encoding layer. The decoding module includes a decoding submodule, a fusion submodule, and a second feature acquisition submodule. The decoding submodule is used to upsample and decode the image features output by the last encoding layer through the first decoding layer in the cascaded multiple decoding layers to obtain decoded image features. The fusion submodule is used to fuse the upsampled and decoded image features with the image features output by the encoding layer corresponding to the first decoding layer to obtain fused image features. The fused image features are then output as the final decoded image features of the first decoding layer to subsequent cascaded decoding layers, so that upsampling decoding, fusion, and final output of decoded image features continue in subsequent cascaded decoding layers until the last decoding layer. The second feature acquisition submodule is used to acquire the image features output by at least two decoding layers in the cascaded multiple decoding layers to obtain at least two second image features, wherein each of the at least two decoding layers is a decoding layer cascaded with an encoding layer that outputs the first image features.

[0010] In one possible implementation, the coding network includes multiple convolutional residual modules, and each coding layer consists of at least one convolutional residual module.

[0011] In one possible implementation, the loss calculation module is further configured to: perform contrast loss calculation on the first target feature and the second target feature to obtain the contrast loss of the image classification model; perform classification calculation on the image features output by the last coding layer using the image classification model to obtain the classification result of the sample image; determine the classification loss of the image classification model based on the classification result and classification label of the sample image; and determine the model loss of the image classification model based on the classification loss and the contrast loss.

[0012] In one possible implementation, the loss calculation module is further configured to: perform a contrast loss calculation on the first target feature and the second target feature to obtain the contrast loss of the image classification model; perform a classification calculation on the first target feature or the second target feature using the image classification model to obtain the classification result of the sample image; determine the classification loss of the image classification model based on the classification result and classification identifier of the sample image; and determine the model loss of the image classification model based on the classification loss and the contrast loss.

[0013] In one possible implementation, the processing module includes a first processing submodule, a first stitching submodule, a second processing submodule, and a second stitching submodule. The first processing submodule is used to perform dimensionality reduction processing on the plurality of first image features respectively, obtaining a plurality of dimensionality-reduced first image features, each with the same dimension. The first stitching submodule is used to stitch the plurality of dimensionality-reduced first image features together to obtain a first target feature. The second processing submodule is used to perform dimensionality reduction processing on the plurality of second image features respectively, obtaining dimensionality-reduced second image features, each with the same dimension as the dimensionality-reduced first image features. The second stitching submodule is used to stitch the plurality of dimensionality-reduced second image features together to obtain a second target feature, and then stitch these second target features together to obtain a second target feature.

[0014] In one possible implementation, the training module is further configured to adjust the parameters of the image classification model based on the model loss to minimize the model loss and obtain a trained image classification model.

[0015] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory; one or more programs are stored in the memory and configured to be executed by the processor to implement the above-described method.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing program code, wherein the above-described method is executed when the program code is run by a processor.

[0017] Fifthly, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device retrieves the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described above.

[0018] This application provides an image classification method, apparatus, electronic device, and storage medium. The method includes: acquiring an image to be classified; inputting the image to be classified into a trained image classification model to obtain a classification result for the image to be classified. The model loss during the training process of the image classification model is calculated based on a first target feature, a second target feature, and a classification identifier of a sample image. The first target feature is obtained by encoding multiple first image features of the sample image using an initial image classification model, with each first image feature corresponding to a feature map of different sizes. The second target feature is obtained by decoding multiple target image features from the multiple first image features using an initial image classification model, with each second image feature corresponding to a feature map of different sizes. By employing the method of this application, the model loss of the trained image classification model used in the image classification process is calculated jointly based on multiple image features obtained during encoding and decoding. Therefore, the model loss used in the model training stage has high accuracy, making the trained image classification model based on the model loss more accurate. Consequently, the classification result obtained by using the trained image classification model to classify the image is also more accurate. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This illustration shows an application scenario diagram of an image classification model training method provided in an embodiment of this application;

[0021] Figure 2 A flowchart illustrating an image classification method proposed in an embodiment of this application is shown;

[0022] Figure 3 This illustration shows a schematic diagram of classifying images to be classified according to an embodiment of this application;

[0023] Figure 4 This paper shows another schematic flowchart of an image classification training method proposed in an embodiment of this application;

[0024] Figure 5 A schematic diagram of the structure of an encoding network proposed in an embodiment of this application is shown;

[0025] Figure 6 This paper shows a schematic diagram of the structure of an image classification model proposed in an embodiment of this application;

[0026] Figure 7 This illustration shows another flowchart of an image classification training method proposed in an embodiment of this application;

[0027] Figure 8 This paper shows a schematic diagram of the structure of a residual module provided in an embodiment of this application;

[0028] Figure 9 This illustrates another structural intent of an image classification model provided in an embodiment of this application;

[0029] Figure 10 This diagram shows a connection block diagram of an image classification module device provided in an embodiment of this application;

[0030] Figure 11 Another connection block diagram of an image classification device provided in an embodiment of this application is shown;

[0031] Figure 12 A structural block diagram of an electronic device for performing the methods of embodiments of this application is shown. Detailed Implementation

[0032] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0033] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0034] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0035] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0036] It should be noted that "multiple" as mentioned in this article refers to two or more.

[0037] With the research and advancement of artificial intelligence technology, it has been studied and applied in multiple fields, playing an increasingly important role.

[0038] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Let's take the application of AI in machine learning as an example:

[0039] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. The solution in this application primarily utilizes machine learning for image classification.

[0040] Currently, when using machine learning to train image classification models for image classification, the convolutional networks used in machine learning are mostly focused on higher-level semantic analysis. However, for images with fewer features (such as simple drawings, emojis, illustrations, animations, and comics), there may be feature omissions when extracting higher-level semantic features during the training process of existing image classification models. This leads to poor generalization performance of the trained image classification model, and consequently, inaccurate classification results when using the trained image classification model to classify such images with fewer features.

[0041] Therefore, this application provides an image classification model training method. The method encodes sample images using an image classification model to obtain multiple first image features. It then decodes the target image features from these first image features using the same model to obtain multiple second image features. A model loss is calculated based on the multiple first image features, the multiple second image features, and the classification label of the sample image. This model loss is then used to train the classification model. During model training, the model loss is calculated jointly using multiple image features obtained during downsampling and decoding to avoid missing features during feature extraction. This improves the accuracy of the model loss and the accuracy of the image classification model trained based on the model loss, resulting in more accurate classification results for subsequent image classification based on the trained model. It should be understood that the above model can also be applied to most products that require training deep neural networks for classification, improving the model's generalization performance and accuracy on test sets and online data.

[0042] Before proceeding with specific explanations, the terms used in this application are explained as follows:

[0043] Sample images: Sample images can be images identified with category information, and the images should include image features corresponding to the category information. Different images contain different image features, and the number of features also varies. In one possible implementation of this application, the sample image can specifically be an image with fewer image features but containing category information, such as a line drawing image, a sketch image, an animation image, an expression image, a comic image, a game screen, or a model figurine image, etc.

[0044] Image to be classified: An image to be classified refers to any image that needs to be classified, such as an image with fewer image features as mentioned above. It should be understood that the image to be classified can also be an image with more image features.

[0045] Image features include color features, texture features, shape features, and spatial relationship features. Color features are global features that describe the surface properties of objects within an image or image region; they are pixel-based. Texture features are also global features, describing the surface properties of objects within an image or image region. Unlike color features, texture features are not pixel-based; they require statistical calculations within a region containing multiple pixels. Shape features have two representation methods: contour features and region features. Contour features primarily target the outer boundaries of objects, while region features relate to the entire shape region. Spatial relationship features refer to the spatial positions or relative directions between multiple targets segmented from an image. These relationships can be categorized as connectivity / adjacency, overlap / intersection, and containment / enclosure. Spatial position information is typically divided into two categories: relative spatial position information and absolute spatial position information. The former emphasizes the relative positions between targets, such as up / down and left / right relationships, while the latter emphasizes the distance and orientation between targets.

[0046] Image classification models refer to the end-to-end training of deep learning models (such as convolutional neural network models) on a large number of labeled sample images. A well-trained image classification model can classify images with relatively high accuracy. For example, it can distinguish the types of animals, the types of flowers and plants, and whether an image is classified as normal or indecent. It should be understood that indecent images can include several different categories.

[0047] The following describes an exemplary application of the device provided in the embodiments of the present invention for performing the above-described image classification model training method. The image classification model training method provided in the embodiments of the present invention can be applied to, for example... Figure 1 In the server within the application environment shown.

[0048] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this application, such as... Figure 1 As shown, this application scenario includes a terminal device 10 and a server 20 that communicates with the terminal device 10 via a network.

[0049] Terminal device 20 may be equipped with a client for displaying images to be classified, such as a browser client, instant messaging client, educational client, social networking client, shopping client, and audio / video playback client.

[0050] The network can be a wide area network (WAN), a local area network (LAN), or a combination of both. Terminal device 10 can be a smartphone, smart TV, tablet, laptop, or desktop computer, etc.

[0051] Server 10 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0052] If using such Figure 1 The terminal device 10 and server 20 in the process of training an image classification model and classifying images to obtain the category of the image to be classified can be performed as follows: The user can upload a sample image to the server 20 through the terminal device 10. When the server 20 obtains the sample image, it uses the image classification model to encode the sample image, obtaining multiple first image features, each with a different feature map size. The image classification model then decodes the target image features among the multiple first image features, obtaining multiple second image features, each with a different feature map size. The multiple first image features are processed to obtain a first target feature, and the multiple second image features are processed to obtain a second target feature. The model loss is calculated based on the first target feature, the second target feature, and the classification label of the sample image. The image classification model is then trained based on the model loss. When the image classification model converges, the training of the image classification model is complete. At this point, the user can also send the image to be classified to the server 20 through the terminal device 10. After obtaining the image to be classified, the server 20 uses the trained image classification model to classify the image and then sends the classification result back to the terminal device 10. So that the terminal device 10 can display the classification result of the image to be classified.

[0053] For example, if the category label corresponding to the sample image includes normal image or abnormal image, and the abnormal image specifically includes first type image, second type image and third type image, then the classification result obtained by using the trained image classification model to classify the image to be classified can be normal image or abnormal image, and the abnormal image can specifically be first type image, second type image or third type image. Figure 2 The image shown is a simple line drawing of a dog. If the image to be classified is classified using the trained image classification model described above, the result is a normal image.

[0054] It should be understood that the above-described method steps can also be executed by only the terminal device 10 or only by the server 20. That is, the method steps described above are merely illustrative and are not intended to limit this solution.

[0055] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0056] Please see Figure 2 , Figure 2 The present application also provides an image classification method, which includes:

[0057] Step S110: Obtain the image to be classified.

[0058] The images to be classified can be user-drawn images, images captured using a terminal device, images obtained from web pages, etc., without specific limitations. The sample images mentioned above can be images with fewer image features, such as user-drawn sketches, illustrations, comics, and emoticons, or images with more image features, such as photographs of scenery, without specific limitations.

[0059] In one possible implementation of this application, the image to be classified may be a user-drawn sketch, illustration, comic, or emoticon, or an image with few image features.

[0060] Step S120: Input the image to be classified into the trained image classification model to obtain the classification result of the image to be classified.

[0061] In the training process of the image classification model, the model loss is calculated based on the first target feature, the second target feature, and the classification label of the sample image. The first target feature is obtained by encoding multiple first image features of the sample image according to the initial image classification model. The feature map corresponding to each first image feature has a different size. The second target feature is obtained by decoding multiple target image features among the multiple first image features according to the initial image classification model. The feature map corresponding to each second image feature has a different size.

[0062] The image classification model mentioned above can be a convolutional neural network model capable of performing image classification. Specifically, the convolutional neural network mentioned above can be ResNet (Residual Neural Network), DenseNet (Densely Connected Convolutional Networks), SENet (Squeeze-and-Extraction Networks), etc. There are no specific limitations here; it can be selected according to actual needs.

[0063] The sample image is encoded using an initial image classification model. Specifically, this can be achieved by encoding the sample image using the encoding network within the initial image classification model. The encoding network can include multiple encoding layers. Specifically, the sample image can be encoded sequentially using multiple encoding layers (i.e., when encoding sequentially using multiple encoding layers, the output of the previous layer becomes the input of the next layer connected to it), thereby obtaining the image features output by each encoding layer. At least two image features are selected from the image features output by each encoding layer as the first image feature. The first target feature can be obtained by dimensionality reduction of the selected at least two first image features to the same dimension and then concatenating them; alternatively, it can be obtained by fusing the selected at least two first image features. No specific limitation is made here.

[0064] The target image features are decoded using the initial image classification model. Specifically, this can be achieved by using the decoding network within the initial image classification model to decode the target image features. When decoding the target image features to obtain multiple second image features, the decoding network can include multiple decoding layers. Specifically, the sample image can be decoded sequentially using multiple decoding layers (i.e., when using multiple decoding layers for sequential encoding, the output of the previous decoding layer is the input of the next decoding layer connected to it), thereby obtaining the image features output by each decoding layer. At least two image features are selected from the image features output by each decoding layer as second image features. The second target features can be obtained by selecting at least two second image features, reducing their dimensionality to the same level, and then concatenating them; alternatively, they can be obtained by fusing the selected at least two second image features. No specific limitation is made here, as long as the method of obtaining the first target features is the same.

[0065] When calculating the model loss based on the first target feature, the second target feature, and the classification label of the sample image, the contrastive loss of the image classification model can be calculated based on the first target feature and the second target feature, and the classification loss of the image classification model can be calculated based on the first target feature, the second target feature, at least one first image feature from multiple first image features, or at least one second image feature from multiple second image features, and the classification label of the sample image. Thus, the model loss of the image classification model can be obtained based on the contrastive loss and the classification loss of the image classification model.

[0066] It should be understood that after obtaining the loss of the image classification model, the model parameters can be adjusted according to the model loss to minimize the model loss. As the model is trained, the model loss will gradually decrease. Consequently, the image classification model after training will be more accurate, and the results obtained by using the trained image classification model to train on the images to be classified will also be more accurate.

[0067] Specifically, step S120 can involve using the encoding network of the trained image classification model to extract image features from the image to be classified, and then using the classifier of the trained image classification model to determine the classification result of the image to be classified based on the extracted image features. For a detailed description of how the encoding module of the trained image classification model extracts image features from the image to be classified, please refer to the detailed description of step S120 in the foregoing embodiments. However, it should be noted that the extracted image features of the image to be classified are the image features output by the last encoding layer of the encoding network.

[0068] For example, such as Figure 3 As shown, if the category label corresponding to the sample image includes normal image or abnormal image, and the abnormal image specifically includes first-class image, second-class image and third-class image, then the classification result obtained by using the trained image classification model to classify the image to be classified can be normal image or abnormal image, and the abnormal image can specifically be first-class image, second-class image or third-class image. Figure 3 This diagram illustrates how an image to be classified is obtained from a dataset to be classified, and how the image is input into a trained image classification model to classify the image into a normal image, a first-class image, a second-class image, or a third-class image.

[0069] The trained image classification model, when classifying images, fully considers the potential for feature loss when using the same-sized feature maps obtained during the encoding and decoding processes. That is, the first and second image features corresponding to the same-sized feature maps obtained during encoding and decoding may differ, leading to different first and second target features. Therefore, based on the first and second target features, the model can accurately capture the differences in features generated during the encoding and decoding processes of the same sample image (the model's contrastive loss). This results in a more accurate and better generalization-performance image classification model trained using this model loss, effectively improving the accuracy of classification results when using the trained model to classify images.

[0070] Please see Figure 4 , Figure 4 This application provides an image classification method that can be applied to electronic devices, such as... Figure 1In the scenario shown, the server or terminal device, when executing the image classification method, specifically performs steps S210 to S270:

[0071] Step S210: Obtain sample images with classification labels.

[0072] The sample images can be user-drawn images, images captured using a terminal device, images obtained from web pages, etc., without specific limitations. The sample images can be user-drawn sketches, illustrations, comics, and emoticons with fewer image features, or they can be photographs of scenery with more image features.

[0073] In one possible implementation of this application, the above-mentioned sample images are images with fewer image features, and each sample image has a classification identifier.

[0074] For example, the classification labels mentioned above can be labels used to characterize the specific categories of items included in the sample image. For instance, a classification label can identify a sample image as belonging to the categories of cat, dog, elephant, etc.; it can also identify a sample image as belonging to the categories of animal, plant, etc.; it can also identify the style type of the building in the sample image; or it can identify a sample image as belonging to the categories of normal image, uncivilized image, etc. It should be understood that when it is an uncivilized image, the uncivilized image category label can also include several uncivilized type labels. It should be understood that the above examples are merely illustrative and should not be considered as limitations of this solution.

[0075] It should be understood that the aforementioned sample images should be a massive number of sample images, each with its own classification label. For example, if there are multiple sample images, these images may include those labeled as normal images, and may also include those labeled as uncivilized behavior, and the specific types of uncivilized behavior can be varied. The method for obtaining sample images may specifically be to obtain at least one sample image from a sample library. That is, when training the image classification model once, one sample image or multiple sample images can be used.

[0076] Step S220: Encode the sample image using an image classification model to obtain multiple first image features, each with a different feature map size.

[0077] It should be understood that since the size of the feature maps corresponding to each first image feature is different, the number and / or types of features included in each first image feature may be different. For example, among the feature maps corresponding to multiple image features extracted from the same image, the feature map with a larger size may contain more image features than the feature map with a smaller size.

[0078] Specifically, the image classification model is used to encode the sample images. This can be achieved by using an encoding network within the image classification model. The encoding network can include multiple encoding layers. The encoding process described above can involve sequentially encoding the sample images using multiple encoding layers (i.e., when using multiple encoding layers sequentially, the output of the previous encoding layer becomes the input of the next encoding layer connected to it), thereby obtaining the image features output by each encoding layer. At least two image features are then selected from the image features output by each encoding layer as the first image feature.

[0079] Specifically, the aforementioned coding layer can be composed of convolutional layers, residual modules, SE modules (squeezing and extracting modules), or dense blocks (closely connected convolutional blocks), depending on the actual needs.

[0080] Step S230: Use an image classification model to decode the target image features among multiple first image features to obtain multiple second image features, each with a different feature map size.

[0081] Specifically, the target image feature mentioned above can be the first image feature corresponding to the smallest feature map among multiple first image features, or it can be any one of the first image features selected from the first image features selected after sorting the feature maps corresponding to multiple first image features in ascending order of size. The setting can be made according to actual needs, and no specific limitation is made here.

[0082] In one possible implementation of this application, the target image feature is the first image feature corresponding to the feature map with the smallest size in the feature map corresponding to each of the plurality of first image features.

[0083] Specifically, decoding the target image features using an image classification model can be achieved by using a decoding network within the image classification model. When decoding the target image features to obtain multiple second image features, the decoding network can include multiple decoding layers. Specifically, decoding the sample image can be performed sequentially using multiple decoding layers (i.e., when using multiple decoding layers for sequential encoding, the output of the previous decoding layer becomes the input of the next decoding layer connected to it), thereby obtaining the image features output by each decoding layer. At least two image features are then selected from the image features output by each decoding layer as the second image features.

[0084] It should be understood that the above decoding process is the reverse of the encoding process. For example, when the encoding layer in the encoder is a convolutional layer and the encoding process involves convolutional processing of the input features, the decoding process involves deconvolutional processing of the input features. If the encoding layer is composed of residual modules, the decoding layer can also be composed of modules that perform the reverse operation of the residual modules. If the encoding layer is composed of SE modules, the decoding layer is composed of modules that perform the reverse operation of the SE modules. Similarly, if the encoding layer is composed of Dense Blocks, the decoding layer is composed of modules that perform the reverse operation of the Dense Blocks.

[0085] Among the multiple second image features obtained by decoding the target image features using an image classification model, at least two of the second image features can have feature maps with the same size as the feature map corresponding to one of the first image features. Alternatively, each second image feature can have a feature map with the same size as the feature map corresponding to one of the first image features. The setting can be adjusted according to actual needs.

[0086] Step S240: Process multiple first image features to obtain first target features, and process multiple second image features to obtain second target features.

[0087] There are multiple ways to process multiple first image features to obtain the first target feature. It should be noted that the way to process multiple first image features can be the same as the way to process multiple second image features.

[0088] In one possible embodiment of this application, step S140 may include: selecting at least two first image features from a plurality of first image features, and selecting at least two second image features from a plurality of second image features; performing dimensionality reduction processing on each selected first image feature and then concatenating them to obtain a first target feature; and performing dimensionality reduction processing on each selected second image feature and then concatenating them to obtain a second target feature. The number of selected first image features and second image features is the same. The dimensions of each first image feature after dimensionality reduction are the same, the dimensions of each second image feature after dimensionality reduction are the same, and the dimensions of each second image feature after dimensionality reduction are the same as the dimensions of each first image feature after dimensionality reduction. This ensures that both the first target feature and the second target feature simultaneously possess features extracted at different encoding sizes, thereby guaranteeing the integrity of the image features included in both the first and second target features.

[0089] In this implementation, the size of the feature map corresponding to each selected first image feature can be the same as the size of the feature map corresponding to a selected second image feature.

[0090] For example, taking a sample image with a size of 224*224 (width*height), the sample image is encoded to obtain multiple first image features including image features of 56*56*64 (width*height*number of channels), 28*28*128, 14*14*256, and 7*7*512. Correspondingly, the multiple second image features obtained by decoding the smallest feature among the multiple first features include image features of 56*56*64, 28*28*128, 14*14*256, and 7*7*512. The above method of obtaining the first target feature based on multiple first image features can be: selecting at least two first image features from the above multiple first image features, performing unified dimensionality reduction, and then stitching them together; and selecting at least two second image features from the multiple second image features, performing dimensionality reduction, and then stitching them together. For example, two first image features, 56*56*64 and 7*7*512, can be selected, their dimensions reduced to 1024, and then concatenated to obtain a 1024*2 feature vector (i.e., the first target feature). Similarly, two second image features, 56*56*64 and 7*7*512, can be selected, their dimensions reduced to 1024, and then concatenated to obtain a 1024*2 feature vector (i.e., the second target feature).

[0091] In another possible implementation, the above steps may also include selecting at least one first image feature from a plurality of first image features as a first target feature, selecting at least one second image feature from a plurality of second image features as a second target feature, wherein the number of selected first image features is the same as the number of second image features, and the size of the feature map corresponding to each selected first image feature is the same as the size of the feature map corresponding to one selected second image feature.

[0092] In another possible embodiment of this application, step S140 may be: selecting at least two first image features from a plurality of first image features, and selecting at least two second image features from a plurality of second image features; performing a fusion process on the selected first image features to obtain a first target feature; and performing a fusion process on the selected second image features to obtain a second target feature. The number of selected first image features and second image features is the same. By employing this method, the first target feature and the second target feature simultaneously possess features extracted at different encoding sizes, thereby ensuring the integrity of the image features included in the first target feature and the second target feature.

[0093] In this implementation, the size of the feature map corresponding to each selected first image feature can be the same as the size of the feature map corresponding to a selected second image feature.

[0094] Step S250: Calculate the model loss based on the first target feature, the second target feature, and the classification label of the sample image.

[0095] Specifically, the model loss mentioned above can be obtained from the model's classification loss and the model's contrastive loss.

[0096] The classification loss is specifically calculated by using the classifier of the image classification model to obtain the predicted classification result of the sample image, and then using the classification loss function to calculate the loss based on the predicted classification result of the sample image and the true classification result of the training sample.

[0097] The classification loss function is a function used to measure the difference between the predicted classification result and the true classification result. The smaller the value of the function, the closer the predicted classification result is to the true classification result, and the higher the accuracy of the model; conversely, the larger the value of the function, the less close the predicted classification result is to the true classification result, and the lower the accuracy of the model.

[0098] Contrast loss is used to indicate the degree of difference between the first target feature obtained during encoding and the second target feature obtained during decoding. This degree of difference can be represented by distance or similarity. Optionally, the smaller the value of contrast loss, the fewer features are lost during the encoding and decoding of the sample image; conversely, the larger the value of contrast loss, the more features are lost during encoding and decoding. This application enhances the robustness of the encoder by reducing contrast loss, thereby improving the classification performance of the classifier, and ultimately improving the classification performance and robustness of the image classification model.

[0099] Specifically, step S250 above may involve calculating the contrast loss of the image classification model based on the first target feature and the second target feature, and calculating the classification loss of the image classification model based on the first target feature, the second target feature, at least one first image feature from a plurality of first image features or at least one second image feature from a plurality of second image features, and the classification label of the sample image. The model loss of the image classification model is thus obtained based on the contrast loss and the classification loss of the image classification model.

[0100] In one possible implementation, if the encoding network includes multiple encoding layers, the decoding network includes a decoding layer, and each encoding layer is used to output a first image feature, and each decoding network is used to output a decoded feature, then to make the model loss obtained above more accurate, step S250 may include: calculating the contrast loss of the first target feature and the second target feature to obtain the contrast loss of the image classification model; using the classifier of the image classification model to perform classification calculation on the image features output by the last encoding layer to obtain the classification result of the sample image; determining the classification loss of the image classification model based on the classification result and classification label of the sample image; and determining the model loss of the image classification model based on the classification loss and the contrast loss.

[0101] In another possible implementation, step S250 may also be: performing a contrast loss calculation on the first target feature and the second target feature to obtain the contrast loss of the image classification model; using the classifier of the image classification model to perform classification calculation on the first target feature or the second target feature to obtain the classification result of the sample image; determining the classification loss of the image classification model based on the classification result and classification label of the sample image; and determining the model loss of the image classification model based on the classification loss and the contrast loss.

[0102] It should be noted that, under the two implementation methods described above, the classification loss can be calculated using a classification loss function. When a sample image has multiple categories (i.e., multiple classification labels), the losses caused by misclassification of different categories are different. For example, misclassifying an image of the first category as the fourth category is more serious than misclassifying an image of the second category as the fourth category. In this case, it is necessary to introduce the concept of loss weights for different categories. In the embodiments of this application, the following cross-entropy loss (i.e., classification loss) can be used, wherein... Where N is the number of samples, C is the number of categories, i is a specific sample, and y is a variable. ic Let P be the true value of the i-th sample belonging to the c-th class, which is either 0 or 1. ic Predict the value of the i-th sample as the c-th class.

[0103] To make the prediction results more accurate, this application can also predict the value P in the aforementioned cross-entropy loss function when performing classification prediction. ic Add weight W before c That is, this application can also use the following cross-entropy loss function L cls :and in, y∈[0, C-1], P i =(p i0 ,p i1 ,...p iC-1 ), argmax(P i ) refers to the position where the weight is the maximum, M refers to the sum of distances between the categories, N is the number of samples, C is the number of categories, i is a sample, and y is the sum of distances between the categories. ic Let P be the true value of the i-th sample belonging to the c-th class, which is either 0 or 1. ic The prediction value for the i-th sample is the c-th class, where y refers to the position of the class.

[0104] For example, if the original predicted values ​​for [Category 1, Category 2, Category 3, Category 4] are [0.6, 0.25, 0.1, 0.05], then the predicted values ​​for each category are corrected. Using the formula for M, the corresponding M value is M = |0-0|+1+|0-1|+1+|0-2|+1+|1-0|+1+|1-1+1+|1-2|+1+|2-0+1+|2-1|+1+|2-2|+1 = 17. The calculated w at this point... c They are respectively:

[0105] Therefore, by adopting the above method, for two similar categories, such as the first category and the second category being quite similar, the second category, which is closest to the first category, will be assigned a lower weight, which means that the loss will be smaller, thus achieving the effect that the closer the distance, the lower the loss.

[0106] The contrastive loss can be calculated using the pixel loss function, where the contrastive loss function is... Where classes is the number of sample images input to the model for each training iteration, and y true As the second target feature, y pred This is the primary target feature.

[0107] The model loss of an image classification model can be obtained by summing the contrast loss and the classification loss, or by setting weight coefficients for the contrast loss and the classification loss respectively, and then summing the contrast loss and the classification loss to obtain the model loss of the image classification model.

[0108] Step S260: Train the image classification model based on the model loss.

[0109] Specifically, step S260 can be: adjusting the model parameters of the image classification model based on the model loss to obtain the adjusted image classification model. If the model loss of the adjusted image classification model does not converge, the process returns to acquiring sample images until the model converges or the training iterations reach a preset number, thus obtaining a fully trained image classification model.

[0110] It should be noted that the above model convergence refers to the model loss being minimized. In other words, the above steps can specifically be: adjusting the parameters of the image classification model based on the model loss to minimize the model loss, thereby obtaining a fully trained image classification model.

[0111] Step S270: Obtain the image to be classified.

[0112] Step S280: Input the image to be classified into the trained image classification model to obtain the classification result of the image to be classified.

[0113] For a detailed description of steps S270 and S280, please refer to the detailed description of steps S110 and S120 in the foregoing embodiments. They will not be repeated in this embodiment.

[0114] By employing the image classification method of this application, during the training phase of the image classification model, the potential for feature loss when using the image classification model to obtain first and second image features corresponding to feature maps of the same size during the encoding and decoding processes of sample images is fully considered. That is, the first and second image features corresponding to feature maps of the same size obtained during the encoding and decoding processes may be different. Correspondingly, the obtained first target features and second target features may also be different. Therefore, based on the first and second target features, the differences in features generated during the encoding and decoding processes of the same sample image during the training of the image classification model (the model's contrastive loss) can be accurately obtained. In other words, this ensures that the model loss calculated based on the first target feature, the second target feature, and the classification label is more accurate, thereby making the trained image classification model more accurate and having better generalization performance. Consequently, when using the trained image classification model to classify images, the classification results obtained are also more accurate.

[0115] Please refer to the following: Figure 5 , Figure 6 as well as Figure 7 Another embodiment of this application provides an image classification method applicable to electronic devices, wherein, as Figure 5 and Figure 6 As shown, the image classification model includes an encoding network and a decoding network. The encoding network includes multiple cascaded encoding layers, and the decoding network includes multiple cascaded decoding layers, with each decoding layer corresponding to one encoding layer. The method includes:

[0116] Step S310: Obtain sample images with classification labels.

[0117] Step S320: The sample image is downsampled and encoded to obtain image features through the first coding layer of the cascaded multiple coding layers.

[0118] Step S330: Output the image features encoded by the first coding layer to the subsequent cascaded coding layers, so that downsampling coding and output of the encoded image features can continue in the subsequent cascaded coding layers until the last coding layer.

[0119] Step S340: Obtain image features from the output of at least two coding layers in the cascaded multiple coding layers to obtain at least two first image features.

[0120] Among them, at least two of the first image features include the image features output by the last coding layer.

[0121] By employing the steps S320-S340 described above, after receiving the sample image in the first coding layer of the coding network, the sample image is downsampled and encoded to obtain the coding result of the first coding layer, i.e., the downsampled image features. The coding result of the first coding layer is then output to the subsequent cascaded coding layer, i.e., the second coding layer, so that downsampling and encoding results can continue to be output in the subsequent cascaded coding layers until the last coding layer. After the last coding layer performs downsampling and encoding, the resulting coding result, i.e., the downsampled image features output by the last coding layer, is used as the downsampled image features output by the coding network. This allows for the extraction of the first image features (downsampled image features) from the feature images of the same sample image at different sizes.

[0122] For example, the sample image is downsampled and encoded by the first coding layer in a series of cascaded coding layers to obtain the first downsampled image feature, which is then output as the coding result. The (n-1)th downsampled image feature output by the (n-1)th coding layer is downsampled and encoded by the nth coding layer in a series of cascaded coding layers to obtain the ith downsampled image feature, which is then output as the coding result. Here, n takes the value 2 ≤ n ≤ N, and N is the number of coding layers in the coding network.

[0123] It should be noted that the coding layer of the coding network in steps S220-S240 above can be a convolutional layer as described in the above embodiments, or it can be a residual module, or it can be an SE module (squeeze and extract module), or it can be a Dense Block (closely connected convolutional block).

[0124] In one possible implementation of this application, the coding layer consists of a residual module, such as Figure 8 As shown, the residual module contains convolution branches and residual branches, and the residual branches in the residual module are from the input of the convolution branch to the output of the convolution branch.

[0125] When the encoding layer consists of residual modules, the corresponding image classification model can be a residual network, such as a deep residual network. Figure 9 As shown, the residual network also includes an initial convolutional layer preceding multiple residual modules, the output of which serves as the input to the first residual module (i.e., the first encoding layer) among the multiple residual modules.

[0126] The initial convolutional layers can contain multiple layers, and the encoding network can also include multiple residual modules. Figure 9The diagram illustrates a configuration including three initial convolutional layers and six residual modules in the encoding network. This allows an input sample image to undergo a change in width (w) and height (h) and the number of channels after passing through three convolutional layers, forming a new feature image. This feature image is then sequentially processed by a sub-network (multi-layer encoding layer) composed of six residual modules for feature extraction, resulting in each residual network generating new image features (multiple first image features). For example, the width (w) and height (h) of the input image might be reduced to 1 / 4 of their original size after three convolutional layers, and the number of channels might increase from 3 to 128, forming a w / 4*h / 4*128 feature image, which is then input into the multi-layer encoding layer for feature extraction.

[0127] In this embodiment, for a residual module, the feature maps corresponding to its input and output image features are of different sizes. Therefore, a convolution operation is needed to map the input and output to the same size. Typically, the first residual module in each convolution stage needs to use a non-identical mapping (i.e., adding a convolutional layer) residual branch to ensure the consistency of the input and output of the residual module. Meanwhile, since the convolution operation in the convolutional layer has no bias term, a BN (Batch Normalization) layer can be added before the convolutional layer to add a bias term, thereby ensuring optimal processing results. This allows the residual module to perform downsampling encoding while ensuring that no feature points are skipped, thus preventing loss of the feature network's representational ability. This ensures the accuracy of image feature extraction and improves the accuracy of subsequent image classification model training based on the extracted features.

[0128] Step S350: The image features output by the last encoding layer are upsampled and decoded through the first decoding layer of the cascaded multiple decoding layers to obtain the decoded image features.

[0129] Step S360: The image features obtained by upsampling decoding are fused with the image features output by the encoding layer connected to the first decoding layer to obtain fused image features. The fused image features are output as the final decoded image features of the first decoding layer to the subsequent cascaded decoding layers, so that upsampling decoding, fusion and final output of decoded image features can continue in the subsequent cascaded decoding layers until the last decoding layer.

[0130] Step S370: Obtain image features output from at least two of the cascaded decoding layers to obtain at least two second image features.

[0131] In this configuration, each of the at least two decoding layers is a decoding layer cascaded with an encoding layer that outputs the first image features.

[0132] In steps S350-S370 above, the number of coding layers is one more than the number of decoding layers. That is, when the coding network for current sampling includes multiple cascaded coding layers, the decoding network includes multiple cascaded decoding layers, one fewer than the number of coding layers. For each decoding layer, the feature map corresponding to the image features output by that decoding layer and the corresponding coding layer is the same size; that is, please refer again... Figure 4 When the encoding layer is N layers and the decoding layers are N-1 layers, the Nth encoding layer is connected to the 1st decoding layer, the N-1th encoding layer is connected to the second decoding layer, the nth encoding layer is connected to the N-n+1th decoding layer, and the 1st encoding layer is connected to the N-1th decoding layer.

[0133] By connecting the encoding layer of the encoding network to the decoding layer of the decoding network, the second image features obtained by the decoding network can fully retain the information of each layer in the encoding and decoding process (image features obtained in the encoding and decoding processes of the encoding and decoding layers). This allows the decoding network to learn all the information of the sample image, making the image features of the sample image obtained by the decoding network more accurate, and making the obtained second image features more accurate.

[0134] It should be understood that when the encoding layer in the encoding network specifically uses a convolutional layer for encoding, the corresponding decoding layer in the decoding network uses a deconvolutional layer for decoding; similarly, if the encoding layer specifically includes a residual module to use the residual module for encoding, the corresponding decoding layer in the decoding network can use the decoding module corresponding to the residual module for decoding.

[0135] Step S380: Process multiple first image features to obtain first target features, and process multiple second image features to obtain second target features.

[0136] Step S390: Calculate the model loss based on the first target feature, the second target feature, and the classification label of the sample image.

[0137] Step S400: Train the image classification model based on the model loss.

[0138] For a detailed description of steps S380-S400, please refer to the detailed description of steps S240-S260 in the foregoing embodiments. In this embodiment, they will not be described in detail.

[0139] Step S410: Obtain the image to be classified.

[0140] Step S420: Input the image to be classified into the trained image classification model to obtain the classification result of the image to be classified.

[0141] By employing the image classification method of this application, during the training phase of the image classification model, multiple cascaded encoding layers in the image classification model are used to downsample and encode sample images to obtain at least two first image features. Then, the first decoding layer in a cascaded series of decoding layers upsamples and decodes the image features output by the last encoding layer to obtain decoded image features. The upsampled and decoded image features are fused with the image features output by the corresponding encoding layer of the first decoding layer to obtain fused image features. These fused image features are then output as the final decoded image features of the first decoding layer to subsequent cascaded decoding layers. Upsampling, decoding, fusion, and final output of decoded image features continue in subsequent cascaded decoding layers until the last decoding layer, resulting in at least two second image features. This ensures that each second image feature obtained by the decoding network fully retains information from each level of the encoding and decoding process, making the decoded second image features closer to the true features extracted from the sample images, i.e., the extracted second image features are more accurate. Correspondingly, the second target features obtained based on at least two second image features are also more accurate. Therefore, when calculating the model loss using the first target feature, the second target feature, and the classification label, the calculated model loss can be more accurate, and the trained image classification model obtained by training the image classification model based on the model loss can be more accurate. Consequently, when using the trained image classification model to classify the image to be classified, the classification result obtained is also more accurate.

[0142] Please refer to it again. Figure 6 The sample images are simple line drawings with a size of 224*224 (width*height), and the image classification model is a residual network model (i.e., ...). Figure 9 The network model shown is a residual network model that includes an encoding network and a decoding network. The encoding layer in the encoding network is composed of residual modules. The classification labels of the sample images include anomaly classification labels and normal classification labels. The anomaly classification labels include a first category, a second category, and a third category. This will be explained using an example.

[0143] When training an image classification model using this residual network model, the specific process can be as follows:

[0144] Sample line drawings are obtained from the sample database. Each sample line drawing image has a classification label. The classification label for each sample line drawing image is either Category 1, Category 2, Category 3, or Normal Category.

[0145] First, a 224*224 pixel sketch image is processed through three initial convolutional layers of a residual network model, and then input into multiple cascaded encoding layers of the residual network model. The first encoding layer of the cascaded encoding layers in the residual network model downsamples and encodes the sample sketch image to obtain image features, where the corresponding image feature size is 56*56*64. The image features encoded by the first encoding layer are output to subsequent cascaded encoding layers to continue downsampling and encoding, until the last encoding layer. At least two image features output from the cascaded encoding layers are obtained to acquire at least two first image features.

[0146] Among them, at least two of the first image features include the image features output by the last coding layer. The feature images corresponding to the multiple first image features obtained above include image features corresponding to at least two of the following: size 56*56*64, 28*28*128, 14*14*256, and 7*7*512. And the feature map corresponding to the image features output by the last coding layer has the smallest size.

[0147] After obtaining the image features output by each coding layer, the encoding process of the sample image is completed. At this point, further decoding processing of the sample image is required. During the decoding process, the image features output by the last coding layer can be input into the decoding network of the residual network model.

[0148] In the residual network model, the first decoding layer of a series of cascaded decoding layers upsamples and decodes the image features output by the last encoding layer to obtain a decoded image feature of size 14*14*256. This image feature can then be fused with the 14*14*256 image features output by the encoding layer to obtain a fused image feature, which can be 14*14*512. This fused image feature is then used as the final decoded image feature of the first decoding layer and output to subsequent cascaded decoding layers. Upsampling, fusion, and final output of decoded image features continue in subsequent cascaded decoding layers until the last decoding layer is reached. At least two second image features are obtained by acquiring the image features output by at least two of the cascaded decoding layers.

[0149] Among them, at least two first image features include the image features output by the last coding layer; each of the at least two decoding layers is a decoding layer cascaded with a coding layer that outputs the first image features.

[0150] The obtained at least two second image features may include image features corresponding to at least two feature images with sizes of 14*14*512, 28*28*256, and 56*56*128.

[0151] Multiple first image features are dimensionality reduced separately to obtain multiple dimensionality-reduced first image features. These multiple dimensionality-reduced first image features are then concatenated to obtain a first target feature. Similarly, multiple second image features are dimensionality reduced separately to obtain dimensionality-reduced second image features. These multiple dimensionality-reduced second image features are then concatenated to obtain a second target feature. Finally, these second target features are concatenated to obtain a second target feature.

[0152] For example, the three first image features (such as the image features corresponding to the encoded feature images with sizes of 56*56*64, 28*28*128, and 14*14*256 respectively) can be uniformly reduced to 1024 dimensions and then concatenated to obtain a 1024*3 first target feature. Similarly, the three second image features (such as the image features corresponding to the decoded feature images with sizes of 56*56*64, 28*28*128, and 14*14*256 respectively) can be uniformly reduced to 1024 dimensions and then concatenated to obtain a 1024*3 second target feature.

[0153] In this case, the dimension of each second image feature after dimensionality reduction is the same as the dimension of the first image feature after dimensionality reduction.

[0154] The contrastive loss of the residual network model is obtained by calculating the contrastive loss of the first target feature and the second target feature using the contrastive loss function.

[0155] The residual network model is used to classify the first target feature to obtain the classification result of the sample line drawing image. Based on the classification result and classification label of the sample line drawing image, the cross-entropy loss function is used to calculate the classification loss of the residual network model.

[0156] The model loss of the residual network model is obtained by summing the classification loss and the contrastive loss. The parameters of the residual network model are then adjusted based on this model loss to minimize it, resulting in a fully trained image classification model.

[0157] It should be noted that the device embodiments in this application correspond to the aforementioned method embodiments. The specific principles in the device embodiments can be found in the content of the aforementioned method embodiments, and will not be repeated here.

[0158] Figure 10The image classification device 500 shown in one embodiment is as follows: Figure 10 As shown, the image classification device 500 includes an image acquisition module 510 and a classification module 520.

[0159] The image acquisition module 510 is used to acquire the image to be classified; the classification module 520 is used to input the image to be classified into the image classification model trained by the above-mentioned image classification model training method to obtain the classification result of the image to be classified.

[0160] like Figure 11 As shown, in one possible implementation, the image classification device 500 further includes: a sample acquisition module 530, an encoding module 540, a decoding module 550, a processing module 560, a loss calculation module 570, and a training module 580.

[0161] The sample acquisition module 530 is used to acquire sample images, which have classification labels; the encoding module 540 is used to encode the sample images using an image classification model to obtain multiple first image features, each with a different size of its corresponding feature map; the decoding module 550 is used to decode the target image features among the multiple first image features using an image classification model to obtain multiple second image features, each with a different size of its corresponding feature map; the processing module 560 is used to process the multiple first image features to obtain first target features and process the multiple second image features to obtain second target features; the loss calculation module 570 is used to calculate the model loss based on the first target features, the second target features, and the classification labels of the sample images; and the training module 580 is used to train the image classification model based on the model loss.

[0162] In one implementation, the image classification model includes an encoding network comprising multiple cascaded encoding layers. The encoding module 540 includes an encoding submodule, a feature output submodule, and a first feature acquisition submodule. The encoding submodule is used to downsample and encode image features from a sample image through the first encoding layer of the cascaded encoding layers. The feature output submodule is used to output the image features encoded by the first encoding layer to subsequent cascaded encoding layers, so that downsampling and encoding of the encoded image features can continue in subsequent cascaded encoding layers until the last encoding layer. The first feature acquisition submodule is used to acquire image features output by at least two of the cascaded encoding layers to obtain at least two first image features, wherein the at least two first image features include the image features output by the last encoding layer.

[0163] In this implementation, the image classification model further includes a decoding network, which includes multiple cascaded decoding layers, and each decoding layer is connected to a corresponding encoding layer. The decoding module 550 includes a decoding submodule, a fusion submodule, and a second feature acquisition submodule.

[0164] The decoding submodule is used to upsample and decode the image features output by the last encoding layer through the first decoding layer in a series of cascaded decoding layers to obtain the decoded image features. The fusion submodule is used to fuse the image features obtained by upsampling and decoding with the image features output by the encoding layer corresponding to the first decoding layer to obtain fused image features. The fused image features are then output as the final decoded image features of the first decoding layer to subsequent cascaded decoding layers, so that upsampling, decoding, fusion, and final output of decoded image features can continue in subsequent cascaded decoding layers until the last decoding layer is reached. The second feature acquisition submodule is used to acquire the image features output by at least two decoding layers in a series of cascaded decoding layers to obtain at least two second image features, wherein each of the at least two decoding layers is a decoding layer cascaded with an encoding layer that outputs the first image features.

[0165] In one possible implementation of this application, the coding network includes multiple convolutional residual modules, and each coding layer consists of at least one convolutional residual module.

[0166] In one possible implementation of this application, the processing module 560 includes a first processing submodule, a first splicing submodule, a second processing submodule, and a second splicing submodule.

[0167] The first processing submodule is used to perform dimensionality reduction processing on multiple first image features respectively to obtain multiple dimensionality-reduced first image features, wherein the dimensions of each dimensionality-reduced first image feature are the same; the first stitching submodule is used to stitch together the multiple dimensionality-reduced first image features to obtain a first target feature; the second processing submodule is used to perform dimensionality reduction processing on multiple second image features respectively to obtain dimensionality-reduced second image features, wherein the dimensions of each dimensionality-reduced second image feature are the same as the dimensions of the dimensionality-reduced first image features; the second stitching submodule is used to stitch together the multiple dimensionality-reduced second image features to obtain a second target feature.

[0168] In one possible implementation of this application, the loss calculation module 570 is further configured to: perform contrast loss calculation on the first target feature and the second target feature to obtain the contrast loss of the image classification model; perform classification calculation on the image features output by the last coding layer using the image classification model to obtain the classification result of the sample image; determine the classification loss of the image classification model based on the classification result and classification label of the sample image; and determine the model loss of the image classification model based on the classification loss and contrast loss.

[0169] In another possible implementation of this application, the loss calculation module 570 is further configured to: perform contrast loss calculation on the first target feature and the second target feature to obtain the contrast loss of the image classification model; perform classification calculation on the first target feature or the second target feature using the image classification model to obtain the classification result of the sample image; determine the classification loss of the image classification model based on the classification result and classification label of the sample image; and determine the model loss of the image classification model based on the classification loss and contrast loss.

[0170] In one possible implementation, the training module 580 is further configured to adjust the parameters of the image classification model based on the model loss in order to minimize the model loss and obtain a trained image classification model.

[0171] The following will combine Figure 12 This application describes an electronic device.

[0172] Please see Figure 12 Based on the image classification model training method provided in the above embodiments, this application embodiment also provides another electronic device 100 including a processor 102 capable of executing the aforementioned method. The electronic device 100 can be a server or a terminal device, and the terminal device can be a smartphone, tablet computer, computer, or portable computer, etc.

[0173] The electronic device 100 also includes a memory 104. The memory 104 stores a program that can execute the contents of the foregoing embodiments, and the processor 102 can execute the program stored in the memory 104.

[0174] The processor 102 may include one or more cores for data processing and message matrix units. The processor 102 connects to various parts within the electronic device 100 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 104, and by calling data stored in the memory 104. Optionally, the processor 102 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 102 may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 102 and may be implemented separately using a communication chip.

[0175] The memory 104 may include random access memory (RAM) or read-only memory (ROM). The memory 104 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 104 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described below, etc. The data storage area may also store data acquired by the electronic device 100 during use (e.g., signature fields and asymmetric keys).

[0176] The electronic device 100 may also include a network module and a screen. The network module is used to receive and transmit electromagnetic waves, converting electromagnetic waves into electrical signals, thereby enabling communication with communication networks or other devices, such as audio playback devices. The network module may include various existing circuit elements used to perform these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, SIM cards, memory, etc. The network module can communicate with various networks such as the Internet, corporate intranets, and wireless networks, or communicate with other devices via wireless networks. The aforementioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The screen can display interface content and facilitate data interaction.

[0177] In some embodiments, the electronic device 100 may further include a peripheral interface and at least one peripheral device. The processor 102, memory 104, and peripheral interface 106 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral interface via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency component 108, a positioning component 112, a camera 114, an audio component 116, a display screen 118, and a power supply 122.

[0178] Peripheral interface 106 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 102 and memory 104. In some embodiments, processor 102, memory 104 and peripheral interface 106 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 102, memory 104 and peripheral interface 106 can be implemented on separate chips or circuit boards, and this application embodiment does not limit this.

[0179] The radio frequency (RF) component 108 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF component 108 communicates with communication networks and other communication devices via electromagnetic signals. The RF component 108 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF component 108 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF component 108 can communicate with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF component 108 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0180] Positioning component 112 is used to locate the current geographic location of an electronic device to enable navigation or LBS (Location Based Service). Positioning component 112 can be a positioning component based on the US GPS (Global Positioning System), BeiDou system, or Galileo system.

[0181] Camera 114 is used to capture images or videos. Optionally, camera 114 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the electronic device 100, and the rear-facing camera is located on the back of the electronic device 100. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, camera 114 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0182] Audio component 116 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input to processor 102 for processing, or input to radio frequency component 108 for voice communication. For stereo acquisition or noise reduction purposes, there may be multiple microphones, each located at a different part of electronic device 100. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from processor 102 or radio frequency component 108 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into sound waves that humans can hear, but also into sound waves that humans cannot hear for purposes such as ranging. In some embodiments, audio component 114 may also include a headphone jack.

[0183] Display screen 118 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 118 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 102 for processing. In this case, display screen 118 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 118, which serves as the front panel of electronic device 100; in other embodiments, there may be at least two display screens, respectively disposed on different surfaces of electronic device 100 or in a folded design; in still other embodiments, display screen 118 may be a flexible display screen, disposed on a curved or folded surface of electronic device 100. Furthermore, display screen 118 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 118 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0184] Power supply 122 is used to supply power to various components in electronic device 100. Power supply 122 can be alternating current, direct current, a disposable battery, or a rechargeable battery. When power supply 122 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0185] This application also provides a structural block diagram of a computer-readable storage medium. The computer-readable medium stores program code, which can be called by a processor to execute the methods described in the above method embodiments.

[0186] Computer-readable storage media can be electronic storage devices such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, computer-readable storage media includes non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for program code that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code can be compressed, for example, in a suitable form.

[0187] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods described in the various optional implementations above.

[0188] In summary, the image classification method, apparatus, electronic device, and storage medium provided in this application fully consider the potential feature loss during the image classification model training phase. This is because the first and second image features corresponding to the same size feature maps obtained during the encoding and decoding processes of the image classification model for sample images may be missing. Specifically, the first and second image features corresponding to the same size feature maps obtained during encoding and decoding may differ. Correspondingly, the obtained first and second target features may also differ. Therefore, based on the first and second target features, the differences in features generated during the encoding and decoding processes of the same sample image during image classification model training (the model's contrastive loss) can be accurately obtained. This ensures the accuracy of the model loss calculated based on the first target feature, the second target feature, and the classification label, thereby making the trained image classification model more accurate and exhibiting better generalization performance. Consequently, when using the trained image classification model to classify images, the accuracy of the classification results can be effectively improved.

[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image classification method, characterized in that, The method includes: Obtain the image to be classified; The image to be classified is input into the trained image classification model to obtain the classification result of the image to be classified; the model loss during the training process of the image classification model is determined based on the classification loss and the contrastive loss; the image classification model includes an encoding network, and the encoding network includes multiple cascaded encoding layers; The contrast loss is calculated by comparing the first target feature with the second target feature. The first target feature is obtained by encoding the sample image according to the initial image classification model to obtain multiple first image features, and the feature map corresponding to each first image feature has a different size. The second target feature is obtained by decoding the target image feature among the multiple first image features according to the initial image classification model to obtain multiple second image features, and the feature map corresponding to each second image feature has a different size. The classification loss is determined based on the classification result and classification label of the sample image; the classification result is obtained by classifying the image features output by the last encoding layer using the image classification model, or by classifying the first target feature or the second target feature using the image classification model; wherein, the process of obtaining the image features output by the last encoding layer includes: downsampling and encoding the sample image to obtain image features through the first encoding layer of the cascaded multiple encoding layers; outputting the image features encoded by the first encoding layer to the subsequent cascaded encoding layers, so as to continue downsampling and encoding and outputting the encoded image features in the subsequent cascaded encoding layers, until the last encoding layer.

2. The method according to claim 1, characterized in that, Before inputting the image to be classified into the trained image classification model to obtain the classification result of the image to be classified, the method further includes: Acquire sample images, which have classification labels; The sample image is encoded using an image classification model to obtain multiple first image features, each of which has a different feature map size. The target image features among the plurality of first image features are decoded using the image classification model to obtain a plurality of second image features, and the feature maps corresponding to each second image feature are of different sizes. The plurality of first image features are processed to obtain first target features, and the plurality of second image features are processed to obtain second target features; Loss is calculated based on the first target feature, the second target feature, and the classification label of the sample image to obtain classification loss and contrast loss; The model loss of the image classification model is determined based on the classification loss and the contrast loss. The image classification model is trained based on the model loss.

3. The method according to claim 2, characterized in that, The image classification model is used to encode the sample image to obtain multiple first image features, including: Image features are obtained by downsampling and encoding the sample image through the first coding layer of the cascaded multiple coding layers; The image features encoded by the first coding layer are output to the subsequent cascaded coding layers, so that downsampling coding and output of the encoded image features can continue in the subsequent cascaded coding layers until the last coding layer. Image features output from at least two of the cascaded coding layers are obtained to obtain at least two first image features, wherein the at least two first image features include the image features output from the last coding layer.

4. The method according to claim 3, characterized in that, The image classification model further includes a decoding network, which comprises multiple cascaded decoding layers, with each decoding layer corresponding to an encoding layer. The image classification model is used to decode the target image features from the multiple first image features to obtain multiple second image features, including: The image features output by the last encoding layer are upsampled and decoded through the first decoding layer of the cascaded multiple decoding layers to obtain the decoded image features; The image features obtained by upsampling and decoding are fused with the image features output by the encoding layer connected to the first decoding layer to obtain fused image features. The fused image features are then output as the final decoded image features of the first decoding layer to subsequent cascaded decoding layers. Upsampling, decoding, fusion, and final output of decoded image features are then performed in subsequent cascaded decoding layers until the last decoding layer is reached. Image features output by at least two of the cascaded decoding layers are obtained to obtain at least two second image features, wherein each of the at least two decoding layers is a decoding layer cascaded with an encoding layer that outputs the first image features.

5. The method according to claim 3, characterized in that, The coding network includes multiple convolutional residual modules, and each coding layer consists of at least one convolutional residual module.

6. The method according to claim 2, characterized in that, The process of processing the plurality of first image features to obtain the first target feature includes: The multiple first image features are subjected to dimensionality reduction processing respectively to obtain multiple dimensionality-reduced first image features, and the dimensions of each dimensionality-reduced first image feature are the same. The first image features after multiple dimensionality reduction processes are concatenated to obtain the first target features; The process of processing the plurality of second image features to obtain the second target features includes: The multiple second image features are subjected to dimensionality reduction processing to obtain dimensionality-reduced second image features, and the dimension of each dimensionality-reduced second image feature is the same as the dimension of the dimensionality-reduced first image feature. The second image features after multiple dimensionality reduction processes are stitched together to obtain the second target features.

7. The method according to claim 2, characterized in that, Training the image classification model based on the model loss includes: The parameters of the image classification model are adjusted based on the model loss to minimize the model loss, thereby obtaining a fully trained image classification model.

8. An image classification device, characterized in that, The device includes: The image acquisition module is used to acquire images to be classified. A classification module is used to input the image to be classified into a trained image classification model to obtain the classification result of the image to be classified; the model loss during the training process of the image classification model is determined based on classification loss and contrastive loss; the image classification model includes an encoding network, and the encoding network includes multiple cascaded encoding layers; The contrast loss is calculated by comparing the first target feature with the second target feature. The first target feature is obtained by encoding the sample image according to the initial image classification model to obtain multiple first image features, and the feature map corresponding to each first image feature has a different size. The second target feature is obtained by decoding the target image feature among the multiple first image features according to the initial image classification model to obtain multiple second image features, and the feature map corresponding to each second image feature has a different size. The classification loss is determined based on the classification result and classification label of the sample image; the classification result is obtained by classifying the image features output by the last encoding layer using the image classification model, or by classifying the first target feature or the second target feature using the image classification model; wherein, the process of obtaining the image features output by the last encoding layer includes: downsampling and encoding the sample image to obtain image features through the first encoding layer of the cascaded multiple encoding layers; outputting the image features encoded by the first encoding layer to the subsequent cascaded encoding layers, so as to continue downsampling and encoding and outputting the encoded image features in the subsequent cascaded encoding layers, until the last encoding layer.

9. An electronic device, characterized in that, include: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be invoked by a processor to execute the method as described in any one of claims 1-7.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-7.

Images