Image classification method and apparatus, and method and apparatus for training image classification model

Abstract

Disclosed in the embodiments of the present application are an image classification method and apparatus, and a method and apparatus for training an image classification model. The main technical solution comprises: acquiring an image to be classified; performing feature extraction on said image, so as to obtain a feature representation of said image; using the feature representation of said image to perform cross attention processing on initial representations of a plurality of clustering centers which are obtained by means of pre-training, so as to obtain a plurality of clustering center representations; performing decoding processing by using the feature representation of said image and the plurality of clustering center representations, so as to obtain a category allocation matrix; and performing classification by using the plurality of clustering center representations and the category allocation matrix, so as to obtain a classification result indicating whether said image belongs to a target category. By means of the present application, an image classification result can be more accurate.

Description

Image classification method, method and device for training image classification model

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on July 4, 2023, with application number 202310822213.1 and application name “Image classification method, method and device for training image classification model”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer vision technology, and in particular to a method for image classification, and a method and device for training an image classification model. Background Art

[0003] Image classification, which distinguishes images of different categories based on their semantic information, is a fundamental problem in computer vision. Image classification has a wide range of applications, such as traffic scene recognition in transportation, automatic classification of product images in e-commerce, and image recognition in medicine.

[0004] In some special fields, high accuracy and recall rates are required for image classification. Although there are already technologies that use deep learning models to classify images, the accuracy of the classification results still needs to be improved.

[0005] Summary of the Invention

[0006] In view of this, the present application provides a method for image classification, a method and apparatus for training an image classification model, so as to improve the accuracy of image classification results.

[0007] This application provides the following solutions:

[0008] In a first aspect, a method for image classification is provided, the method comprising:

[0009] Get the image to be classified;

[0010] Performing feature extraction on the image to be classified to obtain a feature representation of the image;

[0011] Using the feature representation of the image, cross-attention processing is performed on the initial representations of the multiple cluster centers obtained by pre-training to obtain multiple cluster center representations;

[0012] Performing decoding processing using the feature representation of the image and the multiple cluster center representations to obtain a category assignment matrix;

[0013] Classification is performed using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the image to be classified belongs to a target category.

[0014] According to an achievable embodiment of the present application, the method further includes:

[0015] The image to be classified is segmented using the category assignment matrix to obtain image regions of preset categories, where the preset categories include the target category.

[0016] According to an achievable method in an embodiment of the present application, performing feature extraction on the image to be classified to obtain a feature representation of the image includes:

[0017] Feature extraction is performed on the image to be classified to obtain feature representations of each element Token at multiple resolutions, and the feature representation of each Token at the highest resolution is used as the feature representation of the image.

[0018] According to an achievable method in an embodiment of the present application, performing cross-attention processing on the initial representations of the plurality of cluster centers obtained by pre-training using the feature representation of the image includes:

[0019] The query matrix is ​​obtained using the initial representations of multiple cluster centers, and the query matrix is ​​input into a multi-layer serialized transformer network. Each layer of the transformer network corresponds to each resolution in order from low to high resolution. Each layer of the transformer network uses the feature representation of each token at the corresponding resolution to obtain the key matrix and value matrix. The query matrix input to the transformer network of this layer is processed by cross attention to obtain the query matrix output by the transformer network of this layer.

[0020] The query matrix output by the last layer of the Transformer network is used to obtain the multiple cluster center representations.

[0021] According to an achievable method in an embodiment of the present application, performing classification using the multiple cluster center representations and the category assignment matrix includes:

[0022] averaging the plurality of cluster center representations to obtain a cluster average representation;

[0023] Performing pooling processing on the category assignment matrix to obtain clustering pooling features;

[0024] The cluster average representation and the cluster pool representation are integrated, and the integrated feature representation is used for classification to obtain a classification result of whether the image to be classified belongs to the target category.

[0025] In a second aspect, a method for training an image classification model is provided, the method comprising:

[0026] Acquire training data comprising a plurality of training samples, wherein the training samples include image samples and labels indicating whether the image samples belong to a target category;

[0027] An image classification model is trained using the training data, and the image classification model includes: a feature extraction network, a first decoding network, a second decoding network and a classification network; the feature extraction network extracts features from the image sample to obtain a feature representation of the image sample; the first decoding network uses the feature representation of the image sample to perform cross-attention processing on the initial representations of multiple cluster centers to obtain multiple cluster center representations; the second decoding network uses the feature representation of the image sample and the multiple cluster center representations to perform decoding processing to obtain a category assignment matrix; the classification network uses the multiple cluster center representations and the category assignment matrix to perform classification to obtain a classification result of whether the image sample belongs to the target category; the training objectives include: minimizing the difference between the classification result and the corresponding label.

[0028] According to an achievable method in an embodiment of the present application, the training sample further includes a region mask of a preset category annotated on the image sample; the image classification model further includes a segmentation network;

[0029] The segmentation network segments the image sample using the category assignment matrix to obtain image regions of preset categories, where the preset categories include the target category;

[0030] The training goal also includes minimizing the difference between the image region of the preset category and the corresponding region mask.

[0031] According to an achievable method in an embodiment of the present application, the feature extraction network extracts features from the image sample to obtain a feature representation of the image sample, including: extracting features from the image sample to sequentially obtain feature representations of each element token at multiple resolutions, and using the feature representation of each token at the highest resolution as the feature representation of the image sample;

[0032] The first decoding network includes multiple layers of serially connected Transformer networks. The initial representations of the multiple cluster centers are input into the first layer of Transformer network as a query matrix. Each layer of Transformer network corresponds to each resolution one by one in order from low to high resolution. Each layer of Transformer network uses the feature representation of each Token at the corresponding resolution to obtain a key matrix and a value matrix, and performs cross-attention processing on the query matrix input into the current layer of Transformer network to obtain the query matrix output by the current layer of Transformer network; the query matrix output by the last layer of Transformer network is used to obtain the multiple cluster center representations.

[0033] According to an implementable manner in an embodiment of the present application, the classification network performs classification using the multiple cluster center representations and the category assignment matrix, including:

[0034] The classification network performs averaging processing on the multiple cluster center representations to obtain a cluster average representation;

[0035] Performing pooling processing on the category assignment matrix to obtain a cluster pooling representation;

[0036] The cluster average representation and the cluster pool representation are integrated, and the integrated feature representation is used for classification to obtain a classification result of whether the image sample belongs to the target category.

[0037] In a third aspect, an image classification method is provided, which is executed by a cloud server, and the method includes:

[0038] Obtaining an image to be classified from a user terminal;

[0039] Performing feature extraction on the image to be classified to obtain a feature representation of the image;

[0040] Using the feature representation of the image, cross-attention processing is performed on the initial representations of the multiple cluster centers obtained by pre-training to obtain multiple cluster center representations;

[0041] Performing decoding processing using the feature representation of the image and the multiple cluster center representations to obtain a category assignment matrix;

[0042] Performing classification using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the image to be classified belongs to a target category;

[0043] The classification result is returned to the user terminal.

[0044] In a fourth aspect, a method for computer-aided diagnosis of cancer is provided, wherein the method comprises:

[0045] Obtain medical imaging for target organ harvesting;

[0046] performing feature extraction on the medical image to obtain a feature representation of the medical image;

[0047] Using the feature representation of the medical image, cross-attention processing is performed on the initial representations of multiple cluster centers obtained by pre-training to obtain multiple cluster center representations;

[0048] Performing decoding processing using the feature representation of the medical image and the multiple cluster center representations to obtain a category assignment matrix;

[0049] Classification is performed using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the medical image belongs to a target lesion.

[0050] A fifth aspect provides a computer-aided diagnosis method for gastric cancer, the method comprising:

[0051] Obtain medical images of the stomach;

[0052] performing feature extraction on the medical image to obtain a feature representation of the medical image;

[0053] Using the feature representation of the medical image, cross-attention processing is performed on the initial representations of multiple cluster centers obtained by pre-training to obtain multiple cluster center representations;

[0054] Performing decoding processing using the feature representation of the medical image and the multiple cluster center representations to obtain a category assignment matrix;

[0055] Classification is performed using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the medical image belongs to gastric cancer.

[0056] In a sixth aspect, a computer-aided diagnosis system for cancer is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to perform a computer-aided diagnosis method for cancer, the method comprising:

[0057] Obtain medical imaging for target organ harvesting;

[0058] performing feature extraction on the medical image to obtain a feature representation of the medical image;

[0059] Using the feature representation of the medical image, cross-attention processing is performed on the initial representations of multiple cluster centers obtained by pre-training to obtain multiple cluster center representations;

[0060] Performing decoding processing using the feature representation of the medical image and the multiple cluster center representations to obtain a category assignment matrix;

[0061] Classification is performed using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the medical image belongs to a target lesion.

[0062] In a seventh aspect, an image classification device is provided, the device comprising:

[0063] an image acquisition unit, configured to acquire an image to be classified;

[0064] a feature extraction unit configured to extract features from the image to be classified to obtain a feature representation of the image;

[0065] A first decoding unit is configured to perform cross-attention processing on the initial representations of the plurality of cluster centers obtained by pre-training using the feature representation of the image to obtain a plurality of cluster center representations;

[0066] a second decoding unit configured to perform decoding processing using the feature representation of the image and the plurality of cluster center representations to obtain a category assignment matrix;

[0067] The image classification unit is configured to perform classification using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the image to be classified belongs to a target category.

[0068] In an eighth aspect, a device for training an image classification model is provided, the device comprising:

[0069] a sample acquisition unit configured to acquire training data comprising a plurality of training samples, wherein the training samples include image samples and labels indicating whether the image samples belong to a target category;

[0070] A model training unit is configured to use the training data to train an image classification model, wherein the image classification model includes: a feature extraction network, a first decoding network, a second decoding network and a classification network; the feature extraction network extracts features from the image sample to obtain a feature representation of the image sample; the first decoding network uses the feature representation of the image sample to perform cross-attention processing on the initial representations of multiple cluster centers to obtain multiple cluster center representations; the second decoding network uses the feature representation of the image sample and the multiple cluster center representations to perform decoding processing to obtain a category assignment matrix; the classification network uses the multiple cluster center representations and the category assignment matrix to perform classification to obtain a classification result of whether the image sample belongs to a target category; the training objectives include: minimizing the difference between the classification result and the corresponding label.

[0071] In a ninth aspect, an image classification device is provided, the device comprising:

[0072] an image acquisition unit configured to acquire a medical image for a target organ;

[0073] a feature extraction unit, configured to extract features from the medical image to obtain a feature representation of the medical image;

[0074] A first decoding unit is configured to perform cross-attention processing on the initial representations of the plurality of cluster centers obtained by pre-training using the feature representation of the medical image to obtain a plurality of cluster center representations;

[0075] A second decoding unit is configured to perform decoding processing using the feature representation of the medical image and the multiple cluster center representations to obtain a category assignment matrix;

[0076] The image classification unit is configured to perform classification using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the medical image belongs to the target lesion. In a tenth aspect, a computer-readable storage medium is provided, having stored thereon a computer program, which, when executed by a processor, implements the steps of the method described in any one of aspects 1 to 5 above.

[0077] According to an eleventh aspect, an electronic device is provided, comprising:

[0078] one or more processors; and

[0079] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the first to fifth aspects above.

[0080] In a twelfth aspect, a computer program is provided, which, when executed in a computer, causes the computer to execute the steps of any one of the methods described in the first to fifth aspects above.

[0081] In the thirteenth aspect, a computer program product is provided, comprising a computer program, which, when executed in a computer, causes the computer to execute the steps of any one of the methods described in the first to fifth aspects above.

[0082] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0083] 1) This application decodes the high-level semantics contained in multiple cluster centers obtained through pre-training together with the feature representation of the image to obtain a category assignment matrix, so that the category assignment matrix can reflect the image features associated with the cluster centers, that is, match the image features to the corresponding cluster centers, thereby making the image classification results obtained based on the cluster center representation and the category assignment matrix have higher accuracy and recall rate.

[0084] 2) The present application can further utilize the category assignment matrix to segment the image to be classified, and obtain image regions of preset categories, thereby providing an interpretable reference for image classification.

[0085] 3) In this application, multi-resolution features are extracted and the initial representations of multiple cluster centers of multi-resolution features are used for cross-attention processing, so that the image texture can be perceived from multiple scales in the process of obtaining multiple cluster center representations, and the feature representations of multiple cluster centers are redistributed with the feature representation of the image, so that the classification process and segmentation process are both sensitive to local semantics and have global awareness, further improving the accuracy and recall rate of image classification and image segmentation.

[0086] 4) In the process of training the image classification model, the difference between the image area of ​​the preset category obtained by segmenting the image sample using the segmentation network and the area mask of the preset category annotated for the image sample can be used to assist in training the image classification model, thereby further improving the effect and performance of the image classification model.

[0087] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0089] FIG1 is a diagram of a system architecture applicable to an embodiment of the present application;

[0090] FIG2 is a flow chart of an image classification method provided in an embodiment of the present application;

[0091] FIG3 is a schematic diagram of the principle of the image classification model provided in an embodiment of the present application;

[0092] FIG4 is a flow chart of a method for training an image classification model provided in an embodiment of the present application;

[0093] FIG5 is a schematic diagram showing the principle of a training image classification model provided in an embodiment of the present application;

[0094] FIG6 is a flowchart of an image classification method applied to the medical field provided by an embodiment of the present application;

[0095] FIG7 is a schematic block diagram of an image classification device provided in an embodiment of the present application;

[0096] FIG8 is a schematic block diagram of an apparatus for training an image classification model according to an embodiment of the present application;

[0097] FIG9 is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0098] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0099] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "an", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0100] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0101] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0102] Traditional image classification methods based on deep neural networks mostly extract feature representations from images and then directly classify them using these features. However, this approach suffers from low accuracy and recall in complex scenarios, failing to meet practical needs.

[0103] In view of this, the present application provides a new image classification approach. To facilitate understanding of the present application, a brief description of the system architecture to which the present application applies is first given. FIG1 shows an exemplary system architecture to which an embodiment of the present application can be applied. As shown in FIG1 , the system architecture includes a model training device and an image classification device on the server side, and may also include a user terminal.

[0104] The model training device is used to perform model training in an offline stage. That is, after obtaining training data, the method provided in the embodiment of the present application can be used to perform model training to obtain an image classification model.

[0105] The image classification device is used to classify the image to be classified online using the trained image classification model to obtain a classification result of whether the image belongs to the target category.

[0106] The model training device and image classification device can be configured as separate servers, on the same server or server group, or on separate or shared cloud servers. A cloud server, also known as a cloud computing server or cloud host, is a host product within a cloud computing service system that addresses the management difficulties and poor scalability of traditional physical hosts and virtual private servers (VPs). The model training device and image segmentation device can also be configured on a computer terminal with significant computing power.

[0107] As one possible implementation, a user can send an image to be classified to an image classification device on a server via a network through a user terminal. The image classification device classifies the image to be classified using the method provided in the embodiment of the present application and returns the classification result to the user terminal.

[0108] The user terminals mentioned above may include, but are not limited to, smart mobile terminals, smart home devices, wearable devices, smart medical devices, and PCs (Personal Computers). Smart mobile devices may include, for example, mobile phones, tablet computers, laptop computers, PDAs (Personal Digital Assistants), and internet-connected cars. Smart home devices may include, for example, smart TVs and smart refrigerators. Wearable devices may include, for example, smart watches, smart glasses, smart bracelets, VR (Virtual Reality) devices, AR (Augmented Reality) devices, and mixed reality devices (i.e., devices that support both virtual reality and augmented reality).

[0109] It should be noted that, in addition to performing image classification online, the above-mentioned image classification device may also perform image classification offline, for example, performing image classification on batches of images to be classified.

[0110] It should be understood that the number of model training devices, image classification devices, image classification models, and user terminals in Figure 1 is merely illustrative. Depending on the implementation requirements, any number of model training devices, image classification devices, image classification models, and user terminals may be provided.

[0111] It should be noted that the terms "first" and "second" used in this disclosure do not restrict size, order, or quantity; they are merely used to distinguish between different names. For example, "first image segmentation model" and "second image segmentation model" are used to distinguish between two speech sounds. Another example is "first segmentation result" and "second segmentation result" used to distinguish between two segmentation results. Another example is "first target" and "second target" used to distinguish between two targets. And so on.

[0112] FIG2 is a flow chart of an image classification method provided in an embodiment of the present application, which can be performed by the image classification device in the system shown in FIG1. ​​As shown in FIG2, the method may include the following steps:

[0113] Step 202: Obtain an image to be classified.

[0114] Step 204: extract features from the image to be classified to obtain feature representation of the image.

[0115] Step 206: Using the feature representation of the image, cross-attention processing is performed on the initial representations of the multiple cluster centers obtained by pre-training to obtain multiple cluster center representations.

[0116] Step 208: Perform decoding processing using the feature representation of the image and the multiple cluster center representations to obtain a category assignment matrix.

[0117] Step 210: Classification is performed using multiple cluster center representations and a category assignment matrix to obtain a classification result of whether the image to be classified belongs to the target category.

[0118] It can be seen from the above process that the high-level semantics contained in the multiple cluster centers obtained through pre-training in this application are decoded together with the feature representation of the image to obtain a category assignment matrix, so that the category assignment matrix can reflect the image features associated with the cluster centers, that is, match the image features to the corresponding cluster centers, and thus make the image classification results obtained based on the cluster center representation and the category assignment matrix have a higher accuracy.

[0119] The above steps are described in detail below. First, the above step 202, ie, "obtaining an image to be classified", is described in detail.

[0120] The images to be classified in this application may be two-dimensional images or three-dimensional images, grayscale images or color images.

[0121] In different application fields, images to be classified may contain different contents. For example, in the field of transportation, images to be classified are usually images containing traffic elements such as vehicles, pedestrians, roads, and traffic facilities. The purpose of image classification may be to determine whether the image belongs to a specific traffic scene. For another example, in the field of e-commerce, images to be classified are usually images containing product information. The purpose of image classification may be to determine whether the image belongs to a specific product category. For another example, in the field of medicine, images are usually medical images, such as CT (Computed Tomography), MRI (Magnetic Resonance Imaging), ultrasound examination images, etc., containing organs such as lungs, liver, pancreas, colon, etc. The purpose of image classification may be to determine whether the image belongs to a specific category of organ area, or whether there is a specific category of abnormality.

[0122] Steps 204 to 210 in the above process can be implemented by a pre-trained image classification model. As shown in FIG3 , the image classification model provided by the embodiment of the present application can include: a feature extraction network, a first decoding network, a second decoding network, and a classification network.

[0123] The above step 204, namely "extracting features from the image to be classified to obtain feature representation of the image", can be performed by the above feature extraction network.

[0124] The feature extraction network can be implemented based on a Transformer network to extract image features and obtain the characteristic representation of each token (element) in the image to be classified. For example, the feature extraction network can adopt VIT (Vision Transformer), RAN (Residual Attention Network), U-Net (U-Net is a variant of FCN (Fully Convolutional Network), which was proposed to solve problems in biomedical imaging and has been widely used in various fields of image segmentation due to its excellent results).

[0125] Each token in an image is an element that makes up the image. For an image, if it is divided into a sequence of non-overlapping blocks, both the blocks in the image and the start of the sequence are tokens. For a 2D image, a block can be composed of one or more pixels. For a 3D image, a block can be composed of one or more voxels.

[0126] As one of the preferred implementations, a U-Net network can be used as a feature extraction network to obtain feature representations of each Token at multiple resolutions, and the accuracy of image classification can be improved through this multi-scale feature. Specifically, U-Net adopts an Encoder-Decoder architecture, where the Encoder performs downsampling and the Decoder performs upsampling and feature splicing to obtain feature representations of multiple different resolutions, that is, the feature representation of each Token at a low resolution is first obtained, and then the feature representation of each Token at a higher resolution is obtained in turn. The feature representation of each Token at the highest resolution is then used as the feature representation of the image.

[0127] As shown in Figure 3, assuming that the input image to be classified is represented as X, the feature representation F of the image is obtained through the feature extraction network, F∈R A×HWD Where A is the feature dimension, H, W, and D represent the height, width, and length of a three-dimensional image (taking the input image to be classified as a three-dimensional image as an example).

[0128] The above step 206, i.e., "using the feature representation of the image to perform cross-attention processing on the initial representations of multiple cluster centers obtained by pre-training to obtain multiple cluster center representations", can be performed by the first decoding network in the image classification model.

[0129] In this step, the first decoding network actually transforms the initial representation of a set of cluster centers into a cluster center representation that integrates the image semantics through the cross-attention of the image feature representation.

[0130] If the feature extraction network obtains multiple feature representations of different resolutions, then as one of the preferred implementations, the first decoding network may include a multi-layer serial Transformer network. The initial representations of multiple cluster centers can be used to obtain a query matrix, which is then input into the multi-layer serial Transformer network. Each layer of the Transformer network corresponds to each resolution in order from low to high resolution. Each layer of the Transformer network uses the feature representation of each token at the corresponding resolution to obtain a key matrix and a value matrix. The query matrix input to the Transformer network of this layer is subjected to cross-attention processing to obtain the query matrix output by the Transformer network of this layer. The query matrix output by the last layer of the Transformer network is used to obtain multiple cluster center representations.

[0131] The processing performed by each layer of the Transformer network can be expressed as follows:

[0132] Among them, C n Indicates the corresponding matrix of multiple cluster centers output by the n-th layer Transformer network, C n-1 The corresponding matrix represents the multiple cluster centers output by the n-1th layer Transformer network. For the first layer Transformer network, its input C n-1 The matrix C corresponding to the initial representation of multiple cluster centers initial .Q by C n-1 K and V are obtained from the feature representation of each token at the corresponding resolution of the Transformer network at this layer. T represents the transposition process. N is the number of cluster centers.

[0133] From the above process, we can see that the first decoder is equivalent to grouping the tokens in the image according to the initial cluster centers. The cross attention process is similar to the K-means clustering algorithm, using argmax processing instead of the softmax processing in the traditional Transformer network. The matrix corresponding to the multiple cluster centers is represented by C∈R N×A .

[0134] The above step 208, i.e., "using the feature representation of the image and the multiple cluster center representations to perform decoding processing to obtain a category assignment matrix" can be executed by the above-mentioned second decoding network.

[0135] This step actually redistributes the feature representation of each token based on N cluster centers and assigns them to different clusters. Therefore, the result of this step is called the category assignment matrix.

[0136] The second decoding network can perform matrix multiplication of the matrix C corresponding to the multiple cluster center representations and the feature representation F of the image, and then perform Softmax to obtain the matrix M corresponding to the category assignment matrix, which can be expressed as: M = Softmax N (CF) (2)

[0137] Among them, Softmax N () represents the Softmax (normalized exponential function) processing performed on the N cluster center dimensions.

[0138] The above step 210, i.e., “classifying using multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the image to be classified belongs to the target category”, can be performed by the classification network in the image classification model shown in FIG. 3 .

[0139] Since the learned multiple cluster center representations have high-level semantics and reflect the differences between clusters and the similarities within clusters, combining multiple cluster center representations with the category assignment matrix for classification can significantly improve the classification accuracy and recall rate compared to directly using the feature representation of the image for classification.

[0140] As one of the feasible ways, multiple cluster center representations can be averaged to obtain a cluster average representation; and the category assignment matrix can be pooled to obtain a cluster pooled representation; then the cluster average representation and the cluster pooled representation are integrated, and the integrated feature representation is used for classification to obtain a classification result of whether the image to be classified belongs to the target category.

[0141] For example, the cluster average representation can be obtained by averaging multiple cluster center representations in the channel dimension. In addition to averaging, other processing methods can also be used to convert the matrix C into a vector. In Figure 3, the cluster average representation is

[0142] When performing pooling on the class assignment matrix, a global maximum pooling method can be used. In Figure 3, the cluster pooling obtained after pooling is represented as

[0143] The integration of cluster average representation and cluster pooling representation can be spliced, as shown in Figure 3, and After splicing, it passes through several layers of MLP (Multilayer Perceptron) to obtain the classification result of whether the image to be classified belongs to the target category

[0144] In the embodiment of the present application, the classification result can be binary classification or multi-classification. Taking binary classification as an example, the output classification result can be either belonging to the target category or not belonging to the target category.

[0145] Furthermore, in an embodiment of the present application, the segmentation network in the image classification model may also use the category assignment matrix to segment the image to be classified to obtain image regions of preset categories.

[0146] The category assignment matrix is ​​obtained by redistributing the feature representation of each token based on N cluster centers to assign it to different clusters, so that the tokens belonging to the same cluster can be treated as a whole. Therefore, the category assignment matrix can be projected onto K channels to obtain the image region of K categories, i.e. the segmentation result. K is a preset positive integer greater than 1. The K categories include the target category corresponding to the classification network. The segmentation results of the image to be classified can provide interpretability for the classification results, allowing users to refer to and understand the region of the target category in the image to be classified.

[0147] The following describes in detail the training process of the image classification model used in the above embodiment. Figure 4 is a flow chart of the method for training an image classification model provided in an embodiment of the present application. This method flow can be executed by the model training device in the system shown in Figure 1. As shown in Figure 4, the method may include the following steps:

[0148] Step 402: Acquire training data including a plurality of training samples, where the training samples include image samples and labels indicating whether the image samples belong to a target category.

[0149] In the embodiments of the present application, some images known to belong to or not belong to the target category can be obtained as image samples, and labels indicating whether the image samples belong to the target category can be annotated. Alternatively, some images can be obtained as image samples, and labels indicating whether the image samples belong to the target category can be manually annotated. These image samples can be obtained from specific application fields based on actual needs.

[0150] Furthermore, image samples can be annotated with region masks of preset categories. This means that regions of preset categories in image samples are annotated to form region masks of specific categories. This can be done manually, or by obtaining image samples of regions with known preset categories and annotating the regions of the preset categories to form region masks. The aforementioned preset categories include target categories.

[0151] For example, the training data set S can be expressed as: {(X i ,Y i ,P i)|i=1,2,…,m}, where X i is the image sample, P i It can be expressed as X i The label of whether it belongs to the target category, Y i For X i The region mask of the labeled preset category, m is the number of training samples.

[0152] Step 404: Use the training data to train an image classification model, where the image classification model includes: a feature extraction network, a first decoding network, a second decoding network, and a classification network; the feature extraction network extracts features from the image sample to obtain a feature representation of the image sample; the first decoding network uses the feature representation of the image sample to perform cross-attention processing on the initial representations of multiple cluster centers to obtain multiple cluster center representations; the second decoding network uses the feature representation of the image sample and the multiple cluster center representations to perform decoding processing to obtain a category assignment matrix; the classification network uses the multiple cluster center representations and the category assignment matrix to perform classification to obtain a classification result of whether the image sample belongs to the target category; the training objectives include: minimizing the difference between the classification result and the corresponding label.

[0153] As shown in Figure 5, the feature extraction network is used to extract the image sample X i Perform feature extraction to obtain the feature representation F of the image sample i The feature extraction network can be implemented based on the Transformer network to extract image features and obtain the feature representation of each token in the image sample.

[0154] As one of the feasible ways, the feature extraction network can extract features from image samples, obtain the feature representation of each element token at multiple resolutions in turn, and use the feature representation of each token at the highest resolution as the feature representation of the image sample. For example, a network such as U-Net can be used as a feature extraction network to obtain the feature representation of each token at multiple resolutions, and improve the accuracy of image classification through such multi-scale features. Specifically, U-Net adopts an Encoder (encoder)-Decoder (decoder) architecture, where the Encoder performs downsampling and the Decoder performs upsampling and feature splicing to obtain feature representations of multiple different resolutions, that is, first obtain the feature representation of each token at a low resolution, and then obtain the feature representation of each token at a higher resolution in turn. The feature representation of each token at the highest resolution is then used as the feature representation of the image.

[0155] The first decoding network uses the feature representation F of the image sample i The initial representation of multiple cluster centers (the corresponding matrix is ​​represented by C initial) to perform cross attention processing and obtain multiple cluster center representations (the corresponding matrix is ​​represented as C i ).

[0156] If the feature extraction network obtains multiple feature representations of different resolutions, then as one of the preferred implementations, the first decoding network can include a multi-layer serially connected Transformer network, and the initial representations of multiple cluster centers are input into the first layer Transformer network as a query matrix. Each layer of the Transformer network corresponds to each resolution one by one in order from low to high resolution. Each layer of the Transformer network uses the feature representation of each Token at the corresponding resolution to obtain a key matrix and a value matrix, and performs cross-attention processing on the query matrix input into the Transformer network of this layer to obtain the query matrix output by the Transformer network of this layer; and the query matrix output by the last layer of the Transformer network is used to obtain multiple cluster center representations.

[0157] The second decoding network uses the feature representation F of the image samples i and multiple cluster centers represent C i Perform decoding processing to obtain the category assignment matrix M i In fact, the feature representation of each token is redistributed based on multiple cluster centers and assigned to different clusters. Therefore, the result obtained by the second decoding network is called the category assignment matrix.

[0158] The classification network uses multiple cluster centers to represent C i and the category assignment matrix M i Perform classification to obtain the classification result of whether the image sample belongs to the target category.

[0159] As one of the feasible ways, the classification network can average multiple cluster center representations to obtain a cluster average representation; pool the category assignment matrix to obtain a cluster pool representation; integrate the cluster average representation and the cluster pool representation, and use the integrated feature representation for classification to obtain the classification result of whether the image sample belongs to the target category.

[0160] For example, the cluster average representation can be obtained by averaging the representations of multiple cluster centers in the channel dimension. In addition to averaging, other processing methods can also be used to convert the matrix C i In Figure 5, the cluster average is represented as

[0161] When performing pooling on the class assignment matrix, a global maximum pooling method can be used. In Figure 5, the cluster pooling obtained after pooling is represented as

[0162] The integration of cluster average representation and cluster pooling representation can be spliced, as shown in Figure 5, and After splicing, it passes through several layers of MLP (Multilayer Perceptron) to obtain the classification result of whether the image to be classified belongs to the target category

[0163] In the embodiment of the present application, the classification result can be binary classification or multi-classification. Taking binary classification as an example, the output classification result can be either belonging to the target category or not belonging to the target category.

[0164] The training objectives in the above training process include minimizing the classification results and the training samples for X i Annotated label P i The difference between.

[0165] Furthermore, in an embodiment of the present application, the segmentation network in the image classification model may also use the category assignment matrix to segment the image samples to obtain image regions of preset categories.

[0166] The category assignment matrix is ​​obtained by redistributing the feature representation of each token based on multiple cluster centers to assign it to different clusters, so that each token belonging to the same cluster can be treated as a whole. Therefore, the category assignment matrix can be projected onto K channels to obtain the image region of K categories, i.e. the segmentation result. K is a preset positive integer greater than 1. The K categories include the target category corresponding to the classification network.

[0167] The training objectives in the above training process may further include: minimizing the image area of ​​the preset category and the corresponding area mask Y i The goal of this training can be used to assist in learning the classification results during the model training process.

[0168] As one possible implementation, in the embodiments of this specification, a loss function can be constructed based on the aforementioned training objective. In each iteration, the value of the loss function is used to update the model parameters using methods such as gradient descent until a preset training termination condition is met. The training termination condition may include, for example, the value of the loss function being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold.

[0169] If the above two training objectives are adopted, the total loss function L can be constructed, for example:

[0170] Where b is the number of training samples in a batch. reflect and Y i The difference between reflect and P i The difference between.

[0171] The image classification method provided by the above-mentioned embodiment of the present application can be applied to a variety of scenarios. For example, remote sensing images can be used as images to be classified for mineral crystal detection, oil exploration, etc., that is, the remote sensing images can be classified using the method provided by the embodiment of the present application to determine whether the remote sensing images belong to a specific mineral crystal category. At the same time, the remote sensing images can be segmented to obtain various categories of areas, including specific mineral crystal areas, thereby providing an interpretability reference. For another example, it can be used for equipment anomaly detection in some harsh environments, that is, equipment images can be collected using equipment such as drones and inspection robots, and the equipment images can be used as images to be classified. The equipment images can be classified using the method provided by the embodiment of the present application to determine whether they belong to an abnormal category, that is, whether there is an abnormality. At the same time, the equipment image can be segmented to obtain background areas, equipment areas, abnormal areas, etc. to provide an interpretability reference.

[0172] The present invention can also be applied to other scenarios, but one of the more important scenarios is the medical scenario, where the method provided in the embodiment of the present application can be used to classify medical images to determine whether a specific lesion exists. The following describes this application scenario as an example.

[0173] FIG6 is a flow chart of an image classification method for medical scenarios provided by an embodiment of the present application. As shown in FIG6 , the method may specifically include the following steps:

[0174] Step 602: Acquire medical images collected for the target organ.

[0175] Medical images collected here are acquired using non-invasive methods, such as computed tomography (CT), magnetic resonance imaging (MRI), and ultrasound. The target organs may be the stomach, lungs, liver, pancreas, colon, and other organs. For example, a non-contrast CT image may include the stomach.

[0176] Taking the plain scan CT image of the stomach as an example, a plain scan CT device usually produces a multi-layer image after scanning a circle. Therefore, the plain scan CT image X∈R H×W×D, where H, W, and D represent the height, width, and number of layers of a three-dimensional image (taking the input image to be classified as a three-dimensional image as an example).

[0177] Step 604: extract features from the medical image to obtain feature representation of the medical image.

[0178] Step 606: Using the feature representation of the medical image, cross-attention processing is performed on the initial representations of the multiple cluster centers obtained by pre-training to obtain multiple cluster center representations.

[0179] For the above-mentioned steps 604 and 606, reference may be made to the description of steps 204 and 206 in the previous embodiment, which will not be repeated here.

[0180] The number of cluster centers, N, is a hyperparameter. It's typically larger than the number of categories, K, used in the subsequent segmentation. This is because lesions like gastric cancer can have multiple subtypes. For example, gastric cancer manifests differently at different stages of development. Therefore, a larger number of cluster centers is often used. N is typically chosen based on empirical or experimental values, such as 8.

[0181] Step 608: Decoding the medical image using its feature representation and multiple cluster center representations to obtain a category assignment matrix.

[0182] Step 610: Perform classification using multiple cluster center representations and category assignment matrices to obtain a classification result of whether the medical image belongs to the target lesion.

[0183] For the above-mentioned steps 608 and 610, reference may be made to the description of steps 208 and 210 in the previous embodiment, which will not be repeated here.

[0184] Taking a plain CT image of the stomach as an example, the classification result obtained in step 610 can be a binary classification, namely normal or gastric cancer. This enables the detection of suspected gastric cancer based on a plain CT image of the stomach, providing a warning and reference for physicians. Cross-attention processing is performed on the initial feature representations of multiple cluster centers based on the multi-resolution feature representations of each token, and the feature representations of the multiple cluster centers are redistributed with the feature representation of the image, making the classification process both sensitive to local tissue and globally aware of organ physiology.

[0185] Furthermore, in embodiments of the present application, the segmentation network in the image classification model can also segment the medical image using a class assignment matrix to obtain image regions of pre-set categories. Pre-set categories can include background, organs, and lesions. Still using a plain CT image of the stomach as an example, the background region, stomach region, and gastric cancer region can be segmented, providing physicians with a reference for interpretability.

[0186] In an embodiment of the present application, feature extraction can also be performed on medical images to obtain feature representations of each element Token at multiple resolutions, and the feature representation of each Token at the highest resolution is used as the feature representation of the medical image.

[0187] If the feature extraction network obtains multiple feature representations of different resolutions, then as one of the preferred implementation methods, the initial representations of multiple cluster centers can be used to obtain a query matrix, and the query matrix is ​​input into a multi-layer serial transformation Transformer network. Each layer of the Transformer network corresponds to each resolution in order from low to high resolution. Each layer of the Transformer network uses the feature representation of each Token at the corresponding resolution to obtain a key matrix and a value matrix, and performs cross-attention processing on the query matrix input to the Transformer network of this layer to obtain the query matrix output by the Transformer network of this layer; the query matrix output by the last layer of the Transformer network is used to obtain multiple cluster center representations.

[0188] As one of the possible implementation methods, multiple cluster center representations can be averaged to obtain a cluster average representation; and the category assignment matrix can be pooled to obtain a cluster pool representation; then the cluster average representation and the cluster pool representation are integrated, and the integrated feature representation is used for classification to obtain a classification result of whether the medical image belongs to the target lesion. The embodiment of the present application also provides a computer-aided diagnosis method for gastric cancer, which includes:

[0189] Obtain medical images of the stomach;

[0190] performing feature extraction on the medical image to obtain a feature representation of the medical image;

[0191] Using the feature representation of the medical image, cross-attention processing is performed on the initial representations of multiple cluster centers obtained by pre-training to obtain multiple cluster center representations;

[0192] Performing decoding processing using the feature representation of the medical image and the multiple cluster center representations to obtain a category assignment matrix;

[0193] Classification is performed using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the medical image belongs to gastric cancer.

[0194] When the image classification model for detecting gastric cancer is trained using the process shown in FIG4 , the image samples can be plain CT images of the stomach that are known to be normal or gastric cancerous as each X i , label each plain scan CT image as normal or gastric cancer as P iSince it is difficult to mark the area of ​​plain scan CT, the doctor can mark the area of ​​each X-ray machine. i After the background area, stomach area and gastric cancer area are annotated on the corresponding contrast CT image, the annotated contrast CT image is aligned with the plain scan CT image to form relatively rough but highly reliable masks for each area.

[0195] Gastric cancer is the third leading cause of cancer-related deaths worldwide, with a 5-year survival rate of approximately 33%. If related symptoms can be detected in the early stages, the 5-year survival rate can be significantly improved. Since early gastric tumors may only invade the mucosa and muscularis, they are difficult to identify without gastric contrast agent injection. Currently, existing detection methods such as barium meal gastrography, endoscopy, and serum pepsinogen level examination are all invasive, costly, and have large side effects, making them difficult to be well applied to early gastric cancer detection. By adopting the above-mentioned method provided in the embodiment of the present application, it is possible to obtain images using non-invasive and low-cost methods such as plain CT images of the stomach, and by using the method provided in the embodiment of the present application, a computer device performs image classification to realize gastric cancer detection, that is, the final output classification result is whether it belongs to gastric cancer, for example, a binary classification result of normal or gastric cancer is output, thereby providing a reference basis or reminder to the physician or patient as intermediate data, so as to facilitate further examination and diagnosis in the future. At the same time, it can segment the plain scan CT image of the stomach into the background area, the stomach area, and the lesion area (i.e., the area where the gastric cancer is located), providing interpretable reference data for physicians and patients. This is clearly a new detection method that is non-invasive, low-cost, and easy to promote, and it also has excellent accuracy.

[0196] After experiments, the method provided in the embodiment of the present application obtained a classification result of gastric cancer with a sensitivity of 85.0% and a specificity of 92.6% on a test set of stomach plain scan CT images of 100 gastric cancer patients and 148 normal patients. The average sensitivity of radiologists in manually distinguishing gastric cancer from gastric plain scan CT images is 73.5% and the specificity is 84.3%. It is obvious that the image classification method based on computer vision technology provided in the embodiment of the present application is more effective. Among them, sensitivity is also called true positive rate, which refers to the probability that the classification result of the test for gastric cancer samples is gastric cancer. Specificity is also called true negative rate, which refers to the probability that the classification result of the test for normal samples is normal.

[0197] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0198] See Table 1 below, which shows a comparative analysis of the image classification method disclosed in this disclosure and three benchmarks. The first two benchmarks use two methods belonging to the "Segmentation for Classification" (S4C) class, using nnUNet and TransUNet. Indicates p<0.05 of the DeLong test (the present disclosure and nnUNet-S4C); * indicates p<0.05 of the permutation test (the present disclosure and nnUNet-S4C and radiologists). If the segmented tumor volume exceeds the threshold that maximizes the sum of the sensitivity and specificity of the validation set, the case is classified as positive. The third baseline (denoted as "nnUNet-Joint") integrates the CNN classification head into UNet and performs end-to-end training. The 95% confidence intervals of the AUC, sensitivity, and specificity values ​​can be obtained from 1000 bootstrap copies of the test data set for statistical analysis. In order to obtain statistical significance, a DeLong test can be performed between two AUCs (the image classification method of the present disclosure and the comparison method), and a permutation test can be performed between two sensitivities or specificities (the image classification method of the present disclosure and the comparison method and radiologists).

[0199] Table 1

[0200] As shown in Table 1, the image classification method of the present invention outperforms the other three baselines in all metrics (Table 1), especially in terms of AUC and sensitivity. The advantage of the method of the present invention is that it captures both local and global information simultaneously by virtue of the unique architecture of the mask transformer. It also extracts high-level semantics from the cluster representation, making it suitable for classification and facilitating the overall decision-making process. In addition, the method of the present invention achieves a considerable specificity of 97.7% on the external test set, which is crucial for reducing false positives and unnecessary human workload in opportunistic screening.

[0201] The ROC curve of the proposed model outperformed the results of the test by two experienced radiologists. The model achieved a sensitivity of 85.0% in detecting gastric cancer, significantly exceeding the average performance of doctors (73.5%) and the best performing doctor (75.0%), while maintaining high specificity.

[0202] Referring to Table 2, the performance of patient-level detection and tumor-level localization stratified by tumor (T) stage is shown. Table 2 compares the performance of the model disclosed herein with the performance of two radiologists. The results show that the model disclosed herein performs better in detecting early-stage tumors (T1, T2) and provides more accurate tumor localization. Specifically, the model disclosed herein detected 60.0% (6 / 10) T1 cancers and 77.8% (7 / 9) T2 cancers, exceeding the best-performing expert (50% T1, 55.6% T2). At the same time, the model disclosed herein maintained a reliable detection rate and reliable localization accuracy for T3 and T4 tumors (2 of 34 T3 tumors were missed).

[0203] Table 2 Patient-level detection and tumor-level localization results of gastric cancer at different T stages (%)

[0204] As shown in Table 3, with a relatively large test patient population (n=1151 by integrating internal and external test sets), the disclosed method surpassed or was comparable to existing screening tools in terms of sensitivity for gastric cancer detection, with similar specificity levels, as shown in Table 3. * indicates that two in situ tumors were removed from the test set. It indicates that only early gastric cancer cases, including in situ tumors, T1 and T2 stages, were considered in this comparison, of which 17 out of 19 cases were successfully detected by the method disclosed herein.

[0205] Table 3 Comparison of the performance of state-of-the-art blood tests for gastric cancer detection in a large population, UGIS and endoscopic screening, and the early gastric cancer detection rate using magnifying endoscopy (ME-NBI) on narrow-band imaging by senior radiologists

[0206] According to another embodiment, an image classification device is provided. FIG7 shows a schematic block diagram of an image classification device according to one embodiment. As shown in FIG7 , the device 700 includes: an image acquisition unit 701, a feature extraction unit 702, a first decoding unit 703, a second decoding unit 704, and an image classification unit 705, and may further include an image segmentation unit 706. The main functions of each component unit are as follows:

[0207] The image acquisition unit 701 is configured to acquire an image to be classified.

[0208] The feature extraction unit 702 is configured to extract features from the image to be classified to obtain a feature representation of the image.

[0209] The first decoding unit 703 is configured to perform cross-attention processing on the initial representations of multiple cluster centers obtained through pre-training using the feature representation of the image to obtain multiple cluster center representations.

[0210] The second decoding unit 704 is configured to perform decoding processing using the feature representation of the image and the multiple cluster center representations to obtain a category assignment matrix.

[0211] The image classification unit 705 is configured to perform classification using multiple cluster center representations and a category assignment matrix to obtain a classification result of whether the image to be classified belongs to a target category.

[0212] Furthermore, the image segmentation unit 706 is configured to segment the image to be classified using the category assignment matrix to obtain image regions of preset categories, where the preset categories include target categories.

[0213] As one of the possible implementation methods, the feature extraction unit 702 may be specifically configured to perform feature extraction on the image to be classified, obtain feature representations of each element Token at multiple resolutions, and use the feature representations of each Token at the highest resolution as the feature representation of the image.

[0214] Correspondingly, the first decoding unit 703 can use the initial representations of multiple cluster centers to obtain a query matrix, and input the query matrix into a multi-layer serialized transformer network. Each layer of the transformer network corresponds to each resolution in order from low to high resolution. Each layer of the transformer network uses the feature representation of each token at the corresponding resolution to obtain a key matrix and a value matrix, and performs cross-attention processing on the query matrix input to the transformer network of this layer to obtain the query matrix output by the transformer network of this layer; the query matrix output by the last layer of the transformer network is used to obtain multiple cluster center representations.

[0215] As one of the feasible ways, the image classification unit 705 can be specifically configured to average the representations of multiple cluster centers to obtain a cluster average representation; perform pooling on the category assignment matrix to obtain cluster pooling features; integrate the cluster average representation and the cluster pooling representation, and use the integrated feature representation for classification to obtain a classification result of whether the image to be classified belongs to the target category.

[0216] The above-mentioned feature extraction unit 702, first decoding unit 703, second decoding unit 704, image classification unit 705 and image segmentation unit 706 respectively correspond to the feature extraction network, first decoding network, second decoding network, classification network and segmentation network in the image classification model shown in Figure 3. For specific details, please refer to the relevant records of Figure 3 in the above-mentioned method embodiment, which will not be repeated here.

[0217] FIG8 is a schematic block diagram of an apparatus for training an image classification model according to an embodiment of the present application. As shown in FIG8 , the apparatus may include: a sample acquisition unit 801 and a model training unit 802. The main functions of each component unit are as follows:

[0218] The sample acquisition unit 801 is configured to acquire training data including a plurality of training samples, where the training samples include image samples and labels indicating whether the image samples belong to a target category.

[0219] The model training unit 802 is configured to train an image classification model using training data, where the image classification model includes: a feature extraction network, a first decoding network, a second decoding network, and a classification network; the feature extraction network extracts features from image samples to obtain feature representations of the image samples; the first decoding network uses the feature representations of the image samples to perform cross-attention processing on the initial representations of multiple cluster centers to obtain multiple cluster center representations; the second decoding network uses the feature representations of the image samples and the multiple cluster center representations to perform decoding processing to obtain a category assignment matrix; the classification network uses the multiple cluster center representations and the category assignment matrix to perform classification to obtain a classification result of whether the image sample belongs to the target category; the training objectives include: minimizing the difference between the classification result and the corresponding label.

[0220] As one possible implementation, the training samples may further include region masks of preset categories annotated on the image samples. The image classification model may also include a segmentation network.

[0221] The segmentation network segments the image sample using the class assignment matrix to obtain image regions of a preset class, where the preset class includes the target class. Accordingly, the training objective may also include minimizing the difference between the image regions of the preset class and the corresponding region masks.

[0222] As one of the feasible ways, the feature extraction network can extract features from image samples, obtain the feature representations of each element token at multiple resolutions in turn, and use the feature representations of each token at the highest resolution as the feature representation of the image sample.

[0223] Correspondingly, the first decoding network may include multiple layers of serially connected Transformer networks, and the initial representations of multiple cluster centers are input into the first layer of Transformer network as query matrices. Each layer of Transformer network corresponds to each resolution one-to-one in order from low to high resolution. Each layer of Transformer network uses the feature representation of each Token at the corresponding resolution to obtain a key matrix and a value matrix, and performs cross-attention processing on the query matrix input into the current layer of Transformer network to obtain the query matrix output by the current layer of Transformer network; and uses the query matrix output by the last layer of Transformer network to obtain multiple cluster center representations.

[0224] As one of the feasible ways, the classification network can average multiple cluster center representations to obtain a cluster average representation; pool the category assignment matrix to obtain a cluster pool representation; integrate the cluster average representation and the cluster pool representation, and use the integrated feature representation for classification to obtain the classification result of whether the image sample belongs to the target category.

[0225] According to another embodiment, an image classification device is provided for use in medical diagnosis. The device includes: an image acquisition unit, a feature extraction unit, a first decoding unit, a second decoding unit, and an image classification unit, and may further include an image segmentation unit. The main functions of each component unit are as follows:

[0226] an image acquisition unit configured to acquire a medical image for a target organ;

[0227] a feature extraction unit, configured to extract features from the medical image to obtain a feature representation of the medical image;

[0228] A first decoding unit is configured to perform cross-attention processing on the initial representations of the plurality of cluster centers obtained by pre-training using the feature representation of the medical image to obtain a plurality of cluster center representations;

[0229] A second decoding unit is configured to perform decoding processing using the feature representation of the medical image and the multiple cluster center representations to obtain a category assignment matrix;

[0230] The image classification unit is configured to perform classification using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the medical image belongs to a target lesion.

[0231] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0232] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0233] In addition, an embodiment of the present application further provides a computer-aided cancer diagnosis system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to perform an image classification method, the method comprising:

[0234] Obtain medical imaging for target organ harvesting;

[0235] performing feature extraction on the medical image to obtain a feature representation of the medical image;

[0236] Using the feature representation of the medical image, cross-attention processing is performed on the initial representations of multiple cluster centers obtained by pre-training to obtain multiple cluster center representations;

[0237] Performing decoding processing using the feature representation of the medical image and the multiple cluster center representations to obtain a category assignment matrix;

[0238] Classification is performed using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the medical image belongs to a target lesion.

[0239] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0240] And an electronic device comprising:

[0241] one or more processors; and

[0242] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0243] The present application also provides a computer program product, comprising a computer program, which implements the steps of any one of the methods described in the aforementioned method embodiments when executed by a processor.

[0244] The present application also provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of any one of the methods described in the aforementioned method embodiments.

[0245] 9 exemplarily illustrates the architecture of an electronic device, which may include a processor 910, a video display adapter 911, a disk drive 912, an input / output interface 913, a network interface 914, and a memory 920. The processor 910, the video display adapter 911, the disk drive 912, the input / output interface 913, the network interface 914, and the memory 920 may be communicatively connected via a communication bus 930.

[0246] Among them, the processor 910 can be implemented by a general CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in this application.

[0247] The memory 920 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 920 can store an operating system 921 for controlling the operation of the electronic device 900, and a basic input and output system (BIOS) 922 for controlling the low-level operations of the electronic device 900. In addition, a web browser 923, a data storage management system 924, and an image classification device / model training device 925, etc. can also be stored. The above-mentioned image classification device / model training device 925 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 920 and is called and executed by the processor 910.

[0248] The input / output interface 913 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0249] The network interface 914 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0250] The bus 930 comprises a pathway for transmitting information between the various components of the device (eg, the processor 910 , the video display adapter 911 , the disk drive 912 , the input / output interface 913 , the network interface 914 , and the memory 920 ).

[0251] It should be noted that although the above device only shows the processor 910, video display adapter 911, disk drive 912, input / output interface 913, network interface 914, memory 920, bus 930, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may also include only the components necessary to implement the solution of the present application, and does not necessarily include all the components shown in the figure.

[0252] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application or certain parts of the embodiments.

[0253] The above is a detailed introduction to the technical solutions provided by this application. Specific examples are used herein to illustrate the principles and implementation methods of this application. The description of the above embodiments is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the contents of this specification should not be understood as limiting this application.

Claims

1. An image classification method, characterized in that: The method comprises: Get the image to be classified; Extracting features of the image to be classified to obtain a feature representation of the image; Using the feature representation of the image, cross-attention processing is performed on the initial representations of multiple cluster centers obtained by pre-training to obtain multiple cluster center representations; Performing decoding processing using the feature representation of the image and the multiple cluster center representations to obtain a category assignment matrix; Classification is performed using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the image to be classified belongs to a target category.

2. The method according to claim 1, characterized in that: The method further comprises: The image to be classified is segmented using the category assignment matrix to obtain image regions of preset categories, where the preset categories include the target category.

3. The method according to claim 1 or 2, characterized in that: Extracting features from the image to be classified to obtain a feature representation of the image includes: Feature extraction is performed on the image to be classified to obtain feature representations of each element Token at multiple resolutions, and the feature representation of each Token at the highest resolution is used as the feature representation of the image.

4. The method according to claim 3, characterized in that Using the feature representation of the image to perform cross-attention processing on the initial representations of the plurality of cluster centers obtained by pre-training includes: The query matrix is ​​obtained by using the initial representation of multiple cluster centers, and the query matrix is ​​input into a multi-layer serialized transformer network. Each layer of the transformer network corresponds to each resolution in order from low to high resolution. Each layer of the transformer network uses the feature representation of each token at the corresponding resolution to obtain the key matrix and value matrix, and performs cross-attention processing on the query matrix input to the transformer network of this layer to obtain the query matrix output by the transformer network of this layer. The query matrix output by the last layer of the Transformer network is used to obtain the multiple cluster center representations.

5. The method according to any one of claims 1 to 4, characterized in that Using the multiple cluster center representations and the category assignment matrix to perform classification includes: Performing an averaging process on the multiple cluster center representations to obtain a cluster average representation; Performing pooling processing on the category assignment matrix to obtain clustering pooling features; The cluster average representation and the cluster pool representation are integrated, and the integrated feature representation is used for classification to obtain a classification result of whether the image to be classified belongs to the target category.

6. A method for training an image classification model, characterized in that: The method comprises: Acquire training data including a plurality of training samples, wherein the training samples include image samples and labels indicating whether the image samples belong to a target category; The training data is used to train an image classification model, wherein the image classification model includes: a feature extraction network, a first decoding network, a second decoding network and a classification network; the feature extraction network extracts features from the image samples, The feature representation of the image sample is obtained; the first decoding network uses the feature representation of the image sample to perform cross-attention processing on the initial representations of multiple cluster centers to obtain multiple cluster center representations; the second decoding network uses the feature representation of the image sample and the multiple cluster center representations to perform decoding processing to obtain a category assignment matrix; the classification network uses the multiple cluster center representations and the category assignment matrix to perform classification to obtain a classification result of whether the image sample belongs to a target category; the training objectives include: minimizing the difference between the classification result and the corresponding label.

7. The method according to claim 6, characterized in that The training sample also includes a region mask of a preset category annotated on the image sample; the image classification model also includes a segmentation network; The segmentation network segments the image sample using the category assignment matrix to obtain image regions of preset categories, where the preset categories include the target category; The training goal also includes: minimizing the difference between the image region of the preset category and the corresponding region mask.

8. The method according to claim 6 or 7, characterized in that: The feature extraction network extracts features from the image sample to obtain feature representations of the image sample, including: extracting features from the image sample to sequentially obtain feature representations of each element Token at multiple resolutions, and using the feature representations of each Token at the highest resolution as the feature representation of the image sample; The first decoding network includes multiple layers of serially connected Transformer networks. The initial representations of the multiple cluster centers are input into the first layer of Transformer networks as query matrices. Each layer of Transformer networks corresponds to each resolution one by one in order from low to high resolution. Each layer of Transformer networks uses the feature representations of each Token at the corresponding resolution to obtain a key matrix and a value matrix, and performs cross-attention processing on the query matrix input into the current layer of Transformer networks to obtain the query matrix output by the current layer of Transformer networks. The query matrix output by the last layer of Transformer networks is used to obtain the multiple cluster center representations.

9. The method according to any one of claims 6 to 8, characterized in that The classification network uses the multiple cluster center representations and the category assignment matrix to perform classification, including: The classification network averages the multiple cluster center representations to obtain a cluster average representation; Performing pooling processing on the category assignment matrix to obtain a cluster pooling representation; The cluster average representation and the cluster pooling representation are integrated, and the integrated feature representation is used for classification to obtain a classification result of whether the image sample belongs to the target category.

10. An image classification method, executed by a cloud server, characterized in that: The method comprises: Acquire an image to be classified from a user terminal; Extracting features of the image to be classified to obtain a feature representation of the image; Using the feature representation of the image, cross-attention processing is performed on the initial representations of multiple cluster centers obtained by pre-training to obtain multiple cluster center representations; Performing decoding processing using the feature representation of the image and the multiple cluster center representations to obtain a category assignment matrix; Using the multiple cluster center representations and the category assignment matrix to perform classification, obtaining a classification result of whether the image to be classified belongs to a target category; The classification result is returned to the user terminal.

11. A method for computer-aided diagnosis of cancer, characterized in that: The method comprises: Obtain medical images for targeted organ harvesting; Extracting features from the medical image to obtain a feature representation of the medical image; Using the feature representation of the medical image, cross-attention processing is performed on the initial representations of multiple cluster centers obtained by pre-training to obtain multiple cluster center representations; Performing decoding processing using the feature representation of the medical image and the multiple cluster center representations to obtain a category assignment matrix; Classification is performed using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the medical image belongs to a target lesion.

12. The method according to claim 11, characterized in that The method further comprises: The medical image is segmented using the category assignment matrix to obtain image regions of preset categories, where the preset categories include the target lesions.

13. The method according to claim 11 or 12, characterized in that: Extracting features from the medical image to obtain a feature representation of the medical image includes: Feature extraction is performed on the medical image to obtain feature representations of each element Token at multiple resolutions, and the feature representation of each Token at the highest resolution is used as the feature representation of the medical image.

14. The method according to claim 13, characterized in that Using the feature representation of the medical image to perform cross-attention processing on the initial representations of multiple cluster centers obtained by pre-training includes: The query matrix is ​​obtained by using the initial representation of multiple cluster centers, and the query matrix is ​​input into a multi-layer serialized transformer network. Each layer of the transformer network corresponds to each resolution in order from low to high resolution. Each layer of the transformer network uses the feature representation of each token at the corresponding resolution to obtain the key matrix and value matrix, and performs cross-attention processing on the query matrix input to the transformer network of this layer to obtain the query matrix output by the transformer network of this layer. The query matrix output by the last layer of the Transformer network is used to obtain the multiple cluster center representations.

15. The method according to any one of claims 11 to 14, characterized in that Using the multiple cluster center representations and the category assignment matrix to perform classification includes: Performing an averaging process on the multiple cluster center representations to obtain a cluster average representation; Performing pooling processing on the category assignment matrix to obtain clustering pooling features; The cluster average representation and the cluster pool representation are integrated, and the integrated feature representation is used for classification to obtain a classification result of whether the medical image belongs to the target lesion.

16. A computer-aided diagnosis method for gastric cancer, the method comprising: Obtain medical images of the stomach; Extracting features from the medical image to obtain a feature representation of the medical image; The initial representations of multiple cluster centers obtained by pre-training are cross-referenced using the feature representation of the medical image. Attention processing, obtaining multiple cluster center representations; Performing decoding processing using the feature representation of the medical image and the multiple cluster center representations to obtain a category assignment matrix; Classification is performed using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the medical image belongs to gastric cancer.

17. A computer-aided cancer diagnosis system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to perform a cancer computer method, the method comprising: Obtain medical images for targeted organ harvesting; Extracting features from the medical image to obtain a feature representation of the medical image; Using the feature representation of the medical image, cross-attention processing is performed on the initial representations of multiple cluster centers obtained by pre-training to obtain multiple cluster center representations; Performing decoding processing using the feature representation of the medical image and the multiple cluster center representations to obtain a category assignment matrix; Classification is performed using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the medical image belongs to a target lesion.

18. An image classification device, characterized in that: The device comprises: An image acquisition unit, configured to acquire an image to be classified; a feature extraction unit, configured to extract features from the image to be classified to obtain a feature representation of the image; A first decoding unit is configured to perform cross-attention processing on initial representations of multiple cluster centers obtained by pre-training using feature representations of the image to obtain multiple cluster center representations; A second decoding unit is configured to perform decoding processing using the feature representation of the image and the plurality of cluster center representations to obtain a category assignment matrix; The image classification unit is configured to perform classification using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the image to be classified belongs to a target category.

19. A device for training an image classification model, characterized in that: The device comprises: A sample acquisition unit is configured to acquire training data including a plurality of training samples, wherein the training samples include image samples and labels indicating whether the image samples belong to a target category; The model training unit is configured to train an image classification model using the training data, wherein the image classification model includes: a feature extraction network, a first decoding network, a second decoding network and a classification network; the feature extraction network extracts features from the image sample to obtain a feature representation of the image sample; the first decoding network uses the feature representation of the image sample to perform cross-attention processing on initial representations of multiple cluster centers to obtain multiple cluster center representations; the second decoding network uses the feature representation of the image sample and the multiple cluster center representations to perform decoding processing to obtain a category assignment matrix; the classification network uses the multiple cluster center representations and the category assignment matrix to perform classification processing on the image sample; Classification, to obtain a classification result of whether the image sample belongs to the target category; the training goal includes: minimizing the difference between the classification result and the corresponding label.

20. An image classification system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to perform an image classification method, the method comprising: Get the image to be classified; Extracting features of the image to be classified to obtain a feature representation of the image; Using the feature representation of the image, cross-attention processing is performed on the initial representations of multiple cluster centers obtained by pre-training to obtain multiple cluster center representations; Performing decoding processing using the feature representation of the image and the multiple cluster center representations to obtain a category assignment matrix; Classification is performed using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the image to be classified belongs to a target category.

21. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

22. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of claims 1 to 16.

23. A computer program product, comprising computer executable instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 16.

24. A computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 16.

Images