Training methods, usage methods, devices, equipment and media for image classification models

Abstract

This application discloses a training method, usage method, apparatus, device, and medium for an image classification model, belonging to the field of artificial intelligence. The image classification model includes a feature extraction network and a multiple instance learning model. The method includes: acquiring a sample image set, wherein each sample image in the sample image set includes at least two instances; training the feature extraction network using the sample images in the sample image set through self-supervised learning based on contrastive learning, obtaining a trained feature extraction network; and training the multiple instance learning model using the sample images in the sample image set through multiple instance learning based on a mutual attention mechanism, obtaining a trained multiple instance learning model. The above scheme can reduce the computational complexity of the image classification model. The embodiments of this application can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training method, usage method, apparatus, computer device, computer-readable storage medium, and computer program product for an image classification model. Background Technology

[0002] In machine learning, Multiple Instance Learning (MIL) is a method that evolved from supervised algorithms.

[0003] Standard multiple instance learning (MIM) assumes that a bag contains several instances. In MIM, only the bag is labeled; instances are unlabeled. If a bag contains at least one positive instance, it is labeled a positive sample. If all instances in a bag are negative instances, the bag is labeled a negative sample. Taking an image as the bag, the animals in the image as instances, and a cat as a positive instance, since an image can contain many animals, regardless of how many other animals are in an image, as long as the image contains one or more cats, it is labeled a positive sample; if the image contains other animals but no cats, it is labeled a negative sample. Related techniques improve the feature extraction capability of MIM models by mining the feature information of all instances within the same image and then processing the feature information between different instances within the same image using a self-attention mechanism.

[0004] However, the self-attention mechanism in related technologies has high computational complexity, consumes a lot of hardware resources and time, and is difficult to train. How to reduce the computational complexity of multi-instance learning methods is an urgent problem to be solved. Summary of the Invention

[0005] This application provides a training method, usage method, apparatus, computer device, computer-readable storage medium, and computer program product for an image classification model, which can improve the prediction accuracy of the image classification model. The technical solution is as follows:

[0006] One aspect provides a training method for an image classification model, which includes a feature extraction network and a multi-instance learning model. This method includes:

[0007] Obtain a sample image set, which includes at least two examples of sample images;

[0008] By using self-supervised learning based on contrastive learning, the feature extraction network is trained using sample images from the sample image set, resulting in a trained feature extraction network.

[0009] By using multi-instance learning based on mutual attention mechanism, the multi-instance learning model is trained using sample images from the sample image set, and the trained multi-instance learning model is obtained.

[0010] The mutual attention mechanism is a mechanism for computing the attention between the feature representations of at least two examples and low-rank latent variables. The number of low-rank latent variables is less than the number of at least two examples, and the low-rank latent variables are set based on the rank of the sample images.

[0011] On another front, a method for using an image classification model is provided, which includes:

[0012] Obtain the input image to be classified, which includes at least two examples;

[0013] The input image is processed by the feature extraction network in the input image classification model to extract features, resulting in a third feature representation sequence. The third feature representation sequence includes feature representations of at least two examples.

[0014] The third feature representation sequence is input into the multi-instance learning model in the image classification model to perform classification prediction based on the mutual attention mechanism, and the classification result of the input image is obtained.

[0015] The mutual attention mechanism is a mechanism for computing the attention between the feature representations of at least two examples and low-rank latent variables. The number of low-rank latent variables is less than the number of at least two examples, and the low-rank latent variables are set based on the rank of the sample images.

[0016] In another aspect, a training apparatus for an image classification model is provided, the image classification model including a feature extraction network and a multi-instance learning model, the apparatus comprising:

[0017] An acquisition module is used to acquire a sample image set, wherein the sample images in the sample image set include at least two examples;

[0018] The self-supervised learning module is used to train the feature extraction network using sample images from the sample image set through self-supervised learning based on contrastive learning, so as to obtain the trained feature extraction network.

[0019] The multi-instance learning module is used to train the multi-instance learning model using sample images from the sample image set through multi-instance learning based on the mutual attention mechanism, so as to obtain the trained multi-instance learning model.

[0020] The mutual attention mechanism is a mechanism for calculating the attention between the feature representations of the at least two examples and the low-rank latent variables, wherein the number of low-rank latent variables is less than the number of the at least two examples, and the low-rank latent variables are set based on the rank of the sample images.

[0021] In another aspect, an apparatus for using an image classification model is provided, the apparatus comprising:

[0022] The acquisition module is used to acquire the input image to be classified, which includes at least two examples;

[0023] The feature extraction module is used to extract features from the feature extraction network in the input image classification model to obtain a third feature representation sequence, which includes feature representations of at least two examples.

[0024] The prediction module is used to perform classification prediction based on mutual attention mechanism on the multi-instance learning model in the image classification model by inputting the third feature representation sequence into the model, and obtain the classification result of the input image.

[0025] The mutual attention mechanism is a mechanism for computing the attention between the feature representations of at least two examples and low-rank latent variables. The number of low-rank latent variables is less than the number of at least two examples, and the low-rank latent variables are set based on the rank of the sample images.

[0026] In another aspect, a computer device is provided, comprising: a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the training method and / or usage method of the image classification model as described above.

[0027] In another aspect, a computer-readable storage medium is provided, which stores a computer program that is loaded and executed by a processor to implement the training method and / or usage method of the image classification model as described above.

[0028] In another aspect, a computer program product is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the training method and / or usage method of the image classification model provided in the above aspects.

[0029] The beneficial effects of the technical solutions provided in this application include at least the following:

[0030] By pre-setting low-rank latent variables according to the rank of the sample images, and using a mutual attention mechanism between at least two instances from the same sample image and the low-rank latent variables, the self-attention mechanism in related techniques is replaced. This avoids directly calculating self-attention, reducing computational cost while maintaining the model's learning ability and prediction accuracy. For example, if the number of instances in multiple examples from the same sample image is n, then the computational complexity of the self-attention mechanism is O(n^2). If the number of low-rank latent variables in this application is r, then the computational complexity of the mutual attention mechanism is... When the sample images are low-rank sample images, r is much smaller than n, thus significantly reducing computational complexity. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 A block diagram of a computer system provided in an exemplary embodiment is shown;

[0033] Figure 2 A schematic diagram of a training method for an image classification model provided in an exemplary embodiment is shown;

[0034] Figure 3 A schematic diagram of a training method for an image classification model provided in an exemplary embodiment is shown;

[0035] Figure 4 A flowchart illustrating a training method for an image classification model provided in an exemplary embodiment is shown;

[0036] Figure 5 A flowchart illustrating a training method for an image classification model provided in an exemplary embodiment is shown;

[0037] Figure 6 A flowchart illustrating a training method for an image classification model provided in an exemplary embodiment is shown;

[0038] Figure 7 illustrates a schematic diagram of a mutual attention network provided in an exemplary embodiment;

[0039] Figure 8 A schematic diagram of a gated attention module provided in an exemplary embodiment is shown;

[0040] Figure 9 shows a flowchart of a training method for an image classification model provided in an exemplary embodiment;

[0041] Figure 10 illustrates a low-rank schematic diagram of a feature vector provided by an exemplary embodiment;

[0042] Figure 11 A flowchart illustrating a training method for an image classification model provided in an exemplary embodiment is shown;

[0043] Figure 12 illustrates a schematic diagram of the rank and probability density of ImageNet pre-trained features and self-supervised learning provided in an exemplary embodiment;

[0044] Figure 13 shows a flowchart of a method for using an image classification model provided in an exemplary embodiment;

[0045] Figure 14 A scene diagram illustrating a training method for an image classification model provided in an exemplary embodiment is shown.

[0046] Figure 15 A scene diagram illustrating a training method for an image classification model provided in an exemplary embodiment is shown.

[0047] Figure 16 A structural block diagram of a training apparatus for an image classification model provided in an exemplary embodiment is shown;

[0048] Figure 17 A structural block diagram of an apparatus for using an image classification model provided in an exemplary embodiment is shown;

[0049] Figure 18 shows a structural block diagram of a computer device provided in an exemplary embodiment. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0051] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0052] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0053] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the input images involved in this application were obtained with the user's authorization or with the full authorization of all parties.

[0054] It should be understood that although the terms first, second, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, a first parameter may also be referred to as a second parameter without departing from the scope of this disclosure, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0055] Figure 1 A structural block diagram of a computer system provided in an exemplary embodiment of this application is shown. This computer system can implement a system architecture that serves as a training method and / or usage method for an image classification model. The computer system may include: a terminal 100 and a server 110.

[0056] Terminal 100 includes, but is not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, and aircraft. A client application for the target application can be installed and run on terminal 100. This target application can be a training and / or usage application for an image classification model, or other applications that provide training and / or usage functions for an image classification model; this application does not limit the specific application. Furthermore, this application does not limit the form of the target application, including but not limited to apps, mini-programs, etc., installed on terminal 100, and can also be in web page form.

[0057] Server 110 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud servers, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Server 110 can be a backend server for the aforementioned target application, used to provide backend services to the clients of the target application.

[0058] The training method and / or usage method of the image classification model provided in this application embodiment can be executed by a computer device, which refers to an electronic device with data computing, processing, and storage capabilities. Figure 1 Taking the implementation environment of the scheme shown as an example, the training method and / or usage method of the image classification model can be executed by the terminal 100. For example, the client of the target application installed and running in the terminal 100 can execute the training method and / or usage method of the image classification model, or the server 110 can execute the training method and / or usage method of the image classification model, or the terminal 100 and the server 110 can interact and cooperate to execute it. This application does not limit this.

[0059] Furthermore, the technical solution of this application can also be combined with blockchain technology. For example, some data involved in the training method and / or usage method of the image classification model disclosed in this application can be stored on the blockchain. Terminal 100 and server 110 can communicate via a network, such as a wired or wireless network. Embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0060] This application relates to the fields of artificial intelligence technology and computer vision technology.

[0061] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0062] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0063] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0064] Related technologies improve the feature extraction capability of multi-example learning models by mining the feature information of all examples in the same image and then processing the feature information between different examples in the same image based on a self-attention mechanism.

[0065] However, the self-attention mechanism in related technologies has high computational complexity, consumes a lot of hardware resources and time, and is difficult to train. How to reduce the computational complexity of multi-instance learning methods is an urgent problem to be solved.

[0066] In this application, a low-rank latent variable is pre-set according to the rank of the sample image. A mutual attention mechanism between at least two instances from the same sample image and the low-rank latent variable replaces the self-attention mechanism in related technologies, avoiding direct computation of self-attention. This reduces computational cost while maintaining the model's learning ability and prediction accuracy. For example, if the number of instances in multiple examples from the same sample image is n, the computational complexity of the self-attention mechanism is O(n^2). If the number of low-rank latent variables in this application is r, then the computational complexity of the mutual attention mechanism is... When the sample images are low-rank sample images, the computational complexity can be significantly reduced because r is much smaller than n.

[0067] Next, an image classification model provided by an exemplary embodiment of this application will be introduced. Figure 2 This diagram illustrates the training principle of an image classification model provided in an exemplary embodiment of this application. The image classification model includes a feature extraction network 220 and a multiple instance learning model 240.

[0068] The multi-instance learning model 240 includes a mutual attention network 242, a nonlocal pooling layer 244, and a linear classifier 246. The mutual attention network 242 includes two cascaded gated attention components 242-a and 242-b.

[0069] The output of the feature extraction network 220 is connected to the input of the gated attention 242-a, the output of the gated attention component 242-b is connected to the input of the nonlocal pooling layer 244, and the output of the nonlocal pooling layer 244 is connected to the input of the linear classifier 246.

[0070] The image classification model's processing of 200 sample images includes: a preprocessing stage, a self-supervised learning stage, and a multi-instance learning stage.

[0071] Preprocessing stage:

[0072] A sample image is an image input to the image classification model. The training sample set includes multiple sample images. Optionally, these sample images are low-rank sample images, such as cell slice images, forest images, starry sky images, etc., images containing a large number of similar or identical elements. Each sample image is preprocessed to obtain at least two examples for each sample image. For example, each sample image can be divided into n image slices of the same size, and each image slice can be considered as an example, then the same sample image has n examples; or, for example, facial regions in each sample image can be identified, and each facial region can be considered as an example. If a sample image has facial regions of 2 animals, then it has 2 examples; if a sample image has facial regions of 4 animals, then it has 4 examples.

[0073] Self-supervised learning phase:

[0074] Feature extraction network 220 extracts features from at least two examples to obtain a first feature representation sequence. The first feature representation sequence includes multiple first feature representations, each of which is a feature representation of an example. If a sample image includes n examples, then the first feature representation sequence of the sample image includes n first feature representations.

[0075] Traditional self-supervised learning in related technologies uses a first feature representation that is identical to the anchor's first feature representation as the positive sample feature representation of the anchor's first feature representation; and uses a first feature representation that is different from the anchor's first feature representation as the negative sample feature representation of the anchor's first feature representation. Here, the anchor's first feature representation is the first feature representation of any example. That is, traditional self-supervised learning assumes that an example has only one positive sample.

[0076] This application improves the sampling method for positive and negative samples. Addressing the limitation of traditional self-supervised learning that only considers a single positive sample, it clusters the first feature representations in the first feature representation sequence. First feature representations belonging to the same cluster as the anchor first feature representation are used as positive sample feature representations of the anchor first feature representation; first feature representations belonging to different clusters are used as negative sample feature representations of the anchor first feature representation. This introduces multiple positive samples, consistent with practical applications. Then, based on the self-supervised learning loss between positive and negative sample feature representations, the feature extraction network 220 is trained. In other words, the self-supervised learning in this application considers an example to have multiple positive samples.

[0077] Multi-example learning phase:

[0078] Traditional multiple instance learning introduces a self-attention mechanism to model the correlation between different instances of the same sample image. Assuming the same sample image has n instances, the self-attention mechanism has high computational complexity. It consumes a lot of hardware resources and time, making training difficult.

[0079] This application focuses on improving the performance degradation caused by insufficient attention to information interaction among input examples in traditional multiple instance learning. Specifically, it introduces a mutual attention mechanism. By learning the mutual attention between r preset low-rank latent variables and n examples, it achieves modeling of relevant information between different examples in the same sample image, with a computational complexity of O(n). This avoids directly performing highly complex calculations, reducing computational load while ensuring the learning ability and prediction accuracy of the image classification model.

[0080] For example, mutual attention is calculated between the first feature representation in the first feature representation sequence and the preset r low-rank latent variables through gating attention components 242-a and 242-b, outputting a second feature representation sequence. The second feature representation sequence is pooled through a non-local pooling layer 244 to obtain the global feature representation of the sample image. A linear classifier 246 predicts the global feature representation to obtain the predicted classification result of the sample image; based on the error loss between the predicted classification result and the label classification result of the sample image 200, the model parameters of the multi-instance learning model 240 are trained.

[0081] The self-supervised learning phase and the multi-instance learning phase can be performed separately. The multi-instance learning phase is executed after the self-supervised learning phase is completed.

[0082] Next, the training method of the above image classification model will be introduced in detail through different embodiments.

[0083] Figure 3 A block diagram of an image classification model provided in an exemplary embodiment of this application is shown. This image classification model is used for classifying low-rank multi-instance images. Rank can be understood as the richness of information contained in an image. The image classification model includes a feature extraction network 220 and a low-rank multi-instance learning (MIL) model 240.

[0084] Feature extraction network 220 is used to extract feature representations of each instance in a low-rank multiple instance image. The input to feature extraction network 220 is the low-rank multiple instance image, and the output of feature extraction network 220 is a sequence of feature representations. This sequence of feature representations includes multiple feature representations. Each feature representation sequence corresponds to one low-rank multiple instance image, and each feature representation corresponds to one instance.

[0085] The low-rank MIL model 240 includes a mutual attention network 242, a pooling layer 244, and a classifier 246. The output of the feature extraction network 220 is connected to the input of the mutual attention network 242, the output of the mutual attention network is connected to the input of the pooling layer, and the output of the pooling layer is connected to the input of the classifier.

[0086] Figure 4 A flowchart illustrating a training method for an image classification model provided in an exemplary embodiment of this application is shown. This method can be executed by a computer device running [a specific program / system]. Figure 3 The image classification model shown. This method includes:

[0087] Step 320: Obtain a sample image set, which includes at least two examples of sample images.

[0088] The sample image set is the set of images used by the image classification model for its learning task. The sample image set includes a large number of low-rank sample images, which will be referred to as sample images below.

[0089] In multi-instance learning, a sample image is considered a bag, and a sample image contains multiple / groups of classifiable objects, with each / group of classifiable objects considered an instance. The same sample image contains at least two instances. Each sample image in the sample image set has an image-level label.

[0090] For a sample image, if there is at least one positive sample in the sample image, regardless of whether the sample image has other negative sample examples, then the image-level label of the sample image is positive; if all the examples in the sample image are negative sample examples, then the image-level label of the sample image is negative.

[0091] Optionally, the sample images in the sample image set are segmented according to the cutting resolution to obtain at least two image slices; at least two image slices belonging to the same sample image are identified as at least two examples of the same sample image.

[0092] Step 340: Through self-supervised learning based on contrastive learning, the feature extraction network is trained using sample images from the sample image set to obtain the trained feature extraction network.

[0093] The training phase in this embodiment is divided into two stages: a self-supervised learning stage and a multi-instance learning stage. First, the feature extraction network is trained using the self-supervised learning stage to enable it to have a certain feature extraction capability. Then, the low-rank MIL model is trained to improve its classification ability.

[0094] When training the feature extraction network, self-supervised learning based on contrastive learning is used. Illustratively, for an anchor point example (any one of the multiple examples) in multiple examples of the same image, positive and negative sample examples of that anchor point example are determined within these multiple examples. The feature extraction network is trained based on the error between the feature representations extracted by the network for the positive and negative sample examples respectively.

[0095] The input to the feature extraction network is multiple examples of the sample image, and the output of the feature extraction network is a first feature representation sequence, which includes multiple first feature representations, with a one-to-one correspondence between the multiple examples and the multiple first feature representations.

[0096] Step 360: Through multi-instance learning based on mutual attention mechanism, the multi-instance learning model is trained using sample images from the sample image set to obtain the trained multi-instance learning model.

[0097] The mutual attention mechanism is a mechanism for computing the attention between the feature representations of at least two examples and low-rank latent variables. The number of low-rank latent variables is less than the number of at least two examples, and the low-rank latent variables are set based on the rank of the sample images.

[0098] Since the rank of the sample images is relatively low, the rank of the sample images can be determined in advance by analyzing the rank of the sample images. r low-rank latent variables are set according to the rank of the sample images. A mutual attention mechanism between at least two examples from the same sample image and the r low-rank latent variables is used to replace the self-attention mechanism in related techniques, avoiding direct calculation of self-attention. This reduces computational cost while maintaining the model's learning ability and prediction accuracy.

[0099] For example, if the number of instances in the same sample image is n, then the computational complexity of the self-attention mechanism is O(n). If the number of low-rank latent variables in this application is r, then the computational complexity of the mutual attention mechanism is... When the sample images are low-rank sample images, the computational complexity can be significantly reduced.

[0100] Based on Figure 3 In an alternative embodiment, such as Figure 5 As shown, step 360 may include steps 362 to 366.

[0101] Step 362: Perform mutual attention calculation between the first feature representation and the low-rank latent variable in the first feature representation sequence through a mutual attention network, and output the second feature representation sequence; the first feature representation sequence is extracted by the feature extraction network from at least two examples.

[0102] For a given sample image, let the first feature representation sequence include the first feature representations of n examples in the sample image. The r-dimensional low-rank latent variables are pre-set based on the rank of the sample image, where r is the rank of the sample image, which can be an estimate.

[0103] Assume the input sample image is After segmenting the sample image X, a set of image slices is obtained. Each image slice A single instance is called an example, and n is the number of instances, a positive integer greater than 1. The feature extraction function F... f After feature extraction for each example, a sequence of feature representations for the corresponding space is obtained. :

[0104]

[0105] in, , .

[0106] Define mutual attention:

[0107]

[0108] in These are three learnable matrix transformations; It is the first feature representing the sequence; is a low-rank latent variable, a learnable parameter; CAtt is the mutual attention calculation function; softmax is the normalized exponential function; Q is a matrix consisting of a set of queries, and K is a matrix consisting of a set of keys. T It is the transpose of K, and V is a matrix consisting of a set of values. Since the dimension of the low-rank latent variable L is... , These are preset hyperparameters. If the first feature representation has a feature dimension, such as 1024 dimensions, then the first feature representation sequence... The dimension is ,in Therefore, the computational complexity of mutual attention is... It is much smaller than the complexity of self-attention. .

[0109] In some embodiments, the mutual attention network described above is a gated attention block (GAB) constructed by combining a gated attention mechanism and a mutual attention mechanism.

[0110] GAB is constructed by combining gating attention mechanisms and mutual attention mechanisms, specifically as follows:

[0111]

[0112] Among them, GSB is a gated output function based on the mutual attention mechanism; Indicates dot product; L It is a low-rank latent variable; It is the first feature representing the sequence; This refers to the Sigmoid Linear Units activation function, which introduces a non-linear factor to help the data be better classified. represents the linear transformation matrix; CAtt is the mutual attention calculation function.

[0113] In some embodiments, the mutual attention network described above includes a first attention component and a second attention component. For example... Figure 6 As shown, step 362 above may include:

[0114] Step 362-a: Using the r-dimensional low-rank latent variable as a query, the first attention component performs the first mutual attention calculation with the input n feature representations to obtain r low-dimensional vectors H, where r is a positive integer less than n.

[0115] Step 362-b involves using n feature representations as queries through the second attention component, and performing a second mutual attention calculation with r low-dimensional vectors H to obtain n second feature representations in the second feature representation sequence. The initial values ​​of these n feature representations are the first feature representations in the first feature representation sequence.

[0116] Taking each attention component as a GAB as an example, Figure 7 The diagram shows a mutual attention network consisting of two cascaded GABs. Figure 8 An exemplary structure for a GAB is shown. It is represented as:

[0117]

[0118] In the formula, ILRA first uses low-rank latent variables. As a query, it focuses on the high-dimensional first feature representation sequence. Perform mutual attention calculation to generate a low-dimensional vector Then the first feature represents the sequence. Then, combine with the newly generated r low-dimensional vectors Perform mutual attention calculation and output the intermediate feature representation sequence. .

[0119] After t iterations, the output yields the second feature representation sequence. :

[0120]

[0121] Where t is determined by the actual design and is generally 6-8 by default; Z0 is the first and second feature representation sequence; ILRA is the output function.

[0122] Step 364: Pool the second feature representation sequence through a pooling layer to obtain the global feature representation of the sample image.

[0123] Step 366: Predict the global feature representation using a classifier to obtain the predicted classification result of the sample image; train the model parameters of the multi-instance learning model based on the error loss between the predicted classification result of the sample image and the label classification result.

[0124] The sequence is represented by the second feature. Image classification is performed using a multiple instance learning model, as shown below:

[0125]

[0126] in It outputs the classification results; It is a classifier; It is a pooling function used to remove redundant information and obtain a low-resolution feature map; It is the scoring function for each example. Let n be the transpose of the matrix, and n be the number of examples, a positive integer greater than 1. An attention score function is typically used, expressed as:

[0127]

[0128] in All of them are learnable parameter matrices; represents the point product; tanh is the hyperbolic tangent function; sigm is the activation function; n is the number of examples, a positive integer greater than 1.

[0129] In some embodiments, the classifier described above can be implemented using a multi-layer perceptron (MLP). The classification prediction process described above can be represented as follows:

[0130]

[0131] The second feature represents the sequence. MLP is a linear classifier. It is a global feature representation with dimension d, and it consists of learnable parameters; This is the prediction result output by the classifier, where n is the number of examples, which is a positive integer greater than 1.

[0132] In summary, this embodiment combines gating attention and mutual attention mechanisms to filter features in the feature representation sequence, thereby improving the classification accuracy of the classifier.

[0133] This embodiment uses two GABs. The first GAB can reduce the n dimensions of the feature representation sequence to r dimensions of the low-rank latent variables, and the second GAB restores the r dimensions back to n dimensions, thus avoiding the situation where the dimensions of the feature representation sequence become smaller and smaller.

[0134] This embodiment avoids the problem of insufficient training in a single round by iterating through t rounds, thereby ensuring that the extracted global feature representation has good feature representation capabilities.

[0135] In some embodiments, the self-supervised learning described above can be implemented using contrastive learning based on multiple positive samples. Figure 9 A flowchart illustrating a training method for an image classification model provided in an exemplary embodiment of this application is shown. This method can be executed by a computer device. Step 340 described above can be implemented as follows:

[0136] Step 342: Extract features from at least two examples using a feature extraction network to obtain a first feature representation sequence; the first feature representation in the first feature representation sequence corresponds one-to-one with at least two examples;

[0137] For any sample image, features are extracted from at least two examples of the sample image using a feature extraction network to obtain a first feature representation sequence of the sample image; the first feature representation in the first feature representation sequence corresponds one-to-one with at least two examples of the sample image.

[0138] Step 344: Cluster the first feature representations in the first feature representation sequence, and take the first feature representations that belong to the same cluster result as the anchor first feature representation as the positive sample feature representation of the anchor first feature representation; take the first feature representations that belong to different cluster results as the negative sample feature representation of the anchor first feature representation.

[0139] The anchor point's first feature representation is any one of at least two examples from the sample image. The first feature representations in the first feature representation sequence are clustered, with no restriction on the clustering method. Multiple first feature representations belonging to the same cluster as the anchor point's first feature representation are considered positive sample feature representations of the anchor point's first feature representation; multiple first feature representations belonging to different clusters than the anchor point's first feature representation are considered negative sample feature representations of the anchor point's first feature representation.

[0140] Step 346: Train the feature extraction network based on the self-supervised learning loss between positive and negative sample feature representations.

[0141] Contrastive learning is a common self-supervised learning paradigm. Given a set of n images... ,in Let n be a positive integer. Each image undergoes two random data augmentations (rotation, cropping, scaling, color transformation, etc.) to obtain 2n images. Two networks are then used to extract features from each image, resulting in the first feature representation sequence, denoted as... The optimization objective of contrastive learning is to minimize the loss function:

[0142]

[0143] Where, if i=j, then Otherwise, it is 0. yes and The cosine similarity function. The contrastive learning loss function assumes that... yes There is only one positive sample, and There are n-1 negative samples.

[0144] Contrastive learning loss functions assume that each image corresponds to only one positive sample, i.e., the same image enhanced by different images. However, in reality, taking whole-slide images (WSI) as an example, there are a large number of similar samples. Besides the images themselves serving as positive samples, we should mine these similar images to increase the number of positive samples, making the image feature learning process more reasonable. Given a sequence of first-feature representations for an image, each image feature can be represented by a feature dictionary. The basis vectors are linearly represented. For example, as shown below... Figure 10 As shown, the input sequence of first feature representations containing 10 sample examples can be decomposed into a linear combination of 6 basis vectors based on their category information. The corresponding coefficient matrix will exhibit the following behavior: Figure 10 The low-rank characteristics shown.

[0145] This application aims to find the minimum rank of these eigenvectors:

[0146]

[0147] in, The low-rank decomposition is represented as:

[0148]

[0149] in, It is a dictionary; It is a diagonal coefficient matrix; This is the error matrix. Since the original data always contains errors and noise, a perfect low-rank matrix does not exist. This involves estimating and eliminating errors in the original data. (Coefficient matrix) The rank of is r. If the image's category labels can be obtained, r can generally be considered equal to the number of image categories. Since the formula for the low-rank decomposition above cannot be directly calculated and solved during contrastive learning training, this application adopts... Figure 11 The method shown finds multiple positive samples for each example.

[0150] Step 344 above may include the following sub-steps, such as Figure 11 As shown:

[0151] Step 344-a: Sort the first feature representations in the first feature representation sequence in descending order of similarity to the anchor first feature representation; the anchor first feature representation is one of the first feature representation sequences.

[0152] Step 344-b: Decompose the sorted first feature representation sequence matrix into r subspaces, where r is a preset rank.

[0153] Step 344-c: Determine the first feature representation in the first subspace of the r subspaces as the positive sample feature representation of the anchor first feature representation; determine the first feature representation in the last subspace of the r subspaces as the negative sample feature representation of the anchor first feature representation.

[0154] Assuming a given anchor point, the first feature representation The anchor point's first feature representation is the feature representation of any one instance, and the remaining first feature representations are the feature representations of the other instances. Based on the remaining first feature representations and... The similarity is sorted to obtain the set:

[0155]

[0156] That is, the remaining first feature representations and The higher the similarity, the higher the ranking. Based on the low-rank decomposition results of the matrix, the above set is decomposed into... , … There are r subspaces in total, respectively with , … Correspondence. Due to sets Similarity ranking has already been performed, so and The positive and negative samples that are furthest apart should satisfy the following:

[0157]

[0158] in, This is the similarity function interval, where p represents positive samples and n represents negative samples. The functional interval is added to further separate positive and negative samples.

[0159] Adding the above formula to the loss function minimizes the result, resulting in the low-rank constrained loss function:

[0160]

[0161] This formula considers similar examples within the same sample image, increasing the number of positive samples for the same example. When all positive and negative samples are correctly separated and a certain functional margin is achieved, the loss function can be minimized. Combining the low-rank constraint function with the traditional contrastive loss function yields the final self-supervised learning loss function:

[0162]

[0163] in These are the weighting coefficients. It's the original contrastive loss function, which can learn relatively good features; further adding... This involves adding more positive and negative samples to further improve performance.

[0164] For example, taking WSI images as an example, based on the special property of low-rank data: as the dimensionality of data increases, there is often more correlation and redundancy between high-dimensional data. Since the information content of the data itself grows much slower than the dimensionality of the data, the higher the dimensionality of the data, the more redundant the data becomes. WSI images belong to ultra-high-dimensional image data, and the size can typically reach 10,000. The number of images exceeds 10,000, but the presence of numerous similar tissues and cells in the WSI images aligns with the prior assumption of low-rank data. This study analyzes 270 WSI images from breast cancer and performs low-rank matrix decomposition to obtain the rank of the corresponding matrix, then plots histograms. Figure 12 As shown in (1), the feature dimension of ImageNet pre-training is 1024, but after matrix factorization, the rank average of the 270 WSI images is 349, which is lower than the full rank case of 1024. Figure 12 (2) is the self-supervised learning model proposed in this application. The average rank of WSI images is 181, which is much smaller than the rank of ImageNet features. This result shows that low rank is a typical phenomenon of WSI images. Furthermore, subsequent experimental analysis shows that self-supervised learning has even lower rank and better image classification performance.

[0165] Figure 13 A flowchart illustrating a method of using an image classification model provided in an exemplary embodiment of this application is shown. This embodiment illustrates the method by way of execution by a computer device. The image classification model is trained according to the method described above, which includes:

[0166] Step 402: Obtain the input image to be classified, which includes at least two examples.

[0167] The input image is segmented according to the cutting resolution to obtain at least two image slices; the at least two image slices belonging to the input image are identified as at least two examples of the input image.

[0168] Reference Figure 14 Assume the above image classification model is used for cancer classification prediction in pathological images. The WSI image X is divided according to a cropping resolution (e.g., 220 pixels). The image is divided into multiple image slices (220 pixels each). n is a positive integer, and each image slice is an example.

[0169] Step 404: Input the input image into the feature extraction network of the image classification model to extract features and obtain a third feature representation sequence, which includes feature representations of at least two examples.

[0170] After feature extraction from each example, a third feature representation sequence of the input image is obtained. :

[0171]

[0172] in, , .

[0173] Step 406: Input the third feature representation sequence into the multi-instance learning model in the image classification model and perform classification prediction based on the mutual attention mechanism to obtain the classification result of the input image.

[0174] The mutual attention mechanism is a mechanism for computing the attention between the feature representations of at least two examples and low-rank latent variables. The number of low-rank latent variables is less than the number of at least two examples, and the low-rank latent variables are set based on the rank of the sample images.

[0175] In some embodiments, mutual attention is calculated between the third feature representation and the low-rank latent variable in the third feature representation sequence through a mutual attention network to output a fourth feature representation sequence; the third feature representation sequence is extracted by the feature extraction network from at least two examples.

[0176] The fourth feature representation sequence is pooled by a pooling layer to obtain the global feature representation of the sample image;

[0177] The global feature representation is predicted by a classifier to obtain the predicted classification result of the sample image; the model parameters of the multi-instance learning model are trained based on the error loss between the predicted classification result of the sample image and the label classification result.

[0178] In some embodiments, the mutual attention network includes a first attention component and a second attention component;

[0179] The first attention component uses an r-dimensional low-rank latent variable as a query and performs the first mutual attention calculation with the input n feature representations to obtain r low-dimensional vectors H, where r is a positive integer less than n;

[0180] Using n feature representations as queries through the second attention component, and performing second mutual attention calculations with r low-dimensional vectors H, n fourth feature representations are obtained in the fourth feature representation sequence;

[0181] The initial value of each of the n feature representations is the third feature representation in the third feature representation sequence.

[0182] In some embodiments, n feature representations are used as queries by a second attention component, and a second mutual attention calculation is performed with r low-dimensional vectors H to obtain n intermediate feature representations.

[0183] If the iteration termination condition is not met, the n intermediate feature representations are used as the input n feature representations, and the first mutual attention calculation is performed again using the r-dimensional low-rank latent variables as the query and the input n feature representations through the first attention component to obtain r low-dimensional vectors H.

[0184] When the iteration termination condition is met, the n intermediate feature representations are output as the fourth feature representation sequence.

[0185] In some embodiments, the iteration termination condition is: the number of iterations reaches a preset number of iterations t.

[0186] Figure 14 This illustration depicts a scenario of a training method for an image classification model provided in an exemplary embodiment of this application. In this application scenario, it is assumed that cancer analysis is performed on digital pathological slides. This application analyzes digital pathological slides using a multi-instance learning model to determine whether cancerous tissue exists in the pathological images, serving as an important basis for subsequent treatment. This embodiment assists doctors in diagnosing digital pathological slides obtained from pathological image segmentation, providing important reference information on whether a patient has cancer. It helps reduce the workload of pathologists in interpreting slides, improves the consistency of doctor interpretations, effectively improves classification accuracy, enhances the intelligence and automation level of digital pathology diagnosis, and reduces the risk of misdiagnosis and missed diagnosis.

[0187] Figure 15 This illustration depicts a scenario of a training method for an image classification model provided in an exemplary embodiment of this application. In this application scenario, it is assumed that ecological environment analysis is performed using image examples. This application analyzes image examples using a multi-example learning model to determine whether target animals such as deer, wild boars, and rabbits exist in the input image, serving as an important basis for ecological environment analysis. This embodiment assists biologists, forest rangers, and other personnel in statistically analyzing image examples obtained from segmented input images, providing crucial reference information on the local population of target animals. This helps reduce the difficulty of statistical data collection for personnel, effectively improves classification accuracy, enhances the intelligence and automation level of ecological environment analysis, and reduces the consumption of manpower and resources for statistical data collection.

[0188] Figure 16 This invention illustrates a structural block diagram of a training apparatus 500 for an image classification model provided in an exemplary embodiment of this application. The image classification model includes a feature extraction network and a multiple instance learning model. The apparatus includes:

[0189] The acquisition module 510 is used to acquire a sample image set, wherein the sample images in the sample image set include at least two examples;

[0190] The self-supervised learning module 520 is used to train the feature extraction network using sample images from the sample image set through self-supervised learning based on contrastive learning, so as to obtain the trained feature extraction network.

[0191] The multi-instance learning module 530 is used to train the multi-instance learning model using sample images from the sample image set through multi-instance learning based on the mutual attention mechanism, so as to obtain the trained multi-instance learning model.

[0192] In an optional design of this embodiment, the acquisition module 510 includes:

[0193] The segmentation submodule is used to segment the sample images in the sample image set according to the cutting resolution to obtain at least two image slices;

[0194] The determination submodule is used to identify at least two image slices belonging to the same sample image as at least two instances of the same sample image.

[0195] In an optional design of this embodiment, the self-supervised learning module 520 includes:

[0196] The feature extraction submodule is used to extract features from at least two examples through a feature extraction network to obtain a first feature representation sequence; the first feature representation in the first feature representation sequence corresponds one-to-one with at least two examples.

[0197] The clustering submodule is used to cluster the first feature representations in the first feature representation sequence, and to take the first feature representation that belongs to the same cluster result as the anchor first feature representation as the positive sample feature representation of the anchor first feature representation; and to take the first feature representation that belongs to a different cluster result than the anchor first feature representation as the negative sample feature representation of the anchor first feature representation.

[0198] The training submodule is used to train the feature extraction network based on the self-supervised learning loss between positive and negative sample feature representations.

[0199] The clustering submodule includes:

[0200] The sorting unit is used to sort the first feature representations in the first feature representation sequence according to their similarity to the anchor first feature representation from high to low; the anchor first feature representation is one of the first feature representations in the sequence.

[0201] The decomposition unit is used to decompose the sorted first feature representation sequence matrix into r subspaces, where r is a preset rank.

[0202] The determining unit is used to determine the first feature representation in the first subspace of the decomposition into r subspaces as the positive sample feature representation of the anchor first feature representation; and to determine the first feature representation in the last subspace of the decomposition into r subspaces as the negative sample feature representation of the anchor first feature representation.

[0203] In an optional design of this embodiment, the multi-instance learning module 530 includes:

[0204] The computation submodule is used to perform mutual attention computation between the first feature representation and the low-rank latent variable in the first feature representation sequence through a mutual attention network, and outputs the second feature representation sequence; the first feature representation sequence is extracted by the feature extraction network from at least two examples.

[0205] The pooling submodule is used to pool the second feature representation sequence through the pooling layer to obtain the global feature representation of the sample image;

[0206] The prediction submodule is used to predict the global feature representation through a classifier to obtain the predicted classification result of the sample image; based on the error loss between the predicted classification result of the sample image and the label classification result, the model parameters of the multi-instance learning model are trained.

[0207] The calculation submodule includes:

[0208] The first computational unit is used to perform a first mutual attention computation with the input n feature representations by using an r-dimensional low-rank latent variable as a query through a first attention component, to obtain r low-dimensional vectors H, where r is a positive integer less than n;

[0209] The second computational unit is used to perform second mutual attention computation with r low-dimensional vectors H using n feature representations as queries through the second attention component, to obtain n second feature representations in the second feature representation sequence.

[0210] The second calculation unit includes:

[0211] The first computational subunit is used to perform a second mutual attention computation with r low-dimensional vectors H using n feature representations as queries through the second attention component to obtain n intermediate feature representations;

[0212] The second computational subunit is used to, when the iteration termination condition is not met, take the n intermediate feature representations as the input n feature representations and start the first mutual attention calculation again using the r-dimensional low-rank latent variables as the query and the input n feature representations to obtain r low-dimensional vectors H.

[0213] The output sub-unit is used to output the n intermediate feature representations as a second feature representation sequence when the iteration termination condition is met.

[0214] Figure 17 A block diagram of an apparatus 600 for using an image classification model provided in an exemplary embodiment of this application is shown. The apparatus includes:

[0215] The acquisition module 610 is used to acquire the input image to be classified, which includes at least two examples;

[0216] Feature extraction module 620 is used to extract features from the feature extraction network in the input image classification model to obtain a third feature representation sequence, which includes feature representations of at least two examples.

[0217] The prediction module 630 is used to perform classification prediction based on mutual attention mechanism on the multi-instance learning model in the input image classification model by inputting the third feature representation sequence into the input image classification model, so as to obtain the classification result of the input image.

[0218] In some embodiments, the prediction module 630 is configured to perform mutual attention calculation between the third feature representation in the third feature representation sequence and the low-rank latent variable through a mutual attention network, and output a fourth feature representation sequence; the third feature representation sequence is extracted by a feature extraction network from at least two examples; the fourth feature representation sequence is pooled through a pooling layer to obtain a global feature representation of the sample image; the global feature representation is predicted by a classifier to obtain a predicted classification result of the sample image; and the model parameters of the multi-example learning model are trained based on the error loss between the predicted classification result and the label classification result of the sample image.

[0219] In some embodiments, the mutual attention network includes a first attention component and a second attention component;

[0220] The prediction module 630 is used to perform a first mutual attention calculation with the input n feature representations by using an r-dimensional low-rank latent variable as a query through the first attention component, to obtain r low-dimensional vectors H, where r is a positive integer less than n;

[0221] Prediction module 630 is used to use n feature representations as queries through the second attention component, and perform second mutual attention calculation with r low-dimensional vectors H to obtain n fourth feature representations in the fourth feature representation sequence;

[0222] The initial value of each of the n feature representations is the third feature representation in the third feature representation sequence.

[0223] In some embodiments, the prediction module 630 is configured to use n feature representations as queries through a second attention component and perform a second mutual attention calculation with r low-dimensional vectors H to obtain n intermediate feature representations; if the iteration termination condition is not met, the n intermediate feature representations are used as input n feature representations, and the process starts again from the step of using r-dimensional low-rank latent variables as queries and input n feature representations through a first attention component to perform a first mutual attention calculation with r low-dimensional vectors H; if the iteration termination condition is met, the n intermediate feature representations are output as a fourth feature representation sequence.

[0224] In some embodiments, the iteration termination condition is: the number of iterations reaches a preset number of iterations t.

[0225] Figure 18 shows a structural block diagram of a computer device provided in an exemplary embodiment of this application. Typically, the computer device 700 includes a processor 701 and a memory 702.

[0226] Processor 701 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 701 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Processor 701 may also include a main processor and a coprocessor. The main processor, also known as a central processing unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 701 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 701 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0227] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 702 are used to store at least one instruction, which is executed by the processor 701 to implement the method and / or usage method of the image classification model provided in the method embodiments of this application.

[0228] In some embodiments, server 700 may also optionally include an input interface 703 and an output interface 704. Processor 701, memory 702, and input interfaces 703 and 704 can be connected via a bus or signal lines. Various peripheral devices can be connected to input interfaces 703 and 704 via a bus, signal lines, or a circuit board. Input interfaces 703 and 704 can be used to connect at least one input / output (I / O) related peripheral device to processor 701 and memory 702. In some embodiments, processor 701, memory 702, and input interfaces 703 and 704 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 701, memory 702, and input interfaces 703 and 704 can be implemented on separate chips or circuit boards, and this application does not limit this aspect.

[0229] Those skilled in the art will understand that the structure shown above does not constitute a limitation on server 700. Server 700 may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0230] In an exemplary embodiment, a chip is also provided, the chip including programmable logic circuitry and / or program instructions, which, when the chip is run on a computer device, are used to implement the training method and / or usage method of the image classification model described above.

[0231] In an exemplary embodiment, a computer program product is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions to implement the training method and / or usage method of the image classification model provided in the above-described method embodiments.

[0232] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores a computer program that is loaded and executed by a processor to implement the training method and / or usage method of the image classification model provided in the above-described method embodiments.

[0233] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0234] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0235] The above are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A training method for an image classification model, characterized in that, The image classification model includes a feature extraction network and a multiple instance learning model, and the method includes: Obtain a sample image set, wherein the sample images in the sample image set include at least two examples; The feature extraction network is trained by using sample images from the sample image set through self-supervised learning based on contrastive learning, resulting in a trained feature extraction network. By using multi-instance learning based on mutual attention mechanism, the multi-instance learning model is trained using sample images from the sample image set to obtain the trained multi-instance learning model. The mutual attention mechanism is a mechanism for calculating the attention between the feature representations of the at least two examples and the low-rank latent variables. The low-rank latent variables are used as a query to perform mutual attention calculation with the feature representations of the at least two examples. The number of low-rank latent variables is less than the number of the at least two examples. The low-rank latent variables are set based on the rank of the sample images.

2. The method according to claim 1, characterized in that, The multi-instance learning model includes: a mutual attention network, a pooling layer, and a classifier; The multi-instance learning method based on mutual attention mechanism, which trains the multi-instance learning model using sample images from the sample image set to obtain the trained multi-instance learning model, includes: The mutual attention network is used to perform mutual attention calculation between the first feature representation in the first feature representation sequence and the low-rank latent variable, and outputs a second feature representation sequence; the first feature representation sequence is extracted by the feature extraction network from the at least two examples. The second feature representation sequence is pooled by the pooling layer to obtain the global feature representation of the sample image; The global feature representation is predicted by the classifier to obtain the predicted classification result of the sample image; the model parameters of the multi-instance learning model are trained based on the error loss between the predicted classification result and the label classification result of the sample image.

3. The method according to claim 2, characterized in that, The mutual attention network includes a first attention component and a second attention component; The step of performing mutual attention calculation between the feature representations in the first feature representation sequence and the low-rank latent variables through the mutual attention network, and outputting the second feature representation sequence, includes: The first attention component uses the r-dimensional low-rank latent variable as a query and performs a first mutual attention calculation with the input n feature representations to obtain r low-dimensional vectors H, where r is a positive integer less than n. Using the n feature representations as queries through the second attention component, a second mutual attention calculation is performed with the r low-dimensional vectors H to obtain n second feature representations in the second feature representation sequence; The initial value of the n feature representations is the first feature representation in the first feature representation sequence.

4. The method according to claim 3, characterized in that, The step involves using the n feature representations as queries through the second attention component, performing a second mutual attention calculation with the r low-dimensional vectors H to obtain n second feature representations in the second feature representation sequence, including: The second attention component uses the n feature representations as queries and performs a second mutual attention calculation with the r low-dimensional vectors H to obtain n intermediate feature representations. If the iteration termination condition is not met, the n intermediate feature representations are used as the input n feature representations, and the first mutual attention calculation is performed again from the step of using the r-dimensional low-rank latent variable as the query and the input n feature representations through the first attention component to obtain r low-dimensional vectors H. When the iteration termination condition is met, the n intermediate feature representations are output as a second feature representation sequence.

5. The method according to claim 3, characterized in that, The first attention component and the second attention component are constructed based on a combination of gating attention mechanism and mutual attention mechanism.

6. The method according to any one of claims 1 to 5, characterized in that, The step involves training the feature extraction network using sample images from the sample image set through self-supervised learning based on contrastive learning, resulting in a trained feature extraction network, including: The feature extraction network is used to extract features from the at least two examples to obtain a first feature representation sequence; the first feature representation in the first feature representation sequence corresponds one-to-one with the at least two examples. Cluster the first feature representations in the first feature representation sequence, and take the first feature representation that belongs to the same cluster result as the anchor first feature representation as the positive sample feature representation of the anchor first feature representation; take the first feature representation that belongs to a different cluster result than the anchor first feature representation as the negative sample feature representation of the anchor first feature representation; The feature extraction network is trained based on the self-supervised learning loss between the positive sample feature representation and the negative sample feature representation.

7. The method according to claim 6, characterized in that, The first feature representation in the first feature representation sequence is clustered, and the first feature representation that belongs to the same cluster result as the anchor first feature representation is taken as the positive sample feature representation of the anchor first feature representation; Using the first feature representation that belongs to a different clustering result from the first feature representation of the anchor point as the negative sample feature representation of the first feature representation of the anchor point, including: The first feature representations in the first feature representation sequence are sorted in descending order of similarity to the first feature representation of the anchor point; the first feature representation of the anchor point is one of the first feature representations in the sequence. The sorted first feature representation sequence matrix is ​​decomposed into r subspaces, where r is a preset rank. The first feature representation in the first subspace of the r subspaces is determined as the positive sample feature representation of the first feature representation of the anchor point; The first feature representation in the last subspace of the r subspaces is determined as the negative sample feature representation of the first feature representation of the anchor point.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: The sample images in the sample image set are divided according to the cutting resolution to obtain at least two image slices; The at least two image slices belonging to the same sample image are identified as the at least two examples of the same sample image.

9. A method for using an image classification model, characterized in that, The image classification model is trained according to the method described in any one of claims 1 to 8, the method comprising: Obtain the input image to be classified, the input image including at least two examples; The input image is input into the feature extraction network in the image classification model to extract features, resulting in a third feature representation sequence, which includes the feature representations of the at least two examples. The third feature representation sequence is input into the multi-instance learning model in the image classification model to perform classification prediction based on the mutual attention mechanism, thereby obtaining the classification result of the input image; The mutual attention mechanism is a mechanism for calculating the attention between the feature representations of the at least two examples and the low-rank latent variables. The low-rank latent variables are used as a query to perform mutual attention calculation with the feature representations of the at least two examples. The number of low-rank latent variables is less than the number of the at least two examples. The low-rank latent variables are set based on the rank of the sample image.

10. A training device for an image classification model, characterized in that, The image classification model includes a feature extraction network and a multiple instance learning model, and the device includes: An acquisition module is used to acquire a sample image set, wherein the sample images in the sample image set include at least two examples; The self-supervised learning module is used to train the feature extraction network using sample images from the sample image set through self-supervised learning based on contrastive learning, so as to obtain the trained feature extraction network. The multi-instance learning module is used to train the multi-instance learning model using sample images from the sample image set through multi-instance learning based on the mutual attention mechanism, so as to obtain the trained multi-instance learning model. The mutual attention mechanism is a mechanism for calculating the attention between the feature representations of the at least two examples and the low-rank latent variables. The low-rank latent variables are used as a query to perform mutual attention calculation with the feature representations of the at least two examples. The number of low-rank latent variables is less than the number of the at least two examples. The low-rank latent variables are set based on the rank of the sample images.

11. An apparatus for using an image classification model, characterized in that, The image classification model is trained using the apparatus according to claim 10, the apparatus comprising: An acquisition module is used to acquire an input image to be classified, wherein the input image includes at least two examples; The feature extraction module is used to input the input image into the feature extraction network of the image classification model to extract features and obtain a third feature representation sequence, wherein the third feature representation sequence includes the feature representations of the at least two examples; The prediction module is used to input the third feature representation sequence into the multi-instance learning model in the image classification model to perform classification prediction based on the mutual attention mechanism, so as to obtain the classification result of the input image; The mutual attention mechanism is a mechanism for calculating the attention between the feature representations of the at least two examples and the low-rank latent variables. The low-rank latent variables are used as a query to perform mutual attention calculation with the feature representations of the at least two examples. The number of low-rank latent variables is less than the number of the at least two examples. The low-rank latent variables are set based on the rank of the sample image.

12. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the training method of the image classification model as described in any one of claims 1 to 8, and / or the method of using the image classification model as described in claim 9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is loaded and executed by a processor to implement the training method of the image classification model as described in any one of claims 1 to 8, and / or the method of using the image classification model as described in claim 9.

14. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium, from which a processor retrieves the computer instructions, causing the processor to load and execute them to implement the training method of the image classification model as described in any one of claims 1 to 8, and / or the method of using the image classification model as described in claim 9.

Images