Cooperative training method of image classification model, image classification method and related device

Abstract

Embodiments of the present application disclose a method for cooperative training of an image classification model, an image classification method and related devices. The training method comprises: dividing an image sample into a plurality of image blocks; extracting image global features based on a first image classification model for the plurality of image blocks, and predicting an image-level class label based on the image global features to obtain image block classification prediction information generated by the first image classification model in the image classification process; performing classification prediction on each image block based on a second image classification model, and determining the image-level class label based on the block-level class label obtained by the classification prediction to obtain image block classification prediction information generated by the second image classification model in the image classification process; and calculating a cooperative training loss value according to the image block classification prediction information generated by the first image classification model and the second image classification model respectively, and updating the parameters of each model based on the cooperative training loss value. Embodiments of the present application can improve the accuracy of image classification.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a collaborative training method for an image classification model, an image classification method, and related apparatus. Background Technology

[0002] Whole-slide imaging (WSI), also known as digital pathology slides, refers to a technique that uses a data scanner to scan traditional pathology slides to acquire high-resolution digital images, and then seamlessly stitches together the fragmented images using a computer to create a visualized digital image. Whole-slide pathology images typically have very high resolution and are accompanied by many noisy regions unrelated to pathology, making it difficult to directly apply deep learning models to the classification and prediction of whole-slide pathology images. Therefore, there is an urgent need for a technical solution that can accurately classify and predict whole-slide pathology images. Summary of the Invention

[0003] To address the aforementioned technical problems, embodiments of this application provide a collaborative training method and apparatus for image classification models, an image classification method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product. The first image classification model and / or the second image classification model trained using embodiments of this application can accurately predict the category labels of whole-section pathological images.

[0004] One aspect of this application provides a collaborative training method for an image classification model. The method includes: dividing an image sample into multiple image blocks; extracting global image features from the multiple image blocks based on a first image classification model, and predicting image-level category labels based on the global image features to obtain image block classification prediction information generated by the first image classification model during image classification; performing classification prediction on each image block based on a second image classification model, and determining image-level category labels based on the block-level category labels obtained from the classification prediction to obtain image block classification prediction information generated by the second image classification model during image classification; calculating a collaborative training loss value based on the image block classification prediction information generated by the first image classification model and the second image classification model respectively, and updating the parameters of the first image classification model and the second image classification model based on the collaborative training loss value.

[0005] In another exemplary embodiment, the step of classifying and predicting each image block based on the second image classification model, and determining the image-level category label based on the block-level category label obtained from the classification prediction, to obtain the image block classification prediction information generated by the second image classification model during the image classification process, includes: generating block-level pseudo-labels corresponding to each image block according to the block-level category label corresponding to each image block and the prototype vector of each category respectively; wherein, the prototype vector corresponding to each category is applied to the category to which it is predicted; and obtaining the block-level pseudo-labels corresponding to each image block as the image block classification prediction information generated by the second image classification model during the image classification process.

[0006] In another exemplary embodiment, the step of generating block-level pseudo-labels for each image block based on the block-level category labels corresponding to each image block and the prototype vectors corresponding to each category includes: selecting typical image blocks for each category based on the block-level category labels corresponding to each image block; generating prototype labels for each image block based on the selected image blocks and the prototype vectors corresponding to each category; wherein the prototype labels for each image block are used to characterize the category of the corresponding image block; and determining the block-level pseudo-labels for each image block based on the updated prototype labels for each image block.

[0007] In another exemplary embodiment, determining the block-level pseudo-labels corresponding to each image block based on the updated prototype labels corresponding to each image block includes: updating the block-level pseudo-labels of each image block stored in the confidence library based on the updated prototype labels corresponding to each image block; and obtaining the block-level pseudo-labels corresponding to each image block from the updated confidence library.

[0008] In another exemplary embodiment, the step of selecting typical image blocks for each category based on the block-level category labels corresponding to each image block includes: determining the probability value of each image block being predicted as each category based on the block-level category labels corresponding to each image block; and selecting typical image blocks for each category based on the probability values ​​of each image block being predicted as each category.

[0009] In another exemplary embodiment, generating prototype labels for each image patch based on the selected image patch and the prototype vectors corresponding to each category includes: updating the prototype vectors corresponding to each category based on the image patches selected for each category; and generating prototype labels for each image patch based on the updated prototype vectors for each category.

[0010] In another exemplary embodiment, updating the prototype vector corresponding to each category based on the image patches selected for each category includes: determining the momentum coefficient corresponding to the current training based on the current training iteration number, and the complementarity coefficient corresponding to the momentum coefficient; updating the prototype vector corresponding to each category based on the momentum coefficient, the complementarity coefficient, and the feature vector of the image patches selected for each category.

[0011] In another exemplary embodiment, generating prototype labels corresponding to each category based on the updated prototype vectors of each category includes:

[0012] For each image patch, the product of the feature vector and the updated prototype vector for each category is calculated to obtain the category probability set corresponding to each image patch;

[0013] The category with the highest probability is selected from the set of category probabilities corresponding to each image patch, and the prototype category corresponding to each image patch is used as the prototype category. Based on the prototype category, the prototype label corresponding to each image patch is obtained.

[0014] In another exemplary embodiment, the step of extracting global image features from the plurality of image patches based on a first image classification model and predicting image-level category labels based on the global image features to obtain image patch classification prediction information generated by the first image classification model during the image classification process includes: obtaining the prediction scores of each image patch output by the second image classification model when extracting global image features from the plurality of image patches; wherein the prediction scores characterize the degree of contribution of each image patch to the global image features; and obtaining the prediction scores of each image patch as image patch classification prediction information generated by the first image classification model during the image classification process.

[0015] In another exemplary embodiment, the step of classifying and predicting each image block based on the second image classification model, and determining the image-level category label based on the block-level category label obtained from the classification prediction, to obtain the image block classification prediction information generated by the second image classification model in the image classification process, includes: determining the image block classification prediction information generated by the second image classification model in the image classification process based on the image-level category label determined by the second image classification model.

[0016] In another exemplary embodiment, the step of calculating a co-training loss value based on the image patch classification prediction information generated by the first image classification model and the second image classification model respectively, and updating the parameters of the first image classification model and the second image classification model based on the co-training loss value, includes: using the image patch classification prediction information generated by the first image classification model as the predicted value, and using the image patch classification prediction information generated by the second image classification model as the true value, calculating the deviation between the predicted value and the true value to obtain the co-training loss value.

[0017] In another exemplary embodiment, the method further includes: extracting the embedding vectors of each image patch, and extracting the feature vectors of each image patch based on the embedding vectors of each image patch; inputting the feature vectors of each image patch into the first image classification model and the second image classification model respectively, so that the first image classification model and the second image classification model predict image-level category labels based on the received feature vectors of multiple image patches.

[0018] In another exemplary embodiment, the step of extracting the embedding vectors of each image patch and extracting the feature vectors of each image patch based on the embedding vectors of each image patch includes: extracting the embedding vectors of each image patch through an encoding network, and extracting the feature vectors of each image patch based on the embedding vectors of each image patch through an extraction network; wherein the encoding network is in a parameter frozen state; the method further includes: updating the parameters of the extraction network based on the training loss value.

[0019] In another aspect of this application, a collaborative training apparatus for an image classification model is provided. The apparatus includes: a first acquisition module configured to extract global image features from a plurality of image patches based on a first image classification model, and predict image-level category labels based on the global image features, thereby acquiring image patch classification prediction information generated by the first image classification model during image classification; a second acquisition module configured to perform classification prediction on each image patch based on a second image classification model, and determine image-level category labels based on the patch-level category labels obtained from the classification prediction, thereby acquiring image patch classification prediction information generated by the second image classification model during image classification; and a collaborative module configured to calculate a collaborative training loss value based on the image patch classification prediction information generated by the first image classification model and the second image classification model respectively, and update the parameters of the first image classification model and the second image classification model based on the collaborative training loss value.

[0020] In another aspect of this application, an image classification method is provided, the method comprising: acquiring a whole-section pathological image to be classified; inputting the whole-section pathological image into a first image classification model and / or a second image classification model obtained by co-training through any of the above methods, so as to obtain an image classification result output by the first image classification model and / or the second image classification model.

[0021] In another aspect of this application, an image classification apparatus is provided, comprising: an acquisition module configured to acquire a whole-slice pathological image to be classified; and a processing module configured to input the whole-slice pathological image into a first image classification model and / or a second image classification model co-trained by a co-training method of any of the above-mentioned image classification models, so as to obtain an image classification result output by the first image classification model and / or the second image classification model.

[0022] Another aspect of this application provides a training method for an image classification model. The method includes: dividing an image sample into multiple image blocks; performing classification prediction on each image block based on the image classification model to obtain block-level category labels corresponding to each image block, and determining image-level category labels based on the block-level category labels corresponding to each image block; generating block-level pseudo-labels corresponding to each image block based on the block-level category labels corresponding to each image block and the prototype vectors of each category; wherein the prototype vectors corresponding to each category act on the predicted category; calculating block-level training loss values ​​based on the block-level pseudo-labels and block-level category labels corresponding to each image block, and updating the parameters of the image classification model based on the block-level training loss values.

[0023] In another aspect of this application, a training apparatus for an image classification model is provided. The apparatus includes: an image segmentation module configured to segment an image sample into multiple image blocks; a model processing module configured to perform classification prediction on each image block based on the image classification model, obtaining block-level category labels corresponding to each image block, and determining image-level category labels based on the block-level category labels corresponding to each image block; a pseudo-label generation module configured to generate block-level pseudo-labels corresponding to each image block based on the block-level category labels corresponding to each image block and the prototype vectors of each category; wherein the prototype vectors corresponding to each category act on the predicted category; and a loss calculation module configured to calculate block-level training loss values ​​based on the block-level pseudo-labels and block-level category labels corresponding to each image block, and update the parameters of the image classification model based on the block-level training loss values.

[0024] Another aspect of this application provides an electronic device, including: one or more processors; and a memory for storing one or more computer programs, which, when executed by the one or more processors, cause the electronic device to implement the co-training method or image classification method of the image classification model as described above.

[0025] Another aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor of an electronic device, causes the electronic device to perform the co-training method or image classification method of the image classification model as described above.

[0026] Another aspect of this application provides a computer program product, including a computer program that, when executed by a processor of an electronic device, implements the co-training method or image classification method of the image classification model as described above.

[0027] In the technical solution provided in the embodiments of this application, a collaborative learning approach is adopted for image classification prediction using a first image classification model and a second image classification model. Since the collaborative training loss value is calculated based on the image patch classification prediction information generated by the first and second image classification models respectively during image classification, the collaborative training process utilizes patch-level information to enhance the first image classification model's extraction of global image features. This makes the extracted global image features more accurate, thereby improving the classification accuracy of the first image classification model. It also makes the second image classification model's prediction of patch-level category labels more accurate, thus improving its classification accuracy as well. Therefore, both the collaboratively trained first and second image classification models can perform more accurate classification predictions for high-resolution images such as whole-section pathological images.

[0028] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the implementation environment involved in this application;

[0030] Figure 2 This is a flowchart illustrating a collaborative training method for an image classification model, as shown in an exemplary embodiment of this application.

[0031] Figure 3 This is a flowchart illustrating a collaborative training method for an image classification model, as shown in another exemplary embodiment of this application;

[0032] Figure 4This is a flowchart illustrating a collaborative training method for an image classification model, as shown in yet another exemplary embodiment of this application;

[0033] Figure 5 This is a schematic diagram of an exemplary collaborative training process for an image classification model;

[0034] Figure 6 yes Figure 5 The diagram shows the processing flow of the label disambiguation module in an exemplary embodiment.

[0035] Figure 7 This is a flowchart illustrating an exemplary image classification method;

[0036] Figure 8 This is a flowchart of an exemplary image classification model training method;

[0037] Figure 9 This is a block diagram illustrating a collaborative training apparatus for an image classification model, as shown in an exemplary embodiment.

[0038] Figure 10 This is a block diagram illustrating an image classification apparatus in an exemplary embodiment;

[0039] Figure 11 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0040] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0041] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0042] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0043] In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0044] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. The terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0045] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0046] First see Figure 1 , Figure 1 This is a schematic diagram of the implementation environment involved in this application. Specifically, the implementation environment is an image classification system, including a terminal 110 and a server 120, which communicate with each other via wired or wireless means.

[0047] Terminal 110 is used to acquire digital images with high resolution. For example, digital images with high resolution can be medical images such as whole-slice pathology images (WSI), computed tomography (CT) images, and magnetic resonance imaging (MRI) images, or digital images from other fields such as remote sensing images, satellite images, electron microscope images, and high-precision scanning images. This embodiment does not limit this.

[0048] Terminal 110 can use high-resolution digital images as images to be classified and input them into an image classification model for classification prediction. For example, in some application scenarios, a model to be trained is deployed in server 120. Terminal 110 acquires a large number of high-resolution digital images as a training sample set. Server 120 uses these training sample sets to perform training processing on the model to be trained, thereby obtaining a trained image classification model. In other application scenarios, server 120 deploys a trained image classification model. Terminal 110 uploads the image to be classified to server 120, allowing server 120 to use the trained image classification model to predict the category label of the image and return the predicted category label to terminal 110. Terminal 110 can then perform processing such as display based on the acquired category label.

[0049] It should be noted that terminal 110 can be a smartphone, tablet, laptop, computer, smart home appliance, smart terminal, or other similar device, and there are no restrictions on its use. Server 220 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services, and there are no restrictions on its use either.

[0050] Taking whole-slice pathological images (WSI) as an example, WSI images typically have very high resolution and are accompanied by many noisy regions that are not related to pathology, making it difficult to directly apply deep learning models to the classification and prediction of WSI images.

[0051] In related technologies, whole-section pathological images are typically divided into many smaller image patches. Image classification results corresponding to the whole-section pathological image are obtained by processing these image patches. These technologies treat the analysis of whole-section pathological images as a multi-instance learning problem. In this framework, the entire image is considered a bag, and the extracted image patches are considered instances. Specific learning strategies are applied to these instances to obtain comprehensive feature vectors for the image, which are then classified.

[0052] Currently, there are two main types of multi-instance learning strategies: package-level and instance-level. Package-level multi-instance learning first extracts feature vectors from instances, then aggregates them into package-level feature vectors for classification prediction. During training, the package-level image classification model is supervised by package-level labels, achieving fully supervised learning. Instance-level multi-instance learning focuses the learning process at the instance level. It trains an instance classifier to obtain instance-level prediction results, and then simply aggregates these instance-level prediction results to obtain package-level prediction results.

[0053] Both of these multi-instance learning strategies have their advantages and disadvantages. For example, bag-level multi-instance learning only requires bag-level category labels, without needing to label each instance, and focuses more on capturing global features of the bag, making it more suitable for complex scenarios. However, because the model may ignore some important instance-level information during the learning process, and because the number of samples in whole-slice pathological images is often limited, this may lead to insufficient model training, thus risking suboptimal optimization and resulting in lower prediction accuracy. Instance-level multi-instance learning requires labeling each image patch, which can capture more detailed information and help improve the model's prediction accuracy. However, it also faces the problems of high labeling costs and high labeling noise, and high labeling noise still reduces the model's prediction accuracy.

[0054] Based on this, the inventors of this application have uniquely conceived of and proposed a collaborative instance-level and package-level multi-instance learning strategy for classification prediction of high-resolution images such as whole-section pathology images. This collaborative learning strategy integrates the advantages of both instance-level and package-level learning strategies while compensating for their respective shortcomings, thereby enabling more accurate classification prediction for high-resolution images such as whole-section pathology images.

[0055] The collaborative instance-level and package-level multi-instance learning strategies proposed in this application can be summarized as follows:

[0056] First, the image samples are divided into multiple image blocks. Then, global image features are extracted from multiple image blocks based on the first image classification model, and image-level category labels are predicted based on the global image features to obtain the image block classification prediction information generated by the first image classification model during the image classification process. Next, each image block is classified and predicted based on the second image classification model, and image-level category labels are determined based on the block-level category labels obtained from the classification prediction to obtain the image block classification prediction information generated by the second image classification model during the image classification process. Then, a co-training loss value is calculated based on the image block classification prediction information generated by the first and second image classification models respectively, so as to update the parameters of the first and second image classification models based on the co-training loss value.

[0057] In the collaborative instance-level and package-level multi-instance learning strategy proposed in this application, the process of the first image classification model predicting image-level category labels is set according to the package-level multi-instance learning strategy, while the process of the second image classification model predicting image-level category labels is set according to the instance-level multi-instance learning strategy. Since the collaborative training loss value is calculated based on the image patch classification prediction information generated by the first and second image classification models respectively during image classification, the collaborative training process can utilize image patch-level information to enhance the first image classification model's extraction of global image features. This makes the extracted global image features more accurate, thereby improving the classification accuracy of the first image classification model. It also makes the second image classification model's prediction of block-level category labels more accurate, thus improving its classification accuracy as well.

[0058] Therefore, the first and second image classification models trained using the multi-instance learning strategy proposed in this application can perform more accurate classification and prediction for high-resolution images such as whole-slice pathological images.

[0059] It should be noted that, in some exemplary application scenarios, the multi-instance learning strategy proposed in this application can treat image samples as bags, image patches as instances, image-level category labels as bag-level labels, and image patch-level category labels as instance-level labels. However, in other exemplary application scenarios, the image samples, image patches, image-level category labels, and image patch-level category labels involved in the multi-instance learning strategy proposed in this application may be independent of bags and instances.

[0060] The multi-instance learning strategy proposed in this application will be described in detail below.

[0061] Please refer to the following first. Figure 2 , Figure 2 This is a flowchart illustrating a collaborative training method for an image classification model, as demonstrated in an exemplary embodiment of this application. This method can be applied to... Figure 1 The implementation environment shown can be executed by server 120, terminal 110, or both terminal 110 and server 120. Of course, this method can also be applied to other implementation environments and executed by terminals or servers in other implementation environments, or by both terminals and servers in other implementation environments; this embodiment does not impose any limitations.

[0062] like Figure 2 As shown, in an exemplary embodiment, the training method for the image classification model includes steps S210-S240, which are described in detail below:

[0063] S210 divides the image sample into multiple image blocks.

[0064] Given that image samples typically have high resolution, directly classifying and predicting the entire image is very difficult. Therefore, it is necessary to divide the image sample into multiple image patches and then predict the class label based on these patches. This can also be understood as an image sample being composed of multiple image patches. Therefore, the image sample can be divided into image patches according to the unit size corresponding to each patch, thus obtaining the multiple image patches contained within the image sample.

[0065] It should also be noted that image samples can be either two-dimensional or three-dimensional images; there are no restrictions here. For example, a three-dimensional whole-section pathological image is a three-dimensional image constructed based on layer-by-layer scanning of pathological tissue. When the image sample is a three-dimensional image, the individual image blocks divided from the image sample are usually also three-dimensional image blocks.

[0066] S220: Extract global image features from multiple image patches based on the first image classification model, and predict image-level category labels based on the global image features, so as to obtain the image patch classification prediction information generated by the first image classification model in the image classification process.

[0067] Global image features refer to the image features corresponding to an image sample. They are used to describe the feature information of the entire image sample, and can also be understood as describing the global feature information of the image sample.

[0068] The first image classification model consists of an image-level aggregator and an image-level classifier. The image-level aggregator extracts global image features corresponding to multiple image patches contained in an image sample, while the image-level classifier predicts the class label of the image sample based on the global image features. The class label of the image sample is called the image-level class label.

[0069] Image-level aggregators obtain global image features corresponding to image samples by aggregating the feature vectors of multiple image patches. For example, feature extraction networks can extract feature vectors from each image patch. Furthermore, a feature extraction network can consist of an encoding network and an extraction network. The encoding network extracts the embedding vectors of each image patch, while the extraction network extracts the feature vectors of each image patch based on these embedding vectors. This can be understood as follows: the embedding vector of each image patch refers to the vector information obtained by the encoding network mapping high-dimensional data to a low-dimensional space for each image patch; the feature vector of each image patch refers to the vector form further represented by the embedding vectors of each image patch by the extraction network in the form processed by the image classification model. During co-training, the encoding network can be set to a parameter-frozen state, meaning its parameters are not updated, to reduce the computational resources required for training, thereby improving training efficiency to some extent. The extraction network can be updated based on the training loss value to optimize the feature vector extraction process for each image patch, which also contributes to improving the prediction accuracy of the first and second image classification models.

[0070] For example, image-level aggregators and image-level classifiers can be implemented using convolutional neural networks (CNNs), Transformer models, graph neural networks (GNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), etc., without limitation.

[0071] Image patch classification prediction information refers to information used to predict the category labels of each image patch. The category labels of image patches are also called block-level category labels. As an exemplary implementation, the block-level category labels corresponding to each image patch can be determined based on the image-level category labels output by the first image classification model, and these block-level category labels are used as the image patch classification prediction information generated by the first image classification model during the image classification process. The block-level category labels can be the same as or different from the image-level category labels; this is not a limitation.

[0072] As another exemplary implementation, the image-level aggregator extracts global image features from the feature vectors of multiple image patches and also outputs a prediction score for each image patch. This prediction score characterizes the degree to which each image patch contributes to the global image features. For example, the image-level aggregator contains an attention module, and the attention score obtained through the attention module is the prediction score for each image patch.

[0073] S230, classify and predict each image block based on the second image classification model, and determine the image-level category label based on the block-level category label obtained from the classification prediction, so as to obtain the image block classification prediction information generated by the second image classification model in the image classification process.

[0074] The second image classification model consists of a block-level classifier and a block-level aggregator. The block-level classifier predicts the block-level category label for each image patch based on its feature vector. The block-level aggregator aggregates the block-level category labels for each image patch to obtain the image-level category label for each image sample. The block-level classifier can be implemented based on convolutional neural networks, VGG (Visual Geometry Group) networks, or some lightweight image classification networks; no restrictions are imposed here.

[0075] For example, the block-level category labels corresponding to each image patch output by the block-level classifier can be used as the image patch classification prediction information generated by the second image classification model during the image classification process. Alternatively, the block-level category labels corresponding to each image patch determined based on the image-level category labels obtained by the second image classification model can also be used as the image patch classification prediction information generated by the second image classification model during the image classification process. The block-level category labels can be the same as or different from the image-level category labels, and this embodiment does not impose any restrictions on this.

[0076] S240, calculate the collaborative training loss value based on the image patch classification prediction information generated by the first image classification model and the second image classification model respectively, and update the parameters of the first image classification model and the second image classification model based on the collaborative training loss value.

[0077] Considering that the first image classification model does not actually predict the block-level category label corresponding to each image block during the image classification process, this embodiment uses the image block classification prediction information generated by the first image classification model as the predicted value and the image block classification prediction information generated by the second image classification model as the true value. By calculating the deviation between the predicted value and the true value, the co-training loss value is obtained.

[0078] It should be noted that the method of updating the parameters of the first image classification model and the second image classification model based on the collaborative training loss value in this embodiment includes, but is not limited to, gradient descent, and is not restricted here.

[0079] Therefore, this embodiment calculates the co-training loss value based on the image patch classification prediction information generated by the first and second image classification models, respectively. This allows the co-training process to utilize image patch-level loss information for model parameter updates. During model parameter updates, the co-training loss enhances the first image classification model's extraction of global image features, making the extracted global features more accurate and thus improving the classification accuracy of the first image classification model. It also makes the second image classification model's prediction of patch-level category labels more accurate, thereby improving its classification accuracy as well. In other words, the co-training process proposed in this embodiment utilizes patch-level information to improve the quality of image-level information, thereby enhancing the prediction accuracy of the image classification model.

[0080] It should also be noted that, in some exemplary embodiments, while maintaining collaborative training, the first image classification model and / or the second image classification model can be trained independently to further improve training efficiency. For example, for the first image classification model, the image-level annotation labels corresponding to the image samples are used as the ground truth values, and the image-level category labels output by the first image classification model are used as the predicted values. The deviation between the predicted values ​​and the ground truth values ​​is calculated to obtain the image-level training loss value, and the model parameters of the first image classification model are updated based on the image-level training loss value. Similarly, for the second image classification model, the block-level annotation labels corresponding to each image block are determined according to the image-level annotation labels corresponding to the image samples. For example, if the block-level annotation labels corresponding to each image block are the same as the image-level annotation labels corresponding to the image samples, the block-level annotation labels corresponding to each image block are used as the ground truth values, and the block-level category labels corresponding to each image block output by the block-level classifier are used as the predicted values. The deviation between the predicted values ​​and the ground truth values ​​is calculated to obtain the block-level training loss value, and the model parameters of the second image classification model are updated based on the block-level training loss value.

[0081] In some embodiments, when all independent training and collaborative training coexist, the sum of the collaborative training loss value, the image-level training loss value, and the block-level training loss value is used as the total training loss value, and the model is trained based on the total training loss value.

[0082] In another exemplary embodiment, for the second image classification model, considering the high cost of labeling image blocks, block-level labels are often lacking in actual training scenarios, which in turn affects training performance.

[0083] In instance-level multi-instance learning strategies, to address the lack of instance labels, a technique is employed where pseudo-labels are assigned to each instance based on the bag-level label. These pseudo-labels are then used to supervise the instance label prediction process, thereby improving training performance. For example, a common pseudo-label generation scheme involves the pseudo-labels being identical to the bag labels.

[0084] This embodiment further considers that the noise in block-level labels is usually high, which can also affect the prediction accuracy of block-level classifiers for block-level category labels. Therefore, this embodiment further reduces the noise in block-level pseudo-labels during the collaborative training process to improve collaborative training performance.

[0085] The technical solution proposed in this embodiment will be described in detail below.

[0086] Please see Figure 3 , Figure 3 This is a flowchart illustrating a co-training method for an image classification model, as shown in another exemplary embodiment of this application. The method is... Figure 2 Based on the illustrated embodiment, the process of S230, which involves classifying and predicting each image patch based on the second image classification model and determining the image-level category label based on the patch-level category label obtained from the classification prediction, to obtain the image patch classification prediction information generated by the second image classification model during the image classification process, further includes the following steps:

[0087] S310, Generate block-level pseudo-labels for each image block based on the block-level category labels corresponding to each image block and the prototype vectors corresponding to each category;

[0088] S320, the block-level pseudo-labels corresponding to each image block are obtained as the image block classification prediction information generated by the second image classification model during the image classification process.

[0089] First, it's important to understand that a pseudo-label, also known as a soft label, is a probability distribution representing the likelihood of a sample belonging to each class. In classification tasks, a pseudo-label is typically a vector containing probability values. When the data contains noise or uncertainty, pseudo-labels can reflect this noise or uncertainty, thus often leading to more accurate predictions of the probability of a sample belonging to each class.

[0090] In this embodiment, block-level pseudo-labels are assigned to each image patch contained in the training samples, and a confidence library is used to store the block-level pseudo-labels corresponding to each training sample. Subsequently, by obtaining the block-level pseudo-labels corresponding to each image patch as image patch classification prediction information generated by the second image classification model during the image classification process, these pseudo-labels are used to participate in the calculation of the co-training loss value. Based on the co-training loss value, the parameters of the image classification model participating in the co-training are updated. This improves the prediction accuracy of the block-level classifier for the block-level category labels corresponding to each image patch, and also improves the accuracy of the image-level aggregator in extracting global image features, thereby further improving the prediction accuracy of the image-level category labels.

[0091] This embodiment also provides a prototype vector for each category, and the prototype vector corresponding to each category is applied to the predicted category. It can also be understood that the prototype vector corresponding to each category describes the important information corresponding to the category; the prototype vector is a vectorization of the important information corresponding to the category.

[0092] It's important to note that the categories corresponding to image-level category labels and block-level category labels should be consistent. Assuming a binary classification scenario, image-level category labels correspond to categories 0 and 1, and block-level category labels also correspond to categories 0 and 1. For easier understanding, consider this example: an image-level category label represents the probability that an image sample belongs to category 0 and category 1 is 60% and 40%, respectively; a block-level category label represents the probability that an image block belongs to category 0 and category 1 is 65% and 35%, respectively. Therefore, both labels correspond to categories 1 and 2.

[0093] It should also be noted that the prototype vectors for each category will be updated during the training process. The initial prototype vectors can be preset or generated based on the content related to each category obtained in a specific training stage. For example, the initial prototype vectors for each category can be generated based on the training results of the first 10 generations, so as to promote more accurate recognition of each category based on the prototype vectors.

[0094] The process of generating block-level pseudo-labels for each image patch in this embodiment is related not only to the block-level category labels for each image patch predicted by the second image classification model, but also to the prototype vectors for each category. Based on the block-level category labels for each image patch and the prototype vectors for each category, the noise in the generated block-level pseudo-labels is reduced, thereby improving the collaborative training performance.

[0095] Please continue reading. Figure 4 , Figure 4 This is a flowchart illustrating a collaborative training method for an image classification model, as shown in another exemplary embodiment of this application. Figure 4As shown, the process of generating block-level pseudo-labels for each image patch in S310 based on the block-level category labels and the prototype vectors of each category includes the following steps:

[0096] S410: Based on the block-level category label corresponding to each image block, select typical image blocks for each category respectively;

[0097] S420, Based on the selected image patches and the prototype vectors corresponding to each category, generate prototype labels for each image patch; wherein, the prototype labels for each image patch are used to represent the category of the corresponding image patch.

[0098] S430, based on the updated prototype labels corresponding to each image block, determines the block-level pseudo-labels corresponding to each image block.

[0099] It is understood that the process of generating block-level pseudo-labels in this embodiment can be summarized into three important stages: selecting typical image blocks, generating prototype labels, and updating the confidence level. S410 corresponds to the stage of selecting typical image blocks, S420 corresponds to the stage of generating prototype labels, and S430 corresponds to the stage of updating the confidence level library.

[0100] In S410, since the block-level category label contains the probability value of each image block belonging to each category, the probability value of each image block being predicted to be each category can be determined according to the block-level category label corresponding to each image block. Then, based on the probability value of each image block being predicted to be each category, typical image blocks are selected for each category.

[0101] A typical image patch can be understood as an image patch that has a high predictive contribution to the image sample. As an exemplary implementation, for each category, the probability values ​​corresponding to each image patch are sorted from largest to smallest, and the image patch corresponding to the probability value with the rank K is selected. K is a positive integer, so K represents the K typical images to be selected for each category for each image sample. Thus, each category can obtain K*O typical image patches. These K*O typical image patches can be further simplified later, without limitation here.

[0102] As another exemplary implementation, considering that model training is usually performed in batches on a large number of image samples, for each category, the probability values ​​corresponding to each image patch contained in the multiple image samples can be sorted from largest to smallest, and the image patch corresponding to the probability value with a ranking of K can be selected as the typical image patch. Thus, K typical image patches can be selected for each category.

[0103] Of course, other methods can also be used to select a preset number of image patches for each category. This embodiment does not limit this, but it should be noted that the predicted value of the selected image patch with respect to the corresponding category should be a large value, so that the prototype label generated by S420 based on the selected image patch is closer to the important information corresponding to the category, thereby enabling the block-level pseudo-label generated by S430 to more accurately reflect the true category of each image patch. Therefore, it can further improve the prediction accuracy of block-level classification labels. Compared with directly determining the block-level pseudo-label based on the image-level annotation label, this embodiment can further reduce the noise in the block-level pseudo-label and generate more accurate block-level pseudo-labels to improve the performance of collaborative training.

[0104] In S420, the process of generating prototype labels for each image patch based on the typical image patches corresponding to each category selected in S410 and the prototype vectors corresponding to each category includes updating the prototype vectors corresponding to each category according to the image patches selected by the category, and generating prototype labels for each image patch according to the updated prototype vectors of each category.

[0105] For example, the prototype vector update can be based on momentum update rules to accelerate model convergence, improve model stability, and optimize resource utilization. For instance, first, the momentum coefficient and its complementary coefficient are determined based on the current training iteration count. Then, the prototype vectors for each category are updated according to the determined momentum coefficient, complementary coefficient, and feature vectors of selected image patches for each category. The complementary coefficient is the difference between 1 and the momentum coefficient. For example, for each category, a first product between the momentum coefficient and the prototype vector before the update is calculated, and a second product between the complementary coefficient and the feature vector of the selected typical image patch is calculated. The sum of the first and second products is used as the updated prototype vector.

[0106] In some exemplary embodiments, the momentum coefficient is related to the current training iteration number. Throughout the training process, the momentum coefficient gradually decreases within its range of variation as the number of training iterations increases. For example, the preset momentum coefficient variation range has a starting endpoint value and an ending endpoint value, with the starting endpoint value being greater than the ending endpoint value. First, the difference between the ending endpoint value and the starting endpoint value is calculated. Then, the product of the total number of iterations and this difference is calculated. Next, the quotient of the current training iteration coefficient and this product is calculated. The sum of this quotient and the starting endpoint value is taken as the momentum coefficient corresponding to the current training iteration. It should be noted that the momentum coefficient should be a relatively large value, that is, a small value closer to 1, to ensure training efficiency. Therefore, the starting endpoint value and the ending endpoint value corresponding to the momentum coefficient variation range should also be closer to 1. For example, the starting endpoint value could be 0.95 and the ending endpoint value could be 0.8, but this does not imply any numerical limitation on the starting endpoint value and the ending endpoint value.

[0107] In some other exemplary embodiments, the momentum coefficient may also be a preset value, and a small value closer to 1; this embodiment does not limit this.

[0108] For example, the product of the feature vector and the updated prototype vector for each class is calculated for each image patch to obtain the class probability set corresponding to each image patch. Then, the class with the highest probability is selected from the class probability set corresponding to each image patch as the prototype class corresponding to each image patch. Based on the prototype class corresponding to each image patch, vector encoding is performed to obtain the prototype label corresponding to each image patch. The vector encoding method can include One-Hot encoding and similar encoding methods, and is not limited here.

[0109] In S430, the block-level pseudo-labels corresponding to each image patch are obtained based on the updated prototype labels corresponding to each image patch. This allows the block-level pseudo-labels corresponding to each image patch to more accurately reflect the true category of each image patch, improving the accuracy of the block-level pseudo-labels and thus enhancing the performance of collaborative training.

[0110] With a confidence library in place, the block-level pseudo-labels of each image block stored in the confidence library are updated based on the updated prototype labels corresponding to each image block, so as to obtain the block-level pseudo-labels corresponding to each image block from the updated confidence library.

[0111] In some exemplary embodiments, the confidence library can be updated based on momentum update rules, thereby accelerating model convergence, improving model stability, and optimizing resource utilization. For example, the momentum coefficients and complementary coefficients corresponding to the current training iterations are determined based on the current training iterations. Then, based on the determined momentum coefficients, complementary coefficients, and prototype labels corresponding to each image patch, block-level pseudo-labels corresponding to each image patch are obtained. For example, for each image patch, the third product between the momentum coefficient and the block-level pseudo-vector before the update is calculated, and the fourth product between the complementary coefficient and the generated prototype label is calculated. The sum of the third and fourth products is used as the block-level pseudo-label.

[0112] In other exemplary embodiments, the momentum coefficient is also related to the current training iteration number. Throughout the training process, the momentum coefficient gradually decreases within its range of variation as the number of training iterations increases. The momentum coefficient update strategy can refer to the process described in the embodiment for updating the prototype vector, and will not be repeated here.

[0113] In other exemplary embodiments, the momentum coefficient may also be a preset fixed value. It should be noted that the momentum coefficient should be a relatively large value, that is, a small value closer to 1, to ensure training efficiency. For example, the momentum coefficient can be 0.99, but this does not imply a numerical limitation on the momentum coefficient.

[0114] Therefore, the block-level pseudo-labels obtained in this embodiment can more accurately reflect noise information, thus further improving the prediction accuracy of block-level classification labels, thereby further reducing noise in block-level pseudo-labels and improving collaborative training performance.

[0115] To further facilitate understanding of the collaborative training scheme proposed in the embodiments of this application, an exemplary training scenario will be described in detail below.

[0116] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating the collaborative training process of an exemplary image classification model. In real-world training scenarios, it is often necessary to train the model in batches using a large number of image samples. Figure 5 The training sample set S shown in the diagram contains multiple image samples. If we assume that the training sample set S contains N image samples, then the training sample set S can be represented as: S = {S1, S2, ..., S...} i ,…,S N}, where S i Let i represent the i-th sample image, where 1 ≤ i ≤ N.

[0117] Each image sample corresponds to an image-level annotation label, which can be represented as Y. i∈{1, ..., C}, where C represents the number of categories. For example, in a binary classification scenario, C = 2, indicating that the image-level annotation labels contain two categories.

[0118] Each image sample is considered to be composed of multiple tiled image blocks, therefore each image sample is represented as: Among them, M i S represents the number of image patches in the i-th sample image, that is, each sample image contains M image patches. j Let M represent the j-th image patch, where 1 ≤ j ≤ M. Figure 5 The S shown in the middle i,j This indicates that the i-th image sample contains the j-th image patch.

[0119] The block-level label for each image patch in an image sample is unknown. At the start of training, the image-level label for each image sample is assigned as a block-level pseudo-label to each image patch contained within that image sample. As training continues, the block-level pseudo-labels are continuously updated; the detailed update strategy will be described later and will not be repeated here.

[0120] like Figure 5 As shown in the illustration, the system training process in this embodiment consists of a feature extraction process and a collaborative supervised training process, which will be described in detail below:

[0121] 1. Feature Extraction Process

[0122] An image pre-training model is used to encode input image patches, obtaining embedding vectors for each patch. Feature vectors are then extracted from these embedding vectors. Therefore, the image pre-training model can be viewed as consisting of an encoding network and an extraction network. The encoding network extracts embedding vectors, and the extraction network extracts feature vectors. The encoding network can be in a parameter-frozen state and does not participate in model parameter training and updates. The extraction network, however, does participate in model parameter training and updates.

[0123] The extraction network can employ a residual Transformer backbone network, which combines the advantages of Transformer networks and residual connections. During the processing of image patch embedding vectors, it can capture contextual information and the correlation between image patches. The residual Transformer backbone network can be formed by concatenating multiple Transformer blocks through residual blocks. The use of residual blocks facilitates gradient propagation in deep networks, mitigating the vanishing or exploding gradient problems encountered during deep network training. Each Transformer block can contain 5 linear blocks and at least 1 Transformer layer. A residual block can consist of 1 linear layer and 1 activation layer.

[0124] The extraction network can also use various other image feature extraction pre-trained models, such as the Efficient-Net model pre-trained based on natural images, the Ctranspath and UNI models pre-trained based on pathological images, etc., without any restrictions.

[0125] For example, the process of extracting feature vectors from the embedding vectors of image patches by the extraction network can be represented as: x i,j =T(e i,j ), where e i,j Let x represent the embedding vector of the j-th image patch in the i-th image sample. i,j Let represent the feature vector of the j-th image patch in the i-th image sample, and T represent the function that maps the embedding vector of the image patch to the feature vector of the image patch.

[0126] 2. Collaborative Supervision Training Process

[0127] The collaborative supervised training process mainly consists of three parts: block-level supervision, image-level supervision, and image-block collaborative supervision.

[0128] It should be noted that the first image classification model and the second image model participating in the collaborative training... Figure 5 While not explicitly shown, the composition of the first image classification model is illustrated, including an image-level aggregator and an image-level classifier, and the block-level classifier included in the second image classification model is also shown.

[0129] 2.1 Block-level supervision

[0130] The feature vectors of each image patch are first input into a block-level classifier to obtain the block-level category label corresponding to each image patch. If the classification prediction process performed by the block-level classifier is represented as a mapping function F, then for the feature vector x of the j-th image patch in the i-th image sample... i,j The corresponding block-level category labels are obtained. The process can be represented as:

[0131] Then, the feature vectors of each image patch and the corresponding block-level category labels are input into the label disambiguation module. The label disambiguation module aims to reduce the noise contained in the block-level pseudo-labels, thereby obtaining more accurate block-level labels. More accurate block-level pseudo-labels are understood as the block-level pseudo-labels obtained by the label disambiguation module being closer to the true category of each image patch.

[0132] The label disambiguation module generates block-level pseudo-labels for each image patch by generating prototype labels for each patch and accessing a stored confidence library. The prototype labels for each image patch represent its category, while the confidence library stores the block-level pseudo-labels for each image patch.

[0133] Please see Figure 6 , Figure 6 yes Figure 5 The diagram illustrates the processing flow of the label disambiguation module in an exemplary embodiment. Figure 6 As shown, the label disambiguation module mainly consists of three parts: selecting typical image patches, generating prototype labels, and updating the confidence library.

[0134] 2.11 Selecting a typical image patch

[0135] To focus image classification models on image patches that contribute significantly to predictions for image samples, it's first necessary to select representative image patches to update prototype information. Specifically, the feature vector x of the j-th image patch in the i-th image sample is... i,j The probability of being predicted as category c is expressed as At time t (i.e., the current time), a set of image patch feature vectors belonging to class c with the highest probability is selected as the feature vectors of the typical image patch. The process of selecting the typical image patch can be represented as follows:

[0136]

[0137] Among them, Set c,t argTop represents the feature vector of the typical image patch selected at time t, K represents the number of image patch feature vectors with high prediction values ​​to be selected, and argTop K This represents a function used to select the feature vectors of K image patches with high prediction values.

[0138] Therefore, for each category, K image patch feature vectors with high prediction values ​​are selected to obtain the feature vectors of K typical image patches corresponding to each category.

[0139] 2.12 Generate prototype tags

[0140] In this process, based on the feature vector Set of the selected image patch c,t And the feature vectors corresponding to each category, to generate prototype labels for each image patch.

[0141] Specifically, at time t-1, if the prototype vector corresponding to category c is P c,t-1 After obtaining K typical image patch feature vectors, a momentum-based update rule is used to update the prototype vector to obtain the prototype vector P corresponding to category c at time t.c,t The process of updating the prototype vector can be represented as follows:

[0142] P c,t =α·P c,t-1 +(1-α)·x i,j , if x i,j ∈Set c,t

[0143] Here, α represents the momentum coefficient, which automatically decreases from α = 0.95 to α = 0.8 at various times. This can be understood as the momentum coefficient changing within a certain range during training, for example, [0.95, 0.8].

[0144] The change in momentum coefficient α can be represented as follows:

[0145]

[0146] Where epoch represents the current training iteration number, epochs represents the total number of iterations, end represents the end point of the momentum coefficient variation range, and start represents the start point of the momentum coefficient variation range. For example, start = 0.95, end = 0.8.

[0147] For each image patch, the updated prototype vector for each category is multiplied by its feature vector to obtain a product. This product represents the probability that the image patch belongs to the corresponding category, thus each image patch will have a set of category probabilities. Then, for each image patch, the category with the highest probability is selected from its corresponding category probability set as the prototype category for that image patch. Based on the prototype category of each image patch, a prototype label for each image patch is obtained.

[0148] The process of updating the prototype tag can be represented as follows:

[0149] z i,j =OneHot(argmax) c (P c,t ·x i,j ))

[0150] Among them, z i,j This represents the prototype label of the j-th image patch in the i-th image sample, which is a one-hot vector, argmax. c (P c,t ·x i,j () represents calculating the prototype vector and its feature vector x for each category of the j-th image patch in the i-th image sample. i,j The product of the largest product is used to select the category corresponding to the largest product as the prototype category of the image patch.

[0151] The block-level pseudo-labels for each image patch are determined based on the prototype labels of each image patch. The prototype labels of each image patch are generated based on the prototype vectors of each category, which makes the prototype labels of each image patch closer to the true category of each image patch. Compared with generating block-level pseudo-labels directly based on the image-level annotation labels of image samples, the block-level pseudo-labels of each image patch have less noise.

[0152] 2.13 Update the confidence database

[0153] After obtaining the prototype label at time t, the confidence library is also updated according to the momentum update rule. The update process is as follows:

[0154] B i,j,t =β·B i,j,t-1 +(1-β)·z i,j

[0155] Where β represents the momentum coefficient, with a default value of 0.99; B i,j,t B represents the block-level pseudo-label of the j-th image patch in the i-th image sample at time t, which is also the updated block-level pseudo-label in the confidence database; i,j,t-1 This represents the block-level pseudo-label of the j-th image block in the i-th image sample at time t-1, which is also the block-level pseudo-label currently stored in the confidence database, or the block-level pseudo-label to be updated in the confidence database.

[0156] This allows us to obtain block-level pseudo-labels for each image patch from the updated confidence database. Furthermore, the process of generating block-level pseudo-labels does not depend on the label annotation information of the image patch, but is calculated based on the feature vector and prototype vector of the image patch. Therefore, it can effectively solve the problem of lack of block-level annotation information.

[0157] Using block-level pseudo-labels to supervise model learning can promote the model to learn more accurate and robust block-level representations.

[0158] Furthermore, the block-level supervision part can also perform gradient updates independently. After obtaining the corresponding block-level training loss value for the block-level supervision part, the second image classification model is updated based on this block-level training loss value. For example, the block-level training loss value is calculated using the following cross-entropy function:

[0159]

[0160] Among them, L b B represents the block-level training loss value. i,j,c and These are the c-th component in the block-level pseudo-label and the c-th component in the block-level category label of the j-th image block in the i-th image sample, respectively. N represents the number of image samples, M represents the number of image blocks contained in the image sample, and C represents the number of categories.

[0161] 2.2 Image-level supervision

[0162] In the image-level supervision process, the feature vectors of each image patch contained in each image sample are first aggregated by an image-level aggregator to obtain the global image features and attention score of each image sample. The attention score represents the degree of contribution of each image patch to the corresponding image features. The processing of the image-level aggregator can be expressed by the following formula:

[0163]

[0164] Among them, f i Let a represent the global features of the i-th image sample. i,j Let x represent the attention score of the j-th image patch in the i-th image sample. i,j Let represent the feature vector of the j-th image block in the i-th image sample.

[0165] The global image features are then used by an image-level classifier to predict image-level category labels. The prediction process can be represented as follows:

[0166]

[0167] in, F represents the image-level category label of the i-th image sample. s f represents an image-level classifier. i Let represent the global features of the i-th image sample, and softmax represent the normalization function.

[0168] The entire process is represented as follows:

[0169] Similarly, the image-level supervision part can also perform gradient updates independently. After obtaining the corresponding image-level training loss value for the image-level supervision part, the first image classification model is updated based on this image-level training loss value. For example, the image-level training loss value can be calculated using the following cross-entropy function:

[0170]

[0171] Among them, L S This represents the image-level training loss value, where N represents the number of image samples. Y represents the image-level category label of the i-th image sample. i This represents the image-level annotation label for the i-th image sample.

[0172] 2.3 Image-Block Collaborative Supervision

[0173] Collaborative supervision combines image-level and block-level supervision to facilitate better interaction between the two. Specifically, we use the attention score 'a' from the image-level classifier output... i,j As the predicted value, the block-level pseudo-label B obtained by the label disambiguation module i,j The formula for calculating the collaborative training loss value is as follows: (This is used as the true value.)

[0174]

[0175] Among them, L S-b The loss value represents the co-training loss, where N represents the number of image samples, M represents the number of image patches contained in an image sample, C represents the number of classes, and B represents the number of classes. i,j,c a represents the block-level pseudo-label of the j-th image patch in the i-th image sample. i,j,c This represents the c-th component of the attention score.

[0176] In some exemplary embodiments, it can be based on L=L S-b Update the model parameters.

[0177] In some exemplary embodiments, it can be based on L=L S-b +L S Update the model parameters.

[0178] In some exemplary embodiments, it can be based on L=L S-b +L b Update the model parameters.

[0179] In some exemplary embodiments, it can be based on L=L S-b +L b +L S Update the model parameters.

[0180] Where L represents the total training loss of the image classification model.

[0181] In some exemplary embodiments, the total training loss L can also be calculated according to the following formula, and the model parameters can be updated based on the total training loss L.

[0182] L = L S +λL b +μL S-b

[0183] Here, λ and μ represent hyperparameters used to control the relative importance of the loss information among the three.

[0184] Therefore, the collaborative training scheme of the image classification model proposed in this application utilizes the label disambiguation module to obtain more accurate block-level pseudo-labels, which can reduce the noise of block-level pseudo-labels. Furthermore, by utilizing block-level features to optimize the attention in the image-level aggregator, more reasonable image-level category labels can be obtained, enabling both the first and second image classification models after training to obtain more accurate image-level category labels.

[0185] It should also be noted that the label disambiguation module may not be used in actual training scenarios. In this case, the image-level annotation labels of the image samples can be used as the ground truth values ​​to calculate the loss value accordingly; the detailed process will not be elaborated further.

[0186] Please continue reading Figure 7 , Figure 7 This is a flowchart illustrating an exemplary image classification method. This method can be applied to... Figure 1 The implementation environment shown can be executed by server 120, terminal 110, or both terminal 110 and server 120. Of course, this method can also be applied to other implementation environments and executed by terminals or servers in other implementation environments, or by both terminals and servers in other implementation environments; this embodiment does not impose any limitations.

[0187] like Figure 7 As shown, in an exemplary embodiment, the image classification method includes the following steps:

[0188] S710, acquire the whole-section pathological image to be classified;

[0189] S720, input the whole-slice pathological image into the first image classification model and / or the second image classification model after collaborative training, so as to obtain the image classification results output by the first image classification model and / or the second image classification model.

[0190] This embodiment employs a first image classification model and / or a second image classification model, trained collaboratively through the above embodiments, to perform classification prediction on whole-slice pathological images (WIS) to obtain the corresponding image classification results output by the first image classification model and / or the second image classification model. As described above, the first image classification model and / or the second image classification model, trained collaboratively through the above embodiments, can more accurately predict image-level category labels. Therefore, this application, by employing the collaboratively trained first image classification model and / or the second image classification model, can perform more accurate classification prediction on whole-slice pathological images, obtaining image classification results with higher accuracy. For example, the image classification result corresponding to the whole-slice pathological image includes the probability that the whole-slice pathological image is predicted to have a gene mutation, and the probability that it is predicted not to have a gene mutation.

[0191] It should also be noted that the first image classification model after collaborative training can be preferred for classification prediction of whole-slice pathological images, because the image-level aggregator contained in the first image classification model globally summarizes the information of all image patches during the training phase, so that the extracted global image features have high accuracy, thereby enabling the image-level classifier to output highly accurate image classification results.

[0192] The inventors of this application also conducted verification experiments on the classification prediction performance of various image classification model training schemes on different WIS datasets. Specifically, in addition to the technical solution of this application, the various image classification model training schemes also include CNN-ML, DSMIL, CLAM-SB, CLAM-MB, ViT-MIL, TransMIL, SETMIL, WEMO, LCSB-MIL, and IIB-MIL. The WIS datasets include the TGGA-NSCLC dataset, the TGGA-RCC dataset, and the CAMELYON16 dataset. Among them, TGGA-NSCLC and TGGA-RCC are cancer genome atlas datasets, and CAMELYON16 is a dataset used for the detection and classification of lymph node metastatic cancer. The verification experiments correspondingly obtained the AUC (Area Under the Curve) and Accuracy of the image classification model obtained by each image classification model training scheme for each dataset.

[0193] Experiments show that the technical solution of this application performs better on each dataset. The AUC and Accuracy of the technical solution of this application are higher than those of other technical solutions for the corresponding datasets.

[0194] For example, the AUC and Accuracy of the technical solution of this application on the TGGA-NSCLC dataset are 0.999 and 0.9936, respectively, while the highest AUC and Accuracy of other technical solutions on this dataset are 0.9811 and 0.9091. The AUC and Accuracy of the technical solution of this application on the TGGA-RCC dataset are 0.9994 and 0.9787, respectively, while the highest AUC and Accuracy of other technical solutions on this dataset are 0.9902 and 0.9524. The AUC and Accuracy of the technical solution of this application on the TGGA-RCC dataset are 0.9681 and 0.8837, respectively, while the highest AUC and Accuracy of other technical solutions on this dataset are 0.9107 and 0.8494.

[0195] Therefore, it can be demonstrated that the collaborative training scheme proposed in this application can significantly improve the classification prediction accuracy of high-resolution images similar to whole-section pathological images.

[0196] Please continue reading. Figure 8 , Figure 8 This is a flowchart of an exemplary image classification method. This method can be applied to... Figure 1 The implementation environment shown can be executed by server 120, terminal 110, or both terminal 110 and server 120. Of course, this method can also be applied to other implementation environments and executed by terminals or servers in other implementation environments, or by both terminals and servers in other implementation environments; this embodiment does not impose any limitations.

[0197] like Figure 8 As shown, in another exemplary embodiment, the training method for the image classification model includes the following steps:

[0198] S810 divides the image sample into multiple image blocks;

[0199] S820: Based on the image classification model, classify and predict each image block to obtain the block-level category label corresponding to each image block, and determine the image-level category label based on the block-level category label corresponding to each image block.

[0200] S830: Based on the block-level category label corresponding to each image patch and the prototype vector of each category, generate the block-level pseudo-label corresponding to each image patch; wherein, the prototype vector corresponding to each category is applied to the predicted category.

[0201] S840 calculates the block-level training loss value based on the block-level pseudo-label and block-level category label corresponding to each image patch, and updates the parameters of the image classification model based on the block-level training loss value.

[0202] The image classification model training method provided in this embodiment can be understood as the block-level supervision strategy described above. Based on the aforementioned block-level supervision strategy, less noisy block-level pseudo-labels are generated for each image block contained in the image sample. This allows the block-level training loss value to be calculated based on the block-level pseudo-labels and block-level category labels corresponding to each image block. The parameters of the image classification model are then updated based on the block-level training loss value, which improves the accuracy of the image classification model in predicting block-level category labels. Consequently, the trained image classification model can perform more accurate image classification predictions.

[0203] It should also be noted that since the relevant content of this embodiment has been described in detail in the foregoing embodiments, it will not be repeated here.

[0204] Correspondingly, after obtaining the whole-section pathological image to be classified, the image classification method can also input the whole-section pathological image into the trained image classification model to obtain the image classification result output by the image classification model.

[0205] Please see Figure 9 , Figure 9 This is a block diagram illustrating a collaborative training apparatus for an image classification model, as shown in an exemplary embodiment of this application. The apparatus can be deployed in... Figure 1 The implementation environment shown can be deployed, for example, on server 120, on terminal 110, or on both terminal 110 and server 120. Of course, the device can also be deployed in other implementation environments, specifically on terminals or servers in other implementation environments, or jointly on terminals and servers in other implementation environments. This embodiment does not impose any limitations.

[0206] like Figure 9 As shown, in an exemplary embodiment, the co-training apparatus for the image classification model includes:

[0207] The segmentation module 910 is configured to divide the image samples into multiple image blocks;

[0208] The first acquisition module 920 is configured to extract global image features from multiple image patches based on the first image classification model, and predict image-level category labels based on the global image features, so as to obtain image patch classification prediction information generated by the first image classification model in the image classification process.

[0209] The second acquisition module 930 is configured to perform classification prediction on each image block based on the second image classification model, and determine the image-level category label based on the block-level category label obtained from the classification prediction, so as to obtain the image block classification prediction information generated by the second image classification model in the image classification process.

[0210] The collaborative module 940 is configured to calculate a collaborative training loss value based on the image patch classification prediction information generated by the first image classification model and the second image classification model respectively, so as to update the parameters of the first image classification model and the second image classification model based on the collaborative training loss value.

[0211] In another exemplary embodiment, the second acquisition module 930 includes:

[0212] The generation unit is configured to generate block-level pseudo-labels for each image patch based on the block-level category label corresponding to each image patch and the prototype vector of each category; wherein the prototype vector of each category is applied to the predicted category.

[0213] The acquisition unit is configured to acquire the block-level pseudo-labels corresponding to each image block as image block classification prediction information generated by the second image classification model during the image classification process.

[0214] In another exemplary embodiment, the generation unit includes:

[0215] Select sub-units and configure them to select typical image blocks for each category based on the block-level category labels corresponding to each image block;

[0216] The generation subunit is configured to generate prototype labels for each image patch based on the selected image patch and the prototype vectors corresponding to each category; wherein, the prototype labels for each image patch are used to represent the category of the corresponding image patch.

[0217] Determine the sub-units and configure them to determine the block-level pseudo-labels corresponding to each image block based on the updated prototype labels corresponding to each image block.

[0218] In another exemplary embodiment, the determined subunit is further configured to perform the following steps:

[0219] Based on the updated prototype labels corresponding to each image patch, the block-level pseudo-labels of each image patch stored in the confidence database are updated.

[0220] Obtain the block-level pseudo-labels corresponding to each image patch from the updated confidence library.

[0221] In another exemplary embodiment, the generation subunit is further configured to perform the following steps:

[0222] Based on the block-level category label corresponding to each image patch, the probability value of each image patch being predicted as each category is determined.

[0223] Based on the probability values ​​of each image patch being predicted to belong to each category, typical image patches are selected for each category.

[0224] In another exemplary embodiment, the selection subunit is further configured to perform the following steps:

[0225] Update the prototype vector corresponding to each category based on the image patches selected for each category;

[0226] Based on the updated prototype vectors for each category, generate prototype labels for each image patch.

[0227] In another exemplary embodiment, the selection subunit is further configured to perform the following steps:

[0228] The momentum coefficient corresponding to the current training iteration number and the complement coefficient corresponding to the momentum coefficient are determined based on the current training iteration number.

[0229] The prototype vectors for each category are updated based on the momentum coefficient, complementarity coefficient, and feature vectors of the image patches selected for each category.

[0230] In another exemplary embodiment, the selection subunit is further configured to perform the following steps:

[0231] For each image patch, the product of the feature vector and the updated prototype vector for each category is calculated to obtain the category probability set corresponding to each image patch;

[0232] The category with the highest probability is selected from the set of category probabilities corresponding to each image patch, and is used as the prototype category for each image patch. Based on the prototype category, vector encoding is performed to obtain the prototype label for each image patch.

[0233] In another exemplary embodiment, the first acquisition module 920 is further configured to perform the following steps:

[0234] The second image classification model also outputs the prediction scores of each image patch when extracting global features from multiple image patches; where the prediction score represents the degree of contribution of each image patch to the global features of the image.

[0235] The predicted scores of each image patch are obtained as the image patch classification prediction information generated by the first image classification model during the image classification process.

[0236] In another exemplary embodiment, the second acquisition module 930 is further configured to perform the following steps:

[0237] Based on the image-level category labels determined by the second image classification model, the image patch classification prediction information generated by the second image classification model during the image classification process is determined.

[0238] In another exemplary embodiment, the collaboration module 940 is further configured to perform the following steps:

[0239] The image patch classification prediction information generated by the first image classification model is used as the predicted value, and the image patch classification prediction information generated by the second image classification model is used as the true value. The deviation between the predicted value and the true value is calculated to obtain the co-training loss value.

[0240] In another exemplary embodiment, the co-training apparatus for the image classification model further includes an extraction module configured to perform the following steps:

[0241] Extract the embedding vector of each image patch, and extract the feature vector of each image patch based on the embedding vector of each image patch;

[0242] The feature vectors of each image patch are input into the first image classification model and the second image classification model respectively, so that the first image classification model and the second image classification model can predict the image-level category label based on the received feature vectors of multiple image patches.

[0243] In another exemplary embodiment, the extraction module is further configured to perform the following steps:

[0244] The embedding vectors of each image patch are extracted through an encoding network, and the feature vectors of each image patch are extracted through an extraction network based on the embedding vectors of each image patch; wherein, the encoding network is in a parameter frozen state.

[0245] The collaboration module 940 is further configured to perform the following steps:

[0246] The parameters of the extraction network are updated based on the training loss value.

[0247] Please see Figure 10 , Figure 10 This is a block diagram illustrating an image classification apparatus according to an exemplary embodiment of this application. The apparatus can be deployed in… Figure 1 The implementation environment shown can be deployed, for example, on server 120, on terminal 110, or on both terminal 110 and server 120. Of course, the device can also be deployed in other implementation environments, specifically on terminals or servers in other implementation environments, or jointly on terminals and servers in other implementation environments. This embodiment does not impose any limitations.

[0248] like Figure 10 As shown, in one exemplary embodiment, the image classification apparatus includes:

[0249] The acquisition module 1010 is configured to acquire whole-slice pathological images to be classified;

[0250] The processing module 1020 is configured to input the whole-slice pathological image into a first image classification model and / or a second image classification model that have been co-trained by a co-training method of the image classification model, so as to obtain the image classification result output by the first image classification model and / or the second image classification model.

[0251] Another exemplary embodiment of this application also proposes a training apparatus for an image classification model. This apparatus can be deployed in... Figure 1 The implementation environment shown can be deployed, for example, on server 120, on terminal 110, or on both terminal 110 and server 120. Of course, the device can also be deployed in other implementation environments, specifically on terminals or servers in other implementation environments, or jointly on terminals and servers in other implementation environments. This embodiment does not impose any limitations.

[0252] The training apparatus for the image classification model includes:

[0253] The image segmentation module is configured to divide image samples into multiple image blocks;

[0254] The model processing module is configured to perform classification prediction on each image patch based on the image classification model, obtain the block-level category label corresponding to each image patch, and determine the image-level category label based on the block-level category label corresponding to each image patch.

[0255] The pseudo-label generation module is configured to generate block-level pseudo-labels for each image patch based on the block-level category label corresponding to each image patch and the prototype vector of each category; wherein, the prototype vector corresponding to each category is applied to the predicted category.

[0256] The loss calculation module is configured to calculate the block-level training loss value based on the block-level pseudo-label and block-level category label corresponding to each image patch, so as to update the parameters of the image classification model based on the block-level training loss value.

[0257] It should be noted that the apparatus and method provided in the above embodiments belong to the same concept, and the specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the collaborative training device or image classification device for the image classification model provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above, and this is not a limitation here.

[0258] Embodiments of this application also provide an electronic device, including: one or more processors; and a memory for storing one or more computer programs, which, when executed by the one or more processors, cause the electronic device to implement the artificial intelligence-based video understanding model training method provided in the above embodiments.

[0259] Figure 11 A schematic diagram of a computer system suitable for implementing an electronic device according to embodiments of this application is shown. It should be noted that the electronic device can be... Figure 1 The terminal 110 or server 120 in the implementation environment shown can also be a terminal or server in other implementation environments; no restrictions are imposed here. It should also be noted that... Figure 11 The computer system 1100 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0260] like Figure 11 As shown, the computer system 1100 includes a Central Processing Unit (CPU) 1101, which can perform various appropriate actions and processes based on a computer program stored in Read-Only Memory (ROM) 1102 or a computer program loaded from storage portion 1108 into Random Access Memory (RAM) 1103, such as performing the methods described in the above embodiments. Various computer programs and data required for system operation are also stored in RAM 1103. The CPU 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. An Input / Output (I / O) interface 1105 is also connected to bus 1104.

[0261] The following components are connected to I / O interface 1105: an input section 1106 including a keyboard, mouse, etc.; an output section 1107 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to I / O interface 1105 as needed. Removable media 1111, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1110 as needed so that computer programs read from them can be installed into storage section 1108 as needed.

[0262] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1109, and / or installed from removable medium 1111. When the computer program is executed by central processing unit (CPU) 1101, it performs various functions defined in the system of this application.

[0263] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0264] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0265] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0266] Another aspect of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor of an electronic device, implements the method described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0267] Another aspect of this application provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the methods provided in the various embodiments described above.

[0268] The above description is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

[0269] It is understood that in the specific embodiments of this application, data such as images and tags are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. A collaborative training method for an image classification model, characterized in that, The method includes: The image samples are divided into multiple image blocks; Based on the first image classification model, global image features are extracted from the multiple image patches, and image-level category labels are predicted based on the global image features, so as to obtain the image patch classification prediction information generated by the first image classification model in the image classification process; The second image classification model is used to classify and predict each image block, and the image-level category label is determined based on the block-level category label obtained from the classification prediction, so as to obtain the image block classification prediction information generated by the second image classification model in the image classification process. The image patch classification prediction information generated by the first image classification model is used as the predicted value, and the image patch classification prediction information generated by the second image classification model is used as the true value. The deviation between the predicted value and the true value is calculated to obtain the co-training loss value. The parameters of the first image classification model and the second image classification model are updated based on the co-training loss value. The step of classifying and predicting each image patch based on the second image classification model, and determining the image-level category label based on the patch-level category label obtained from the classification prediction, to obtain the image patch classification prediction information generated by the second image classification model during the image classification process, includes: Based on the block-level category labels corresponding to each image block, typical image blocks are selected for each category. Based on the selected image patches and the prototype vectors corresponding to each category, prototype labels are generated for each image patch. The prototype vectors corresponding to each category are applied to the predicted category, and the prototype labels for each image patch are used to characterize the category of the corresponding image patch. Based on the updated prototype labels corresponding to each image block, determine the block-level pseudo-labels corresponding to each image block; The block-level pseudo-labels corresponding to each image block are obtained as the image block classification prediction information generated by the second image classification model during the image classification process.

2. The method according to claim 1, characterized in that, The step of determining the block-level pseudo-labels corresponding to each image block based on the updated prototype labels of each image block includes: Based on the updated prototype labels corresponding to each image patch, the block-level pseudo-labels of each image patch stored in the confidence database are updated. Obtain the block-level pseudo-labels corresponding to each image patch from the updated confidence library.

3. The method according to claim 2, characterized in that, The step of selecting typical image blocks for each category based on the block-level category label corresponding to each image block includes: Based on the block-level category label corresponding to each image patch, the probability value of each image patch being predicted as each category is determined. Based on the probability values ​​of each image patch being predicted to belong to each category, typical image patches are selected for each category.

4. The method according to claim 1, characterized in that, The step of generating prototype labels for each image patch based on the selected image patch and the prototype vectors corresponding to each category includes: Update the prototype vector corresponding to each category based on the image patches selected for each category; Based on the updated prototype vectors for each category, generate prototype labels for each image patch.

5. The method according to claim 4, characterized in that, The step of updating the prototype vector corresponding to each category of the image patch selected according to each category includes: The momentum coefficient corresponding to the current training iteration number and the complementarity coefficient corresponding to the momentum coefficient are determined based on the current training iteration number. The prototype vectors corresponding to each category are updated based on the momentum coefficient, the complementarity coefficient, and the feature vectors of the image patches selected for each category.

6. The method according to claim 4, characterized in that, The step of generating prototype labels for each category based on the updated prototype vectors of each category includes: For each image patch, the product of the feature vector and the updated prototype vector for each category is calculated to obtain the category probability set corresponding to each image patch; The category with the highest probability is selected from the set of category probabilities corresponding to each image patch, and is used as the prototype category corresponding to each image patch. Based on the prototype category, vector encoding is performed to obtain the prototype label corresponding to each image patch.

7. The method according to claim 1, characterized in that, The step of extracting global image features from the multiple image patches based on the first image classification model and predicting image-level category labels based on the global image features to obtain image patch classification prediction information generated by the first image classification model during the image classification process includes: The second image classification model also outputs the prediction scores of each image patch when extracting global image features from the multiple image patches; wherein the prediction scores characterize the degree of contribution of each image patch to the global image features. The prediction scores of each image patch are obtained as the image patch classification prediction information generated by the first image classification model during the image classification process.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: Extract the embedding vector of each image patch, and extract the feature vector of each image patch based on the embedding vector of each image patch; The feature vectors of each image patch are input into the first image classification model and the second image classification model, respectively, so that the first image classification model and the second image classification model can predict the image-level category label based on the received feature vectors of multiple image patches.

9. An image classification method, characterized in that, The method includes: Obtain whole-section pathological images to be classified; The whole-section pathological image is input into a first image classification model and / or a second image classification model that have been trained by the method described in any one of claims 1-8, so as to obtain the image classification results output by the first image classification model and / or the second image classification model.

10. A collaborative training device for an image classification model, characterized in that, The device includes: The segmentation module is configured to divide image samples into multiple image blocks. The first acquisition module is configured to extract global image features from the multiple image patches based on the first image classification model, and predict image-level category labels based on the global image features, so as to obtain image patch classification prediction information generated by the first image classification model during the image classification process; The second acquisition module is configured to perform classification prediction on each image block based on the second image classification model, and determine the image-level category label based on the block-level category label obtained from the classification prediction, so as to obtain the image block classification prediction information generated by the second image classification model in the image classification process. The collaborative module is configured to use the image patch classification prediction information generated by the first image classification model as the predicted value and the image patch classification prediction information generated by the second image classification model as the true value, calculate the deviation between the predicted value and the true value to obtain a collaborative training loss value, and update the parameters of the first image classification model and the second image classification model based on the collaborative training loss value. The second acquisition module is further configured to perform the following steps: Based on the block-level category labels corresponding to each image block, typical image blocks are selected for each category. Based on the selected image patches and the prototype vectors corresponding to each category, prototype labels are generated for each image patch. The prototype vectors corresponding to each category are applied to the predicted category, and the prototype labels for each image patch are used to characterize the category of the corresponding image patch. Based on the updated prototype labels corresponding to each image block, determine the block-level pseudo-labels corresponding to each image block; The block-level pseudo-labels corresponding to each image block are obtained as the image block classification prediction information generated by the second image classification model during the image classification process.

11. An electronic device, characterized in that, include: One or more processors; A memory for storing one or more computer programs that, when executed by one or more processors, cause the electronic device to perform the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the processor of the electronic device, causes the electronic device to perform the method of any one of claims 1-9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor of the electronic device, it implements the method as described in any one of claims 1-9.

Images