Cooperative training method of image classification model, image classification method and related device

Abstract

The embodiment of the invention discloses a collaborative training method of an image classification model, an image classification method and a related device. The training method comprises the following steps: dividing an image sample into a plurality of image blocks; extracting image global features of the plurality of image blocks based on the first image classification model, and predicting an image level category label based on the image global features to obtain image block classification prediction information generated by the first image classification model in an image classification process; performing classification prediction on each image block based on a second image classification model, and determining an image level category label based on a block level category label obtained by the classification prediction so as to obtain image block classification prediction information generated by the second image classification model in an image classification process; and calculating a cooperative training loss value according to the image block classification prediction information generated by the first image classification model and the second image classification model, thereby performing parameter updating on each model based on the cooperative training loss value. According to the embodiment of the invention, the image classification accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a collaborative training method for an image classification model, an image classification method, and related devices. Background Art

[0002] Whole Slide Imaging (WSI), also known as digital pathology slides, refers to a technology that uses a data scanner to scan traditional pathology slides to acquire high-resolution digital images, and then seamlessly stitches and integrates the obtained fragmented images through a computer to produce a visual digital image. Whole-slide pathology images usually have very high resolution and are accompanied by many noise areas that are irrelevant to pathology, which makes it difficult to directly apply deep learning models to the classification prediction of whole-slide pathology images. Therefore, there is an urgent need to provide a technical solution that can accurately classify and predict whole-slide pathology images. Summary of the invention

[0003] To solve the above technical problems, the embodiments of the present application provide a collaborative training method and device for an image classification model, an image classification method and device, an electronic device, a computer-readable storage medium, and a computer program product. The first image classification model and / or the second image classification model trained by the embodiments of the present application can accurately predict the category label of the whole-slice pathology image.

[0004] According to one aspect of an embodiment of the present application, a collaborative training method for an image classification model is provided, the method comprising: dividing an image sample into a plurality of image blocks; extracting global image features from the plurality of image blocks based on a first image classification model, and predicting image-level category labels based on the image global features to obtain image block classification prediction information generated by the first image classification model during an image classification process; performing classification prediction on each image block based on a second image classification model, and determining image-level category labels based on block-level category labels obtained by classification prediction to obtain image block classification prediction information generated by the second image classification model during an image classification process; and calculating a collaborative training loss value based on the image block classification prediction information respectively generated by the first image classification model and the second image classification model to perform parameter update on the first image classification model and the second image classification model based on the collaborative training loss value.

[0005] In another exemplary embodiment, the classification prediction is performed on each image block based on the second image classification model, and the image-level category label is determined based on the block-level category label obtained by the classification prediction to obtain the image block classification prediction information generated by the second image classification model during the image classification process, including: generating block-level pseudo-labels corresponding to each image block according to the block-level category labels corresponding to each image block and the prototype vectors of each category; wherein the prototype vectors corresponding to each category act on the category predicted to belong to; and the block-level pseudo-labels corresponding to each image block are obtained as the image block classification prediction information generated by the second image classification model during the image classification process.

[0006] In another exemplary embodiment, the method of generating block-level pseudo-labels corresponding to each image block according to the block-level category labels corresponding to each image block and the prototype vectors corresponding to each category includes: selecting typical image blocks for each category according to the block-level category labels corresponding to each image block; generating prototype labels corresponding to each image block based on the selected image blocks and the prototype vectors corresponding to each category; wherein the prototype labels corresponding to each image block are used to characterize the category of the corresponding image block; and determining the block-level pseudo-labels corresponding to each image block based on the updated prototype labels corresponding to each image block.

[0007] In another exemplary embodiment, the method of determining the block-level pseudo-label corresponding to each image block based on the updated prototype label corresponding to each image block includes: updating the block-level pseudo-label of each image block stored in a confidence library based on the updated prototype label corresponding to each image block; and obtaining the block-level pseudo-label corresponding to each image block from the updated confidence library.

[0008] In another exemplary embodiment, the selecting typical image blocks for each category according to the block-level category labels corresponding to each image block includes: determining the probability value of each image block being predicted as each category according to the block-level category labels corresponding to each image block; and selecting typical image blocks for each category based on the probability value of each image block being predicted as each category.

[0009] In another exemplary embodiment, the method of generating a prototype label corresponding to each image block based on the selected image blocks and the prototype vectors corresponding to each category includes: updating the prototype vectors corresponding to each category according to the image blocks selected from each category; and generating a prototype label corresponding to each image block according to the updated prototype vectors of each category.

[0010] In another exemplary embodiment, the updating of the prototype vectors corresponding to the respective categories according to the image blocks selected from the respective categories includes: determining the momentum coefficient corresponding to the current training and the complementary coefficient corresponding to the momentum coefficient based on the number of iterations of the current training; and updating the prototype vectors corresponding to the respective categories according to the momentum coefficient, the complementary coefficient and the feature vectors of the image blocks selected from the respective categories.

[0011] In another exemplary embodiment, generating prototype labels corresponding to each category according to the updated prototype vectors of each category includes:

[0012] For each image block, the product of the feature vector and the updated prototype vector of each category is calculated to obtain the category probability set corresponding to each image block;

[0013] A category corresponding to the maximum probability is selected from the category probability set corresponding to each image block as the prototype category corresponding to each image block, and the prototype category is encoded to obtain the prototype label corresponding to each image block.

[0014] In another exemplary embodiment, the extracting of image global features from the multiple image blocks based on the first image classification model, and predicting image-level category labels based on the image global features to obtain image block classification prediction information generated by the first image classification model during the image classification process, includes: obtaining the prediction scores of each image block output by the second image classification model when extracting image global features from the multiple image blocks; wherein the prediction scores represent the degree of contribution of each image block to the image global features; and obtaining the prediction scores of each image block as the image block classification prediction information generated by the first image classification model during the image classification process.

[0015] In another exemplary embodiment, the classification prediction is performed on each image block based on the second image classification model, and the image-level category label is determined based on the block-level category label obtained by the classification prediction to obtain the image block classification prediction information generated by the second image classification model during the image classification process, including: determining the image block classification prediction information generated by the second image classification model during the image classification process based on the image-level category label determined by the second image classification model.

[0016] In another exemplary embodiment, the collaborative training loss value is calculated according to the image block classification prediction information respectively generated by the first image classification model and the second image classification model to update the parameters of the first image classification model and the second image classification model based on the collaborative training loss value, including: taking the image block classification prediction information generated according to the first image classification model as the prediction value, and taking the image block classification prediction information generated according to the second image classification model as the true value, and calculating the deviation between the prediction value and the true value to obtain the collaborative training loss value.

[0017] In another exemplary embodiment, the method further includes: extracting an embedding vector of each image block, and extracting a feature vector of each image block based on the embedding vector of each image block; inputting the feature vector of each image block into the first image classification model and the second image classification model, respectively, so that the first image classification model and the second image classification model predict image-level category labels based on the received feature vectors of multiple image blocks.

[0018] In another exemplary embodiment, the extracting of the embedding vectors of each image block and extracting the feature vectors of each image block based on the embedding vectors of each image block includes: extracting the embedding vectors of each image block through an encoding network, and extracting the feature vectors of each image block with respect to the embedding vectors of each image block through an extraction network; wherein the encoding network is in a parameter freezing state; and the method further includes: updating the parameters of the extraction network based on the training loss value.

[0019] In another aspect of an embodiment of the present application, a collaborative training device for an image classification model is provided, the device comprising: a first acquisition module, configured to extract image global features for the multiple image blocks based on the first image classification model, and predict image-level category labels based on the image global features, so as to obtain image block classification prediction information generated by the first image classification model during the image classification process; a second acquisition module, configured to perform classification prediction on each image block based on the second image classification model, and determine the image-level category label based on the block-level category label obtained by the classification prediction, so as to obtain the image block classification prediction information generated by the second image classification model during the image classification process; a collaborative module, configured to calculate a collaborative training loss value based on the image block classification prediction information respectively generated by the first image classification model and the second image classification model, so as to update the parameters of the first image classification model and the second image classification model based on the collaborative training loss value.

[0020] In another aspect of an embodiment of the present application, an image classification method is provided, the method comprising: obtaining a whole-slice pathology image to be classified; inputting the whole-slice pathology image into a first image classification model and / or a second image classification model obtained by collaborative training via any of the above methods, so as to obtain an image classification result output by the first image classification model and / or the second image classification model.

[0021] In another aspect of an embodiment of the present application, an image classification device is provided, comprising: an acquisition module configured to acquire a full-slice pathology image to be classified; and a processing module configured to input the full-slice pathology image into a first image classification model and / or a second image classification model that has been collaboratively trained by any of the collaborative training methods of the above-mentioned image classification models, so as to obtain an image classification result output by the first image classification model and / or the second image classification model.

[0022] On the other hand, an embodiment of the present application provides a training method for an image classification model, the method comprising: dividing an image sample into a plurality of image blocks; performing classification prediction on each image block based on an image classification model to obtain a block-level category label corresponding to each image block, and determining an image-level category label based on the block-level category label corresponding to each image block; generating a block-level pseudo-label corresponding to each image block according to the block-level category label corresponding to each image block and the prototype vectors of each category; wherein the prototype vectors corresponding to each category act on the category predicted to belong to; calculating a block-level training loss value based on the block-level pseudo-label and the block-level category label corresponding to each image block, so as to update the parameters of the image classification model based on the block-level training loss value.

[0023] On the other hand, an embodiment of the present application provides a training device for an image classification model, the device comprising: an image division module, configured to divide an image sample into multiple image blocks; a model processing module, configured to perform classification prediction on each image block based on the image classification model, obtain a block-level category label corresponding to each image block, and determine an image-level category label based on the block-level category label corresponding to each image block; a pseudo-label generation module, configured to generate a block-level pseudo-label corresponding to each image block according to the block-level category label corresponding to each image block and the prototype vectors of each category; wherein the prototype vectors corresponding to each category act on the category predicted to belong to; a loss calculation module, configured to calculate a block-level training loss value based on the block-level pseudo-label and block-level category label corresponding to each image block, so as to update the parameters of the image classification model based on the block-level training loss value.

[0024] Another aspect of an embodiment of the present application provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein when the one or more computer programs are executed by the one or more processors, the electronic device implements the collaborative training method of the image classification model or the image classification method as described above.

[0025] Another aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor of an electronic device, the electronic device executes the collaborative training method of the image classification model or the image classification method as described above.

[0026] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor of an electronic device, implements the collaborative training method of the image classification model or the image classification method as described above.

[0027] In the technical solution provided in the embodiment of the present application, a collaborative learning method of the first image classification model and the second image classification model is used for image classification prediction. Since the collaborative training loss value is calculated based on the image block classification prediction information generated by the first image classification model and the second image classification model respectively during the image classification process, the information at the image block level can be used to enhance the extraction of global image features by the first image classification model during the collaborative training process, thereby making the global image features extracted by the first image classification model more accurate, thereby improving the classification accuracy of the first image classification model, and making the second image classification model more accurate in predicting block-level category labels, thereby also improving the classification accuracy of the second image classification model. Therefore, the first image classification model and the second image classification model after collaborative training can both make more accurate classification predictions for high-resolution images such as full-slice pathology images.

[0028] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the implementation environment involved in this application;

[0030] Figure 2 is a flow chart of a collaborative training method of an image classification model shown in an exemplary embodiment of the present application;

[0031] Figure 3 is a flowchart of a collaborative training method for an image classification model shown in another exemplary embodiment of the present application;

[0032] Figure 4is a flowchart of a collaborative training method for an image classification model shown in another exemplary embodiment of the present application;

[0033] Figure 5 is a schematic diagram of an exemplary collaborative training process of an image classification model;

[0034] Figure 6 yes Figure 5 A schematic diagram of the processing flow of the tag disambiguation module in an exemplary embodiment is shown;

[0035] Figure 7 is a flow chart of an image classification method shown in an exemplary embodiment;

[0036] Figure 8 is a flowchart of an exemplary method for training an image classification model;

[0037] Fig. 9 is a block diagram of a collaborative training device for an image classification model shown in an exemplary embodiment;

[0038] Fig.10 is a block diagram of an image classification device shown in an exemplary embodiment;

[0039] Fig.11 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0040] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0041] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0042] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0043] The term "multiple" as used in this application refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0044] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. The terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0045] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the function of the module or unit.

[0046] See first Figure 1 , Figure 1 1 is a schematic diagram of an implementation environment involved in the present application. The implementation environment is specifically an image classification system, including a terminal 110 and a server 120, and the terminal 110 and the server 120 communicate with each other in a wired or wireless manner.

[0047] Terminal 110 is used to acquire digital images with high resolution. Exemplarily, the digital images with high resolution may be medical images such as whole slice pathology images (WSI), computed tomography (CT) images, and magnetic resonance images (MRI). They may also be digital images in other fields such as remote sensing images, satellite images, electron microscope images, and high-precision scanning images. This embodiment is not limited to this.

[0048] The terminal 110 can use a digital image with high resolution as an image to be classified and input it into an image classification model for classification prediction. For example, in some application scenarios, a model to be trained is deployed in the server 120, and the terminal 110 obtains a large number of digital images with high resolution accordingly, and uses them as training sample sets. The server 120 uses these training sample sets to perform training processing of the model to be trained, thereby obtaining a trained image classification model. For another example, in other application scenarios, a trained image classification model is deployed in the server 120, and the terminal 110 uploads the image to be classified to the server 120, so that the server 120 uses the trained image classification model to predict the category label of the image to be classified, and returns the predicted category label to the terminal 110, so that the terminal 110 can perform display and other processing for the obtained category label.

[0049] It should be noted that the terminal 110 can be a smart phone, a tablet, a laptop, a computer, a smart home appliance, a smart terminal, etc., which is not limited here. The server 220 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services, which is not limited here.

[0050] Taking whole-slice pathology images (WSI) as an example, whole-slice pathology images usually have very high resolution and are accompanied by many noise areas that are irrelevant to pathology, which makes it difficult to directly apply deep learning models to classification predictions of whole-slice pathology images.

[0051] In the related art, the whole-slice pathology image is usually divided into many smaller image patches, and the image classification results corresponding to the whole-slice pathology image are obtained by processing the image patches. The related art regards the analysis of the whole-slice pathology image as a multi-instance learning problem. In the multi-instance learning framework, the whole image is regarded as a package, and the extracted image patches are regarded as instances. Specific learning strategies are applied to the instances to obtain the comprehensive feature vector of the image, and then classification is performed.

[0052] There are two main types of multi-instance learning strategies currently in existence: one is the package-level multi-instance learning strategy, and the other is the instance-level multi-instance learning strategy. The package-level multi-instance learning strategy first extracts the feature vector of the instance, and then aggregates it into a package-level feature vector for classification prediction. During the training process, the package-level image classification model is supervised by the package-level label to achieve fully supervised learning. The instance-level multi-instance learning strategy focuses the learning process on the instance level, obtains the instance-level prediction results by training the instance classifier, and then simply aggregates the instance-level prediction results to obtain the package-level prediction results.

[0053] These two multi-instance learning strategies have their own advantages and disadvantages. For example, the package-level multi-instance learning strategy only needs to provide the category label at the package level, does not need to annotate each instance, and focuses more on capturing the global features of the package, which is more suitable for complex scenarios. However, since the model may ignore some important instance-level information during the learning process, and since the number of samples of full-slice pathology images is often limited, this may cause insufficient model training, thus falling into the risk of sub-optimization, resulting in low prediction accuracy of the model. The instance-level multi-instance learning strategy requires annotating each image patch, which can capture more detailed information and help improve the prediction accuracy of the model, but it also faces the problems of high annotation cost and high annotation noise, and the problem of high annotation noise will still reduce the prediction accuracy of the model.

[0054] Based on this, the inventor of this application has uniquely come up with a collaborative instance-level and package-level multi-instance learning strategy for classification prediction of high-resolution images such as full-slice pathology images. This collaborative learning strategy can integrate the advantages of instance-level and package-level learning strategies, while making up for the shortcomings of each learning strategy, thereby enabling more accurate classification prediction of high-resolution images such as full-slice pathology images.

[0055] The collaborative instance-level and package-level multi-instance learning strategy proposed in this application can be summarized as follows:

[0056] First, the image sample is divided into multiple image blocks; then, based on the first image classification model, the image global features are extracted from the multiple image blocks, and the image-level category labels are predicted based on the image global features to obtain the image block classification prediction information generated by the first image classification model during the image classification process; and based on the second image classification model, classification prediction is performed on each image block, and the image-level category labels are determined based on the block-level category labels obtained by the classification prediction to obtain the image block classification prediction information generated by the second image classification model during the image classification process; then, the collaborative training loss value is calculated according to the image block classification prediction information generated by the first image classification model and the second image classification model respectively, so as to update the parameters of the first image classification model and the second image classification model based on the collaborative training loss value.

[0057] In the collaborative instance-level and package-level multi-instance learning strategies proposed in the present application, the process of the first image classification model predicting image-level category labels is set according to the package-level multi-instance learning strategy, and the process of the second image classification model predicting image-level category labels is set according to the instance-level multi-instance learning strategy. Since the collaborative training loss value is calculated based on the image block classification prediction information generated by the first image classification model and the second image classification model respectively during the image classification process, the image block-level information can be used to enhance the extraction of global image features by the first image classification model during the collaborative training process, thereby making the global image features extracted by the first image classification model more accurate, thereby improving the classification accuracy of the first image classification model, and making the second image classification model more accurate in predicting block-level category labels, thereby improving the classification accuracy of the second image classification model.

[0058] Therefore, the first image classification model and the second image classification model trained based on the multi-instance learning strategy proposed in this application can perform more accurate classification prediction for high-resolution images such as whole-slice pathology images.

[0059] It should be noted that in some exemplary application scenarios, the multi-instance learning strategy proposed in this application may regard image samples as packages, image blocks as instances, image-level category labels as package-level labels, and image block-level category labels as instance-level labels. In other exemplary application scenarios, the image samples, image blocks, image-level category labels, and image block-level category labels involved in the multi-instance learning strategy proposed in this application may also be independent of packages and instances.

[0060] The following is a detailed introduction to the multi-instance learning strategy proposed in this application.

[0061] First see Figure 2 , Figure 2 is a flowchart of a collaborative training method for an image classification model shown in an exemplary embodiment of the present application. The method can be applied to Figure 1 The implementation environment shown, for example, can be specifically executed by the server 120, can also be specifically executed by the terminal 110, or can be jointly executed by the terminal 110 and the server 120. Of course, the method can also be applied to other implementation environments, and can be executed by a terminal or a server in other implementation environments, or can be jointly executed by a terminal and a server in other implementation environments, and this embodiment is not limited thereto.

[0062] like Figure 2 As shown, in an exemplary embodiment, the training method of the image classification model includes S210-S240, which are described in detail as follows:

[0063] S210, dividing the image sample into a plurality of image blocks.

[0064] Considering that image samples usually have a high resolution, it is very difficult to directly perform classification prediction on the entire image. Therefore, it is necessary to divide the image sample into multiple image blocks, and then predict the category label of the image sample based on these image blocks. It can also be understood that the image sample is an image composed of multiple image blocks. Therefore, the image sample can be divided into image blocks according to the unit size corresponding to the image block, so as to obtain the multiple image blocks contained in the image sample accordingly.

[0065] It should also be noted that the image sample can be a two-dimensional image or a three-dimensional image, which is not limited here. For example, a three-dimensional full-slice pathological image is a three-dimensional image constructed based on scanning the pathological tissue layer by layer. When the image sample is a three-dimensional image, each image block divided for the image sample is usually also a three-dimensional image block.

[0066] S220, extracting global image features from multiple image blocks based on the first image classification model, and predicting image-level category labels based on the global image features to obtain image block classification prediction information generated by the first image classification model during the image classification process.

[0067] The global image feature refers to the image feature corresponding to the image sample, which is used to describe the feature information on the entire image sample. It can also be understood as describing the global feature information of the image sample.

[0068] The first image classification model consists of an image-level aggregator and an image-level classifier. The image-level aggregator is used to extract the image global features corresponding to the image sample from the multiple image blocks contained in the image sample, and the image-level classifier is used to predict the category label of the image sample based on the image global features. The category label of the image sample is called the image-level category label.

[0069] The image-level aggregator performs feature aggregation processing on the feature vectors of multiple image blocks to obtain the global image features corresponding to the image sample. For example, the feature vectors of each image block can be extracted through a feature extraction network. For another example, the feature extraction network can be composed of an encoding network and an extraction network, the encoding network is used to extract the embedding vectors of each image block, and the extraction network is used to extract the feature vectors of each image block according to the embedding vectors of each image block. It can be understood that the embedding vector of each image block refers to the vector information obtained by the encoding network mapping high-dimensional data to a low-dimensional space for the image block, and the feature vector of each image block refers to the embedding vector of each image block further represented by the extraction network as a vector processed by the image classification model. During the collaborative training process, the encoding network can be set to a parameter freezing state, that is, the parameters of the encoding network will not be updated to reduce the computing resources required for training, thereby improving the training efficiency to a certain extent; the extraction network can update parameters based on the training loss value to optimize the extraction network for the feature vector extraction process of each image block, which will also help to improve the prediction accuracy of the first image classification model and the second image classification model to a certain extent.

[0070] Exemplarily, the image-level aggregator and image-level classifier can be implemented by convolutional neural networks (CNN), transformer models, graph neural networks (GNN), deep neural networks (DNN), recurrent neural networks (RNN), etc., without limitation herein.

[0071] Image block classification prediction information refers to relevant information used to predict the category label of each image block. The category label of the image block is also called the block-level category label. As an exemplary implementation, the block-level category label corresponding to each image block can be determined based on the image-level category label output by the first image classification model, so that the block-level category label corresponding to each image block can be used as the image block classification prediction information generated by the first image classification model during the image classification process. The block-level category label can be the same as the image-level category label or different, and is not limited here.

[0072] As another exemplary embodiment, the image-level aggregator extracts the global features of the image from the feature vectors of multiple image blocks and also outputs the prediction score of each image block, which represents the contribution of each image block to the global features of the image. For example, the image-level aggregator contains an attention module, and the attention score obtained by the attention module is the prediction score of each image block.

[0073] S230, performing classification prediction on each image block based on the second image classification model, and determining the image-level category label based on the block-level category label obtained by the classification prediction, so as to obtain the image block classification prediction information generated by the second image classification model during the image classification process.

[0074] The second image classification model consists of a block-level classifier and a block-level aggregator. The block-level classifier is used to predict the block-level category label corresponding to each image block based on the feature vector of each image block. The block-level aggregator is used to aggregate the block-level category labels corresponding to each image block to obtain the image-level category label corresponding to the image sample. The block-level classifier can be implemented based on a convolutional neural network, a VGG (Visual Geometry Group) network, or some lightweight image classification network, etc., which is not limited here.

[0075] Exemplarily, the block-level category labels corresponding to each image block output by the block-level classifier can be used as the image block classification prediction information generated by the second image classification model during the image classification process, and the block-level category labels corresponding to each image block determined according to the image-level category labels obtained by the second image classification model can be used as the image block classification prediction information generated by the second image classification model during the image classification process. The block-level category label can be the same as the image-level category label or different, and this embodiment does not limit this.

[0076] S240, calculating a collaborative training loss value according to the image block classification prediction information respectively generated by the first image classification model and the second image classification model, so as to update the parameters of the first image classification model and the second image classification model based on the collaborative training loss value.

[0077] Taking into account that the first image classification model does not actually predict the block-level category labels corresponding to each image block during the image classification process, this embodiment uses the image block classification prediction information generated by the first image classification model as the prediction value, and uses the image block classification prediction information generated by the second image classification model as the true value, and calculates the deviation between the predicted value and the true value to obtain the collaborative training loss value.

[0078] It should be noted that the method of updating the parameters of the first image classification model and the second image classification model based on the collaborative training loss value in this embodiment includes but is not limited to the gradient descent method, which is not limited here.

[0079] It can be obtained that this embodiment calculates the collaborative training loss value according to the image block classification prediction information respectively generated by the first image classification model and the second image classification model, so that the collaborative training process uses the loss information at the image block level to update the model parameters. In the process of updating the model parameters, based on the collaborative training loss, the first image classification model can enhance the extraction of global image features, thereby making the global image features extracted by the first image classification model more accurate, thereby improving the classification accuracy of the first image classification model, and can also make the second image classification model more accurate in predicting block-level category labels, thereby also improving the classification accuracy of the second image classification model. It can also be understood that in the collaborative training process proposed in this embodiment, block-level information is used to improve the quality of image-level information, thereby improving the prediction accuracy of the image classification model.

[0080] It should also be noted that, in some exemplary embodiments, on the basis of maintaining collaborative training, the first image classification model and / or the second image classification model can also be trained independently to further improve the training efficiency. For example, for the first image classification model, the image-level annotation label corresponding to the image sample is taken as the true value, the image-level category label output by the first image classification model is taken as the predicted value, and the deviation between the predicted value and the true value is calculated to obtain the image-level training loss value, thereby updating the model parameters of the first image classification model based on the image-level training loss value. Similarly, for the second image classification model, the block-level annotation label corresponding to each image block is determined according to the image-level annotation label corresponding to the image sample, for example, the block-level annotation label corresponding to each image block is the same as the image-level annotation label corresponding to the image sample, the block-level annotation label corresponding to each image block is taken as the true value, the block-level category label corresponding to each image block output by the block-level classifier is taken as the predicted value, and the deviation between the predicted value and the true value is calculated to obtain the block-level training loss value, thereby updating the model parameters of the second image classification model based on the block-level training loss value.

[0081] In some embodiments, when all independent training and collaborative training coexist, the sum of the collaborative training loss value, the image-level training loss value, and the block-level training loss value is taken as the total training loss value, so that model training is performed based on the total training loss value.

[0082] In another exemplary embodiment, for the second image classification model, considering that the cost of labeling image blocks is high, block-level annotation labels are usually lacking in actual training scenarios, which in turn affects training performance.

[0083] In the instance-level multi-instance learning strategy, in order to solve the problem of lack of instance labels, a technical means is adopted to assign pseudo labels to each instance according to the package-level labels, and then supervise the prediction process of instance labels based on pseudo labels, thereby improving training performance. For example, a conventional pseudo-label generation scheme is that the pseudo-label is consistent with the package label.

[0084] This embodiment further takes into account that the noise of block-level annotation labels is usually high, which will also affect the prediction accuracy of the block-level classifier for the block-level category labels. Therefore, during the collaborative training process, this embodiment further reduces the noise in the block-level pseudo-labels to improve the collaborative training performance.

[0085] The technical solution proposed in this embodiment will be introduced in detail below.

[0086] See also Figure 3 , Figure 3 FIG. 1 is a flowchart of a collaborative training method for an image classification model shown in another exemplary embodiment of the present application. Figure 2 On the basis of the illustrated embodiment, it is proposed that S230 performs classification prediction on each image block based on the second image classification model, and determines the image level category label based on the block level category label obtained by the classification prediction, so as to obtain the image block classification prediction information generated by the second image classification model in the image classification process, and the process further includes the following steps:

[0087] S310, generating a block-level pseudo label corresponding to each image block according to the block-level category label corresponding to each image block and the prototype vector corresponding to each category;

[0088] S320, obtaining the block-level pseudo-label corresponding to each image block as image block classification prediction information generated by the second image classification model during the image classification process.

[0089] First of all, it is important to understand that a pseudo label, also known as a soft label, is a probability distribution that indicates the probability that a sample belongs to each category. In classification tasks, a pseudo label is usually a vector containing probability values. When there is noise or uncertainty in the data, the pseudo label can reflect the noise or uncertainty, so using a pseudo label can often help to more accurately predict the probability of a sample for each category.

[0090] In this embodiment, a block-level pseudo-label is provided for each image block included in the training sample, and a confidence library is provided to store the block-level pseudo-label corresponding to each training sample. Subsequently, by obtaining the block-level pseudo-label corresponding to each image block as the image block classification prediction information generated by the second image classification model during the image classification process, participating in the calculation of the collaborative training loss value, and updating the parameters of the image classification model participating in the collaborative training based on the collaborative training loss value, the prediction accuracy of the block-level classifier for the block-level category label corresponding to each image block can be promoted, and the accuracy of the image-level aggregator in extracting the global features of the image can be promoted, thereby further improving the prediction accuracy of the image-level category label.

[0091] This embodiment also configures a prototype vector for each category, and the prototype vector corresponding to each category acts on the predicted category. It can also be understood that the prototype vector corresponding to each category describes the important information corresponding to the category, and the prototype vector is the vectorization of the important information corresponding to the category.

[0092] It should be noted that the categories corresponding to the image-level category labels should be consistent with the categories corresponding to the block-level category labels. Assume that in a binary classification scenario, the categories corresponding to the image-level category labels include category 0 and category 1, and the categories corresponding to the block-level category labels also include category 0 and category 1. To make it easier to understand, for example, a certain image-level category label represents that the probability that an image sample belongs to category 0 and category 1 is 60% and 40%, and a certain block-level category label represents that the probability that an image block belongs to category 0 and category 1 is 65% and 35%. It can be seen that both labels correspond to category 1 and category 2.

[0093] It should also be noted that the prototype vectors corresponding to each category will be updated as the training process progresses. The initial prototype vectors can be preset or generated based on the content related to each category obtained during a specific training phase. For example, the initial prototype vectors for each category can be generated based on the training results of the first 10 generations to promote more accurate recognition of each category based on the prototype vectors.

[0094] The process of generating block-level pseudo-labels corresponding to each image block in this embodiment is not only related to the block-level category labels corresponding to each image block predicted by the second image classification model, but also related to the prototype vectors corresponding to each category. Based on the block-level category labels corresponding to each image block and the prototype vectors corresponding to each category, the noise in the generated block-level pseudo-labels is reduced, thereby improving the collaborative training performance.

[0095] Please continue reading Figure 4 , Figure 4 FIG. 1 is a flowchart of a collaborative training method of an image classification model shown in another exemplary embodiment of the present application. Figure 4As shown, the process of generating a block-level pseudo label corresponding to each image block according to the block-level category label corresponding to each image block and the prototype vector corresponding to each category in S310 includes the following steps:

[0096] S410, selecting typical image blocks for each category according to the block-level category labels corresponding to each image block;

[0097] S420, based on the selected image blocks and the prototype vectors corresponding to the categories, generating prototype labels corresponding to the image blocks; wherein the prototype labels corresponding to the image blocks are used to represent the categories of the corresponding image blocks;

[0098] S430: Determine a block-level pseudo label corresponding to each image block based on the updated prototype label corresponding to each image block.

[0099] It can be understood that the process of generating block-level pseudo labels in this embodiment can be summarized as including three important stages: selecting typical image blocks, generating prototype labels, and updating confidence. S410 corresponds to the stage of selecting typical image blocks, S420 corresponds to the stage of generating prototype labels, and S430 corresponds to the stage of updating the confidence library.

[0100] In S410, since the block-level category label includes the probability values ​​of the image blocks belonging to each category, the probability values ​​of each image block being predicted as each category can be determined according to the block-level category labels corresponding to each image block, and then based on the probability values ​​of each image block being predicted as each category, typical image blocks are selected for each category.

[0101] A typical image block can be understood as an image block with a high degree of prediction contribution to an image sample. As an exemplary implementation, for each category, the probability values ​​corresponding to each image block are sorted from large to small, and the image block corresponding to the probability value ranked as K is selected, where K is a positive integer. Therefore, K means that for each image sample, K typical images need to be selected for each category, so that K*O typical image blocks can be obtained for each category. These K*O typical image blocks can be further simplified later, which is not limited here.

[0102] As another exemplary implementation, considering that model training is usually performed in batches for a large number of image samples, for each category, the probability values ​​corresponding to the image blocks contained in the multiple image samples may be sorted from large to small, and the image blocks corresponding to the probability values ​​ranked as K are selected as typical image blocks. Thus, K typical image blocks may be selected for each category.

[0103] Of course, other methods may also be used to select a preset number of image blocks for each category, and this embodiment does not limit this. However, it should be noted that the prediction value of the selected image block for the corresponding category should be a larger value, so that the prototype label generated by S420 based on the selected image block is closer to the important information corresponding to the category, and then the block-level pseudo-label generated by S430 can more accurately reflect the true category of each image block, thereby further promoting the prediction accuracy of the block-level classification label. Compared with directly determining the block-level pseudo-label based on the image-level annotation label, this embodiment can further reduce the noise in the block-level pseudo-label and generate more accurate block-level pseudo-label to improve the collaborative training performance.

[0104] In S420, based on the typical image blocks corresponding to the respective categories selected in S410 and the prototype vectors corresponding to the respective categories, the process of generating the prototype labels corresponding to the respective image blocks includes updating the prototype vectors corresponding to the respective categories according to the image blocks selected by the categories, and generating the prototype labels corresponding to the respective image blocks according to the updated prototype vectors of the respective categories.

[0105] Exemplarily, the updating of the prototype vector can be performed based on the momentum updating rule to achieve the advantages of accelerating model convergence, improving model stability, optimizing resource utilization, etc. For example, the momentum coefficient corresponding to the current training and the complementary coefficient corresponding to the momentum coefficient are first determined based on the number of iterations of the current training, and then the prototype vector corresponding to each category is updated according to the determined momentum coefficient, complementary coefficient and the feature vector of the image block selected from each category. The complementary coefficient corresponding to the momentum coefficient refers to the difference between 1 and the momentum coefficient. Exemplarily, for each category, the first product between the momentum coefficient and the prototype vector before the update is calculated, and the second product between the complementary coefficient and the feature vector of the selected typical image block is calculated, and the sum of the first product and the second product is used as the updated prototype vector.

[0106] In some exemplary embodiments, the momentum coefficient is related to the number of iterations of the current training. During the entire training process, the momentum coefficient gradually decreases within the momentum coefficient variation range as the number of training increases. Exemplarily, the preset momentum coefficient variation range has a starting endpoint value and an ending endpoint value, and the starting endpoint value is greater than the ending endpoint value. First, the difference between the ending endpoint value and the starting endpoint value is calculated, and then the product of the total number of iterations and the difference is calculated, and then the quotient of the iteration coefficient of the current training and the product is calculated, and the sum of the quotient and the starting endpoint value is used as the momentum coefficient corresponding to the current training. It should be noted that the momentum coefficient should be a larger value, that is, the momentum coefficient should be a small value closer to 1 to ensure training efficiency. Therefore, the starting endpoint value and the ending endpoint value corresponding to the momentum coefficient variation range should also be closer to 1. For example, the starting endpoint value can be 0.95 and the ending endpoint value can be 0.8, but it does not represent a numerical limit for the starting endpoint value and the ending endpoint value.

[0107] In other exemplary embodiments, the momentum coefficient may also be a preset value, and a small value closer to 1, which is not limited in this embodiment.

[0108] Exemplarily, the product of the feature vector and the updated prototype vector of each category is calculated for each image block to obtain a category probability set corresponding to each image block, and then the category corresponding to the maximum probability is selected from the category probability set corresponding to each image block as the prototype category corresponding to each image block, and vector encoding is performed based on the prototype category corresponding to each image block to obtain the prototype label corresponding to each image block. The vector encoding method may include One-Hot encoding and other similar encoding methods, which are not limited here.

[0109] In S430, the block-level pseudo-labels corresponding to each image block are obtained based on the updated prototype labels corresponding to each image block, so that the block-level pseudo-labels corresponding to each image block can more accurately reflect the true category of each image block, thereby improving the accuracy of the block-level pseudo-labels and further improving the performance of collaborative training.

[0110] In the case where a confidence library is provided, the block-level pseudo-labels of each image block stored in the confidence library are updated based on the updated prototype labels corresponding to each image block, so as to obtain the block-level pseudo-labels corresponding to each image block from the updated confidence library.

[0111] In some exemplary embodiments, the confidence library can be updated based on the momentum update rule, which can correspondingly achieve the advantages of accelerating model convergence, improving model stability, optimizing resource utilization, etc. For example, the momentum coefficient corresponding to the current training and the complementary coefficient corresponding to the momentum coefficient are determined based on the number of iterations of the current training, and then the block-level pseudo-label corresponding to each image block is obtained according to the determined momentum coefficient, complementary coefficient and the prototype label corresponding to each image block. Exemplarily, for each image block, the third product between the momentum coefficient and the block-level pseudo-vector before the update is calculated, and the fourth product between the complementary coefficient and the generated prototype label is calculated, and the sum of the third product and the fourth product is used as the block-level pseudo-label.

[0112] In other exemplary embodiments, the momentum coefficient is also related to the number of iterations of the current training. During the entire training process, the momentum coefficient gradually decreases within the momentum coefficient variation range as the number of training increases. The update strategy of the momentum coefficient can refer to the process recorded in the embodiment of updating the prototype vector, which will not be described in detail here.

[0113] In other exemplary embodiments, the momentum coefficient may also adopt a preset fixed value. It should be noted that the momentum coefficient should be a larger value, that is, the momentum coefficient should be a small value closer to 1 to ensure training efficiency. For example, the momentum coefficient may be 0.99, but this does not represent a numerical limit for the momentum coefficient.

[0114] Therefore, the block-level pseudo-labels obtained in this embodiment can more accurately reflect the noise information, and thus can further promote the prediction accuracy of the block-level classification labels, thereby further reducing the noise in the block-level pseudo-labels to improve the collaborative training performance.

[0115] In order to further facilitate the understanding of the collaborative training scheme proposed in the embodiment of the present application, an exemplary training scenario is described in detail below.

[0116] See also Figure 5 , Figure 5 This is a schematic diagram of an exemplary collaborative training process of an image classification model. In actual training scenarios, it is often necessary to train the model in batches using a large number of image samples. Figure 5 The training sample set S shown in FIG. 1 includes multiple image samples. If it is assumed that the training sample set S includes N image samples, the training sample set S can be expressed as: S = {S1, S2, …, S i ,…,S N}, where S i represents the i-th sample image, 1≤i≤N.

[0117] Each image sample corresponds to an image-level annotation label, which can be expressed as Y i∈{1,…,C}, where C represents the number of categories. For example, if it is a binary classification scenario, C=2, which means that the image-level annotation label contains two categories.

[0118] Considering each image sample as consisting of multiple image tiles, each image sample is represented as: Among them, M i represents the number of image blocks contained in the i-th sample image, that is, each sample image contains M image blocks. j represents the j-th image block, 1≤j≤M. Figure 5 The S indicated in i,j Represents the jth image block contained in the i-th image sample.

[0119] The block-level annotation label of each image block in the image sample is unknown. At the beginning of training, the image-level annotation label of the image sample is assigned to each image block contained in the image sample as a block-level pseudo label. With continuous training, the block-level pseudo label will be continuously updated. For detailed update strategies, please refer to the records in the following text, which will not be repeated here.

[0120] like Figure 5 As shown, the system training process illustrated in this embodiment consists of a feature extraction process and a collaborative supervision training process, which will be described in detail below:

[0121] 1. Feature extraction process

[0122] The image pre-training model is used to encode the input image block to obtain the embedding vector corresponding to the image block, and the feature vector corresponding to each image block is extracted based on the embedding vector corresponding to each image block. Therefore, the image pre-training model can be regarded as consisting of an encoding network and an extraction network, where the encoding network is used to extract the embedding vector and the extraction network is used to extract the feature vector. The encoding network can be in a parameter-freezing state and does not participate in the training and updating of the model parameters. The extraction network can participate in the training and updating of the model parameters.

[0123] The extraction network can adopt a residual Transformer backbone network, which combines the advantages of the Transformer network and residual connections. In the process of processing the embedding vector of the image block, it can capture the context information and the correlation between the image blocks. The residual Transformer backbone network can be formed by connecting multiple Transformer blocks in series through residual blocks. The use of residual blocks helps the propagation of gradients in deep networks to alleviate the problem of gradient vanishing or gradient exploding when training deep networks. Each Transformer block can contain 5 linear blocks and at least 1 Transformer layer. The residual block can consist of 1 linear layer and 1 activation layer.

[0124] The extraction network may also adopt various other image feature extraction pre-trained models, such as the Efficient-Net model pre-trained based on natural images, the Ctranspath and UNI models pre-trained based on pathological images, etc., which are not limited here.

[0125] Exemplarily, the process of extracting a feature vector from an embedding vector of an image block by the extraction network is expressed as: i,j =T(e i,j ), where e i,j represents the embedding vector of the jth image block in the i-th image sample, x i,j represents the feature vector of the jth image block in the ith image sample, and T represents the function that maps the embedding vector of the image block to the feature vector of the image block.

[0126] 2. Collaborative supervision training process

[0127] The collaborative supervision training process mainly consists of three parts: block-level supervision, image-level supervision and image-block-level collaborative supervision.

[0128] It should be noted here that the first image classification model and the second image model participating in the collaborative training are Figure 5 It is not explicitly illustrated, but the composition structure of the first image classification model is illustrated, that is, it includes an image-level aggregator and an image-level classifier, and the block-level classifier included in the second image classification model is illustrated.

[0129] 2.1 Block-level supervision

[0130] The feature vector of each image block is first input into the block-level classifier to obtain the block-level category label corresponding to each image block. If the classification prediction process performed by the block-level classifier is represented as a mapping function F, for the feature vector x of the jth image block in the i-th image sample, i,j , and obtain the corresponding block-level category label The process can be expressed as:

[0131] Then, the feature vectors of each image block and the block-level category labels corresponding to each image block are input into the label disambiguation module. The label disambiguation module aims to reduce the noise contained in the block-level pseudo-labels, so as to obtain more accurate block-level labels. More accurate block-level pseudo-labels are understood as the block-level pseudo-labels of each image block obtained by the label disambiguation module are closer to the true categories of each image block.

[0132] The label disambiguation module generates block-level pseudo labels corresponding to each image block by generating prototype labels of each image block and accessing the stored confidence library. The prototype labels of each image block are used to represent the category of each image block, and the confidence library is used to store the block-level pseudo labels of each image block.

[0133] See also Figure 6 , Figure 6 yes Figure 5 FIG. 1 is a schematic diagram of a processing flow of a tag disambiguation module in an exemplary embodiment. Figure 6 As shown in the figure, the label disambiguation module mainly includes three parts: selecting typical image blocks, generating prototype labels, and updating the confidence library.

[0134] 2.11 Selecting Typical Image Blocks

[0135] In order to focus the image classification model on image patches with high prediction contribution to the image sample, it is first necessary to select typical image patches to update the prototype information through typical image patches. Specifically, the feature vector x of the jth image patch in the i-th image sample is i,j The probability of being predicted as category c is expressed as At time t (i.e., the current time), a set of image block feature vectors with the highest probability belonging to category c are selected as feature vectors of typical image blocks. The process of selecting typical image blocks can be expressed as follows:

[0136]

[0137] Among them, Set c,t represents the feature vector of the typical image block selected at time t, K represents the number of feature vectors of image blocks with high prediction values ​​to be selected, argTop K Represents a function for selecting K image block feature vectors with high prediction values.

[0138] Thus, K image block feature vectors with high prediction values ​​are selected for each category to obtain feature vectors of K typical image blocks corresponding to each category.

[0139] 2.12 Generate prototype tags

[0140] In this process, according to the feature vector Set of the selected image block c,t , and the feature vectors corresponding to each category, generate the prototype label of each image block.

[0141] Specifically, at time t-1, if the prototype vector corresponding to category c is P c,t-1 After obtaining K typical image block feature vectors, the momentum-based update rule is used to update the prototype vector to obtain the prototype vector corresponding to category c at time t as Pc,t The updating process of the prototype vector can be expressed as follows:

[0142] P c,t =α·P c,t-1 +(1-α)·x i,j , if x i,j ∈Set c,t

[0143] Wherein, α represents the momentum coefficient, which automatically decreases from α=0.95 to α=0.8 at each moment. It can be understood that during the training process, the momentum coefficient will change within the momentum coefficient variation range, and the momentum coefficient variation range is, for example, [0.95, 0.8].

[0144] The change process of momentum coefficient α can be expressed as follows:

[0145]

[0146] Among them, epoch represents the number of iterations corresponding to the current training, epochs represents the total number of iterations, end represents the end value of the momentum coefficient change range, and start represents the start value of the momentum coefficient change range. For example, start = 0.95, end = 0.8.

[0147] For each image block, the updated prototype vectors of each category are multiplied by their feature vectors to obtain the corresponding products, which represent the probability that the image block belongs to the corresponding category. Therefore, each image block will obtain a category probability set. Then, for each image block, the category with the maximum probability is selected from its corresponding category probability set as the prototype category corresponding to the corresponding image block. Based on the vector encoding of the prototype category of each image block, the prototype label corresponding to each image block is obtained.

[0148] The process of updating the prototype tag can be expressed as follows:

[0149] z i,j =OneHot(argmax c (P c,t ·x i,j ))

[0150] Among them, z i,j Represents the prototype label of the jth image patch in the i-th image sample, which is a one-hot vector, argmax c (P c,t ·x i,j ) means that for the jth image block in the i-th image sample, the prototype vector and its feature vector x of each category are calculated. i,j The product of , and selects the category corresponding to the maximum product as the prototype category of the image block.

[0151] The block-level pseudo-label of each image block is determined based on the prototype label of each image block. The prototype label of each image block is generated according to the prototype vector of each category, so that the prototype label of each image block can be closer to the real category of each image block. Compared with directly generating block-level pseudo-labels according to the image-level annotation labels of image samples, the noise of the block-level pseudo-labels of each image block can be smaller.

[0152] 2.13 Update confidence library

[0153] After obtaining the prototype label at time t, the confidence library is also updated through the momentum update rule. The update process is expressed as follows:

[0154] B i,j,t =β·B i,j,t-1 +(1-β)·z i,j

[0155] Where β represents the momentum coefficient, and its default value is 0.99; B i,j,t represents the block-level pseudo-label of the j-th image block in the i-th image sample at time t, which is also the updated block-level pseudo-label in the confidence library; B i,j,t-1 It represents the block-level pseudo-label of the j-th image block in the i-th image sample at time t-1, that is, the block-level pseudo-label currently stored in the confidence library, and can also be understood as the block-level pseudo-label to be updated in the confidence library.

[0156] In this way, the block-level pseudo-label corresponding to each image block can be obtained from the updated confidence library. Moreover, the process of generating block-level pseudo-labels does not need to rely on the label annotation information of the image block, but is calculated based on the feature vector and prototype vector of the image block. Therefore, it can effectively solve the problem of lack of block-level annotation information.

[0157] Using block-level pseudo-labels to supervise model learning can help the model learn more accurate and robust block-level representations.

[0158] In addition, the block-level supervision part can also perform gradient update independently. After obtaining the corresponding block-level training loss value for the block-level supervision part, the second image classification model is updated based on the block-level training loss value. For example, the block-level training loss value is calculated by the following cross entropy function:

[0159]

[0160] Among them, L b represents the block-level training loss value, B i,j,c and are the cth component in the block-level pseudo-label and the cth component in the block-level category label of the jth image block in the ith image sample, respectively. N represents the number of image samples, M represents the number of image blocks contained in the image sample, and C represents the number of categories.

[0161] 2.2 Image-level Supervision

[0162] In the image-level supervision process, the feature vectors of each image block contained in each image sample are first aggregated through the image-level aggregator to obtain the global image features and attention scores of each image sample. The attention score indicates the contribution of each image block to the corresponding image features. The processing of the image-level aggregator can be expressed as the following formula:

[0163]

[0164] Among them, f i represents the global image feature of the i-th image sample, a i,j represents the attention score of the jth image block in the i-th image sample, x i,j Represents the feature vector of the jth image block in the i-th image sample.

[0165] The global features of the image are then used to predict the image-level category labels through an image-level classifier. The prediction process can be expressed as follows:

[0166]

[0167] in, represents the image-level category label of the i-th image sample, F s represents the image-level classifier, f i represents the global image feature of the i-th image sample, and softmax represents the normalization function.

[0168] The whole process is expressed as:

[0169] Similarly, the image-level supervision part can also perform gradient update independently. After obtaining the corresponding image-level training loss value for the image-level supervision part, the first image classification model is updated based on the image-level training loss value. For example, the image-level training loss value is calculated by the following cross entropy function:

[0170]

[0171] Among them, L S represents the image-level training loss value, N represents the number of image samples, represents the image-level category label of the i-th image sample, Y i Represents the image-level annotation label of the i-th image sample.

[0172] 2.3 Image-Block-Level Collaborative Supervision

[0173] Co-supervision combines the image-level and patch-level to promote better interaction between them. Specifically, we use the attention score a output by the image-level classifier i,j As the prediction value, the block-level pseudo label B obtained by the label disambiguation module i,j As the true value. The calculation formula of the collaborative training loss value is as follows:

[0174]

[0175] Among them, L S-b represents the collaborative training loss value, N represents the number of image samples, M represents the number of image blocks contained in the image sample, C represents the number of categories, and B i,j,c represents the block-level pseudo label of the jth image block in the i-th image sample, a i,j,c represents the c-th component of the attention score.

[0176] In some exemplary embodiments, based on L=L S-b Update the model parameters.

[0177] In some exemplary embodiments, based on L=L S-b +L S Update the model parameters.

[0178] In some exemplary embodiments, based on L=L S-b +L b Update the model parameters.

[0179] In some exemplary embodiments, based on L=L S-b +L b +L S Update the model parameters.

[0180] Among them, L represents the total training loss value based on the image classification model.

[0181] In some exemplary embodiments, the total training loss value L may also be calculated according to the following formula, so that the model parameters are updated according to the total training loss value L.

[0182] L=L S +λL b +μL S-b

[0183] Among them, λ and μ represent hyperparameters, which are used to control the relative importance of the three loss information.

[0184] It can be concluded that the collaborative training scheme of the image classification model proposed in this application uses a label disambiguation module to obtain more accurate block-level pseudo-labels, which can reduce the noise of block-level pseudo-labels, and uses block-level features to optimize the attention in the image-level aggregator, which can obtain more reasonable image-level category labels, so that the trained first image classification model and the second image classification model can both obtain more accurate image-level category labels.

[0185] It should also be noted that in actual training scenarios, the label disambiguation module may not be used. In this case, the image-level annotation labels of the image samples are used as the true values ​​to calculate the loss values ​​accordingly, and the detailed process will not be repeated.

[0186] Please continue to see Figure 7 , Figure 7 is a flowchart of an image classification method shown in an exemplary embodiment. The method can be applied to Figure 1 The implementation environment shown, for example, can be specifically executed by the server 120, can also be specifically executed by the terminal 110, or can be jointly executed by the terminal 110 and the server 120. Of course, the method can also be applied to other implementation environments, and can be executed by a terminal or a server in other implementation environments, or can be jointly executed by a terminal and a server in other implementation environments, and this embodiment is not limited thereto.

[0187] like Figure 7 As shown, in an exemplary embodiment, the image classification method includes the following steps:

[0188] S710, obtaining a full-slice pathological image to be classified;

[0189] S720, inputting the whole-slice pathological image into the first image classification model and / or the second image classification model after collaborative training to obtain an image classification result output by the first image classification model and / or the second image classification model.

[0190] This embodiment uses the first image classification model and / or the second image classification model that have been collaboratively trained as in the above embodiments to perform classification prediction on the whole-slice pathology image (WIS) to obtain the image classification results outputted by the first image classification model and / or the second image classification model. Based on the foregoing records, it can be known that the first image classification model and / or the second image classification model that have been collaboratively trained as in the above embodiments can more accurately predict image-level category labels. Therefore, this application uses the first image classification model and / or the second image classification model that have been collaboratively trained to perform more accurate classification prediction on the whole-slice pathology image and obtain image classification results with higher accuracy. Exemplarily, the image classification results corresponding to the whole-slice pathology image include the probability that the whole-slice pathology image is predicted to have a genetic mutation, and the probability that it is predicted to have no genetic mutation.

[0191] It should also be noted that the first image classification model after collaborative training can preferably be used to perform classification prediction on the whole-slice pathology image, because the image-level aggregator contained in the first image classification model globally summarizes the information of all image blocks during the training phase, so that the extracted global features of the image have high accuracy, so that the image-level classifier can output high-accuracy image classification results.

[0192] The inventors of the present application also conducted verification experiments on the classification prediction performance of various image classification model training schemes on different WIS data sets. Specifically, in addition to the technical scheme of the present application, the various image classification model training schemes also include CNN-ML, DSMIL, CLAM-SB, CLAM-MB, ViT-MIL, TransMIL, SETMIL, WEMO, LCSB-MIL, and IIB-MIL. The WIS data set includes the TGGA-NSCLC data set, the TGGA-RCC data set, and the CAMELYON16 data set. Among them, TGGA-NSCLC and TGGA-RCC are cancer genome atlas data sets, and CAMELYON16 is a data set for the detection and classification of lymph node metastasis. The verification experiment correspondingly obtains the AUC (Area Under the Curve) and Accuracy (accuracy) of the image classification model obtained by using each image classification model training scheme for each data set.

[0193] Experiments show that the technical solution of the present application has better performance on each data set. The AUC and Accuracy of the technical solution of the present application for each data set are higher than the AUC and Accuracy of other technical solutions for the corresponding data sets.

[0194] For example, the AUC and Accuracy of the technical solution of the present application on the TGGA-NSCLC data set are 0.999 and 0.9936, while the highest AUC and highest Accuracy of other technical solutions on this data set are 0.9811 and 0.9091. The AUC and Accuracy of the technical solution of the present application on the TGGA-RCC data set are 0.9994 and 0.9787, while the highest AUC and highest Accuracy of other technical solutions on this data set are 0.9902 and 0.9524. The AUC and Accuracy of the technical solution of the present application on the TGGA-RCC data set are 0.9681 and 0.8837, while the highest AUC and highest Accuracy of other technical solutions on this data set are 0.9107 and 0.8494.

[0195] Therefore, it can be proved that the collaborative training scheme proposed in this application can significantly improve the classification prediction accuracy of high-resolution images such as full-slice pathology images.

[0196] Please continue reading Figure 8 , Figure 8 is a flowchart of an exemplary image classification method. The method can be applied to Figure 1 The implementation environment shown, for example, can be specifically executed by the server 120, can also be specifically executed by the terminal 110, or can be jointly executed by the terminal 110 and the server 120. Of course, the method can also be applied to other implementation environments, and can be executed by a terminal or a server in other implementation environments, or can be jointly executed by a terminal and a server in other implementation environments, and this embodiment is not limited thereto.

[0197] like Figure 8 As shown, in another exemplary embodiment, the training method of the image classification model includes the following steps:

[0198] S810, dividing the image sample into a plurality of image blocks;

[0199] S820, performing classification prediction on each image block based on the image classification model to obtain a block-level category label corresponding to each image block, and determining an image-level category label based on the block-level category label corresponding to each image block;

[0200] S830, generating block-level pseudo labels corresponding to each image block according to the block-level category labels corresponding to each image block and the prototype vectors of each category; wherein the prototype vectors corresponding to each category act on the category to which the image block is predicted to belong;

[0201] S840, calculating a block-level training loss value based on the block-level pseudo-label and the block-level category label corresponding to each image block, so as to update parameters of the image classification model based on the block-level training loss value.

[0202] The training method of the image classification model provided in this embodiment can be understood as the block-level supervision strategy described above. Based on the aforementioned loaded block-level supervision strategy, a block-level pseudo-label with less noise is generated for each image block contained in the image sample, so that the block-level training loss value is calculated based on the block-level pseudo-label and block-level category label corresponding to each image block, and the parameters of the image classification model are updated based on the block-level training loss value, which can improve the accuracy of the image classification model in predicting the block-level category label, thereby enabling the trained image classification model to perform more accurate image classification predictions.

[0203] It should also be noted that, since the relevant contents of this embodiment have been described in detail in the aforementioned embodiments, they will not be repeated here.

[0204] Correspondingly, after obtaining the whole-slice pathology image to be classified, the image classification method can also input the whole-slice pathology image into the trained image classification model to obtain the image classification result output by the image classification model.

[0205] See also Fig. 9 , Fig. 9 is a block diagram of a collaborative training device for an image classification model shown in an exemplary embodiment of the present application. The device can be deployed in Figure 1 The implementation environment shown, for example, can be deployed on the server 120, also deployed on the terminal 110, or deployed on both the terminal 110 and the server 120. Of course, the device can also be deployed in other implementation environments, and specifically deployed on the terminal or server in other implementation environments, or deployed on the terminal and server in other implementation environments, and this embodiment is not limited thereto.

[0206] like Fig. 9 As shown, in an exemplary embodiment, the collaborative training device of the image classification model includes:

[0207] A division module 910 is configured to divide the image samples into a plurality of image blocks;

[0208] A first acquisition module 920 is configured to extract global image features from a plurality of image blocks based on a first image classification model, and predict image-level category labels based on the global image features, so as to obtain image block classification prediction information generated by the first image classification model during an image classification process;

[0209] A second acquisition module 930 is configured to perform classification prediction on each image block based on the second image classification model, and determine the image-level category label based on the block-level category label obtained by the classification prediction, so as to obtain the image block classification prediction information generated by the second image classification model during the image classification process;

[0210] The collaborative module 940 is configured to calculate a collaborative training loss value based on the image block classification prediction information generated by the first image classification model and the second image classification model respectively, so as to update the parameters of the first image classification model and the second image classification model based on the collaborative training loss value.

[0211] In another exemplary embodiment, the second acquisition module 930 includes:

[0212] A generating unit is configured to generate a block-level pseudo label corresponding to each image block according to the block-level category label corresponding to each image block and the prototype vector of each category; wherein the prototype vector corresponding to each category acts on the category predicted to belong to;

[0213] The acquisition unit is configured to acquire the block-level pseudo-label corresponding to each image block as the image block classification prediction information generated by the second image classification model during the image classification process.

[0214] In another exemplary embodiment, the generating unit includes:

[0215] A selection subunit is configured to select typical image blocks for each category according to the block-level category labels corresponding to each image block;

[0216] A generating subunit is configured to generate a prototype label corresponding to each image block based on the selected image block and the prototype vector corresponding to each category; wherein the prototype label corresponding to each image block is used to characterize the category of the corresponding image block;

[0217] The determination subunit is configured to determine the block-level pseudo-label corresponding to each image block based on the updated prototype label corresponding to each image block.

[0218] In another exemplary embodiment, the determining subunit is further configured to perform the following steps:

[0219] Based on the updated prototype labels corresponding to the image blocks, the block-level pseudo labels of the image blocks stored in the confidence library are updated;

[0220] Obtain the block-level pseudo-label corresponding to each image block from the updated confidence library.

[0221] In another exemplary embodiment, the generating subunit is further configured to perform the following steps:

[0222] According to the block-level category labels corresponding to each image block, the probability values ​​of each image block being predicted as each category are determined;

[0223] Based on the probability values ​​of each image block being predicted as each category, a typical image block is selected for each category.

[0224] In another exemplary embodiment, the selection subunit is further configured to perform the following steps:

[0225] According to the image blocks selected for each category, the prototype vector corresponding to each category is updated;

[0226] According to the updated prototype vectors of each category, the prototype labels corresponding to each image block are generated.

[0227] In another exemplary embodiment, the selection subunit is further configured to perform the following steps:

[0228] Determine the momentum coefficient corresponding to the current training and the complementary coefficient corresponding to the momentum coefficient based on the number of iterations of the current training;

[0229] According to the momentum coefficient, the complementary coefficient and the feature vector of the image block selected for each category, the prototype vector corresponding to each category is updated.

[0230] In another exemplary embodiment, the selection subunit is further configured to perform the following steps:

[0231] For each image block, the product of the feature vector and the updated prototype vector of each category is calculated to obtain the category probability set corresponding to each image block;

[0232] The category corresponding to the maximum probability is selected from the category probability set corresponding to each image block as the prototype category corresponding to each image block, and the prototype label corresponding to each image block is obtained based on vector encoding of the prototype category.

[0233] In another exemplary embodiment, the first acquisition module 920 is further configured to perform the following steps:

[0234] Obtaining a prediction score of each image block output by the second image classification model when extracting the global image features from the multiple image blocks; wherein the prediction score represents the contribution of each image block to the global image features;

[0235] The prediction score of each image block is obtained as the image block classification prediction information generated by the first image classification model during the image classification process.

[0236] In another exemplary embodiment, the second acquisition module 930 is further configured to perform the following steps:

[0237] Based on the image-level category label determined by the second image classification model, image block classification prediction information generated by the second image classification model during the image classification process is determined.

[0238] In another exemplary embodiment, the collaboration module 940 is further configured to perform the following steps:

[0239] The image block classification prediction information generated according to the first image classification model is used as the prediction value, and the image block classification prediction information generated according to the second image classification model is used as the true value, and the deviation between the prediction value and the true value is calculated to obtain the collaborative training loss value.

[0240] In another exemplary embodiment, the collaborative training apparatus for the image classification model further includes an extraction module, and the extraction module is configured to perform the following steps:

[0241] Extracting an embedding vector of each image block, and extracting a feature vector of each image block based on the embedding vector of each image block;

[0242] The feature vectors of each image block are respectively input into the first image classification model and the second image classification model, so that the first image classification model and the second image classification model predict the image-level category label based on the feature vectors of the received multiple image blocks.

[0243] In another exemplary embodiment, the extraction module is further configured to perform the following steps:

[0244] Extracting an embedding vector of each image block through the encoding network, and extracting a feature vector of each image block through the extraction network for the embedding vector of each image block; wherein the encoding network is in a parameter freezing state;

[0245] The collaboration module 940 is further configured to perform the following steps:

[0246] The parameters of the extraction network are updated based on the training loss value.

[0247] See also Fig.10 , Fig.10 is a block diagram of an image classification device shown in an exemplary embodiment of the present application. The device can be deployed in Figure 1 The implementation environment shown, for example, can be deployed on the server 120, also deployed on the terminal 110, or deployed on both the terminal 110 and the server 120. Of course, the device can also be deployed in other implementation environments, and specifically deployed on the terminal or server in other implementation environments, or deployed on the terminal and server in other implementation environments, and this embodiment is not limited thereto.

[0248] like Fig.10 As shown, in an exemplary embodiment, the image classification device includes:

[0249] An acquisition module 1010 is configured to acquire a full-slice pathological image to be classified;

[0250] The processing module 1020 is configured to input the whole-slice pathology image into the first image classification model and / or the second image classification model that have been collaboratively trained by the collaborative training method of the image classification model to obtain the image classification results output by the first image classification model and / or the second image classification model.

[0251] Another exemplary embodiment of the present application also proposes a training device for an image classification model. The device can be deployed in Figure 1 The implementation environment shown, for example, can be deployed on the server 120, also deployed on the terminal 110, or deployed on both the terminal 110 and the server 120. Of course, the device can also be deployed in other implementation environments, and specifically deployed on the terminal or server in other implementation environments, or deployed on the terminal and server in other implementation environments, and this embodiment is not limited thereto.

[0252] The training setup for the image classification model includes:

[0253] An image partitioning module, configured to partition an image sample into a plurality of image blocks;

[0254] A model processing module is configured to perform classification prediction on each image block based on the image classification model, obtain a block-level category label corresponding to each image block, and determine an image-level category label based on the block-level category label corresponding to each image block;

[0255] A pseudo-label generation module is configured to generate a block-level pseudo-label corresponding to each image block according to the block-level category label corresponding to each image block and the prototype vector of each category; wherein the prototype vector corresponding to each category acts on the category predicted to belong to;

[0256] The loss calculation module is configured to calculate the block-level training loss value based on the block-level pseudo-label and the block-level category label corresponding to each image block, so as to update the parameters of the image classification model based on the block-level training loss value.

[0257] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here. In practical applications, the collaborative training device or image classification device of the image classification model provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0258] An embodiment of the present application also provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs. When the one or more computer programs are executed by the one or more processors, the electronic device implements the artificial intelligence-based video understanding model training method provided in the above-mentioned embodiments.

[0259] Fig.11 The structure diagram of the computer system of the electronic device suitable for implementing the embodiment of the present application is shown. It should be noted that the electronic device can be Figure 1 The terminal 110 or server 120 in the implementation environment shown may also be a terminal or server in other implementation environments, and is not limited here. It should also be noted that Fig.11 The computer system 1100 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0260] like Fig.11 As shown, the computer system 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage part 1108 to a random access memory (RAM) 1103, such as executing the method described in the above embodiment. Various computer programs and data required for system operation are also stored in RAM 1103. CPU 1101, ROM 1102 and RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0261] The following components are connected to the I / O interface 1105: an input section 1106 including a keyboard, a mouse, etc.; an output section 1107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the I / O interface 1105 as needed. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1110 as needed so that a computer program read therefrom is installed into the storage section 1108 as needed.

[0262] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1109, and / or installed from a removable medium 1111. When the computer program is executed by a central processing unit (CPU) 1101, various functions defined in the system of the present application are executed.

[0263] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer program contained in the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0264] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0265] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0266] Another aspect of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor of an electronic device, the method described above is implemented. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0267] Another aspect of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the electronic device executes the method provided in each of the above embodiments.

[0268] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

[0269] It is understandable that in the specific implementation of this application, data related to images, labels, etc. are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

Claims

1. A collaborative training method for an image classification model, characterized in that: The method comprises: Divide the image sample into a plurality of image blocks; Extracting global image features from the multiple image blocks based on the first image classification model, and predicting image-level category labels based on the global image features, so as to obtain image block classification prediction information generated by the first image classification model during the image classification process; Performing classification prediction on each image block based on the second image classification model, and determining the image-level category label based on the block-level category label obtained by the classification prediction, so as to obtain the image block classification prediction information generated by the second image classification model during the image classification process; A collaborative training loss value is calculated according to the image block classification prediction information respectively generated by the first image classification model and the second image classification model, so as to update the parameters of the first image classification model and the second image classification model based on the collaborative training loss value.

2. The method according to claim 1, characterized in that The method of performing classification prediction on each image block based on the second image classification model, and determining the image-level category label based on the block-level category label obtained by the classification prediction, so as to obtain the image block classification prediction information generated by the second image classification model in the image classification process, includes: Generate block-level pseudo labels corresponding to each image block according to the block-level category labels corresponding to each image block and the prototype vectors of each category; wherein the prototype vectors corresponding to each category act on the category to which it is predicted; The block-level pseudo-label corresponding to each image block is obtained as the image block classification prediction information generated by the second image classification model during the image classification process.

3. The method according to claim 2, characterized in that The step of generating a block-level pseudo label corresponding to each image block according to the block-level category label corresponding to each image block and the prototype vector corresponding to each category includes: According to the block-level category labels corresponding to each image block, typical image blocks are selected for each category; Based on the selected image blocks and the prototype vectors corresponding to the categories, a prototype label corresponding to each image block is generated; wherein the prototype label corresponding to each image block is used to characterize the category of the corresponding image block; Based on the updated prototype labels corresponding to the image blocks, the block-level pseudo labels corresponding to the image blocks are determined.

4. The method according to claim 3, characterized in that The step of determining the block-level pseudo-label corresponding to each image block based on the updated prototype label corresponding to each image block includes: Based on the updated prototype labels corresponding to the image blocks, the block-level pseudo labels of the image blocks stored in the confidence library are updated; Obtain the block-level pseudo-label corresponding to each image block from the updated confidence library.

5. The method according to claim 3, characterized in that: The selecting typical image blocks for each category according to the block-level category labels corresponding to each image block includes: According to the block-level category labels corresponding to each image block, the probability values ​​of each image block being predicted as each category are determined; Based on the probability values ​​of each image block being predicted as each category, a typical image block is selected for each category.

6. The method according to claim 3, characterized in that: The method of generating a prototype label corresponding to each image block based on the selected image block and the prototype vector corresponding to each category includes: According to the image blocks selected for each category, the prototype vector corresponding to each category is updated; According to the updated prototype vectors of each category, the prototype labels corresponding to each image block are generated.

7. The method according to claim 6, characterized in that The updating of the prototype vectors corresponding to the respective categories according to the image blocks selected from the respective categories comprises: Determining a momentum coefficient corresponding to the current training and a complementary coefficient corresponding to the momentum coefficient based on the number of iterations of the current training; The prototype vector corresponding to each category is updated according to the momentum coefficient, the complementary coefficient and the feature vector of the image block selected from each category.

8. The method according to claim 6, characterized in that The method of generating prototype labels corresponding to each category according to the updated prototype vectors of each category includes: For each image block, the product of the feature vector and the updated prototype vector of each category is calculated to obtain the category probability set corresponding to each image block; A category corresponding to the maximum probability is selected from the category probability set corresponding to each image block as the prototype category corresponding to each image block, and a prototype label corresponding to each image block is obtained based on vector encoding of the prototype category.

9. The method according to claim 1, characterized in that: The extracting global image features from the multiple image blocks based on the first image classification model, and predicting image-level category labels based on the global image features to obtain image block classification prediction information generated by the first image classification model during the image classification process, includes: Obtaining a prediction score of each image block output by the second image classification model when extracting the global image features from the multiple image blocks; wherein the prediction score represents the contribution of each image block to the global image features; The prediction score of each image block is obtained as the image block classification prediction information generated by the first image classification model during the image classification process.

10. The method according to claim 1, characterized in that The method of performing classification prediction on each image block based on the second image classification model, and determining the image-level category label based on the block-level category label obtained by the classification prediction, so as to obtain the image block classification prediction information generated by the second image classification model in the image classification process, includes: Based on the image-level category label determined by the second image classification model, image block classification prediction information generated by the second image classification model during the image classification process is determined.

11. The method according to any one of claims 1 to 10, characterized in that The calculating the collaborative training loss value according to the image block classification prediction information respectively generated by the first image classification model and the second image classification model, so as to update the parameters of the first image classification model and the second image classification model based on the collaborative training loss value, comprises: The image block classification prediction information generated according to the first image classification model is used as the prediction value, and the image block classification prediction information generated according to the second image classification model is used as the true value, and the deviation between the prediction value and the true value is calculated to obtain the collaborative training loss value.

12. The method according to any one of claims 1 to 10, characterized in that The method further comprises: Extracting an embedding vector of each image block, and extracting a feature vector of each image block based on the embedding vector of each image block; The feature vectors of each image block are respectively input into the first image classification model and the second image classification model, so that the first image classification model and the second image classification model predict image-level category labels based on the feature vectors of the received multiple image blocks.

13. An image classification method, characterized in that: The method comprises: Acquire a whole-slice pathological image to be classified; The whole-slice pathology image is input into the first image classification model and / or the second image classification model that has been collaboratively trained by the method described in any one of claims 1-12 to obtain an image classification result output by the first image classification model and / or the second image classification model.

14. A training method for an image classification model, characterized in that: The method comprises: Divide the image sample into a plurality of image blocks; Based on the image classification model, classification prediction is performed on each image block to obtain a block-level category label corresponding to each image block, and an image-level category label is determined based on the block-level category label corresponding to each image block; Generate block-level pseudo labels corresponding to each image block according to the block-level category labels corresponding to each image block and the prototype vectors of each category; wherein the prototype vectors corresponding to each category act on the category to which it is predicted; A block-level training loss value is calculated based on the block-level pseudo-labels and block-level category labels corresponding to each image block, so as to update the parameters of the image classification model based on the block-level training loss value.

15. A collaborative training device for an image classification model, characterized in that: The device comprises: A division module, configured to divide the image samples into a plurality of image blocks; A first acquisition module is configured to extract image global features from the multiple image blocks based on a first image classification model, and predict image-level category labels based on the image global features, so as to obtain image block classification prediction information generated by the first image classification model during the image classification process; a second acquisition module configured to perform classification prediction on each image block based on a second image classification model, and determine an image-level category label based on a block-level category label obtained by the classification prediction, so as to obtain image block classification prediction information generated by the second image classification model during the image classification process; The collaborative module is configured to calculate a collaborative training loss value based on the image block classification prediction information respectively generated by the first image classification model and the second image classification model, so as to update the parameters of the first image classification model and the second image classification model based on the collaborative training loss value.

16. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more computer programs, which, when executed by the one or more processors, enables the electronic device to implement the method according to any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 14.

18. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor of an electronic device, the method according to any one of claims 1 to 14 is implemented.

Images