Image classification model training methods, image classification methods, devices and equipment

Abstract

The application provides a training method and an image classification method and device of an image classification model, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a first sample data set; acquiring a second sample data set; pre-training an image classification model in a self-supervised manner based on the first sample data set to obtain a first image classification model; and training the first image classification model based on the second sample data set to obtain a target image classification model. According to the technical solution, the image type label and the staining fixed label are introduced during pre-training, so that the first image classification model can learn the features of the sample staining image itself, the first image classification model can accurately classify cells, the first image classification model is trained based on the sample cell image and the cell label, the classification accuracy of the model can be further improved, and the target image classification model obtained through training has high classification accuracy.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training method for an image classification model, an image classification method, an apparatus, and a device. Background Technology

[0002] With the development of artificial intelligence technology, image classification models trained on large-scale data can classify images. The classification accuracy of these models often depends on the quality and complexity of the dataset; therefore, improving the accuracy of image classification models is a problem that requires further research.

[0003] Currently, when training image classification models, image augmentation is typically performed on sample images in the dataset. These augmented images are then compared with other images to improve the classification accuracy. Image classification models trained in this way can classify natural images relatively accurately, such as classifying them as animal images, landscape images, or architectural images.

[0004] However, because medical images are imaged differently from natural images, and medical images contain more classification features, the image classification model trained using the above scheme has low classification accuracy when classifying medical images. Summary of the Invention

[0005] This application provides a training method, an image classification method, an apparatus, and a device for an image classification model, resulting in a trained image classification model with high classification accuracy. The technical solution is as follows:

[0006] On the one hand, a method for training an image classification model is provided, the method comprising:

[0007] Obtain a first sample dataset, which includes multiple sample stained images, image type labels for the sample stained images, and staining fixation labels. The sample stained images include multiple stained cells, and the staining fixation labels are used to indicate the staining fixation method of the corresponding sample stained images.

[0008] Obtain a second sample dataset, which includes multiple sample cell images and cell type labels for the sample cell images;

[0009] Based on the first sample dataset, the image classification model is pre-trained in a self-supervised manner to obtain the first image classification model;

[0010] Based on the second sample dataset, the first image classification model is trained to obtain a target image classification model, which is used to determine the cell type corresponding to the input cell image.

[0011] On the other hand, an image classification method is provided, the method comprising:

[0012] Cell sampling is performed on the stained image to be processed to obtain multiple cell images, wherein the stained image includes multiple stained cells;

[0013] The multiple cell images are classified based on a target image classification model to obtain classification result information. The target image classification model is trained on a first sample dataset and a second sample dataset. The first sample dataset includes multiple stained sample images, image type labels for the stained sample images, and staining fixation labels. The stained sample images include multiple stained cells, and the staining fixation labels are used to indicate the staining fixation method of the corresponding stained sample images. The second sample dataset includes multiple sample cell images and cell type labels for the sample cell images. The classification result information is used to indicate the cell type of the multiple cell images.

[0014] Based on the classification result information, a classification result image is generated, in which the cell types of the multiple cell images are labeled.

[0015] On the other hand, a training apparatus for an image classification model is provided, the apparatus comprising:

[0016] The data acquisition module is used to acquire a first sample dataset, which includes multiple sample staining images, image type labels of the sample staining images, and staining fixation labels. The sample staining images include multiple stained cells, and the staining fixation labels are used to indicate the staining fixation method of the corresponding sample staining images.

[0017] The data acquisition module is further configured to acquire a second sample dataset, which includes multiple sample cell images and cell type labels for the sample cell images;

[0018] The first training module is used to pre-train the image classification model based on the first sample dataset in a self-supervised manner to obtain the first image classification model.

[0019] The second training module is used to train the first image classification model based on the second sample dataset to obtain a target image classification model, which is used to determine the cell type corresponding to the input cell image.

[0020] In some embodiments, the first training module includes:

[0021] The image processing submodule is used to perform image sampling and image enhancement on the sample stained image input in the i-th iteration based on the image classification model in the i-th iteration, to obtain a first enhanced image, a second enhanced image and a third enhanced image, where i is a positive integer greater than 1;

[0022] The model training submodule is used to train the image classification model in the i-th iteration based on the first enhanced image, the second enhanced image, and the third enhanced image;

[0023] The model training submodule is further configured to obtain the first image classification model in response to the satisfaction of the training termination condition.

[0024] In some embodiments, the model training submodule includes:

[0025] The first training unit is used to perform feature comparison on the first enhanced image and the second enhanced image based on the image classification model of the (i-1)th iteration and the image classification model of the ith iteration, and obtain the first loss of the ith iteration. The first loss is used to indicate the difference before and after the image classification model is updated.

[0026] The second training unit is used to determine the second loss of the i-th iteration based on the classification result of the image classification model in the i-th iteration of the second enhanced image, the image type label of the sample stained image, and the stained fixed label of the sample stained image. The second loss is used to indicate the classification loss of the image classification model.

[0027] The third training unit is used to perform feature comparison on the anchor image, positive sample image, and negative sample image based on the image classification model of the i-th iteration, and obtain the third loss of the i-th iteration. The anchor image is the second enhanced image, the positive sample image is the third enhanced image, and the negative sample image is obtained by randomly sampling the sample stained image based on the anchor image. The third loss is a triplet loss, which is used to indicate the difference between the enhanced images of the sample stained image.

[0028] The fourth training unit is used to adjust the model parameters of the image classification model in the i-th iteration based on the first loss, the second loss, and the third loss.

[0029] In some embodiments, the first training unit is configured to extract features from the first enhanced image based on the image classification model in the (i-1)th iteration to obtain a first enhanced feature; extract features from the second enhanced image based on the image classification model in the i-th iteration to obtain a second enhanced feature; and determine a first loss in the i-th iteration based on the difference between the first enhanced feature and the second enhanced feature.

[0030] In some embodiments, the second training unit is configured to classify the second enhanced image based on the image classification model of the i-th iteration to obtain first classification information and second classification information. The first classification information is used to indicate the probability that the second enhanced image belongs to different image types, and the second classification information is used to indicate the probability that the second enhanced image belongs to different staining fixation methods. Based on the first classification information and the image type label of the sample stained image, a first classification loss is determined. Based on the second classification information and the staining fixation label of the sample stained image, a second classification loss is determined. The first classification loss and the second classification loss are summed to obtain the second loss of the i-th iteration.

[0031] In some embodiments, the third training unit is used to extract features from the anchor image, the positive sample image, and the negative sample image based on the image classification model of the i-th iteration, to obtain a third enhanced feature, a fourth enhanced feature, and a fifth enhanced feature; and to determine the third loss of the i-th iteration based on the differences between the third enhanced feature, the fourth enhanced feature, and the fifth enhanced feature.

[0032] In some embodiments, the fourth training unit is used to perform a weighted summation of the first loss, the second loss, and the third loss to obtain a fourth loss; and to adjust the model parameters of the image classification model in the i-th iteration based on the fourth loss.

[0033] In some embodiments, the image processing submodule is used to perform image sampling on the sample staining image to obtain a sample sampling image; and to enhance the sample sampling image using different image enhancement methods to obtain a first enhanced image, a second enhanced image, and a third enhanced image.

[0034] In some embodiments, the second training module is used to classify sample cell images in the second sample dataset based on the first image classification model to obtain sample classification information, wherein the sample classification information is used to indicate the probability that multiple sample cell images belong to different cell types; and to determine the target image classification model based on the sample classification information and the cell type labels of the sample cell images.

[0035] On the other hand, an image classification apparatus is provided, the apparatus comprising:

[0036] A cell sampling module is used to sample cells from the stained image to be processed, thereby obtaining multiple cell images, wherein the stained image includes multiple stained cells;

[0037] An image classification module is used to classify the multiple cell images based on a target image classification model to obtain classification result information. The target image classification model is trained based on a first sample dataset and a second sample dataset. The first sample dataset includes multiple stained sample images, image type labels for the stained sample images, and staining fixation labels. The stained sample images include multiple stained cells, and the staining fixation labels are used to indicate the staining fixation method of the corresponding stained sample images. The second sample dataset includes multiple sample cell images and cell type labels for the sample cell images. The classification result information is used to indicate the cell type of the multiple cell images.

[0038] An image generation module is used to generate a classification result image based on the classification result information, wherein the classification result image is labeled with the cell types of the multiple cell images.

[0039] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory being used to store at least one computer program, the at least one computer program being loaded and executed by the processor to implement the training method of the image classification model in the embodiments of this application, or the at least one computer program being loaded and executed by the processor to implement the image classification method in the embodiments of this application.

[0040] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, the at least one computer program being loaded and executed by a processor to implement the training method of the image classification model as described in the embodiments of this application, or the at least one computer program being loaded and executed by a processor to implement the image classification method as described in the embodiments of this application.

[0041] On the other hand, a computer program product is provided, which includes computer program code stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the computer device to perform a training method for an image classification model provided in various alternative implementations of the above-mentioned aspects, or causing the computer device to perform an image classification method provided in various alternative implementations of the above-mentioned aspects.

[0042] The beneficial effects of the technical solutions provided in this application are:

[0043] The training scheme for the image classification model provided in this application introduces easily obtainable image type labels and staining fixed labels during pre-training, enabling the trained first image classification model to learn the features of the sample stained image itself. Since the sample stained image includes multiple stained cells, the first image classification model can classify cells more accurately. Further training the first image classification model based on the sample cell image and cell labels can further improve the classification accuracy of the model, resulting in a target image classification model with high classification accuracy. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram illustrating the implementation environment of a training method for an image classification model according to an embodiment of this application;

[0046] Figure 2 This is a flowchart of a training method for an image classification model according to an embodiment of this application;

[0047] Figure 3 This is a flowchart of a training method for another image classification model provided according to an embodiment of this application;

[0048] Figure 4 This is a schematic diagram of a cell detection result provided according to an embodiment of this application;

[0049] Figure 5 This is a schematic diagram of an iterative process provided according to an embodiment of this application;

[0050] Figure 6 This is an input / output comparison diagram provided according to an embodiment of this application;

[0051] Figure 7 This is a flowchart of an image classification method provided according to an embodiment of this application;

[0052] Figure 8 This is a block diagram of a training device for an image classification model according to an embodiment of this application;

[0053] Figure 9This is a block diagram of a training apparatus for another image classification model provided according to an embodiment of this application;

[0054] Figure 10 This is a block diagram of an image classification device provided according to an embodiment of this application;

[0055] Figure 11 This is a schematic diagram of the structure of a terminal according to an embodiment of this application;

[0056] Figure 12 This is a schematic diagram of the structure of a server according to an embodiment of this application. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0058] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0059] In this application, the term "at least one" means one or more, and "multiple" means two or more.

[0060] The following is an explanation of the terms used in this application.

[0061] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0062] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0063] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), autonomous driving, intelligent transportation, and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0064] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.

[0065] Triplet loss is a loss function in deep learning used to train on samples with low dissimilarity. The triplet consists of an anchor example, a positive example, and a negative example. It calculates sample similarity by optimizing the distance between the anchor example and the positive example to be smaller than the distance between the anchor example and the negative example.

[0066] The Rectified Linear Unit (ReLU), also known as the Modified Linear Unit, is a commonly used activation function in artificial neural networks. It usually refers to nonlinear functions represented by the ramp function and its variants.

[0067] The F1 score is a statistical metric used to measure the precision of a binary classification model. It takes into account both precision and recall. The F1 score can be viewed as a harmonic average of the model's precision and recall, with a maximum value of 1 and a minimum value of 0.

[0068] The image classification model training method provided in this application can be executed by a computer device, which can be a terminal or a server. The following describes the implementation environment of the image classification model training method provided in this application, using a computer device as a server as an example. Figure 1 This is a schematic diagram illustrating the implementation environment of a training method for an image classification model according to an embodiment of this application. See also... Figure 1 The implementation environment includes terminal 101 and server 102.

[0069] Terminal 101 and server 102 can be connected directly or indirectly via wired or wireless communication, and this application does not impose any restrictions on this.

[0070] In some embodiments, terminal 101 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Terminal 101 has an application installed and running, which enables image uploading and image display. Those skilled in the art will understand that the number of terminals may be more or less. For example, there may be only one terminal, or there may be dozens or hundreds, or even more. This application does not limit the number or type of terminals.

[0071] In some embodiments, server 102 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Server 102 is used to provide background services for applications. In some embodiments, server 102 undertakes the main computing work, and terminal 101 undertakes the secondary computing work; or, server 102 undertakes the secondary computing work, and terminal 101 undertakes the main computing work; or, server 102 and terminal 101 collaborate on computing using a distributed computing architecture.

[0072] The image classification method provided in this application embodiment can be executed by the aforementioned terminal. The terminal obtains a trained target image classification model from the server and then implements the image classification scheme provided in this application embodiment based on the target image classification model.

[0073] Figure 2 This is a flowchart of a training method for an image classification model according to an embodiment of this application, such as... Figure 2 As shown in the embodiment of this application, the training method of the image classification model is illustrated using the server as an example. The training method of this image classification model includes the following steps:

[0074] 201. The server obtains the first sample dataset, which includes multiple sample stained images, image type labels for the sample stained images, and staining fixation labels. The sample stained images include multiple stained cells, and the staining fixation labels are used to indicate the staining fixation method of the corresponding sample stained images.

[0075] In this embodiment of the application, the server is Figure 1 The server 102 shown here generates the first sample dataset based on an existing public dataset, or the first sample dataset is a pre-generated dataset that the server directly obtains. The image type labels and staining fixation labels of the stained images in this first sample dataset do not require manual annotation; they can be obtained by detecting the stained images.

[0076] 202. The server obtains a second sample dataset, which includes multiple sample cell images and cell type labels for the sample cell images.

[0077] In this embodiment of the application, the second sample dataset is a dataset with manually labeled cell type tags. The second sample dataset is a publicly available dataset directly obtained by the server, or the second sample dataset is maintained by the server and can be directly obtained by the server from the database.

[0078] It should be noted that the first and second sample datasets mentioned above can be stored in a blockchain system, and the server can obtain the first and second sample datasets from the blockchain system.

[0079] 203. Based on the first sample dataset, the server pre-trains the image classification model in a self-supervised manner to obtain the first image classification model.

[0080] In this embodiment, the server can pre-train an initial image classification model based on the aforementioned first sample dataset, using self-supervised learning. By performing self-supervised learning on the image classification model based on the sample stained images, the image type labels of the sample stained images, and the fixed stained labels, a first image classification model can be obtained.

[0081] 204. The server trains the first image classification model based on the second sample dataset to obtain a target image classification model, which is used to determine the cell type corresponding to the input cell image.

[0082] In this embodiment of the application, after the first image classification model is pre-trained, the server can further train the first image classification model based on the second sample dataset to improve the classification accuracy of the model, so that the trained target image classification model can classify the input cell image more accurately and obtain the cell type of the cell image.

[0083] The solution provided in this application introduces easily obtainable image type labels and staining fixed labels during pre-training, enabling the trained first image classification model to learn the features of the sample stained image itself. Since the sample stained image includes multiple stained cells, the first image classification model can classify cells more accurately. Further training the first image classification model based on the sample cell image and cell labels can further improve the classification accuracy of the model, resulting in a target image classification model with high classification accuracy.

[0084] Figure 2 The main flow of the training method for the image classification model provided in this application embodiment is illustrated by way of example. The training method for this image classification model will be further described below. Figure 3 This is a flowchart of another image classification model training method provided according to an embodiment of this application, such as... Figure 3 As shown in the embodiment of this application, the training method of the image classification model is illustrated using the server as an example. The training method of this image classification model includes the following steps:

[0085] 301. The server obtains a first sample dataset, which includes multiple sample stained images, image type labels for the sample stained images, and staining fixation labels. The sample stained images include multiple stained cells, and the staining fixation labels are used to indicate the staining fixation method of the corresponding sample stained images.

[0086] In this embodiment, the example of the first sample dataset being generated by a server is used for illustration. The server is... Figure 1 The server 102 shown above generates this first sample dataset based on an existing publicly available dataset.

[0087] For example, the server obtains 23,087 stained slide images (1000×1000 pixels, 40x magnification) from the large-scale public dataset TCGA (The Cancer Genome Atlas). The Hover-net cell detection and segmentation algorithm is used to detect and segment the cell nuclei in these stained slide images, resulting in a first sample dataset. This first sample dataset includes not only the stained images themselves, but also image type labels and staining fixation labels. It should be noted that this application does not limit the cell detection and segmentation algorithm used. Figure 4 This is a schematic diagram of a cell detection result provided according to an embodiment of this application. See also... Figure 4 As shown, the cells in the stained slide before detection are not labeled, while the results of cell detection and segmentation are marked with different colors in the stained slide after detection, with each cell corresponding to a different color.

[0088] 302. The server obtains a second sample dataset, which includes multiple sample cell images and cell type labels for the sample cell images.

[0089] In this embodiment of the application, the second sample dataset is a dataset with cell type labels, which are annotated by domain professionals. For example, the second sample dataset is the CoNSeP (Colorectal Nuclear Segmentation and Phenotypes) dataset.

[0090] It should be noted that the numbers 301 and 302 mentioned above are only assigned for the convenience of description and do not restrict the order in which the first sample dataset and the second sample dataset are obtained. That is, the server can obtain the first sample dataset first and then obtain the second sample dataset; the server can also obtain the second sample dataset first and then obtain the first sample dataset; or the server can obtain the first sample dataset and the second sample dataset at the same time.

[0091] It should be noted that the server can execute a multi-branch task based on the first sample dataset to obtain a first image classification model. This multi-branch task is implemented based on different modules in the initial image classification model. In this embodiment, taking the image classification model executing a three-branch task as an example, each branch corresponds to a training loss during training: a first loss, a second loss, and a third loss. The first loss is the training loss of the main branch of self-supervised learning, used to indicate the difference before and after the image classification model update. The second loss is the training loss of the auxiliary classification branch, used to indicate the classification loss of the image classification model. The third loss is the training loss of the three-sample branch, used to indicate the difference between the enhanced images of the stained sample images. The server adjusts the model parameters of the image classification model based on the first, second, and third losses of the above three branches.

[0092] It should be noted that this application embodiment uses the image classification model performing a three-branch task as an example for illustration. In different application scenarios, the image classification model can also add more branches and perform more branch tasks based on the correlation between data. This application embodiment does not limit this.

[0093] It should be noted that the server can train the first image classification model based on multiple iterations, with each iteration following the same method. In the i-th iteration, executing the tasks of the three branches mentioned above will yield the first, second, and third losses for the i-th iteration. The server adjusts the image classification model based on these losses to complete the i-th iteration, where i is a positive integer greater than 1. Steps 303 to 307 are illustrated using the i-th iteration as an example.

[0094] 303. For the sample stained image input in the i-th iteration, the server performs image sampling and image enhancement on the sample stained image based on the image classification model of the i-th iteration to obtain the first enhanced image, the second enhanced image and the third enhanced image, where i is a positive integer greater than 1.

[0095] In this embodiment, the image classification model for the i-th iteration is the image classification model whose parameters have been adjusted after the (i-1)-th iteration. For the sample staining image input in the i-th iteration, the server first samples the sample staining image based on the image classification model of the i-th iteration to obtain a sampled image. Then, it uses different image enhancement methods to enhance the sampled image, obtaining a first enhanced image, a second enhanced image, and a third enhanced image. The first enhanced image, the second enhanced image, and the third enhanced image are not completely identical. Here, different enhancement methods refer to using the same enhancement technique but different enhancement parameters.

[0096] For example, if the image enhancement method is rotation, the server rotates the sampled image clockwise by 90°, 180°, and 270° counterclockwise, respectively, to obtain the first, second, and third enhanced images. Alternatively, if the image enhancement method is translation, the server uses the sampled image as a reference and translates the sampling frame of the sampled image in three different directions by a certain distance to obtain the first, second, and third enhanced images. Image enhancement methods can also include Gaussian blur, contrast adjustment, etc., and this application embodiment does not limit these methods.

[0097] It should be noted that the embodiments of this application take the example of the server performing image sampling and image enhancement on the sample stained image based on the image classification model. In some embodiments, the server can also perform image sampling and image enhancement on the sample stained image based on other modules besides the image classification model, and then input the obtained first enhanced image, second enhanced image and third enhanced image into the image classification model. The embodiments of this application do not limit this.

[0098] 304. Based on the image classification model in the (i-1)th iteration and the image classification model in the i-th iteration, the server performs feature comparison on the first enhanced image and the second enhanced image to obtain the first loss in the i-th iteration. The first loss is used to indicate the difference before and after the image classification model is updated.

[0099] In this embodiment, the first loss of the i-th iteration is the training loss of the backbone branch of the self-supervised learning in the i-th iteration, and the first loss of the i-th iteration can indicate the difference between the image classification model and the i-th iteration.

[0100] In some embodiments, the server extracts features from the first enhanced image based on the image classification model of the (i-1)th iteration to obtain first enhanced features. The server also extracts features from the second enhanced image based on the image classification model of the i-th iteration to obtain second enhanced features. Since the first and second enhanced images originate from the same image and contain identical content, the server can determine the first loss for the i-th iteration based on the difference between the first and second enhanced features. By extracting features from the first and second enhanced images based on the image classification models of the (i-1)th and i-th iterations respectively, and then calculating the first loss based on the difference between the extracted features, the first loss can represent the difference between the image classification models of the (i-1)th and i-th iterations, and avoids the training collapse problem caused by extracting features from the image enhancement results of the same image.

[0101] In some embodiments, the first loss of the i-th iteration can be calculated based on formula (1), as shown in formula (1):

[0102] (1);

[0103] in, This indicates the first loss; Indicates the first enhancement feature; Indicates the second enhancement feature; Indicates L2 normal form; This indicates the inner product operation.

[0104] 305. Based on the result of classifying the second enhanced image by the image classification model in the i-th iteration, the image type label of the sample stained image, and the stained fixed label of the sample stained image, the server determines the second loss in the i-th iteration. The second loss is used to indicate the classification loss of the image classification model.

[0105] In this embodiment, the second loss of the i-th iteration is the training loss of the auxiliary classification branch in the i-th iteration, and the second loss of the i-th iteration can indicate the classification loss for classifying the second enhanced image.

[0106] In some embodiments, the server classifies the second augmented image based on the image classification model of the i-th iteration, obtaining first classification information and second classification information. The first classification information indicates the probability that the second augmented image belongs to different image types, and the second classification information indicates the probability that the second augmented image belongs to different staining fixation methods. Then, the server determines a first classification loss based on the first classification information and the image type label of the sample stained image. This first classification loss represents the loss when classifying the second augmented image by image type. In some embodiments, the first classification information is in vector form, and the elements in the first classification information represent the probability that the sample stained image belongs to different image types. The server determines a second classification loss based on the second classification information and the staining fixation label of the sample stained image. This second classification loss represents the loss when classifying the second augmented image by staining fixation method. In some embodiments, the second classification information is in vector form, and the elements in the second classification information represent the probability that the sample stained image belongs to different staining fixation methods. The server sums the first classification loss and the second classification loss to obtain the second loss of the i-th iteration. Since each stained image in the first sample dataset has an image type label and a stained label, the server can use these two types of labels to assist in training the image classification model, thereby improving the classification accuracy of the image classification model based on the classification features of the stained images themselves.

[0107] For example, the stained images in the first sample dataset have 32 image type labels, corresponding to 32 disease types: adrenocortical carcinoma, bladder urothelial carcinoma, invasive breast carcinoma, cervical squamous cell carcinoma and cervical endometrial adenocarcinoma, bile duct carcinoma, colon adenocarcinoma, lymphoid tumors, diffuse large B-cell lymphoma, esophageal cancer, glioblastoma, head and neck squamous cell carcinoma, kidney chromosome, kidney clear cell carcinoma, kidney papillary cell carcinoma, low-grade glioma, hepatocellular carcinoma, lung adenocarcinoma, lung squamous cell carcinoma, mesothelioma, ovarian serous cystadenocarcinoma, pancreatic cancer, pheochromocytoma and paraganglioma, prostate adenoma, rectal adenoma, sarcoma, skin melanoma, gastric adenocarcinoma, testicular germ cell tumor, thyroid cancer, thymoma, endometrial cancer, uterine carcinosarcoma, and uveal melanoma. The stained images in the first sample dataset have two staining and fixation methods: physical methods, such as cryotherapy; and chemical methods, such as formalin fixation. This application does not impose restrictions on image type labels and color fixation labels.

[0108] In some embodiments, the second loss of the i-th iteration can be calculated based on formulas (2)-(4). As shown in formulas (2)-(4):

[0109] (2);

[0110] (3);

[0111] (4);

[0112] in, This represents the first category loss; Represents the cross-entropy loss function; Indicates the first category of information; Indicates the image type label; Indicates the second category loss; This indicates information categorized as a second category; Indicates a dyed and fixed label; This indicates the second loss.

[0113] 306. Based on the image classification model of the i-th iteration, the server performs feature comparison on the anchor image, positive sample image and negative sample image to obtain the third loss of the i-th iteration. The anchor image is the second enhanced image, the positive sample image is the third enhanced image, and the negative sample image is obtained by randomly sampling the sample stained image based on the anchor image. The third loss is a triplet loss and is used to indicate the difference between the enhanced images of the sample stained image.

[0114] In the embodiments of this application, the third loss of the i-th iteration is the training loss of the three sample branches in the i-th iteration, the third loss of the i-th iteration is the triplet loss, and the third loss of the i-th iteration can indicate the difference between the enhanced images of the sample stained images.

[0115] In some embodiments, the server extracts features from the anchor image, positive sample image, and negative sample image based on the image classification model in the i-th iteration, obtaining third, fourth, and fifth enhancement features. These third, fourth, and fifth enhancement features originate from the same stained sample image. The third and fourth enhancement features originate from different states of the same sample image, i.e., features of the enhanced image obtained by enhancing the sample image with different enhancement parameters. The server determines the third loss for the i-th iteration based on the differences between the third, fourth, and fifth enhancement features. By calculating the triplet loss, the features of the anchor image are made as close as possible to the positive sample image and as far away from the negative sample image as possible. In some embodiments, the positive sample image can be obtained by enhancing the anchor image.

[0116] In some embodiments, the third loss of the i-th iteration can be calculated based on formula (5). As shown in formula (5):

[0117] (5);

[0118] in, Indicates the third loss; Indicates the activation function; Indicates L2 normal form; This indicates the third enhancement feature; This indicates the fourth enhancement feature; This indicates the fifth enhancement feature; Indicates the edge parameter.

[0119] It should be noted that this application embodiment uses the example of randomly sampling a stained image to determine a negative sample image. In some embodiments, the positive sample image is the nearest neighbor sample of the anchor image, and the negative sample image is a distant sample of the anchor image. This application embodiment does not limit the sampling method.

[0120] 307. The server adjusts the model parameters of the image classification model for the i-th iteration based on the first loss, second loss, and third loss of the i-th iteration.

[0121] In this embodiment, the server can perform a weighted summation of the first loss, the second loss, and the third loss of the i-th iteration to obtain the fourth loss of the i-th iteration, which is the total loss of the i-th iteration. Based on the fourth loss of the i-th iteration, the server adjusts the model parameters of the image classification model for the i-th iteration to complete the i-th iteration.

[0122] In some embodiments, the fourth loss of the i-th iteration can be calculated based on formula (6). As shown in formula (6):

[0123] (6);

[0124] in, This indicates the fourth loss; Indicates the first loss. and This represents the loss weight, with a value ranging from 0.1 to 2. This represents the first category loss; Indicates the second category loss; Indicates the second loss; This indicates the third loss.

[0125] It should be noted that, in order to make the iterative process described in steps 303 to 307 above easier to understand, embodiments of this application show... Figure 5 , Figure 5 This is a schematic diagram of an iterative process provided according to an embodiment of this application. See also... Figure 5As shown, the image classification model is a module that performs three branches of tasks. The main branch includes encoder 0, projector 0, projector 1, and predictor; the auxiliary classification branch includes encoder 1, classifier 0, and classifier 1; and the three-sample branch includes encoder 1 and projector 2. One iteration of this image classification model includes the following steps: 501. Image sampling and image enhancement are performed on the stained sample image to obtain a first enhanced image, a second enhanced image, and a third enhanced image, where the image enhancement method is translation; random sampling is performed on the stained sample image to obtain a negative sample image. 502. Feature extraction is performed on the first enhanced image based on encoder 0 and projector 0 to obtain the first enhanced feature; feature extraction is performed on the second enhanced image based on encoder 1, projector 1, and predictor to obtain the second enhanced feature; the first loss is determined based on the first and second enhanced features. Here, encoder 0 and projector 0 are the encoder and projector in the previous iteration of the image classification model; encoder 1 and projector 1 are the encoder and projector in the current iteration of the image classification model, and the predictor is used to further extract predictive features. 503. Classify the second enhanced image based on encoder 1 and classifier 0 to obtain first classification information. Determine the first classification loss based on the first classification information and image classification labels. Classify the second enhanced image based on encoder 1 and classifier 1 to obtain second classification information. Determine the second classification loss based on the second classification information and color-fixed labels. Determine the second loss based on the first and second classification losses. 504. Extract features from the anchor image based on encoder 1 and projector 2 to obtain third enhanced features. Extract features from the positive sample image based on encoder 1 and projector 2 to obtain fourth enhanced features. Extract features from the negative sample image based on encoder 1 and projector 2 to obtain fifth enhanced features. Calculate the triplet loss based on the third, fourth, and fifth enhanced features to obtain the third loss.

[0126] It should be noted that in some embodiments, when the server adjusts the model parameters of the image classification model, it adjusts the parameters of the encoder in the image classification model.

[0127] It should be noted that steps 303 to 307 above are illustrated using the i-th round as an example. If the image classification model after parameter adjustment meets the training termination condition after the i-th round, the server executes step 308; otherwise, it continues to execute the (i+1)-th round of iteration. Of course, the server can also use the number of iteration rounds as the training termination condition; this embodiment does not impose any restrictions on the training termination condition.

[0128] 308. The server responds to the fact that the training termination condition is met and obtains the first image classification model.

[0129] In this embodiment of the application, in response to the image classification model satisfying the training termination condition after the i-th iteration, the pre-training ends and the image classification model is determined as the first image classification model obtained by pre-training; if the image classification model satisfies the non-training termination condition, the next iteration continues until the training termination condition is met.

[0130] 309. Based on the second sample dataset, the server trains the first image classification model to obtain a target image classification model, which is used to determine the cell type corresponding to the input cell image.

[0131] In this embodiment of the application, the server can fine-tune the first image classification model based on the second sample dataset to improve the classification accuracy of the first image classification model.

[0132] In some embodiments, the server retains the encoder in the first image classification model, removes the structure other than the encoder in the first image classification model, adds a fully connected layer to the first image classification model to obtain an adjusted first image classification model, and then trains the adjusted first image classification model based on the second sample dataset to obtain the target image classification model.

[0133] In some embodiments, the server classifies sample cell images in the second sample dataset based on the first image classification model to obtain sample classification information. This sample classification information indicates the probability that multiple sample cell images belong to different cell types. Then, based on the sample classification information and the cell type labels of the sample cell images, the server determines the target image classification model. Specifically, for any sample cell image in the second sample dataset, the server classifies the sample cell image based on the first image classification model to obtain the probability that the sample cell image belongs to different cell types. Then, based on the probability and the cell type labels of the sample cell images, the server adjusts the parameters of the first image classification model. Based on the adjusted first image classification model, the server classifies other sample cell images, repeating the above classification and parameter adjustment process until the target image classification model is obtained.

[0134] 310. Based on the target image classification model, the server classifies the stained image to be processed and obtains a classification result image, which is labeled with the cell types of the multiple cell images.

[0135] In this embodiment, the server receives a stained image to be processed uploaded by the terminal. The server performs cell sampling on the stained image to obtain multiple cell images, each including stained cells. Then, the server classifies each of the multiple cell images based on a target image classification model, obtaining classification result information indicating the cell type of each cell image. Based on this classification result information, the server generates a classification result image, which labels the cell types of the multiple cell images. The server returns the classification result image to the terminal for display. By sampling cells from the stained image, multiple cell images can be obtained as input images for the target image classification model. The model then classifies these cell images, and the classification results are labeled in the stained image to obtain the classification result image. This classification result image visually displays the cell type of each cell in the stained image.

[0136] For example, Figure 6 This is an input / output comparison diagram provided according to an embodiment of this application. See also... Figure 6 As shown, the input image is a stained image containing multiple cells. In the output image, cells of the same type are labeled with the same color, and cells of different types are labeled with different colors.

[0137] It should be noted that the solution provided in this application fully utilizes large-scale data and the image type labels and color-fixed labels inherent in the data itself to pre-train the image classification model, thereby improving the model's classification accuracy. Table 1 exemplarily illustrates the advantages of the solution provided in this application embodiment compared with other solutions in terms of accuracy and F1 score. The dataset used for comparison is the CoNSeP dataset.

[0138] Table 1

[0139]

[0140] The solution provided in this application introduces easily obtainable image type labels and staining fixed labels during pre-training, enabling the trained first image classification model to learn the features of the sample stained image itself. Since the sample stained image includes multiple stained cells, the first image classification model can classify cells more accurately. Further training the first image classification model based on the sample cell image and cell labels can further improve the classification accuracy of the model, resulting in a target image classification model with high classification accuracy.

[0141] Figure 7This is a flowchart of an image classification method provided according to an embodiment of this application, such as... Figure 7 As shown in the embodiment of this application, the method is described using an example executed by a terminal. The image classification method includes the following steps:

[0142] 701. The terminal performs cell sampling on the stained image to be processed, obtaining multiple cell images, each of which includes multiple stained cells.

[0143] In this embodiment of the application, the terminal can use a cell sampling algorithm to sample the stained image to be processed to obtain multiple cell images, and each cell image includes a stained cell in the stained image.

[0144] 702. The terminal classifies multiple cell images based on the target image classification model to obtain classification result information, which is used to indicate the cell type of the multiple cell images.

[0145] In this embodiment, the target image classification model is the target image classification model trained by the server based on the image classification model training scheme in the above embodiments. The terminal can obtain the target image classification model from the server.

[0146] 703. The terminal generates a classification result image based on the classification result information. The classification result image is labeled with the cell types of multiple cell images.

[0147] In this embodiment, the terminal can annotate multiple cell types indicated in the classification result information onto the stained image to obtain a classification result image. A comparison between the stained image and the classification result image is provided above. Figure 6 As shown.

[0148] Figure 8 This is a block diagram of a training apparatus for an image classification model according to an embodiment of this application. The apparatus is used to perform the steps in the above-described image classification model training method, see [link to relevant documentation]. Figure 8 The device includes: a data acquisition module 81, a first training module 82, and a second training module 83.

[0149] The data acquisition module 81 is used to acquire a first sample dataset, which includes multiple sample staining images, image type labels of the sample staining images, and staining fixation labels. The sample staining images include multiple stained cells, and the staining fixation labels are used to indicate the staining fixation method of the corresponding sample staining images.

[0150] The data acquisition module 81 is also used to acquire a second sample dataset, which includes multiple sample cell images and cell type labels for the sample cell images;

[0151] The first training module 82 is used to pre-train the image classification model based on the first sample dataset in a self-supervised manner to obtain the first image classification model.

[0152] The second training module 83 is used to train the first image classification model based on the second sample dataset to obtain a target image classification model, which is used to determine the cell type corresponding to the input cell image.

[0153] In some embodiments, Figure 9 This is a block diagram of a training apparatus for another image classification model provided according to an embodiment of this application. See also... Figure 9 As shown, the first training module 82 includes:

[0154] The image processing submodule 821 is used to perform image sampling and image enhancement on the sample stained image input in the i-th iteration based on the image classification model of the i-th iteration, to obtain a first enhanced image, a second enhanced image and a third enhanced image, where i is a positive integer greater than 1;

[0155] The model training submodule 822 is used to train the image classification model for the i-th iteration based on the first enhanced image, the second enhanced image, and the third enhanced image;

[0156] The model training submodule 822 is also used to obtain the first image classification model in response to the satisfaction of the training termination condition.

[0157] In some embodiments, see Figure 9 As shown, the model training submodule 822 includes:

[0158] The first training unit 8221 is used to perform feature comparison on the first enhanced image and the second enhanced image based on the image classification model of the (i-1)th iteration and the image classification model of the i-th iteration, and obtain the first loss of the i-th iteration. The first loss is used to indicate the difference before and after the image classification model is updated.

[0159] The second training unit 8222 is used to determine the second loss of the i-th iteration based on the classification result of the image classification model of the i-th iteration on the second enhanced image, the image type label of the sample stained image, and the stained fixed label of the sample stained image. The second loss is used to indicate the classification loss of the image classification model.

[0160] The third training unit 8223 is used to perform feature comparison on the anchor image, positive sample image and negative sample image based on the image classification model of the i-th iteration, and obtain the third loss of the i-th iteration. The anchor image is the second enhanced image, the positive sample image is the third enhanced image, and the negative sample image is obtained by randomly sampling the sample stained image based on the anchor image. The third loss is a triplet loss and is used to indicate the difference between the enhanced images of the sample stained image.

[0161] The fourth training unit 8224 is used to adjust the model parameters of the image classification model in the i-th iteration based on the first loss, the second loss and the third loss.

[0162] In some embodiments, the first training unit 8221 is used to extract features from the first enhanced image based on the image classification model in the (i-1)th iteration to obtain a first enhanced feature; to extract features from the second enhanced image based on the image classification model in the i-th iteration to obtain a second enhanced feature; and to determine a first loss in the i-th iteration based on the difference between the first enhanced feature and the second enhanced feature.

[0163] In some embodiments, the second training unit 8222 is configured to classify the second enhanced image based on the image classification model of the i-th iteration to obtain first classification information and second classification information. The first classification information is used to indicate the probability that the second enhanced image belongs to different image types, and the second classification information is used to indicate the probability that the second enhanced image belongs to different staining fixation methods. Based on the first classification information and the image type label of the sample stained image, a first classification loss is determined. Based on the second classification information and the staining fixation label of the sample stained image, a second classification loss is determined. The first classification loss and the second classification loss are summed to obtain the second loss of the i-th iteration.

[0164] In some embodiments, the third training unit 8223 is used to extract features from the anchor image, the positive sample image, and the negative sample image based on the image classification model of the i-th iteration, to obtain a third enhanced feature, a fourth enhanced feature, and a fifth enhanced feature; and to determine the third loss of the i-th iteration based on the differences between the third enhanced feature, the fourth enhanced feature, and the fifth enhanced feature.

[0165] In some embodiments, the fourth training unit 8224 is used to perform a weighted summation of the first loss, the second loss, and the third loss to obtain a fourth loss; and to adjust the model parameters of the image classification model in the i-th iteration based on the fourth loss.

[0166] In some embodiments, the image processing submodule 821 is used to perform image sampling on the sample staining image to obtain a sample sampling image; and to enhance the sample sampling image using different image enhancement methods to obtain the first enhanced image, the second enhanced image, and the third enhanced image.

[0167] In some embodiments, the second training module 83 is used to classify sample cell images in the second sample dataset based on the first image classification model to obtain sample classification information, which is used to indicate the probability that multiple sample cell images belong to different cell types; and to determine the target image classification model based on the sample classification information and the cell type label of the sample cell image.

[0168] The apparatus provided in this application introduces easily obtainable image type labels and staining fixed labels during pre-training, enabling the trained first image classification model to learn the features of the sample stained image itself. Since the sample stained image includes multiple stained cells, the first image classification model can classify the cells more accurately. Further training the first image classification model based on the sample cell image and cell labels can further improve the classification accuracy of the model, resulting in a target image classification model with high classification accuracy.

[0169] It should be noted that the image classification model training device provided in the above embodiments is only illustrated by the division of the above functional modules when training the image classification model. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image classification model training device and the image classification model training method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0170] Figure 10 This is a block diagram of an image classification apparatus according to an embodiment of this application. The apparatus is used to perform the steps in the image classification method described above. See also... Figure 10 The device includes: a cell sampling module 1001, an image classification module 1002, and an image generation module 1003.

[0171] The cell sampling module 1001 is used to sample cells from the stained image to be processed, thereby obtaining multiple cell images, which include multiple stained cells.

[0172] The image classification module 1002 is used to classify the multiple cell images based on the target image classification model to obtain classification result information. The target image classification model is trained based on a first sample dataset and a second sample dataset. The first sample dataset includes multiple sample stained images, image type labels of the sample stained images, and staining fixation labels. The sample stained images include multiple stained cells. The staining fixation labels are used to indicate the staining fixation method of the corresponding sample stained images. The second sample dataset includes multiple sample cell images and cell type labels of the sample cell images. The classification result information is used to indicate the cell type of the multiple cell images.

[0173] The image generation module 1003 is used to generate a classification result image based on the classification result information, wherein the classification result image is labeled with the cell types of the multiple cell images.

[0174] It should be noted that the image classification device provided in the above embodiments is only illustrated by the division of the above functional modules when performing image classification. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image classification device and the image classification method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0175] In the embodiments of this application, the computer device can be configured as a terminal or a server. When the computer device is configured as a terminal, the terminal can act as the execution subject to implement the technical solutions provided in the embodiments of this application. When the computer device is configured as a server, the server can act as the execution subject to implement the technical solutions provided in the embodiments of this application. Alternatively, the technical solutions provided in this application can be implemented through the interaction between the terminal and the server. The embodiments of this application do not limit this.

[0176] When the computer device is a terminal, Figure 11 This is a structural block diagram of a terminal 1100 provided according to an embodiment of this application. The terminal 1100 can be a portable mobile terminal, such as a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The terminal 1100 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0177] Typically, terminal 1100 includes a processor 1101 and a memory 1102.

[0178] Processor 1101 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1101 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1101 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1101 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1101 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0179] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1102 are used to store at least one computer program, which is executed by the processor 1101 to implement the image classification method or the image classification model training method provided in the method embodiments of this application.

[0180] In some embodiments, the terminal 1100 may also optionally include a peripheral device interface 1103 and at least one peripheral device. The processor 1101, memory 1102, and peripheral device interface 1103 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1103 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 1104, a display screen 1105, a camera assembly 1106, an audio circuit 1107, and a power supply 1109.

[0181] Peripheral device interface 1103 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1101 and memory 1102. In some embodiments, processor 1101, memory 1102 and peripheral device interface 1103 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1101, memory 1102 and peripheral device interface 1103 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0182] The radio frequency (RF) circuit 1104 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1104 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1104 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. In some embodiments, the RF circuit 1104 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1104 can communicate with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1104 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0183] Display screen 1105 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1105 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1101 for processing. In this case, display screen 1105 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1105, disposed on the front panel of terminal 1100; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 1100 or in a folded design; in still other embodiments, display screen 1105 may be a flexible display screen, disposed on a curved or folded surface of terminal 1100. Furthermore, display screen 1105 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1105 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0184] The camera assembly 1106 is used to acquire images or videos. In some embodiments, the camera assembly 1106 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1106 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash is a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0185] The audio circuit 1107 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1101 for processing, or input to the radio frequency circuit 1104 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 1100. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1101 or the radio frequency circuit 1104 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1107 may also include a headphone jack.

[0186] Power supply 1109 is used to power the various components in terminal 1100. Power supply 1109 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1109 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0187] In some embodiments, the terminal 1100 further includes one or more sensors 1110. The one or more sensors 1110 include, but are not limited to: an acceleration sensor 1111, a gyroscope sensor 1112, a pressure sensor 1113, an optical sensor 1115, and a proximity sensor 1116.

[0188] Accelerometer 1111 can detect the magnitude of acceleration along the three axes of a coordinate system established with terminal 1100. For example, accelerometer 1111 can be used to detect the components of gravitational acceleration along the three axes. Processor 1101 can control display screen 1105 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 1111. Accelerometer 1111 can also be used for collecting game or user motion data.

[0189] The gyroscope sensor 1112 can detect the orientation and rotation angle of the terminal 1100. The gyroscope sensor 1112 can work in conjunction with the accelerometer sensor 1111 to collect the user's 3D movements on the terminal 1100. Based on the data collected by the gyroscope sensor 1112, the processor 1101 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0190] The pressure sensor 1113 can be disposed on the side bezel of the terminal 1100 and / or on the lower layer of the display screen 1105. When the pressure sensor 1113 is disposed on the side bezel of the terminal 1100, it can detect the user's grip signal on the terminal 1100, and the processor 1101 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1113. When the pressure sensor 1113 is disposed on the lower layer of the display screen 1105, the processor 1101 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1105. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0191] An optical sensor 1115 is used to collect ambient light intensity. In one embodiment, the processor 1101 can control the display brightness of the display screen 1105 based on the ambient light intensity collected by the optical sensor 1115. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1105 is increased; when the ambient light intensity is low, the display brightness of the display screen 1105 is decreased. In another embodiment, the processor 1101 can also dynamically adjust the shooting parameters of the camera assembly 1106 based on the ambient light intensity collected by the optical sensor 1115.

[0192] The proximity sensor 1116, also known as a distance sensor, is typically mounted on the front panel of the terminal 1100. The proximity sensor 1116 is used to detect the distance between the user and the front of the terminal 1100. In one embodiment, when the proximity sensor 1116 detects that the distance between the user and the front of the terminal 1100 is gradually decreasing, the processor 1101 controls the display screen 1105 to switch from a screen-on state to a screen-off state; when the proximity sensor 1116 detects that the distance between the user and the front of the terminal 1100 is gradually increasing, the processor 1101 controls the display screen 1105 to switch from a screen-off state to a screen-on state.

[0193] Those skilled in the art will understand that Figure 11 The structure shown does not constitute a limitation on terminal 1100 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0194] When the computer device is a server, Figure 12This is a schematic diagram of a server structure according to an embodiment of this application. The server 1200 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1201 and one or more memories 1202. The memory 1202 stores at least one computer program, which is loaded and executed by the processor 1201 to implement the image classification model training method or image classification method provided in the above-described method embodiments. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated here.

[0195] This application also provides a computer-readable storage medium storing at least one computer program. This computer program is loaded and executed by a processor of a computer device to implement the operations performed by the computer device in the image classification model training method of the above embodiments, or to implement the operations performed by the computer device in the image classification method of the above embodiments. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0196] In some embodiments, the computer program involved in the present application embodiments may be deployed and executed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network. Multiple computer devices distributed in multiple locations and interconnected through a communication network may constitute a blockchain system.

[0197] This application also provides a computer program product including computer program code stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the computer device to perform the training method for the image classification model provided in the various optional implementations described above, or causing the computer device to perform the image classification method provided in the various optional implementations described above.

[0198] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0199] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for training an image classification model, the method comprising: The method comprises: obtaining a first sample data set, the first sample data set comprising a plurality of sample staining images, image type labels of the sample staining images, and staining fixation labels, the sample staining images comprising a plurality of stained cells, and the staining fixation labels being used to indicate staining fixation manners of corresponding sample staining images; obtaining a second sample data set, the second sample data set comprising a plurality of sample cell images and cell type labels of the sample cell images; for a sample staining image input in an i-th iteration, performing image sampling and image enhancement on the sample staining image based on an image classification model of the i-th iteration to obtain a second enhanced image, i being a positive integer greater than 1; determining a second loss of the i-th iteration based on a result of classifying the second enhanced image by the image classification model of the i-th iteration, the image type label of the sample staining image, and the staining fixation label of the sample staining image, the second loss being used to indicate a classification loss of the image classification model; and adjusting model parameters of the image classification model of the i-th iteration based on the second loss; in response to a training end condition being met, obtaining a first image classification model; training the first image classification model based on the second sample data set to obtain a target image classification model, the target image classification model being used to determine a cell type corresponding to an input cell image.

2. The method of claim 1, wherein, The method further comprises: for a sample staining image input in an i-th iteration, performing image sampling and image enhancement on the sample staining image based on an image classification model of the i-th iteration to obtain a first enhanced image, a second enhanced image, and a third enhanced image; The method further comprises: performing feature comparison on the first enhanced image and the second enhanced image based on an image classification model of an (i-1)-th iteration and the image classification model of the i-th iteration to obtain a first loss of the i-th iteration, the first loss being used to indicate differences before and after the image classification model is updated; performing feature comparison on an anchor image, a positive sample image, and a negative sample image based on the image classification model of the i-th iteration to obtain a third loss of the i-th iteration, the anchor image being the second enhanced image, the positive sample image being the third enhanced image, the negative sample image being obtained by randomly sampling the sample staining image based on the anchor image, the third loss being a triplet loss, and the third loss being used to indicate differences between enhanced images of the sample staining image; The method further comprises: adjusting the model parameters of the image classification model of the i-th iteration based on the first loss, the second loss, and the third loss.

3. The method of claim 2, wherein, The first loss of the i-th iteration is obtained by comparing features of the first enhanced image and the second enhanced image based on the image classification model of the i-1-th iteration and the image classification model of the i-th iteration, and the first loss comprises: The first enhanced feature is obtained by extracting features of the first enhanced image based on the image classification model of the i-1-th iteration; The second enhanced feature is obtained by extracting features of the second enhanced image based on the image classification model of the i-th iteration; The first loss of the i-th iteration is determined based on the difference between the first enhanced feature and the second enhanced feature.

4. The method of claim 1, wherein, The second loss of the i-th iteration is determined based on a result of classifying the second enhanced image by the image classification model of the i-th iteration, an image type label of the sample staining image, and a staining fixation label of the sample staining image, and the second loss comprises: The first classification information and the second classification information are obtained by classifying the second enhanced image based on the image classification model of the i-th iteration, the first classification information is used to indicate a probability that the second enhanced image belongs to different image types, and the second classification information is used to indicate a probability that the second enhanced image belongs to different staining fixation modes; The first classification loss is determined based on the first classification information and the image type label of the sample staining image; The second classification loss is determined based on the second classification information and the staining fixation label of the sample staining image; The second loss of the i-th iteration is obtained by summing the first classification loss and the second classification loss.

5. The method of claim 2, wherein, The third loss of the i-th iteration is obtained by comparing features of an anchor image, a positive sample image and a negative sample image based on the image classification model of the i-th iteration, and the third loss comprises: The third enhanced feature, the fourth enhanced feature and the fifth enhanced feature are obtained by extracting features of the anchor image, the positive sample image and the negative sample image respectively based on the image classification model of the i-th iteration; The third loss of the i-th iteration is determined based on the difference between the third enhanced feature, the fourth enhanced feature and the fifth enhanced feature.

6. The method of claim 2, wherein, The model parameters of the image classification model of the i-th iteration are adjusted based on the first loss, the second loss and the third loss, and the adjustment comprises: The fourth loss is obtained by weighted summing the first loss, the second loss and the third loss; The model parameters of the image classification model of the i-th iteration are adjusted based on the fourth loss.

7. The method of claim 2, wherein, The first enhanced image, the second enhanced image and a third enhanced image are obtained by image sampling and image enhancement on the sample staining image, and the image sampling and the image enhancement comprise: The sample sampling image is obtained by image sampling on the sample staining image; The first enhanced image, the second enhanced image and the third enhanced image are obtained by enhancing the sample sampling image by using different image enhancement manners.

8. The method of claim 1, wherein, The target image classification model is obtained by training the first image classification model based on the second sample data set, and the training comprises: based on the first image classification model, respectively classifying sample cell images in the second sample data set to obtain sample classification information, the sample classification information being used to indicate probabilities of the sample cell images belonging to different cell types; based on the sample classification information and cell type labels of the sample cell images, determining the target image classification model.

9. An image classification method characterized by, The method comprises: performing cell sampling on a to-be-processed stained image to obtain a plurality of cell images, the stained image comprising a plurality of dyed cells; based on a target image classification model, respectively classifying the plurality of cell images to obtain classification result information, the target image classification model being obtained by training a first image classification model based on a second sample data set, the first image classification model being obtained by training an image classification model based on a first sample data set, the first sample data set comprising a plurality of sample stained images, image type labels of the sample stained images, and a staining fixation label, the sample stained images comprising a plurality of dyed cells, the staining fixation label being used to indicate a staining fixation mode of a corresponding sample stained image, the second sample data set comprising a plurality of sample cell images and cell type labels of the sample cell images, the classification result information being used to indicate cell types of the plurality of cell images; based on the classification result information, generating a classification result image, the classification result image being annotated with the cell types of the plurality of cell images; wherein the training process of the image classification model comprises: for a sample stained image input in an i-th iteration, based on an image classification model in the i-th iteration, performing image sampling and image enhancement on the sample stained image to obtain a second enhanced image, i being a positive integer greater than 1; based on a result of classifying the second enhanced image by the image classification model in the i-th iteration, the image type label of the sample stained image, and the staining fixation label of the sample stained image, determining a second loss of the i-th iteration, the second loss being used to indicate a classification loss of the image classification model; based on the second loss, adjusting model parameters of the image classification model in the i-th iteration; in response to satisfying a training end condition, obtaining the first image classification model.

10. A training device for an image classification model, characterized in that, The device comprises: a data acquisition module configured to acquire a first sample data set, the first sample data set comprising a plurality of sample stained images, image type labels of the sample stained images, and a staining fixation label, the sample stained images comprising a plurality of dyed cells, the staining fixation label being used to indicate a staining fixation mode of a corresponding sample stained image; the data acquisition module is further configured to acquire a second sample data set, the second sample data set comprising a plurality of sample cell images and cell type labels of the sample cell images; a first training module configured to, for a sample stained image input in an i-th iteration, based on an image classification model in the i-th iteration, perform image sampling and image enhancement on the sample stained image to obtain a second enhanced image, i being a positive integer greater than 1; The first training module is further configured to determine a second loss of the i th iteration based on a result of classifying the second enhanced image by the image classification model of the i th iteration, an image type label of the sample stained image, and a staining fixation label of the sample stained image, the second loss being used to indicate a classification loss of the image classification model; and adjust model parameters of the image classification model of the i th iteration based on the second loss. The first training module is further configured to obtain the first image classification model in response to satisfying a training end condition. The second training module is configured to train the first image classification model based on the second sample data set to obtain a target image classification model, the target image classification model being used to determine a cell type corresponding to an input cell image.

11. An image classification apparatus characterized by comprising: The device comprises: The cell sampling module is configured to perform cell sampling on a to-be-processed stained image to obtain a plurality of cell images, the stained image comprising a plurality of dyed cells. The image classification module is configured to classify the plurality of cell images based on a target image classification model to obtain classification result information, the target image classification model being obtained by training a first image classification model based on a second sample data set, the first image classification model being obtained by training an image classification model based on a first sample data set, the first sample data set comprising a plurality of sample stained images, image type labels of the sample stained images, and staining fixation labels, the sample stained images comprising a plurality of dyed cells, the staining fixation labels being used to indicate staining fixation modes of the corresponding sample stained images, the second sample data set comprising a plurality of sample cell images and cell type labels of the sample cell images, and the classification result information being used to indicate cell types of the plurality of cell images. The image generation module is configured to generate a classification result image based on the classification result information, the classification result image being annotated with the cell types of the plurality of cell images. The training process of the image classification model comprises: For a sample stained image input in an i th iteration, performing image sampling and image enhancement on the sample stained image based on an image classification model of the i th iteration to obtain a second enhanced image, i being a positive integer greater than 1. Determine a second loss of the i th iteration based on a result of classifying the second enhanced image by the image classification model of the i th iteration, an image type label of the sample stained image, and a staining fixation label of the sample stained image, the second loss being used to indicate a classification loss of the image classification model; and adjust model parameters of the image classification model of the i th iteration based on the second loss. Obtain the first image classification model in response to satisfying a training end condition.

12. A computer device, comprising: The computer device comprises a processor and a memory, the memory is used to store at least one computer program, the at least one computer program is loaded by the processor and executes the training method of the image classification model as claimed in any one of claims 1 to 8, or the at least one computer program is loaded by the processor and executes the image classification method as claimed in claim 9.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium is used to store at least one computer program, the at least one computer program is used to execute the training method of the image classification model as claimed in any one of claims 1 to 8, or the at least one computer program is used to execute the image classification method as claimed in claim 9.

14. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the training method of the image classification model as claimed in any one of claims 1 to 8, or the computer program is executed by the processor to implement the image classification method as claimed in claim 9.

Images