Image classification model training method, device, equipment, storage medium and product

Abstract

This application discloses a training method, apparatus, device, storage medium, and product for an image classification model, belonging to the field of artificial intelligence technology. The method comprises: determining multiple hard-to-understand images at at least two scales from multiple sample slice images, where the sample slice images are images of sample slices containing sample cells, and the hard-to-understand images are images containing hard-to-understand cells; generating target slice images based on the multiple hard-to-understand images at at least two scales, each target slice image containing hard-to-understand images at at least one scale; and training an image classification model based on the target slice images, where the image classification model is used to predict the category to which an input slice image belongs. This method can improve the prediction performance of the image classification model for hard-to-understand samples and new samples.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, equipment, storage medium and product for an image classification model. Background Art

[0002] In the field of medical diagnosis, in addition to doctors' subjective diagnosis based on experience, many auxiliary diagnostic methods exist. For example, in cancer diagnosis, automated microscopy can be used to collect digital whole-slide images (WSIs) of specific tissue sites. The WSIs can then be analyzed to determine the presence of positive tissue cells.

[0003] In related technologies, artificial intelligence has been applied to pathology-assisted diagnosis. By extracting cell-level features from WSI and aggregating these features, slice-level features are generated. By training an image classification model with slice-level features and slice labels, an image classification model capable of slice classification can be developed. This image classification model can then be used to assist in screening for positive or negative results in scanned slice images.

[0004] Obviously, when using image classification models for pathology-assisted diagnosis, the accuracy of the model's category predictions is closely related to the training samples used to train the model. Setting up training samples is a crucial issue to improve the performance of image classification models. Summary of the Invention

[0005] This application provides a training method, apparatus, device, storage medium, and product for an image classification model, which can improve the prediction accuracy of the image classification model. The technical solution is as follows:

[0006] According to one aspect of the present application, a method for training an image classification model is provided, the method comprising:

[0007] Determining a plurality of hard-example images at at least two scales from a plurality of sample slice images, wherein the sample slice images are images of sample slices, the sample slices contain sample cells, and the hard-example images are images containing hard-example cells;

[0008] generating a target slice image based on the multiple hard-example images at the at least two scales, each of the target slice images including a hard-example image at at least one scale;

[0009] Based on the target slice image, an image classification model is trained, and the image classification model is used to predict the category to which the input slice image belongs.

[0010] According to another aspect of the present application, a training device for an image classification model is provided, the device comprising:

[0011] a determination module, configured to determine a plurality of hard-example images of at least two scales from a plurality of sample slice images, wherein the sample slice images are images of sample slices containing sample cells, and the hard-example images are images containing hard-example cells;

[0012] a generating module, configured to generate a target slice image based on the plurality of hard-example images at the at least two scales, wherein each of the target slice images includes a hard-example image at at least one scale;

[0013] A training module is used to train an image classification model based on the target slice image, and the image classification model is used to predict the category to which the input slice image belongs.

[0014] According to one aspect of the present application, a computer device is provided, comprising: a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the above-mentioned image classification model training method.

[0015] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the training method of the image classification model as described above.

[0016] According to another aspect of the present application, a computer program product is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described image classification model training method.

[0017] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0018] By mining multi-scale hard examples on sample slice images, multiple hard example images at at least two scales are obtained. Based on these multiple hard example images at at least two scales, new target slice images containing hard example images are generated, and the image classification model is trained using these new target slice images. This allows the image classification model to specifically learn hard example samples, improving the image classification model's prediction performance for both hard and new examples. Furthermore, by mining multi-scale hard example samples, the image classification model can fully learn hard example samples at different scales, further improving the classification performance of the image classification model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 This is a schematic diagram of the principle of category prediction for slice images;

[0021] Figure 2 is a schematic diagram of the training architecture of an image classification model shown in an exemplary embodiment of the present application;

[0022] Figure 3 is a schematic diagram of a computer system shown in an exemplary embodiment of the present application;

[0023] Figure 4 A flowchart of a method for training an image classification model provided by an exemplary embodiment of the present application is shown;

[0024] Figure 5 A flowchart of a method for training an image classification model provided by another exemplary embodiment of the present application is shown;

[0025] Figure 6 A schematic diagram showing a difficult slice image determination process according to an exemplary embodiment of the present application is shown;

[0026] Figure 7 A schematic diagram of a target slice image generation process shown in an exemplary embodiment of the present application is shown;

[0027] Figure 8 A schematic diagram of a target slice image generation process shown in another exemplary embodiment of the present application is shown;

[0028] Figure 9 A schematic diagram of a process for determining a difficult cell region image according to an exemplary embodiment of the present application is shown;

[0029] Figure 10 A schematic diagram of the training architecture of an image classification model shown in another exemplary embodiment of the present application is shown;

[0030] Figure 11 is a schematic diagram of an application scenario of a medical diagnosis scenario shown in an exemplary embodiment of the present application;

[0031] Figure 12 is a schematic diagram of an application scenario of a teaching scenario shown in an exemplary embodiment of the present application;

[0032] Figure 13This is a structural block diagram of a training device for an image classification model provided by an exemplary embodiment of the present application;

[0033] Figure 14 The figure is a schematic structural diagram of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION

[0034] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0035] First, a brief introduction to the terms involved in the embodiments of this application is given:

[0036] Whole Slide Image (WSI): WSI is a visual digital image produced by scanning traditional pathology slices with a digital scanner and collecting high-resolution images, and then seamlessly splicing the collected fragmented images through a computer. WSI can be zoomed in and out at any ratio, moved and browsed in any direction, etc. by using specific software. In the embodiment of the present application, both the sample slice image and the slice image to be classified are WSI. In the related art, when performing slice classification on WSI, WSI is often regarded as a collection composed of multiple cells, and equivalent cell subsets are screened out from WSI, and then cell features are extracted from the equivalent cell subsets to obtain cell-level features. The cell-level features are used to represent the slice features corresponding to the WSI, and subsequent category predictions are performed. For example, Figure 1 , which is a schematic diagram of the principle of class prediction for slice images. A sample slice image 101 is subjected to cell screening to obtain an equivalent cell subset 102 corresponding to the sample slice image 101. The equivalent cell subset 102 includes images of specific sample cells screened from the sample slice (the sample slice corresponding to the sample slice image 101). The equivalent cell subset 102 is input into an image classification model 103, and a prediction result 104 is output by the image classification model 103.

[0037] Hard Example: Hard examples refer to some samples that cause large loss values during model training, or samples with low model prediction accuracy. They can also be called difficult samples or complex samples. In the process of model training, how to better mine hard examples and use the mined hard examples to retrain the model is the key to improving model training. In an embodiment of the present application, by performing multi-scale hard example mining on sample slice images, hard example images of multiple scales are obtained, such as hard example images at the cell scale, hard example images at the slice scale, etc., and then the multi-scale hard example images are applied to the training process of the image classification model to improve the prediction accuracy of the image classification model.

[0038] In this application, we will focus on how to generate multi-scale and multi-instance target slice images so as to use the target slice images to improve the accuracy of image classification model classification for difficult samples. Figure 2 As shown, it is a schematic diagram of the training architecture of the image classification model shown in an exemplary embodiment of the present application. By performing cell screening on the sample slice image 201, an equivalent cell subset 202 is obtained, and the equivalent cell subset 202 contains images of some sample cells screened out from the sample slice (the sample slice corresponding to the sample slice image 201). The equivalent cell subset 202 is input into the image classification model 203 to obtain the prediction result 204 output by the image classification model 203. Further, based on the prediction result 204, the sample slice image 201 is subjected to multi-scale difficult example mining to determine the difficult example cell region image 205 (the difficult example sample obtained by cell scale mining) and the difficult example slice image 206 (the difficult example sample obtained by slice scale mining). Further, by performing multi-example image splicing on the difficult example cell region image 205, the difficult example slice image 206 and the sample slice image 201, a new target slice image containing a difficult example image is generated. And the target slice image is used to train the image classification model 203.

[0039] Figure 3 FIG. 1 is a schematic diagram of a computer system shown in an exemplary embodiment of the present application. Figure 3 As shown, the computer system includes a first device 310 and a second device 320 .

[0040] The first device 310 is a training device for training an image classification model. After the image classification model training is completed, the first device 310 can send the trained image classification model to the second device 320 so that the image classification model can be deployed in the second device 320. The second device 320 is a device that uses the image classification model to perform category prediction.

[0041] During the model training phase, multi-scale hard example mining is performed on sample slice image 311 to obtain a hard example image 312 containing a hard example cell region image 311 and a hard example slice image 314. Image processing is then performed on hard example image 312 and sample slice image 315 to generate a new target slice image 315 containing hard example image 312. This target slice image 315 is then used to train image classification model 316. This allows image classification model 316 to fully learn the multi-scale hard example features, thereby improving the accuracy of image classification model 316's classification predictions for hard example samples and new samples.

[0042] In the model application stage: the slice image can be input into the trained image classification model 316, and the image classification model 316 outputs the category to which the slice image belongs.

[0043] Optionally, the first device 310 and the second device 320 may be computer devices with machine learning capabilities, for example, the computer device may be a terminal or a server.

[0044] Optionally, the first device 310 and the second device 320 may be the same computer device, or the first device 310 and the second device 320 may be different computer devices. Furthermore, when the first device 310 and the second device 320 are different devices, they may be the same type of device, such as both servers; or the first device 310 and the second device 320 may be different types of devices. The server may be an independent physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal may be a smartphone, an in-vehicle terminal, a smart TV, a wearable device, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in this application.

[0045] Please refer to Figure 4 , which shows a flowchart of a training method for an image classification model provided by an exemplary embodiment of the present application, in which the method is applied to Figure 3 Taking the first device 310 as an example, the method includes:

[0046] Step 401 : determining a plurality of hard-example images of at least two scales from a plurality of sample slice images, wherein the sample slice images are images of sample slices containing sample cells, and the hard-example images are images containing hard-example cells.

[0047] The sample slice image, or WSI, is a digital image of a sample slice, which is a slice containing sample cells. In this embodiment, the sample slice is a pathological slice prepared using histopathological methods from diseased or normal tissue. Medical personnel observe the pathological slice under a microscope for further pathological diagnosis.

[0048] Optionally, the sample slices may be pathological slices of plant, microorganism, etc. in addition to pathological slices of diseased tissue or normal tissue of human or animal; this embodiment does not constitute a limitation to this.

[0049] By applying artificial intelligence technology to pathological diagnosis scenarios, an image classification model with pathological diagnosis functions can be pre-trained. So that in the pathological diagnosis process, the image classification model can automatically distinguish whether the slice image is a positive slice image or a negative slice image, or automatically distinguish the positive category indicated by the slice image, etc., without the need for manual analysis of each sample slice image. The accuracy of pathological diagnosis in this application scenario depends on the discrimination accuracy of the image classification model, and the discrimination accuracy of the image classification model depends on the training process of the image classification model, mainly the training samples used to train the image classification model. In order to enable the image classification model to have the classification function of the slice image, in a possible embodiment, it is necessary to prepare in advance a training sample set containing the slice image, and each slice image corresponds to a labeled label (i.e., a labeled category). So that during the model training process, the slice image can be input into the image classification model for category prediction to obtain the category prediction result of the slice image. Further, the model parameters of the image classification model can be updated based on the category prediction result and the labeled category. So that the image classification model can learn the image features of the slice image and thus have the classification function of the slice image.

[0050] However, in the training process of image classification models, existing sample databases are often used to train them, without considering the proportion of simple samples or difficult samples in the sample database (difficult samples are samples with low model prediction accuracy, which can also be called difficult samples or complex samples). As a result, the model may only learn the features of simple samples during the learning process, resulting in low accuracy in the classification and discrimination of difficult samples. Therefore, in order to improve the accuracy of the image classification model in distinguishing difficult samples in the slice classification scenario, in one possible implementation, it is necessary to mine complex samples with low prediction accuracy of the image classification model, i.e., difficult images, from the existing multiple sample slice images, so that the image classification model can be trained on difficult images in a targeted manner based on the difficult images.

[0051] Since the process of using an image classification model to discriminate slice images is a multi-scale judgment process: it is necessary to first select regional images corresponding to some sample cells from the slice image, perform feature extraction on the regional images of these sample cells to obtain regional image features, and then aggregate the regional image features to obtain slice image features, and then perform classification learning based on the slice image features. In other words, the slice image discrimination process involves a multi-scale learning process from regional image features to slice image features. Therefore, in order to enable the image classification model to learn difficult samples from multiple scales, in one possible embodiment, the computer device can mine multiple difficult images of at least two scales from the sample slice image during the process of determining the difficult images from the sample slice image. In other words, the multiple difficult images mined from the multiple sample slice images exist at at least two different scales.

[0052] Optionally, the multiple hard-example images at at least two scales may be hard-example cell region images at the cell scale and hard-example slice images at the slice scale. A hard-example slice image is a sample slice image that is a hard-example sample among the sample slice images, while a hard-example cell region image is an image of the region occupied by a hard-example cell in the sample slice image. Hard-example cells are cells in the sample slice that are hard-examples and for which the image classification model has a low accuracy rate of distinguishing.

[0053] Since the hard cell region image is an image of the area occupied by the hard cell in the sample slice image, that is, the hard cell region image is a local region image corresponding to a single cell in the sample slice image, corresponding to the cell scale. The hard slice image is the sample slice image that belongs to the hard sample in the sample slice image, that is, the hard slice image is the complete sample slice image, corresponding to the slice scale. Therefore, by mining the hard cell region image and the hard slice image from the sample slice image, it is possible to mine hard samples at both the cell scale and the slice scale from the sample slice image.

[0054] Optionally, in addition to performing hard example mining at the cell scale and slice scale, hard example slice images can also be segmented to obtain hard example images of different resolutions and sizes.

[0055] Step 402 : generating a target slice image based on a plurality of hard-example images at at least two scales, wherein each target slice image includes a hard-example image at at least one scale.

[0056] To improve the image classification model's prediction accuracy for difficult examples, in one possible implementation, after a computer device mines multiple difficult examples at different scales from multiple sample slice images, it can generate new training samples, namely target slice images, based on these difficult examples. These target slice images contain difficult examples at at least one scale. The target slice images can then be used to train the image classification model, increasing the model's ability to learn difficult examples.

[0057] Optionally, in order to improve the data diversity of the target slice images used for training and avoid reusing the same training samples for training, in one possible embodiment, after the computer device determines multiple hard-example images of at least two scales from the sample slice images, it can perform image stitching processing on the different hard-example images, or perform image stitching processing on the hard-example images and the sample slice images to generate a target slice image containing hard-example images of at least one scale, and then use the new target slice image to train the image classification model.

[0058] It should be noted that although multiple hard-example images of at least two scales are mined from the sample slice image, when generating the target slice image based on the hard-example image, it is only necessary to ensure that the target slice image contains hard-example images, and there is no restriction on the type and number of hard-example images contained in the target slice image; that is, the target slice image contains hard-example images of at least one scale and contains at least one hard-example image.

[0059] Optionally, the target slice image may be generated in the following ways: the difficult cell region image and the sample slice image may be subjected to image stitching processing; the difficult slice image and the sample slice image may be subjected to image stitching processing; or the difficult cell region image, the difficult slice image and the sample slice image may be subjected to image stitching processing; or multiple difficult cell region images may be subjected to image stitching processing; or multiple difficult slice images may be subjected to image stitching processing; or the difficult cell region image and the difficult slice image may be subjected to image stitching processing, etc. This embodiment does not limit the image stitching processing method, and it is only necessary to ensure that the generated target slice image contains at least a difficult image.

[0060] Optionally, during the image stitching process, an appropriate image stitching method can be used based on the needs of the business personnel or in combination with the current model performance of the image classification model. For example, if the current model performance of the image classification model indicates that its accuracy in distinguishing difficult examples is low, the proportion of difficult example images included in the newly generated target slice image can be appropriately increased, or all difficult example images can be used to generate the new target slice image.

[0061] Step 403: Based on the target slice image, an image classification model is trained. The image classification model is used to predict the category to which the input slice image belongs.

[0062] To improve the image classification model's prediction accuracy for difficult examples, in one possible implementation, a computer device inputs a target slice image containing a difficult example into the image classification model, and outputs a predicted category for the target slice image. The image classification model is then trained based on the loss between the predicted category and the corresponding labeled category of the target slice image to update its model parameters. This allows the image classification model to fully learn the characteristics of difficult examples, thereby improving the model's prediction performance.

[0063] It should be noted that before using the target slice image to train the image classification model, the image classification model has already been trained with the sample slice image. Alternatively, the image classification model has already been trained with other sample slice images that are different from the sample slice images used to mine hard-example images.

[0064] In summary, in the embodiment of the present application, by performing multi-scale hard example mining on the sample slice image, a plurality of hard example images of at least two scales are obtained, and based on the plurality of hard example images of at least two scales, a new target slice image containing the hard example image is generated, so that the new target slice image is used to perform classification training on the image classification model. This allows the image classification model to learn hard example samples in a targeted manner, thereby improving the image classification model's prediction performance for hard example samples and new samples. Moreover, by mining multi-scale hard example samples, the image classification model can also fully learn hard example samples at different scales, thereby further improving the classification performance of the image classification model.

[0065] Difficult example samples are generally samples with low model discrimination accuracy. Therefore, in a possible implementation, in the process of multi-scale difficult example mining of sample slice images, it is necessary to input the sample slice images into the image classification model to obtain the prediction results of the sample slice images output by the image classification model, and then perform multi-scale difficult example mining on the sample slice images based on the prediction results.

[0066] Please refer to Figure 5 , which shows a flowchart of a training method for an image classification model provided by another exemplary embodiment of the present application, in which the method is applied to Figure 3 Taking the first device 310 as an example, the method includes:

[0067] Step 501: Input a plurality of sample slice images into an image classification model to obtain a sample prediction result output by the image classification model. The sample prediction result at least includes the prediction categories to which the plurality of sample slice images belong.

[0068] Since hard examples are generally samples that the image classification model still cannot distinguish after initial training, or samples with low discrimination accuracy, in order to mine hard examples from sample slice images, in one possible implementation, it is necessary to input the sample slice images into the image classification model, and the image classification model performs category prediction on the sample slice images to obtain sample prediction results for each sample slice image output by the image classification model. Then, based on the sample prediction results, it is determined whether the sample slice images are samples with low discrimination accuracy.

[0069] In order to determine the accuracy of the image classification model in distinguishing the sample slice image, the predicted category of the sample slice image output by the image classification model can be compared with the labeled category corresponding to the sample slice image to determine whether the image classification model can accurately classify the sample slice image. The corresponding sample prediction result can at least include the predicted category to which the sample slice image belongs.

[0070] In the process of class prediction of sample slice images, the cell features corresponding to the sample cells in the sample slices are obtained, feature weights are assigned to the cell features, and feature aggregation is performed based on the cell features and feature weights to characterize the image features of the sample slice images. It can be seen that in the class prediction process of sample slice images, the accuracy of setting the feature weights corresponding to the cell features will affect the class prediction results of the sample slice images. Therefore, in a possible implementation, in order to mine difficult cell region images from sample slice images, the image classification model is also required to output the feature weights of the cell features corresponding to each sample cell in the sample slice image. That is to say, in addition to the predicted category to which the sample slice image belongs, the sample prediction results should also include the feature weights of the cell features corresponding to the sample cells.

[0071] Step 502 : Based on the sample prediction results and the labeled categories corresponding to the sample slice images, a plurality of difficult example images of at least two scales are determined from the plurality of sample slice images.

[0072] In one possible implementation, after the computer device obtains the sample prediction results output by the image classification model, it can determine multiple difficult-example images of multiple scales from multiple sample slices based on the sample prediction results and the corresponding annotations of the sample slice images.

[0073] In an embodiment of the present application, the multiple hard-example images at at least two scales include hard-example cell region images at the cell scale and hard-example slice images at the slice scale; wherein the hard-example slice image at the slice scale indicates that the entire sample slice image belonging to the hard-example sample is determined as the hard-example image; and the hard-example cell region image at the cell scale indicates that the image region occupied by the hard-example cell in the sample slice is determined as the hard-example image, that is, the hard-example cell region image only contains a single hard-example cell.

[0074] This embodiment is mainly described by determining a difficult slice image from a sample slice image. Correspondingly, in an exemplary example, step 502 may include step 502A and step 502B.

[0075] Step 502A: Determine the prediction losses corresponding to the plurality of sample slice images based on the sample prediction results and the labeled categories.

[0076] The difficult slice images are a part of the sample slice images with lower prediction accuracy in the sample slice images. Corresponding to a possible implementation method, when the computer device obtains the sample prediction result corresponding to each sample slice image, the prediction loss between the sample prediction result (prediction category) and the labeled category can be determined. The size of the prediction loss represents the accuracy of the image classification model in predicting the category of the sample slice image.

[0077] Optionally, the prediction loss can be calculated using a cross entropy method.

[0078] Step 502B: determining a hard-example slice image from the plurality of sample slice images based on the prediction loss.

[0079] After the computer device obtains the prediction loss corresponding to the sample slice image, it can select sample slice images with lower prediction accuracy from multiple sample slice images based on the prediction loss of the sample slice image, and determine these sample slice images as difficult example slice images.

[0080] In the process of determining difficult slice images based on prediction loss, the sample slice images can be sorted according to the prediction loss to select a portion of the sample slice images with a larger prediction loss in the sorting process and determine them as difficult slice images. Alternatively, a prediction loss threshold can be set to determine a portion of the sample slice images above the prediction loss threshold as difficult slice images. Step 502B can also include the following steps 502B1 or 502B2.

[0081] 502B1 , sorting the plurality of sample slice images from high to low according to the prediction loss to obtain a third image sequence; determining the first k sample slice images in the third image sequence as hard-example slice images, where k is a positive integer.

[0082] It is known that the greater the prediction loss, the greater the difference between the predicted category and the labeled category of the sample slice image, the lower the image classification model's prediction accuracy for the sample slice image, and the image classification model has not yet fully learned the image features of this portion of the sample slice images. Therefore, in one possible implementation, the sample slice images can be sorted from high to low according to the prediction loss to obtain a third image sequence. Then, a portion of sample slice images with higher prediction losses are selected from the third image sequence, for example, the first k sample slice images in the third image sequence, as difficult slice images. This allows the image classification model to subsequently learn the image features of this portion of the sample slice images in a targeted manner.

[0083] Here, k can be a fixed value. For example, if k is 10, the first 10 sample slice images in the third image sequence are selected as difficult slice images. Optionally, the value of k can also be dynamically changed, and can be dynamically adjusted based on the prediction accuracy requirements of the image classification model and the prediction loss of the sample slice images. For example, if the prediction loss of the image classification model is set to be less than 0.2, the number of sample slice images in the third image sequence with a loss below 0.2 is the value of k.

[0084] 502B2, the sample slice image whose prediction loss is higher than the loss threshold is determined as a difficult slice image.

[0085] The loss threshold can be set based on business needs. For example, if the business requires the image classification model's classification accuracy to be higher than a certain indicator, the loss threshold can be set based on that indicator. For example, if the image classification model's classification accuracy needs to reach 90%, the loss threshold can be set to 0.1.

[0086] In one possible implementation, the computer device obtains a loss threshold, compares the relationship between the predicted loss of each sample slice and the loss threshold, and based on the comparison result, selects a sample slice image whose predicted loss is higher than the loss threshold from the sample slice images as a difficult example slice image.

[0087] like Figure 6 As shown, it shows a schematic diagram of the difficult example slice image determination process shown in an exemplary embodiment of the present application. Cell screening is performed on the sample slice image 601 to obtain an equivalent cell subset 602 corresponding to the sample slice image 601. The equivalent cell subset 602 includes images of some sample cells screened from the sample slice (the sample slice corresponding to the sample slice image 601). The equivalent cell subset 602 is input into the image classification model 603 to obtain a prediction result 604 output by the image classification model 603. The prediction result 604 includes a prediction category corresponding to the sample slice image 601. By comparing the prediction result 604 and the labeled category 605, the prediction loss 606 corresponding to the sample slice image 601 is determined. Then, based on the prediction loss 606, the sample slice image 601 is mined at the slice level to determine the difficult example slice image 607.

[0088] It should be noted that when determining hard-example slice images from sample slice images, sample slice images with higher loss can be directly identified as hard-example slice images. To avoid the need to screen images of some sample cells from the sample slice images during subsequent training, the equivalent cell subset corresponding to the sample slice images with higher loss can be directly identified as hard-example slice images. The equivalent cell subset is obtained by performing cell screening on the sample slice images before inputting them into the image classification model.

[0089] The process of determining the difficult cell region image from the sample slice image can be referred to the following embodiment, which will not be described in detail here.

[0090] In the process of generating target slice images containing hard example images based on hard example images of at least two scales, the target slice images can be divided into two categories according to the different categories of images constituting the target slice images. One is that the images constituting the target slice images are all hard example images, and the other is that the images constituting the target slice images include both hard example images and sample slice images. The following embodiments provide a more specific explanation of these two image generation methods.

[0091] Step 503 : selecting at least two hard-example images from the multiple hard-example images at at least two scales, and performing image stitching processing on the at least two hard-example images to generate a target slice image.

[0092] Image stitching, also known as image blending, is used to combine at least two images into a single image without losing detail. Image stitching methods can include simply stitching the at least two images into a single image or filling blank areas of an image with at least one other image.

[0093] Optionally, the computer device may store the hard example images mined from the sample slice images in a multi-scale hard example library so that during subsequent image classification model training, hard example images can be selected from the multi-scale hard example library to generate new target slice images.

[0094] Optionally, when storing difficult example images in a multi-scale difficult example library, they can be stored according to different scales. For example, difficult example images of the same scale can be stored in the same storage area or in the same database; difficult example images of different scales can be stored in different storage areas or in different databases, respectively, to facilitate subsequent computer devices to access difficult example images from the storage area or database corresponding to the scale based on scale requirements.

[0095] In one possible embodiment, each target slice image is composed of hard-example images. The computer device may randomly select at least two hard-example images from a plurality of hard-example images at at least two scales, and perform image stitching processing on the at least two selected hard-example images to generate a new target slice image.

[0096] Optionally, each target slice image can be generated by stitching two hard-to-solve images together. Alternatively, each target slice image can be generated by stitching more than two hard-to-solve images together. This embodiment of the present application does not limit the number of hard-to-solve images included in the target slice image.

[0097] Since the difficult images include difficult images of different scales: difficult cell region images and difficult slice images, and image stitching can be performed between difficult images of different scales, as well as between difficult images of the same scale, to enrich the diversity of the generated target slice images, at least one of the following image stitching methods can be employed during the image stitching of at least two difficult images. That is, step 503 can include at least one of steps 503A to 503C.

[0098] Step 503A: Select at least two difficult cell region images from a plurality of difficult cell region images at at least two scales, and perform image stitching processing on the at least two difficult cell region images to generate a target slice image.

[0099] Optionally, each target slice image may be composed of at least two difficult cell region images. In one possible implementation, the computer device randomly selects at least two difficult cell region images from multiple difficult cell region images at at least two scales, and performs image stitching processing on the at least two selected difficult cell region images to generate a new target slice image.

[0100] like Figure 7 FIG2 is a schematic diagram showing a target slice image generation process according to an exemplary embodiment of the present application. A first difficult cell region image 704 and a second difficult cell region image 705 are acquired, and image stitching processing is performed on the first difficult cell region image 704 and the second difficult cell region image 705 to generate a second target slice image 706.

[0101] Step 503B: selecting at least two difficult example slice images from a plurality of difficult example images at at least two scales, and performing image stitching processing on the at least two difficult example slice images to generate a target slice image.

[0102] Optionally, each target slice image may be composed of at least two difficult slice images. In one possible implementation, the computer device randomly selects at least two difficult slice images from a plurality of difficult images at at least two scales, and performs image stitching processing on the at least two selected difficult slice images to generate a new target slice image.

[0103] like Figure 7 As shown, a first difficult example slice image 701 and a second difficult example slice image 702 are obtained, and image stitching processing is performed on the first difficult example slice image 701 and the second difficult example slice image 702 to generate a first target slice image 703.

[0104] Step 503C: Select at least one difficult cell region image and at least one difficult slice image from multiple difficult images at at least two scales, and perform image stitching processing on the at least one difficult slice image and the at least one difficult cell region image to generate a target slice image.

[0105] Optionally, each target slice image is composed of at least one difficult slice image and at least one difficult cell region image. In one possible embodiment, the computer device randomly selects at least one difficult slice image and at least one difficult cell region image from multiple difficult images at at least two scales, and performs image stitching processing on the selected at least one difficult slice image and at least one difficult cell region image to generate a new target slice image.

[0106] like Figure 7 As shown, a third difficult example slice image 707 and a third difficult example cell region image 708 are obtained, and image stitching processing is performed on the third difficult example slice image 707 and the third difficult example cell region image 708 to generate a third target slice image 709.

[0107] Depend on Figure 7 As can be seen from the image stitching process shown, in the present embodiment, during the image stitching process for difficult example images, only two or more difficult example images are simply stitched together, and the stitching positions are random, so that the target slice image after the image stitching process still retains the image detail features in the original difficult example image, and does not affect the image parameters of the original difficult example image.

[0108] Optionally, in addition to the three image stitching methods mentioned above, namely, difficult cell area image + difficult cell area image, difficult slice image + difficult slice image, and difficult cell area image + difficult slice image, when performing image stitching processing on at least two difficult images, the same difficult image can also be copied and image stitching processing can be performed using at least two identical difficult images to generate a target slice image.

[0109] Step 504 : selecting at least one difficult example image from the multiple difficult example images at at least two scales, and performing image stitching processing on the at least one difficult example image and the at least one sample slice image to generate a target slice image.

[0110] Since normal cell images account for the majority of slice images in actual applications, in order to make the generated target slice image more consistent with the image of an actual biological slice, thereby improving the image classification model's accuracy in distinguishing real slice images, in one possible embodiment, a computer device randomly selects at least one difficult example image from multiple difficult example images at at least two scales, and randomly selects at least one sample slice image from a training sample set, and then performs image stitching processing on the at least one difficult example image and the at least one sample slice image to generate a target slice image. The training sample set includes a plurality of sample slice images.

[0111] The purpose of performing image stitching processing on at least one difficult example image and at least one sample slice image is to generate more difficult example samples so as to perform targeted difficult example sample training on the image classification model. Therefore, in the process of selecting sample slice images for image stitching, non-difficult example images and difficult example images in the sample slice images can be selected for image stitching processing so as to convert simple samples (sample slice images) in the original training sample set into difficult example samples (target slice images), thereby generating more difficult example samples based on the original training sample set and improving the efficiency of generating difficult example samples. Moreover, using such target slice images to train the image classification model can also improve the image classification model's prediction accuracy for difficult example samples.

[0112] Optionally, for determining non-hard-example images among multiple sample slice images, the hard-example slice images can be directly removed from the multiple sample slice images based on the determined hard-example slice images, and the remaining sample slice images are non-hard-example images.

[0113] Optionally, each target slice image can be generated by stitching together one hard-example image and one sample slice image. Alternatively, each target slice image can be generated by stitching together two or more hard-example images and one sample slice image. Alternatively, each target slice image can be generated by stitching together two or more hard-example images and two or more sample slice images. The embodiments of the present application do not limit the number of hard-example images and the number of sample slice images contained in the target slice image.

[0114] Similar to the generation of the target slice image based on at least two hard-example images in the above embodiment, when stitching the sample slice image with the hard-example image, hard-example images of different scales may also be selected to enrich the diversity of the generated target slice image. During the stitching of the at least one hard-example image and the at least one sample slice image, at least one of the following image stitching methods may also be employed. Specifically, step 504 may include at least one of steps 504A through 504C.

[0115] Step 504A: Select at least one difficult slice image from multiple difficult example images at at least two scales, and perform image stitching processing on the at least one difficult slice image and the at least one sample slice image to generate a target slice image.

[0116] Optionally, each target slice image may be composed of at least one difficult slice image and at least one sample slice image. In one possible implementation, the computer device randomly selects at least one difficult slice image from a plurality of difficult slice images at at least two scales, and randomly selects at least one sample slice image from a training sample set, so as to perform image stitching processing on the at least one difficult slice image and the at least one sample slice image to generate a new target slice image.

[0117] like Figure 8 FIG2 is a schematic diagram showing a target slice image generation process shown in another exemplary embodiment of the present application. A first difficult example slice image 801 and a first sample slice image 802 are obtained, and image stitching processing is performed on the first difficult example slice image 801 and the first sample slice image 802 to generate a first target slice image 803.

[0118] Step 504B: Select at least one difficult cell region image from multiple difficult image images at at least two scales, and perform image stitching processing on the at least one difficult cell region image and the at least one sample slice image to generate a target slice image.

[0119] Optionally, each target slice image may be composed of at least one difficult cell region image and at least one sample slice image. In one possible embodiment, the computer device randomly selects at least one difficult cell region image from multiple difficult images at at least two scales, and randomly selects at least one sample slice image from the training sample set, so as to perform image stitching processing on the at least one difficult cell region image and the at least one sample slice image to generate a new target slice image.

[0120] like Figure 8 As shown, a second difficult cell region image 809 and a third sample slice image 808 are obtained, and image stitching processing is performed on the second difficult cell region image 809 and the third sample slice image 808 to generate a third target slice image 810.

[0121] Step 504C: Select at least one difficult slice image and at least one difficult cell region image from multiple difficult images at at least two scales, and perform image stitching processing on the at least one difficult cell region image, the at least one difficult slice image, and the at least one sample slice image to generate a target slice image.

[0122] Optionally, each target slice image may be composed of at least one difficult slice image, at least one difficult cell region image, and at least one sample slice image. In one possible embodiment, the computer device randomly selects at least one difficult slice image and at least one difficult cell region image from a plurality of difficult images at at least two scales, and randomly selects at least one sample slice image from a training sample set, so as to perform image stitching processing on the at least one difficult slice image, the at least one difficult cell region image, and the at least one sample slice image to generate a new target slice image.

[0123] like Figure 8 As shown, a first difficult cell region image 804, a second difficult slice image 805 and a second sample slice image 806 are obtained, and the first difficult cell region image 804, the second difficult slice image 805 and the second sample slice image 806 are image stitched to generate a second target slice image 807.

[0124] It should be noted that this embodiment provides a variety of methods for generating target slice images: difficult cell region image + difficult cell region image, difficult slice image + difficult slice image, difficult cell region image + difficult slice image, difficult cell region image + sample slice image, difficult slice image + sample slice image, difficult slice image + difficult cell region image + sample slice image, etc. In the actual image stitching process, the computer device can adopt at least one of the above generation methods to generate a target slice image based on multiple difficult images at at least two scales.

[0125] Step 505 : training an image classification model based on the target slice image and the annotated category corresponding to the target slice image.

[0126] When training an image classification model, in addition to using the newly generated target slice image, it is also necessary to determine the loss between the annotated category (annotated label) corresponding to the target slice image and the predicted category to update the model parameters of the image classification model. Therefore, in one possible embodiment, after generating a target slice image corresponding to the difficult image based on the difficult image, it is also necessary to determine the annotated category of the generated target slice image based on the annotated category corresponding to the difficult image.

[0127] It can be seen from the above embodiments that the target slice image is generated by performing image stitching processing on at least two difficult example images, or is generated by performing image stitching processing on at least one difficult example image and at least one sample slice image. Then the target slice image contains the image information of at least two images. Therefore, in a possible implementation, when determining the annotation category corresponding to the target slice image, it is also necessary to determine it based on the at least two images that constitute the target slice image. In other words: if the target slice image is generated by performing image stitching processing on at least two difficult example images, then the annotation category corresponding to the target slice image also needs to be comprehensively determined based on the annotation categories of at least two difficult example images. If the target slice image is generated by performing image stitching processing on at least one difficult example image and at least one sample slice image, then the annotation category corresponding to the target slice image also needs to be comprehensively determined based on the annotation categories of at least one target image and at least one sample slice image.

[0128] Taking pathology diagnosis as an example, image classification models are primarily used to predict whether an input slice image is positive or negative. A positive result indicates the presence of a lesion, while a negative result indicates the absence of a lesion. In the training scenario of an image classification model, the image classification model is primarily used to predict whether the input target slice image belongs to the negative or positive category. The corresponding target slice image has two annotated categories (labels): negative or positive.

[0129] In an illustrative example, if the target slice image is generated by stitching at least two difficult-example images, the process of determining the labeled category corresponding to the target slice image may include step one and step two.

[0130] Step 1: When the labeled category of at least one of the at least two difficult example images is positive, determine that the labeled category of the target slice image is positive.

[0131] The labeling category is determined based on the criteria of positive + positive = positive, negative + negative = negative, and negative + positive = positive. If at least one of the at least two difficult images has a labeling category of positive, then the labeling category of the target slice image, which is the result of the stitching of the at least two difficult images, should also be positive if positive cells are present in the target slice image.

[0132] Step 2: When the labeled categories corresponding to at least two difficult example images are negative, determine that the labeled category of the target slice image is negative.

[0133] If there is no difficult example image with a positive labeling category in at least two difficult example images, it means that all images constituting the target slice image are images of negative cells, and the labeling category of the target slice image should also be negative.

[0134] In another exemplary example, if the target slice image is generated by stitching at least one difficult image and at least one sample slice image, the process of determining the labeled category corresponding to the target slice image may include steps three and four.

[0135] Step 3: When the labeled category is positive in at least one difficult image and at least one sample slice image, determine that the labeled category of the target slice image is positive.

[0136] If there is a positive labeled category in at least one difficult example image and at least one sample slice image, then after image stitching processing based on at least one difficult example image and at least one sample slice image, the generated target slice image should also contain an image of positive cells, and the labeled category of the target slice image should also be positive.

[0137] Step 4: When the labeled categories corresponding to at least one difficult image and at least one sample slice image are both negative, determine that the labeled category of the target slice image is negative.

[0138] If there is no image with a positive labeling category in at least one difficult image and at least one sample slice image, it means that all images constituting the target slice image are images of negative cells, and the labeling category of the target slice image should also be negative.

[0139] In one possible implementation, after determining the labeled category corresponding to the target slice image, the target slice image can be input into the image classification model to obtain the predicted category output by the image classification model, and then based on the loss between the predicted category and the labeled category, the parameters of the image classification model are updated to achieve the training purpose of the image classification model.

[0140] In this embodiment, by mining difficult examples from the sample slice images based on the sample prediction results of the image classification model, the mined difficult examples can be made to conform to the current model performance of the image classification model, thereby specifically improving the model performance of the image classification model. In addition, by performing image splicing processing on the difficult examples and sample slice images in a multi-scale and multi-example manner, the image classification model can focus on difficult examples at different scales and fully learn from difficult examples of different proportions, thereby further improving the classification accuracy of the image classification model.

[0141] The above embodiment describes the process of mining difficult example slice images. This embodiment mainly focuses on the process of mining difficult example cell region images. The difficult example cells mined in this embodiment mainly include two categories of difficult example cells, corresponding to two categories of difficult example cell region images.

[0142] In an exemplary example, step 502 may further include steps 502C to 502E, or step 502 may further include steps 502F to 502H.

[0143] Step 502C: determining a first slice image from the plurality of sample slice images based on the annotated categories corresponding to the sample slice images, wherein the annotated category of the sample slice images included in the first slice image is the first category.

[0144] In the process of class prediction of sample slice images, the cell features corresponding to the sample cells in the sample slices are obtained, feature weights are assigned to the cell features, and feature aggregation is performed based on the cell features and feature weights to characterize the image features of the sample slice images. It can be seen that in the class prediction process of sample slice images, the accuracy of setting the feature weights corresponding to the cell features will affect the class prediction results of the sample slice images. Therefore, in a possible implementation, in order to mine difficult cell region images from sample slice images, the image classification model is also required to output the feature weights of the cell features corresponding to each sample cell in the sample slice image. That is to say, in addition to the predicted category to which the sample slice image belongs, the sample prediction results should also include the feature weights of the cell features corresponding to the sample cells.

[0145] For example, using an image classification model to determine whether a slice is positive or negative, the image classification model requires the presence of cells with interpretable significance in the slice when determining a slice as positive. These cells with clear interpretable significance are often cells with higher feature weights. Therefore, in this embodiment of the application, cells labeled as positive, with higher interpretable significance but lower feature weights are identified as the first type of difficult cells.

[0146] Screening is performed based on the feature that the labeled category of the first type of difficult cells is positive. In one possible embodiment, in order to determine the difficult cell region image, the computer device is required to first determine the first slice image with the labeled category of the first category (positive) from the sample slice image, and then determine the difficult cell region image from the first slice image.

[0147] Step 502D: Based on the feature weights of the cell features in the first slice image, determine a first cell region image from the first slice image. The first cell region image is an image of the region occupied by the first type of cells in the first slice image. The first type of cells are sample cells whose feature weights are in the top m feature weights in the first slice image, where m is a positive integer.

[0148] After acquiring the first slice image, screening continues based on the feature of the first type of difficult cells being more significant. Sample cells with greater significance are often assigned higher feature weights for their cellular features, meaning that sample cells with higher feature weights need to be selected from each first slice image. Therefore, in one possible implementation, based on the feature weights of the cellular features in the first slice image, sample cells with higher feature weights are identified from the first slice image as the first type of cells, and the image of the area occupied by the first type of cells in the first slice image is determined as the first cell area image.

[0149] Optionally, for the method of screening out the first type of cells with higher feature weights from the first slice image, for each first slice image, the sample cells in the first slice image can be sorted from high to low according to the feature weights, and then the sample cells in the first m feature weights are determined as the first type of cells.

[0150] The value of m can be a fixed value. For example, if m is 10, the first 10 sample cells with higher feature weights are selected from each first slice image as the first type of cells, and the image of the area occupied by each sample cell of the first type of cells in the first slice image is the first cell area image.

[0151] It should be noted that the first cell area image is an image of the area occupied by multiple first-type cells in the first slice image. For example, if the first-type cells include 10 sample cells, the first cell area image also includes 10 images, each of which contains a first-type cell.

[0152] Optionally, a feature weight threshold may be set, and sample cells having feature weights greater than the feature weight threshold may be determined as first-type cells, and an image of the area occupied by each sample cell of the first type in the first sample slice may be determined as a first cell area image.

[0153] Step 502E: Determine a difficult cell region image from the first cell region image based on the feature weight corresponding to the first cell region image.

[0154] The feature weight corresponding to the first cell region image is determined based on the sample cells contained in the first cell region image, that is, the feature weight of the cell feature corresponding to the sample cells.

[0155] After the first cell region image is determined, the screening can be continued based on the feature that the feature weight of the first type of difficult cell is low. For the selected first cell region image with interpretation significance, if the feature weight of the first cell region image with interpretation significance is still low, it is a difficult cell region image. In one possible embodiment, based on the feature weight corresponding to the first cell region image, a cell region image with a low feature weight can be selected from the first cell region image as a difficult cell region image (first type of difficult cell region image).

[0156] In an illustrative example, step 502E may further include step 502E1 or 502E2.

[0157] 502E1, sorting the feature weights corresponding to the first cell region images from low to high to obtain a first image sequence; determining the first n first cell region images in the first image sequence as difficult cell region images, where n is a positive integer.

[0158] In order to obtain a part of difficult cells with low feature weights, in a possible implementation, the first cell region images can be sorted from low to high according to the feature weights to obtain a first image sequence. The first n first cell region images in the first image sequence are difficult cell region images with low feature weights.

[0159] Optionally, the first cell region images may be sorted from high to low according to the feature weights to obtain a first image sequence, and the last n first cell region images in the first image sequence are difficult cell region images with lower feature weights. This embodiment does not limit the sorting method.

[0160] Optionally, the value of n can be a fixed value, for example, n is 10. Alternatively, the value of n can be a dynamically changing value, and can be dynamically set based on the number of first cell region images screened. If the number of first cell region images is large, the value of n is relatively large; if the number of first cell region images is small, the value of n is relatively small.

[0161] 502E2, determining the first cell region image whose feature weight is lower than the first weight threshold as a difficult cell region image.

[0162] The first weight threshold may be set by business personnel based on demand. Schematically, the first weight threshold may be 0.5.

[0163] In one possible embodiment, after the computer device obtains multiple first cell region images and the feature weight corresponding to each first cell region image, it can compare the feature weight with the first weight threshold, and then, based on the comparison result, determine the first cell region image with a feature weight lower than the first weight threshold as a difficult cell region image, so as to screen out difficult cell region images with lower feature weights.

[0164] Step 502F: determining a second slice image from the plurality of sample slice images based on the labeled category and the predicted category, wherein the predicted category and the labeled category corresponding to the second slice image are different, and the labeled category of the sample slice image included in the second slice image is the second category.

[0165] Taking the image classification model as an example of judging whether a slice is a positive slice or a negative slice, when the image classification model judges a slice as a positive slice, it is necessary for the slice to have cells with interpretation significance, and these cells with clear interpretation significance are often cells with higher feature weights. If the image classification model misinterprets the slice, the main reason is that this type of cell has a clear interpretation significance. Therefore, in the embodiment of the present application, the negative slice is misclassified as a positive slice, and the cells with greater interpretation significance and higher feature weights are determined to be the second type of difficult cells.

[0166] Screening is performed based on the feature that the labeled category of the second type of difficult cells is negative, while the predicted category is positive. In one possible embodiment, in order to determine the second type of difficult cell region image, the computer device first needs to determine a second slice image from the sample slice image whose labeled category is the second category (negative) and whose predicted category is positive (the labeled category is different from the predicted category), and then determine the difficult cell region image (the second type of difficult cell region image) from the second slice image.

[0167] Step 502G: Based on the feature weights of the cell features in the second slice image, determine a second cell region image from the second slice image. The second cell region image is an image of the region occupied by the second type of cells in the second slice image. The second type of cells are sample cells whose feature weights are in the top i feature weights in the sample slice image, where i is a positive integer.

[0168] After acquiring the second slice image, the second type of difficult cells are further screened based on their greater significance. Sample cells with greater significance are often assigned higher feature weights, meaning that sample cells with higher feature weights need to be selected from each second slice image. Therefore, in one possible implementation, based on the feature weights of the cell features in the second slice image, sample cells with higher feature weights are identified from the second slice image as the second type of cells, and the image of the area occupied by the second type of cells in the second slice image is determined as the second cell area image.

[0169] Optionally, for the method of screening out the second type of cells with higher feature weights from the second slice image, for each second slice image, the sample cells in the second slice image can be sorted from high to low according to the feature weights, and then the sample cells with the first i feature weights can be determined as the second type of cells.

[0170] The value of i can be a fixed value. For example, if i is 10, the first 10 sample cells with higher feature weights are selected from each second slice image as the second type of cells, and the image of the area occupied by each sample cell of the second type in the second slice image is the second cell area image.

[0171] Optionally, a feature weight threshold may be set, and sample cells having feature weights greater than the feature weight threshold may be determined as second-type cells, and an image of the area occupied by each sample cell of the second type in the second sample slice may be determined as a second cell area image.

[0172] Step 502H: Determine a difficult cell region image from the second cell region image based on the feature weight corresponding to the second cell region image.

[0173] The feature weight corresponding to the second cell region image is determined based on the sample cells contained in the second cell region image, that is, the feature weight of the cell feature corresponding to the sample cells.

[0174] After the second cell region image is determined, the screening can be continued based on the feature that the feature weight of the second type of difficult cells is higher. In one possible embodiment, based on the feature weight corresponding to the second cell region image, the cell region image with the higher feature weight can be selected from the second cell region image as the difficult cell region image (the second type of difficult cell region image).

[0175] In an illustrative example, step 502H may include step 502H1 or 502H2.

[0176] Step 502H1, sort the feature weights corresponding to the second cell region images from high to low to obtain a second image sequence; determine the first j second cell region images in the second image sequence as difficult cell region images, where j is a positive integer.

[0177] In order to obtain a part of difficult cells with lower feature weights, in a possible implementation, the second cell region images can be sorted from high to low according to the feature weights to obtain a second image sequence. The first j second cell region images in the second image sequence are the images corresponding to difficult cells with higher feature weights, that is, the difficult cell region images.

[0178] Optionally, the second cell region images may be sorted from low to high according to the feature weights to obtain a second image sequence, and the last j second cell region images in the second image sequence are difficult cell region images with higher feature weights. This embodiment does not limit the sorting method.

[0179] Optionally, the value of j can be a fixed value, for example, j is 10. Alternatively, the value of j can be a dynamically changing value, and can be dynamically set according to the number of second cell region images screened. If the number of second cell region images is large, the value of j is relatively large; if the number of second cell region images is small, the value of j is relatively small.

[0180] Step 502H2: Determine the second cell region image whose feature weight is higher than the second weight threshold as a difficult cell region image.

[0181] The second weight threshold may be set by business personnel based on demand. Schematically, the second weight threshold may be 0.6.

[0182] In one possible embodiment, after the computer device obtains the second cell region image and the feature weight corresponding to each second cell region image, it can compare the feature weight with the second weight threshold, and then, based on the comparison result, determine the second cell region image with a feature weight higher than the second weight threshold as a difficult cell region image, so as to screen out difficult cell region images with higher feature weights.

[0183] like Figure 9As shown, it shows a schematic diagram of the process of determining a difficult cell region image shown in an exemplary embodiment of the present application. The sample slice image 901 is subjected to cell screening to obtain an equivalent cell subset 902 corresponding to the sample slice image 901. The equivalent cell subset 902 contains images of some sample cells screened out from the sample slice (the sample slice corresponding to the sample slice image 901). The equivalent cell subset 902 is input into the image classification model 900, and the feature extraction network 903 first performs feature extraction on the cell region image in the equivalent cell subset 902 to obtain cell features 904. The cell features 904 are then input into the weight assignment module 905, and the weight assignment module 905 assigns a feature weight 908 to each cell feature 904. Feature aggregation is then performed based on the feature weight 908 and the cell features 904 to obtain image features corresponding to the sample slice image 901. The image features are input into the multi-layer perceptron classifier 906, which performs category prediction based on the image features to obtain a predicted category 907 (0 or 1) corresponding to the sample slice image 901, where 0 represents negative and 1 represents positive. Further, the sample slice image 901 is subjected to cell-scale difficult example mining based on the predicted category 907 and feature weight 908 to obtain a difficult example cell region image 909 ( Figure 9 909 is a collection of several difficult cell region images mined).

[0184] In this embodiment, by analyzing the prediction process of the image classification model for the sample slice image, the image features of the sample slice image are generated by cell features and feature weights. Corresponding to the prediction category and feature weight based on the sample slice image, two ideas for mining difficult cell region images are provided to achieve the purpose of mining difficult cell region images from the sample slice image.

[0185] Taking a cervical cell slice image as an example, during the interpretation process of a cervical cell slice image, it is often necessary to output positive classification predictions of different categories, indicating that different positive categories of sample cells may exist in the sample slice image. To reflect the clustering relationship between different difficult cells during the hard example mining process, in one possible implementation, a computer device clusters the mined hard example cell region images to obtain multiple different hard example cell sets, and then performs subsequent image stitching processing based on the clustered hard example cell sets.

[0186] In an illustrative example, step 506 may be further included after step 502 .

[0187] Step 506: cluster the difficult cell region image to obtain at least one difficult cell set.

[0188] In order to mine the clustering features of difficult cell region images, in a possible implementation, after several difficult cell region images are determined from the sample slice image, the difficult cell region images can be clustered to obtain at least one category and a set of difficult cells corresponding to the category.

[0189] Optionally, in the process of clustering difficult cell region images, a k-means algorithm, a k-medoids algorithm, a clustering algorithm based on random selection (A Clustering Algorithm based on RandomizedSearch, CLARANS), etc. can be used. The embodiments of the present application do not limit the clustering algorithm used.

[0190] It should be noted that, among several difficult cell sets after clustering processing, the image feature similarity between the difficult cell region images in the same difficult cell set is high, while the image feature similarity between the difficult cell region images in different difficult cell sets is low.

[0191] like Figure 9 As shown, in the difficult cell clustering process, the difficult cell region image 909 can be input into the feature extraction network 903 to obtain the difficult cell features 910 corresponding to each difficult cell region image 909; clustering processing is performed based on the difficult cell features 910, so that n difficult cell sets can be obtained: k1, k2~kn; in the image stitching process, difficult cell region images can be selected from the same difficult cell set or different difficult cell sets for image stitching processing to obtain the target slice image 912, and then the image classification model 900 is trained based on the target slice image 912.

[0192] After the computer device acquires multiple difficult cell sets, during the image stitching process, the difficult cell region image can be selected based on the difficult cell sets. Correspondingly, in an exemplary example, step 503A may include at least one of step 503A1 and step 503A2:

[0193] Step 503A1 : selecting at least two difficult cell region images from the same difficult cell set, and performing image stitching processing on the at least two difficult cell region images to generate a target slice image.

[0194] Optionally, in the process of performing image stitching processing based on at least two difficult cell region images, at least two difficult cell region images may come from the same difficult cell set; in one possible embodiment, the computer device randomly selects a difficult cell set from multiple difficult cell sets, and randomly selects at least two difficult cell region images from the difficult cell set, and performs image stitching processing on the at least two difficult cell region images to generate a target slice image.

[0195] In this embodiment, by performing image stitching processing on multiple difficult cell area images from the same difficult cell set, the generated target slice image can contain only difficult cells of the same type. Then, using this type of target slice image to train the image classification model can enable the image classification model to learn the characteristics of this type of difficult cells in a targeted manner, thereby improving the image classification model's prediction accuracy for a certain category, and is suitable for training scenarios for image classification models that identify specific categories.

[0196] Optionally, since the difficult cell area images in the same difficult cell set all belong to the same category, business personnel can also set the difficult cell set to be selected according to business needs. For example, if a business personnel needs to train an image classification model to mainly detect category A, then when the computer equipment trains the image classification model, in order to improve the prediction accuracy of category A, it can select difficult cell area images from the difficult cell area set corresponding to category A and perform image stitching processing.

[0197] Step 503A2: Select at least two difficult cell region images from different difficult cell sets, and perform image stitching processing on the at least two difficult cell region images to generate a target slice image.

[0198] Optionally, in the process of performing image stitching based on at least two difficult cell region images, the at least two difficult cell region images may come from different difficult cell sets. In one possible embodiment, a computer device randomly selects at least two difficult cell sets from a plurality of difficult cell sets, and randomly selects at least two difficult cell region images from the at least two difficult cell sets, performs image stitching on the at least two difficult cell region images, and generates a target slice image. That is, among the difficult cell region images constituting the target slice image, at least two difficult cell region images come from different difficult cell sets.

[0199] Optionally, in another exemplary example, step 503C may also include at least one of steps 503C1 and 503C2:

[0200] Step 503C1: select at least one difficult cell region image from the same difficult cell set, select at least one difficult slice image from multiple difficult images at at least two scales, and perform image stitching processing on at least one difficult slice image and at least one difficult cell region image to generate a target slice image.

[0201] Optionally, in the process of performing image stitching processing based on at least one difficult cell region image and at least one difficult slice image, the at least one difficult cell region image may be from the same difficult cell set. In one possible embodiment, the computer device randomly selects a difficult cell set from multiple difficult cell sets, randomly selects at least one difficult cell region image from the difficult cell set, and selects at least one difficult slice image from the difficult slice images, and performs image stitching processing on the at least one difficult cell region image and the at least one difficult slice image to generate a target slice image.

[0202] Step 503C2: Select at least two difficult cell region images from different difficult cell sets, select at least one difficult slice image from multiple difficult images at at least two scales, and perform image stitching processing on at least one difficult slice image and at least two difficult cell region images to generate a target slice image.

[0203] Optionally, in the process of performing image stitching processing based on at least one difficult cell region image and at least one difficult slice image, if the number of selected difficult cell region images is greater than or equal to 2, then the at least two difficult cell region images may come from different difficult cell sets. In one possible embodiment, the computer device randomly selects at least two difficult cell sets from multiple difficult cell sets, randomly selects at least two difficult cell region images from the at least two difficult cell sets, and selects at least one difficult slice image from multiple difficult images at at least two scales, and performs image stitching processing on the at least two difficult cell region images and the at least one difficult slice image to generate a target slice image.

[0204] Optionally, in another exemplary example, step 504B may also include at least one of steps 504B1 and 504B2:

[0205] Step 504B1 , selecting at least one difficult cell region image from the same difficult cell set, and performing image stitching processing on the at least one difficult cell region image and at least one sample slice image to generate a target slice image.

[0206] Optionally, during the image stitching process on at least one difficult cell region image and at least one sample slice image, the at least one difficult cell region image may be from the same difficult cell set. In one possible implementation, the computer device randomly selects a difficult cell set from multiple difficult cell sets, randomly selects at least one difficult cell region image from the difficult cell set, and stitches the at least one difficult cell region image and the at least one sample slice image to generate a target slice image.

[0207] Step 504B2: Select at least two difficult cell region images from different difficult cell sets, and perform image stitching processing on the at least two difficult cell region images and at least one sample slice image to generate a target slice image.

[0208] Optionally, during the image stitching process of at least one difficult cell region image and at least one sample slice image, if the number of selected difficult cell region images is greater than or equal to 2, the at least two difficult cell region images may be from different difficult cell sets. In one possible embodiment, the computer device randomly selects at least two difficult cell sets from multiple difficult cell sets, and randomly selects at least two difficult cell region images from the at least two difficult cell sets, and stitches the at least two difficult cell region images with the at least one sample slice image to generate a target slice image.

[0209] Optionally, in another exemplary example, step 504C may also include at least one of steps 504C1 and 504C2:

[0210] Step 504C1: Select at least one difficult cell region image from the same difficult cell set, select at least one difficult slice image from multiple difficult images at at least two scales, and perform image stitching processing on the at least one difficult cell region image, at least one difficult slice image, and at least one sample slice image to generate a target slice image.

[0211] Optionally, in the process of performing image stitching processing on at least one difficult cell region image, at least one difficult slice image, and at least one sample slice image, the at least one difficult cell region image may be from the same difficult cell set. In one possible embodiment, the computer device randomly selects a difficult cell set from multiple difficult cell sets, randomly selects at least one difficult cell region image from the difficult cell set, and selects at least one difficult slice image from multiple difficult images at at least two scales, and performs image stitching processing on the at least one difficult cell region image, the at least one difficult slice image, and the at least one sample slice image to generate a target slice image.

[0212] Step 504C2: Select at least two difficult cell region images from different difficult cell sets, select at least one difficult slice image from multiple difficult images at at least two scales, and perform image stitching processing on the at least two difficult cell region images, at least one difficult slice image, and at least one sample slice image to generate a target slice image.

[0213] Optionally, in the process of performing image stitching processing on at least one difficult cell region image, at least one difficult slice image, and at least one sample slice image, if the number of selected difficult cell region images is greater than or equal to 2, then the at least two difficult cell region images may come from different difficult cell sets. In one possible embodiment, the computer device randomly selects at least two difficult cell sets from multiple difficult cell sets, randomly selects at least two difficult cell region images from the at least two difficult cell sets, and selects at least one difficult slice image from multiple difficult images at at least two scales, and performs image stitching processing on the at least two difficult cell region images, at least one difficult slice image, and at least one sample slice image to generate a target slice image.

[0214] In this embodiment, by clustering the mined difficult cell region images to obtain sets of difficult cells of different categories, when performing image stitching, it is possible to select difficult cell region images of the same category, or to select difficult cell region images of different categories based on actual needs. This allows the target slice image after image stitching to contain images of difficult cell regions of the same category, which is beneficial for subsequent classification training. Furthermore, the image classification model can also focus on features of different categories when learning features, further improving the model performance of the image classification model.

[0215] In combination with the above embodiments, Figure 10As shown, it shows a schematic diagram of the training architecture of the image classification model shown in another exemplary embodiment of the present application. By performing cell screening on the sample slice image 1001, an equivalent cell subset 1002 is obtained, and the equivalent cell subset 1002 includes images of specific sample cells screened from the sample slice (the sample slice corresponding to the sample slice image 1001), and the equivalent cell subset 1002 is input into the image classification model 1003 to obtain a prediction result 1004 output by the image classification model 1003, and the prediction result 1004 includes a prediction category 1008 corresponding to the sample slice image 1001, and a feature weight 1007 corresponding to the specific sample cell of the sample slice image 1001; on the one hand, by comparing the prediction category 1008 in the prediction result 1002, the prediction result 1004 is obtained. 8 and the labeled category 1009, determine the prediction loss corresponding to the sample slice image 1001, and then perform slice-level hard example mining on the sample slice image 1001 based on the prediction loss to obtain a hard example slice image 1006; on the other hand, combine the prediction category 1008, the labeled category 1009 and the feature weight 1007 in the prediction result 1002 to filter out a hard example cell region image 1005 from the equivalent cell subset 1002; thereby obtaining a multi-scale hard example image, and by performing image stitching processing on the hard example image and the sample slice image, generate a new target slice image containing the hard example image, so as to use the target slice image to train the image classification model 1003. Optionally, the hard example cell region image 1005 or the hard example slice image 1006 can also be image stitched with the sample slice image 1001 to generate a new target slice image to train the image classification model 1003.

[0216] The above embodiments mainly describe the process of determining difficult example images and target slice images. The difficult example images and target slice images are mainly used in the training process of the image classification model. After multiple rounds of training of the image classification model using the target slice images, an image classification model trained with the target slice images is obtained. The image classification model has a higher prediction accuracy for difficult example samples than before training, and the trained image classification model can be applied to the category prediction process of the slice image.

[0217] In an exemplary embodiment, after the image classification model is trained using the target slice image, the following steps may be further included:

[0218] Step 507: Obtain slice images to be classified.

[0219] The method for obtaining the slice image to be classified is similar to the method for obtaining the sample slice image, which will not be described in detail in this embodiment.

[0220] In actual application, during the process of pathological diagnosis, medical staff can view pathological slice images in real time, and the pathological slice images are slice images to be classified.

[0221] Optionally, the slice image to be classified may also be any slice image downloaded from the Internet.

[0222] Step 508: Input the slice image to be classified into the image classification model trained with the target slice image, and obtain the target category to which the slice image to be classified belongs as output by the image classification model.

[0223] The trained image classification model has the function of image category prediction. In one possible implementation, the slice image to be classified is input into the image classification model, and the image classification model performs cell feature extraction, cell feature aggregation, and category prediction, thereby outputting the target category corresponding to the slice image to be classified.

[0224] Optionally, during the application process, the image classification model can also output images of key discriminant cells in the slice image to be classified. The key discriminant cells are cells with higher feature weights, so that the user can further analyze the slice image to be classified based on the key discriminant cells.

[0225] Based on the model training process shown in the above embodiment, the application scenario of this embodiment is exemplified.

[0226] (1) Application in medical diagnosis scenarios

[0227] like Figure 11 As shown, it is a schematic diagram of an application scenario of a medical diagnosis scenario shown in an exemplary embodiment of the present application. In the model training stage 1111: in the first device 1110, multi-scale difficult example mining is performed on the sample pathological slice image 1112 to obtain a multi-scale difficult example image 1113, including a difficult example cell region image 1114 at the cell scale and a difficult example slice image 1115 at the slice scale. Further, multi-example image splicing processing is performed on the difficult example image 1113 and the sample pathological slice image 1112 to obtain a target pathological slice image 1116 containing the difficult example image 1113. The target pathological slice image 1116 is then used to train the image classification model 1117, so that the image classification model 1117 has the function of accurately predicting the category to which the pathological slice image belongs.

[0228] The first device 1110 deploys the image classification model 1117 trained with the target pathological slice image 1116 to the second device 1120, which is equipped with a medical auxiliary diagnosis system 1121. During the model application phase, the medical auxiliary diagnosis system 1121 can use the trained image classification model 1117 for auxiliary diagnosis.

[0229] In the medical auxiliary diagnosis system 1121, medical personnel can input real-time acquired pathological slice images 1122 into the image classification model 1117. Image classification model 1117 then predicts the pathological slice image's category and outputs a pathological diagnosis result 1123 corresponding to the pathological slice image 1122. Pathological diagnosis result 1123 can include the slice category 1124 to which the pathological slice image 1122 belongs. Slice category 1124 can indicate whether the pathological slice image 1122 is a positive slice image or a negative slice image; for positive slice images, which positive category it belongs to. Optionally, pathological diagnosis result 1123 can also include key cell interpretation 1125, which are cells with high feature weights in pathological slice image 1122. Optionally, if pathological slice image 1122 is diagnosed as a positive slice image, cells belonging to a specific positive category can also be output. This allows medical personnel to make a more detailed diagnosis based on the interpretation of key cells or cells in a specific positive category, thereby improving diagnostic efficiency.

[0230] (2) Application in teaching scenarios

[0231] like Figure 12 As shown, it is a schematic diagram of an application scenario of a teaching scenario shown in an exemplary embodiment of the present application. In the model training stage 1211: the first device 1110 performs multi-scale hard example mining on the sample pathology slice image 1212 to obtain a multi-scale hard example image 1213, including a cell-scale hard example cell region image 1214 and a slice-scale hard example slice image 1215; further, the hard example image 1213 and the sample pathology slice image 1212 are subjected to multi-example image stitching processing to obtain a target pathology slice image 1216 containing the hard example image 1213, and then the target pathology slice image 1216 is used to train the image classification model 1217, so that the image classification model 1217 has the function of accurately predicting the category to which the pathology slice image belongs.

[0232] The first device 1210 deploys the image classification model 1217 trained with the target pathological section image 1216 to the second device 1220. The second device 1220 is a device equipped with an online question-generating system 1221. In the model application phase, the online question-generating system 1221 can use the trained image classification model 1217 to generate questions online in real time.

[0233] In the online question-setting system 1221, question setters can download existing pathological slice images from a web page and store them in a pathological slice image library 1222. In order to avoid the generated pathological discrimination image option 1226 being similar to the existing pathological slice images and to increase the complexity of the pathological discrimination image option 1226, multi-scale difficult example mining is performed on the pathological slice images in the pathological slice image library 1222 to obtain multi-scale difficult pathological images 1223, including difficult pathological cell region images 1224 at the cell scale and difficult pathological slice images 1225 at the slice scale. Furthermore, multi-example image splicing processing is performed on the difficult pathological images 1223 and the pathological slice images to obtain a pathological discrimination image option 1226 containing the difficult pathological image 1223. The pathological discrimination image option 1226 can be used as an option for a multiple-choice question or as a title for an analytical question, thereby eliminating the need for manual question setting and allowing the online question-setting system to automatically set questions based on the existing pathological slice images, thereby improving question setting efficiency.

[0234] The online question-setting system 1221 not only needs to automatically generate new pathology discrimination image options 1226 through multi-scale mining and multi-example splicing processing, but in order to further improve the efficiency of question setting, the online question-setting system 1222 can also input the generated pathology discrimination image options 1226 into the image classification model 1217, and the image classification model 1217 performs category prediction on the pathology discrimination image options 1226 and outputs the pathology discrimination result 1227 corresponding to the pathology discrimination image option 1226; wherein the pathology discrimination result 1227 can be the slice category to which the pathology discrimination image option 1226 belongs, and the slice category 1124 can indicate whether the pathology slice image 1122 is an image of a positive slice or an image of a negative slice; for the image of a positive slice, which positive category it belongs to, the slice category here is the option answer 122 8; Optionally, the pathology discrimination result 1227 may also include key cells for interpretation, i.e., cells with higher feature weights in the pathology discrimination image option 1226; Optionally, if the pathology discrimination image option 1226 is diagnosed as an image of a positive slice, cells belonging to a specific positive category may also be output, and the key cells for interpretation are the answer analysis 1229 corresponding to the option answer 1228, that is, the slice category corresponding to the pathology discrimination image option 1226 may be determined based on the interpretation of the key cells; Correspondingly, the option answer 1228 and the answer analysis 1229 may be automatically generated by the online question setting system 1221, and the question setter only needs to check whether the option answer 1228 and the answer analysis 1229 correspond to the pathology discrimination image option 1226, which may further improve the question setting efficiency of the question setter.

[0235] Figure 13 This is a structural block diagram of a training device for an image classification model provided by an exemplary embodiment of the present application, the device comprising:

[0236] A determination module 1301 is configured to determine a plurality of hard-example images of at least two scales from a plurality of sample slice images, wherein the sample slice images are images of sample slices containing sample cells, and the hard-example images are images containing hard-example cells;

[0237] A generating module 1302 is configured to generate a target slice image based on the multiple hard-example images at the at least two scales, wherein each of the target slice images includes a hard-example image at at least one scale;

[0238] The training module 1303 is used to train an image classification model based on the target slice image, and the image classification model is used to predict the category to which the input slice image belongs.

[0239] In an optional embodiment, the determining module 1301 is further configured to:

[0240] Inputting the plurality of sample slice images into the image classification model to obtain a sample prediction result output by the image classification model, wherein the sample prediction result at least includes the prediction categories to which the plurality of sample slice images belong;

[0241] Based on the sample prediction result and the labeled categories corresponding to the sample slice images, a plurality of difficult example images of the at least two scales are determined from the plurality of sample slice images.

[0242] In an optional embodiment, the multiple difficult-example images at at least two scales include difficult-example cell region images at a cell scale, and the sample prediction result further includes feature weights of cell features corresponding to the sample cells, and the feature weights and the cell features are used to determine image features of the sample slice image;

[0243] The determining module 1301 is further configured to:

[0244] Determining a first slice image from the plurality of sample slice images based on the annotated categories corresponding to the sample slice images, wherein the annotated category of the sample slice images included in the first slice image is the first category;

[0245] Determining a first cell region image from the first slice image based on the feature weights of the cell features in the first slice image, the first cell region image being an image of a region occupied by a first type of cell in the first slice image, the first type of cell being a sample cell whose feature weight is in the top m feature weights in the first slice image, where m is a positive integer;

[0246] The difficult cell region image is determined from the first cell region image based on the feature weight corresponding to the first cell region image.

[0247] In an optional embodiment, the determining module 1301 is further configured to:

[0248] The feature weights corresponding to the first cell region images are sorted from low to high to obtain a first image sequence; the first n first cell region images in the first image sequence are determined as the difficult cell region images, where n is a positive integer;

[0249] or,

[0250] The first cell region image whose feature weight is lower than a first weight threshold is determined as the difficult cell region image.

[0251] In an optional embodiment, the multiple difficult-example images at at least two scales include difficult-example cell region images at a cell scale, and the sample prediction result further includes feature weights of cell features corresponding to the sample cells, and the feature weights and the cell features are used to determine image features of the sample slice image;

[0252] The determining module 1301 is further configured to:

[0253] Determining a second slice image from the plurality of sample slice images based on the labeled category and the predicted category, wherein the predicted category and the labeled category corresponding to the second slice image are different, and the labeled category of the sample slice image included in the second slice image is the second category;

[0254] Determining a second cell region image from the second slice image based on the feature weights of the cell features in the second slice image, where the second cell region image is an image of a region occupied by a second type of cell in the second slice image, and the second type of cell is a sample cell whose feature weights are in the first i feature weights in the second slice image, where i is a positive integer;

[0255] Based on the feature weight corresponding to the second cell region image, the difficult cell region image is determined from the second cell region image.

[0256] In an optional embodiment, the determining module 1301 is further configured to:

[0257] Sorting the feature weights corresponding to the second cell region images from high to low to obtain a second image sequence; determining the first j second cell region images in the second image sequence as the difficult cell region images, where j is a positive integer;

[0258] or,

[0259] The second cell region image whose feature weight is higher than a second weight threshold is determined as the difficult cell region image.

[0260] In an optional embodiment, the plurality of hard-example images at the at least two scales include hard-example slice images at a slice scale;

[0261] The determining module 1301 is further configured to:

[0262] Determining prediction losses corresponding to a plurality of the sample slice images based on the sample prediction results and the labeled categories;

[0263] Based on the prediction loss, the hard example slice image is determined from a plurality of the sample slice images.

[0264] In an optional embodiment, the determining module 1301 is further configured to:

[0265] sorting the plurality of sample slice images from high to low according to the prediction loss to obtain a third image sequence; determining the first k sample slice images in the third image sequence as the hard-example slice images, where k is a positive integer;

[0266] or,

[0267] The sample slice image whose prediction loss is higher than a loss threshold is determined as the difficult example slice image.

[0268] In an optional embodiment, the generating module 1302 is further configured to:

[0269] Selecting at least two difficult-example images from the plurality of difficult-example images at the at least two scales, and performing image stitching processing on the at least two difficult-example images to generate a target slice image;

[0270] At least one difficult example image is selected from the multiple difficult example images at the at least two scales, and image stitching processing is performed on at least one difficult example image and at least one sample slice image to generate one target slice image.

[0271] In an optional embodiment, the plurality of hard-example images at the at least two scales include hard-example cell region images at the cell scale and hard-example slice images at the slice scale;

[0272] The generating module 1302 is further configured to:

[0273] Selecting at least two of the difficult cell region images from the multiple difficult cell region images at the at least two scales, and performing image stitching processing on the at least two difficult cell region images to generate a target slice image;

[0274] Selecting at least two of the difficult example slice images from the multiple difficult example images at the at least two scales, and performing image stitching processing on the at least two difficult example slice images to generate one of the target slice images;

[0275] At least one of the difficult cell region images and at least one of the difficult slice images are selected from the multiple difficult images at the at least two scales, and image stitching processing is performed on at least one of the difficult slice images and at least one of the difficult cell region images to generate a target slice image.

[0276] In an optional embodiment, the device further comprises:

[0277] A clustering processing module, configured to perform clustering processing on the difficult cell region image to obtain at least one difficult cell set;

[0278] The generating module 1302 is further configured to:

[0279] Selecting at least two images of the difficult cell region from the same difficult cell set;

[0280] Selecting at least two images of the difficult cell region from different difficult cell sets;

[0281] The generating module 1302 is further configured to:

[0282] Selecting at least one of the difficult cell region images from the same difficult cell set, and selecting at least one of the difficult slice images from the multiple difficult images at the at least two scales;

[0283] At least two of the difficult cell region images are selected from different difficult cell sets, and at least one of the difficult slice images is selected from the multiple difficult images at the at least two scales.

[0284] In an optional embodiment, the plurality of hard-example images at the at least two scales include hard-example cell region images at the cell scale and hard-example slice images at the slice scale;

[0285] The generating module 1302 is further configured to:

[0286] Selecting at least one of the difficult example slice images from the multiple difficult example images at the at least two scales, and performing image stitching processing on the at least one difficult example slice image and the at least one sample slice image to generate a target slice image;

[0287] Selecting at least one of the difficult cell region images from the multiple difficult image images at the at least two scales, and performing image stitching processing on the at least one difficult cell region image and the at least one sample slice image to generate a target slice image;

[0288] At least one of the difficult example slice images and at least one of the difficult example cell region images are selected from the multiple difficult example images at the at least two scales, and image stitching processing is performed on at least one of the difficult example cell region image, at least one of the difficult example slice image, and at least one of the sample slice images to generate a target slice image.

[0289] In an optional embodiment, the device further comprises:

[0290] A clustering processing module, configured to perform clustering processing on the difficult cell region image to obtain at least one difficult cell set;

[0291] The generating module 1302 is further configured to:

[0292] Selecting at least one image of the difficult cell region from the same difficult cell set;

[0293] Selecting at least two images of the difficult cell region from different difficult cell sets;

[0294] The generating module 1302 is further configured to:

[0295] Selecting at least one of the difficult cell region images from the same difficult cell set, and selecting at least one of the difficult slice images from the multiple difficult images at the at least two scales;

[0296] At least two of the difficult cell region images are selected from different difficult cell sets, and at least one of the difficult slice images is selected from the multiple difficult images at the at least two scales.

[0297] In an optional embodiment, the image classification model is used to predict whether the input target slice image belongs to negative or positive;

[0298] The device further comprises:

[0299] a label determination module, configured to determine that the labeled category of the target slice image is positive when at least one of the at least two difficult example images has a labeled category corresponding to the difficult example image being positive;

[0300] The label determination module is further configured to determine that the labeled category of the target slice image is negative when the labeled categories corresponding to at least two of the difficult example images are both negative;

[0301] The device further comprises:

[0302] The label determination module is further configured to determine that the labeled category of the target slice image is positive when the labeled category in at least one of the difficult example images and at least one of the sample slice images is positive;

[0303] The label determination module is further configured to determine that the labeled category of the target slice image is negative when the labeled categories corresponding to at least one of the difficult example images and at least one of the sample slice images are both negative;

[0304] The training module 1303 is further configured to:

[0305] The image classification model is trained based on the target slice image and the labeled category corresponding to the target slice image.

[0306] In an optional embodiment, the device further comprises:

[0307] An acquisition module is used to acquire slice images to be classified;

[0308] The processing module is used to input the slice image to be classified into the image classification model trained with the target slice image, and obtain the target category to which the slice image to be classified belongs output by the image classification model.

[0309] In summary, by performing multi-scale hard example mining on sample slice images, multiple hard example images of at least two scales are obtained, and based on the multiple hard example images of at least two scales, new target slice images containing hard example images are generated, so that the new target slice images are used to perform classification training on the image classification model, so that the image classification model can learn hard example samples and improve the classification performance of the image classification model for hard example samples and new samples; moreover, by mining hard example samples at multiple scales, the image classification model can also fully learn hard examples at different scales to further improve the classification performance of the image classification model.

[0310] Figure 14 is a schematic diagram showing the structure of a computer device according to an exemplary embodiment. The computer device 1400 may be Figure 3 The first device 310 may also be Figure 3 The computer device 1400 includes a central processing unit (CPU) 1401, a system memory 1404 including a random access memory (RAM) 1402 and a read-only memory (ROM) 1403, and a system bus 1405 connecting the system memory 1404 and the CPU 1401. The computer device 1400 also includes a basic input / output (I / O) system 1406 that facilitates information transmission between various components within the computer device, and a mass storage device 1407 for storing an operating system 1413, application programs 1414, and other program modules 1415.

[0311] The basic input / output system 1406 includes a display 1408 for displaying information and an input device 1409, such as a mouse or keyboard, for user input. Both the display 1408 and the input device 1409 are connected to the central processing unit 1401 via an input / output controller 1410 connected to the system bus 1405. The basic input / output system 1406 may also include an input / output controller 1410 for receiving and processing input from a variety of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1410 also provides output to a display screen, printer, or other types of output devices.

[0312] The mass storage device 1407 is connected to the central processing unit 1401 via a mass storage controller (not shown) connected to the system bus 1405. The mass storage device 1407 and its associated computer-readable medium provide non-volatile storage for the computer device 1400. In other words, the mass storage device 1407 may include computer-readable media (not shown) such as a hard disk or a CD-ROM drive.

[0313] Without loss of generality, the computer device readable medium may include computer device storage media and communication media. Computer device storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer device readable instructions, data structures, program modules or other data. Computer device storage media include RAM, ROM, Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), CD-ROM, Digital Video Disc (DVD) or other optical storage, tape cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer device storage media is not limited to the above-mentioned ones. The above-mentioned system memory 1404 and mass storage device 1407 can be collectively referred to as memory.

[0314] According to various embodiments of the present disclosure, the computer device 1400 may also be connected to a remote computer device on a network such as the Internet for operation. That is, the computer device 1400 may be connected to the network 1411 via the network interface unit 1412 connected to the system bus 1405, or the network interface unit 1412 may be used to connect to other types of networks or remote computer device systems (not shown).

[0315] The memory also includes one or more programs, which are stored in the memory. The central processing unit 1401 implements all or part of the steps of the above-mentioned image classification model training method by executing the one or more programs.

[0316] The present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set. The at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the training method of the image classification model provided by the above method embodiment.

[0317] The present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image classification model training method provided in the above method embodiment.

[0318] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0319] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0320] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A training method for an image classification model, characterized in that: The method comprises: Inputting a plurality of sample slice images into an image classification model to obtain sample prediction results output by the image classification model, the sample prediction results at least including prediction categories to which the plurality of sample slice images belong; determining a plurality of hard-example images at at least two scales from the plurality of sample slice images based on the sample prediction results and the labeled categories corresponding to the sample slice images, the sample slice images being images of sample slices containing sample cells, the hard-example images being images containing hard-example cells, and the hard-example images at at least two scales including hard-example cell region images at a cell scale and hard-example slice images at a slice scale; Selecting at least two hard-example images from the plurality of hard-example images at the at least two scales, and performing image stitching processing on the at least two hard-example images to generate target slice images, each of the target slice images containing a hard-example image at at least one scale; Based on the target slice image, an image classification model is trained, and the image classification model is used to predict the category to which the input slice image belongs.

2. The method according to claim 1, characterized in that The multiple difficult-example images at at least two scales include difficult-example cell region images at a cell scale, the sample prediction result further includes feature weights of cell features corresponding to the sample cells, and the feature weights and the cell features are used to determine image features of the sample slice image; The determining, based on the sample prediction result and the labeled categories corresponding to the sample slice images, a plurality of difficult images of at least two scales from the plurality of sample slice images comprises: Determining a first slice image from the plurality of sample slice images based on the annotated categories corresponding to the sample slice images, wherein the annotated category of the sample slice images included in the first slice image is the first category; Determining a first cell region image from the first slice image based on the feature weights of the cell features in the first slice image, where the first cell region image is an image of a region occupied by a first type of cell in the first slice image, and the first type of cell is a sample cell whose feature weight is in the top m feature weights in the first slice image, where m is a positive integer; The difficult cell region image is determined from the first cell region image based on the feature weight corresponding to the first cell region image.

3. The method according to claim 2, characterized in that The determining the difficult cell region image from the first cell region image based on the feature weight corresponding to the first cell region image includes: The feature weights corresponding to the first cell region images are sorted from low to high to obtain a first image sequence; the first n first cell region images in the first image sequence are determined as the difficult cell region images, where n is a positive integer; or, The first cell region image whose feature weight is lower than a first weight threshold is determined as the difficult cell region image.

4. The method according to claim 1, wherein The multiple difficult-example images at at least two scales include difficult-example cell region images at a cell scale, the sample prediction result further includes feature weights of cell features corresponding to the sample cells, and the feature weights and the cell features are used to determine image features of the sample slice image; The determining, based on the sample prediction result and the labeled categories corresponding to the sample slice images, a plurality of difficult images of at least two scales from the plurality of sample slice images comprises: Determining a second slice image from the plurality of sample slice images based on the labeled category and the predicted category, wherein the predicted category and the labeled category corresponding to the second slice image are different, and the labeled category of the sample slice image included in the second slice image is the second category; Determining a second cell region image from the second slice image based on the feature weights of the cell features in the second slice image, where the second cell region image is an image of a region occupied by a second type of cell in the second slice image, and the second type of cell is a sample cell whose feature weights are in the first i feature weights in the second slice image, where i is a positive integer; Based on the feature weight corresponding to the second cell region image, the difficult cell region image is determined from the second cell region image.

5. The method according to claim 4, characterized in that The determining the difficult cell region image from the second cell region image based on the feature weight corresponding to the second cell region image includes: Sorting the feature weights corresponding to the second cell region images from high to low to obtain a second image sequence; determining the first j second cell region images in the second image sequence as the difficult cell region images, where j is a positive integer; or, The second cell region image whose feature weight is higher than a second weight threshold is determined as the difficult cell region image.

6. The method according to claim 1, characterized in that The plurality of hard-example images at the at least two scales include hard-example slice images at a slice scale; The determining, based on the sample prediction result and the labeled categories corresponding to the sample slice images, a plurality of difficult images of at least two scales from the plurality of sample slice images comprises: Determining prediction losses corresponding to a plurality of the sample slice images based on the sample prediction results and the labeled categories; Based on the prediction loss, the hard example slice image is determined from a plurality of the sample slice images.

7. The method according to claim 6, characterized in that The determining the difficult slice image from the plurality of sample slice images based on the prediction loss includes: sorting the plurality of sample slice images from high to low according to the prediction loss to obtain a third image sequence; determining the first k sample slice images in the third image sequence as the hard-example slice images, where k is a positive integer; or, The sample slice image whose prediction loss is higher than a loss threshold is determined as the difficult example slice image.

8. The method according to claim 1, characterized in that The selecting at least two difficult-example images from the plurality of difficult-example images at the at least two scales, and performing image stitching processing on the at least two difficult-example images to generate a target slice image, includes at least one of the following: Selecting at least two of the difficult cell region images from the multiple difficult cell region images at the at least two scales, and performing image stitching processing on the at least two difficult cell region images to generate a target slice image; Selecting at least two of the difficult example slice images from the multiple difficult example images at the at least two scales, and performing image stitching processing on the at least two difficult example slice images to generate one of the target slice images; At least one of the difficult cell region images and at least one of the difficult slice images are selected from the multiple difficult images at the at least two scales, and image stitching processing is performed on at least one of the difficult slice images and at least one of the difficult cell region images to generate a target slice image.

9. The method according to claim 8, characterized in that The method further comprises: performing clustering processing on the difficult cell region image to obtain at least one difficult cell set; The selecting at least two of the difficult cell region images from the multiple difficult images at the at least two scales includes at least one of the following: Selecting at least two images of the difficult cell region from the same difficult cell set; Selecting at least two images of the difficult cell region from different difficult cell sets; The selecting at least one of the hard-example cell region images and at least one of the hard-example slice images from the multiple hard-example images at the at least two scales includes at least one of the following: Selecting at least one of the difficult cell region images from the same difficult cell set, and selecting at least one of the difficult slice images from the multiple difficult images at the at least two scales; At least two of the difficult cell region images are selected from different difficult cell sets, and at least one of the difficult slice images is selected from the multiple difficult images at the at least two scales.

10. The method according to claim 1, characterized in that The image classification model is used to predict whether the input target slice image is negative or positive; After performing image stitching processing on at least two of the difficult example images to generate a target slice image, the method further includes: In a case where the labeled category corresponding to at least one of the at least two difficult example images is positive, determining that the labeled category of the target slice image is positive; When the labeled categories corresponding to at least two of the difficult example images are negative, determining that the labeled category of the target slice image is negative; The step of training an image classification model based on the target slice image comprises: The image classification model is trained based on the target slice image and the labeled category corresponding to the target slice image.

11. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: At least one difficult example image is selected from the multiple difficult example images at the at least two scales, and image stitching processing is performed on at least one difficult example image and at least one sample slice image to generate one target slice image.

12. The method according to claim 11, characterized in that The selecting at least one difficult example image from the multiple difficult example images at the at least two scales, and performing image stitching processing on the at least one difficult example image and the at least one sample slice image to generate the target slice image, includes at least one of the following: Selecting at least one of the difficult example slice images from the multiple difficult example images at the at least two scales, and performing image stitching processing on the at least one difficult example slice image and the at least one sample slice image to generate a target slice image; Selecting at least one of the difficult cell region images from the multiple difficult image images at the at least two scales, and performing image stitching processing on the at least one difficult cell region image and the at least one sample slice image to generate a target slice image; At least one of the difficult example slice images and at least one of the difficult example cell region images are selected from the multiple difficult example images at the at least two scales, and image stitching processing is performed on at least one of the difficult example cell region image, at least one of the difficult example slice image, and at least one of the sample slice images to generate a target slice image.

13. The method according to claim 12, characterized in that The method further comprises: performing clustering processing on the difficult cell region image to obtain at least one difficult cell set; The selecting at least one of the difficult cell region images from the multiple difficult images at the at least two scales includes at least one of the following: Selecting at least one image of the difficult cell region from the same difficult cell set; Selecting at least two images of the difficult cell region from different difficult cell sets; The selecting at least one of the hard-example slice images and at least one of the hard-example cell region images from the multiple hard-example images at the at least two scales includes at least one of the following: Selecting at least one of the difficult cell region images from the same difficult cell set, and selecting at least one of the difficult slice images from the multiple difficult images at the at least two scales; At least two of the difficult cell region images are selected from different difficult cell sets, and at least one of the difficult slice images is selected from the multiple difficult images at the at least two scales.

14. The method according to claim 11, characterized in that The image classification model is used to predict whether the input target slice image is negative or positive; After performing image stitching processing on at least one of the difficult example images and at least one of the sample slice images to generate the target slice image, the method further includes: In a case where a labeled category is positive in at least one of the difficult example images and at least one of the sample slice images, determining that the labeled category of the target slice image is positive; When the labeled categories corresponding to at least one of the difficult example images and at least one of the sample slice images are both negative, determining that the labeled category of the target slice image is negative; The step of training an image classification model based on the target slice image comprises: The image classification model is trained based on the target slice image and the labeled category corresponding to the target slice image.

15. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Obtaining slice images to be classified; The slice image to be classified is input into the image classification model trained with the target slice image, and the target category to which the slice image to be classified belongs is obtained as output by the image classification model.

16. A training device for an image classification model, characterized in that: The device comprises: a determination module, configured to input a plurality of sample slice images into an image classification model to obtain sample prediction results output by the image classification model, wherein the sample prediction results include at least prediction categories to which the plurality of sample slice images belong; and determine, based on the sample prediction results and the labeled categories corresponding to the sample slice images, a plurality of hard-example images at at least two scales from the plurality of sample slice images, wherein the sample slice images are images of sample slices containing sample cells, the hard-example images are images containing hard-example cells, and the plurality of hard-example images at at least two scales include hard-example cell region images at a cell scale and hard-example slice images at a slice scale; a generating module, configured to select at least two hard-example images from the plurality of hard-example images at the at least two scales, and perform image stitching processing on the at least two hard-example images to generate target slice images, each of the target slice images containing a hard-example image at at least one scale; A training module is used to train an image classification model based on the target slice image, and the image classification model is used to predict the category to which the input slice image belongs.

17. A computer device, characterized in that: The computer device includes: a processor and a memory, the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the training method of the image classification model as described in any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the training method of the image classification model as described in any one of claims 1 to 15.

19. A computer program product, characterized in that The computer program product stores a computer program, and the computer program is loaded and executed by a processor to implement the training method of the image classification model according to any one of claims 1 to 15.

Images