Image interpretation model training method, device, equipment and storage medium

Abstract

The present application discloses a training method, device, equipment and storage medium for an image interpretation model, and relates to the field of artificial intelligence technology. The method comprises: using unlabeled local images to train a feature extraction network; using unlabeled local images and labeled local images to train a preliminarily trained feature extraction network; using the trained feature extraction network to extract deep learning features of the global image from the labeled global image; obtaining a global interpretation result corresponding to the global image based on the fusion features of the global image through the global interpretation network, the fusion features including deep learning features and manual features; training the global interpretation network based on the global interpretation result and global annotation result corresponding to the global image. The present application provides a hybrid self-supervised and semi-supervised image interpretation method from local to global, which not only improves the accuracy of the final global interpretation result of the model, but also takes into account interpretability.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a training method, apparatus, device, and storage medium for an image interpretation model. Background Art

[0002] Currently, artificial intelligence technology has begun to be applied to assisted diagnosis of whole-film cytology films, and an auxiliary interpretation process has been established from the cell level to the whole-film level.

[0003] Current AI technologies for extracting cell features either rely on manual features based on physicians' experience or train deep learning features from a small number of labeled cells. Manual features can only capture a portion of the physicians' experience, while deep learning features trained on a small number of labeled cells have poor generalization and interpretability. Summary of the Invention

[0004] The present invention provides a method, apparatus, device, and storage medium for training an image interpretation model. The technical solution is as follows:

[0005] According to one aspect of an embodiment of the present application, a method for training an image interpretation model is provided, wherein the image interpretation model includes a feature extraction network and a global interpretation network, and the method includes:

[0006] Using unlabeled local images to train the feature extraction network to obtain a preliminarily trained feature extraction network;

[0007] Using the unlabeled local area image and the labeled local area image, training the preliminarily trained feature extraction network to obtain a trained feature extraction network;

[0008] Extracting deep learning features of the global image from the labeled global image using the trained feature extraction network;

[0009] Obtaining a global interpretation result corresponding to the global image through the global interpretation network according to the fusion features of the global image; wherein the fusion features of the global image include: deep learning features of the global image and manual features of the global image;

[0010] Based on the global interpretation results and global labeling results corresponding to the global image, the global interpretation network is trained to obtain a trained global interpretation network; wherein the trained image interpretation model includes the trained feature extraction network and the trained global interpretation network.

[0011] According to one aspect of an embodiment of the present application, a method for image interpretation is provided, the method comprising:

[0012] Extracting multiple local images from the global image to be interpreted;

[0013] Inputting the multiple local images into a feature extraction network of an image interpretation model respectively to obtain deep learning features corresponding to the multiple local images respectively;

[0014] Merging the deep learning features corresponding to the multiple local images to obtain the deep learning features of the global image;

[0015] fusing the deep learning features of the global image with the manual features of the global image to obtain fused features of the global image;

[0016] A global interpretation result corresponding to the global image is obtained through the global interpretation network of the image interpretation model based on the fusion features of the global image.

[0017] According to one aspect of an embodiment of the present application, a training device for an image interpretation model is provided, wherein the image interpretation model includes a feature extraction network and a global interpretation network, and the device includes:

[0018] A first training module is used to train the feature extraction network using an unlabeled local image to obtain a preliminarily trained feature extraction network;

[0019] A second training module is configured to train the preliminarily trained feature extraction network using the unlabeled local area images and the labeled local area images to obtain a trained feature extraction network;

[0020] A feature extraction module, configured to extract deep learning features of the global image from the labeled global image using the trained feature extraction network;

[0021] A global interpretation module, configured to obtain a global interpretation result corresponding to the global image based on the fusion features of the global image through the global interpretation network; wherein the fusion features of the global image include: deep learning features of the global image and manual features of the global image;

[0022] The third training module is used to train the global interpretation network based on the global interpretation results and global labeling results corresponding to the global image to obtain a trained global interpretation network; wherein the trained image interpretation model includes the trained feature extraction network and the trained global interpretation network.

[0023] According to one aspect of an embodiment of the present application, there is provided an image interpretation device, the device comprising:

[0024] An image capture module, used to capture multiple local images from a global image to be interpreted;

[0025] A feature extraction module, configured to input the plurality of local images into a feature extraction network of an image interpretation model to obtain deep learning features corresponding to the plurality of local images;

[0026] A feature merging module, configured to merge the deep learning features corresponding to the multiple local images to obtain the deep learning features of the global image;

[0027] A feature fusion module, configured to fuse the deep learning features of the global image with the manual features of the global image to obtain fused features of the global image;

[0028] The global interpretation module is used to obtain a global interpretation result corresponding to the global image based on the fusion features of the global image through the global interpretation network of the image interpretation model.

[0029] According to one aspect of an embodiment of the present application, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the above-mentioned image interpretation model training method, or to implement the above-mentioned image interpretation method.

[0030] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the above-mentioned image interpretation model training method, or to implement the above-mentioned image interpretation method.

[0031] According to one aspect of an embodiment of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned image interpretation model training method or the aforementioned image interpretation method.

[0032] The technical solutions provided by the embodiments of the present application include at least the following beneficial effects:

[0033] The present application provides a method for image interpretation from local to global that mixes self-supervision and semi-supervision. First, in order to solve the problem of insufficient labeled data for deep learning feature training, the present application takes into account massive unlabeled local images, allowing the model to learn task-irrelevant features from massive unlabeled local images through self-supervised training, thereby having better generalization. Then, a small number of labeled local images are combined with massive unlabeled local images, and after fine-tuning the model through the semi-supervised training method, the model is allowed to further learn task-related features. Finally, the above-mentioned task-related deep learning features and manual features are combined, and through the model integration method, more representative global-level fusion features are obtained, thereby better interpreting the global image. Through the above process, while improving the accuracy of the final global interpretation result of the model, interpretability is also taken into account. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0035] Figure 1 This is a schematic diagram of an implementation environment for a solution provided by an embodiment of the present application;

[0036] Figure 2 This is a flowchart of a method for training an image interpretation model provided by one embodiment of the present application;

[0037] Figure 3 is a flowchart of a method for training an image interpretation model provided by another embodiment of the present application;

[0038] Figure 4 Schematic diagram of a self-supervised and semi-supervised training feature extraction network provided by one embodiment of the present application;

[0039] Figure 5 is a schematic diagram of a self-supervised training process provided by an embodiment of the present application;

[0040] Figure 6 is a schematic diagram of a semi-supervised training process provided by an embodiment of the present application;

[0041] Figure 7 This is a schematic diagram of the fusion of deep learning features and manual features provided by an embodiment of the present application;

[0042] Figure 8 is a schematic diagram of a global image interpretation process provided by an embodiment of the present application;

[0043] Figure 9 This is a flow chart of an image interpretation method provided by one embodiment of the present application;

[0044] Figure 10 This is a block diagram of a training device for an image interpretation model provided by one embodiment of the present application;

[0045] Figure 11 is a block diagram of an image interpretation device provided by one embodiment of the present application;

[0046] Figure 12 It is a structural diagram of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0048] Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0049] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0050] Computer vision (CV) is the science of making machines "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying, tracking, and measuring objects. Furthermore, it performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems that can extract information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / action recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. It also includes common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0051] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0052] The technical solutions provided in the embodiments of this application involve technologies such as artificial intelligence machine learning and computer vision, and are specifically introduced and explained through the following embodiments.

[0053] Please refer to Figure 1 , which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application. The solution implementation environment may include a model training device 10 and a model use device 20.

[0054] The model training device 10 can be an electronic device such as a computer, server, intelligent robot, or other electronic device with strong computing capabilities. The model training device 10 is used to train the image interpretation model 30. In the embodiment of the present application, the image interpretation model 30 is a neural network model for image interpretation. The model training device 10 can use machine learning to train the image interpretation model 30 to achieve better performance.

[0055] The trained image interpretation model 30 can be deployed in a model-using device 20 to provide image interpretation results. The model-using device 20 can be a terminal device such as a mobile phone, computer, smart TV, multimedia player, wearable device, medical device, or a server, and this application does not limit this.

[0056] In some embodiments, as Figure 1 As shown, the image interpretation model 30 may include: a feature extraction network 31 and a global interpretation network 32. The feature extraction network 31 may be a neural network, which is used to extract deep learning features from the local image. The global interpretation network 32 may also be a neural network, which is used to obtain a global interpretation result corresponding to the global image based on the fusion features of the global image. The fusion features of the global image include: the deep learning features of the global image and the manual features of the global image. The deep learning features of the global image are obtained by merging the deep learning features corresponding to multiple local images, and the multiple local images are obtained by intercepting from the global image.

[0057] In the embodiments of this application, a global image refers to an image containing global information. From a global image, one or more local images can be captured. A local image is a partial area captured from a global image, or an image block captured from a global image. From a global image, different areas can be captured to obtain multiple different local images.

[0058] For example, in the context of cytology-assisted film reading, the global image is a whole-slice image of cells, such as a WSI (Whole Slide Image) image. Accordingly, several different regions can be intercepted from the whole-slice image of cells to obtain several different cell local images. Each cell local image may contain one cell or multiple cells, which is not limited in this application.

[0059] Of course, in some other possible application scenarios, global images and local images will have different implementations. Taking the interpretation scenario of pathological slice images in the medical field as an example, the global image can be a pathological slice image, and the local image can be an image block corresponding to an area intercepted from the pathological slice image. Taking the interpretation scenario of satellite images in the intelligent transportation field as an example, the global image can be a satellite image obtained by photographing a certain area, and the local image can be an image block corresponding to an area intercepted from the satellite image. The technical solution of this application can be applied to image interpretation in different application scenarios.

[0060] Below, the technical solution of this application will be introduced and explained through several method embodiments.

[0061] Please refer to Figure 2 , which shows a flow chart of a method for training an image interpretation model provided by one embodiment of the present application. The execution entity of each step of the method can be the model training device described above. The method can include the following steps (210-250):

[0062] Step 210: Use unlabeled local images to train the feature extraction network to obtain a preliminarily trained feature extraction network.

[0063] The feature extraction network can be a neural network for extracting deep learning features from the local image. In the embodiment of the present application, the structure of the feature extraction network is not limited. For example, the feature extraction network can be a CNN (Convolutional Neural Networks) or other forms of neural networks. The input of the feature extraction network can be a local image, and its output is the deep learning features corresponding to the local image.

[0064] In an embodiment of the present application, first, a feature extraction network is preliminarily trained using unlabeled local images by means of self-supervised learning to obtain a preliminarily trained feature extraction network. Optionally, a contrastive learning strategy is adopted to construct positive and negative sample pairs based on unlabeled local images, and by shortening the distance between positive samples and increasing the distance between negative samples, a loss function can be constructed between positive and negative samples to supervise the learning of the network. Through this contrastive learning strategy, the feature extraction network can learn task-agnostic feature information from a large number of unlabeled local images, and these task-agnostic feature information are also domain-invariant, that is, they are the same at different centers. This is because contrastive learning randomly selects data from different centers to form positive and negative sample pairs when constructing sample pairs.

[0065] In some embodiments, step 210 may include the following sub-steps:

[0066] 1. Based on an unlabeled local image, generate a first sample set; wherein the first sample set includes at least one positive sample pair and at least one negative sample pair.

[0067] A positive sample pair includes two images, which can be two different images generated based on the same unlabeled local image. Optionally, two images with different attributes generated based on the same unlabeled local image are used to construct a positive sample pair. For example, for any unlabeled local image, the local image is transformed to generate a transformed local image that is different from the original local image. The original local image and the transformed local image can constitute a positive sample pair. For another example, for any unlabeled local image, two different methods are used to transform the local image to generate two different transformed local images. The two transformed local images can constitute a positive sample pair. The above-mentioned transformation processing includes but is not limited to flipping, rotation, color conversion, random cropping, noise addition and other processing, which is not limited in this application. Taking rotation as an example, the original local image can be rotated 90 degrees clockwise to generate a transformed local image. The original local image and the transformed local image can constitute a positive sample pair. Alternatively, the original local image can be rotated 90 degrees and 180 degrees clockwise respectively to generate two transformed local images, and the two transformed local images can constitute a positive sample pair.

[0068] The negative sample pair also contains two images, which can be two different unlabeled local images. Optionally, two different unlabeled local images are used to construct a negative sample pair. For example, it contains multiple unlabeled local images, such as local image A, local image B, local image C, etc., where local image A and local image B can constitute a negative sample pair, local image A and local image C can constitute a negative sample pair, and local image B and local image C can also constitute a negative sample pair.

[0069] 2. Use the first sample set to train the feature extraction network to obtain a preliminarily trained feature extraction network.

[0070] Optionally, for each sample pair in the first sample set, the two images in the sample pair are respectively input into the feature extraction network, and the deep learning features corresponding to the two images are output through the feature extraction network; based on the distance between the deep learning features corresponding to the two images, the loss function value is determined; with the convergence of the loss function value as the goal, the parameters of the feature extraction network are adjusted to obtain a preliminarily trained feature extraction network.

[0071] Regardless of whether the pair is positive or negative, the two images in the pair are fed into a feature extraction network, which then outputs the deep learning features corresponding to the two images. When constructing the loss function during the initial training of the feature extraction network, the distance between the deep learning features corresponding to the two images in the same pair can be measured. This loss function is constructed with the goal of narrowing the distance between the deep learning features corresponding to the two images in the positive pair and widening the distance between the deep learning features corresponding to the two images in the negative pair. Subsequently, with the goal of achieving convergence of this loss function, the parameters of the feature extraction network are adjusted using methods such as gradient descent to obtain a preliminarily trained feature extraction network.

[0072] In step 220 , the unlabeled local area image and the labeled local area image are used to train the initially trained feature extraction network to obtain a trained feature extraction network.

[0073] After completing the initial training process, the parameters of the initially trained feature extraction network are fine-tuned using semi-supervised learning with unlabeled local images and a small number of labeled local images, resulting in a fully trained feature extraction network. In this step, the structure of the feature extraction network remains the same as in the previous step, and its parameters are initialized to the parameters of the initially trained feature extraction network obtained in the previous step. For semi-supervised learning, two sets of data are used for learning: the first set is again the unlabeled local images, and the second set is a small number of labeled local images. For the unlabeled local images, two different types of transformations are applied. The feature extraction network is then required to output similar probabilistic predictions for the different transformed versions of the same image, employing a consistency loss to guide network learning. For the labeled local images, traditional supervised learning is employed. The loss functions for both the unlabeled and labeled data are combined to supervise network learning. After this step of semi-supervised learning, task-specific features are obtained.

[0074] In some embodiments, step 220 may include the following sub-steps:

[0075] 1. Based on the unlabeled local image, generate a second sample set; wherein the second sample set includes at least one image pair, each image pair is two images with different attributes generated based on the same unlabeled local image.

[0076] An image pair contains two images, and these two images can be two different images generated based on the same unlabeled local image. Optionally, two images with different attributes generated based on the same unlabeled local image are used to construct an image pair. For example, for any unlabeled local image, the local image is transformed to generate a transformed local image that is different from the original local image. The original local image and the transformed local image can constitute an image pair. For another example, for any unlabeled local image, two different methods are used to transform the local image to generate two different transformed local images. The two transformed local images can constitute an image pair. The above-mentioned transformation processing includes but is not limited to rotation, noise addition and other processing, which is not limited in this application.

[0077] In addition, the image pairs in the second sample set are generated in the same or similar manner as the positive sample pairs in the first sample set. The image pairs in the second sample set can directly use the positive sample pairs in the first sample set or can be regenerated, which is not limited in this application.

[0078] 2. Based on the labeled local image, generate a third sample set; wherein the third sample set includes at least one labeled local image.

[0079] The label of a local image is used to indicate the local annotation result of the local image. The local annotation result is task-related. For example, for a classification task, the local annotation result can be a certain labeled category.

[0080] 3. Use the second sample set and the third sample set to train the initially trained feature extraction network to obtain a fully trained feature extraction network.

[0081] For each image pair in the second sample set, the two images in the pair can be fed into a preliminarily trained feature extraction network. This preliminarily trained feature extraction network outputs the deep learning features corresponding to the two images. Furthermore, the deep learning features corresponding to the two images can be fed into the local interpretation network of the image interpretation model. This local interpretation network outputs the local interpretation results corresponding to the two images. The training goal at this stage is to ensure that the local interpretation results corresponding to the two images in the same image pair are as identical or close as possible.

[0082] For labeled local images in the third sample set, the local images can be input into a preliminarily trained feature extraction network, which then outputs deep learning features corresponding to the local images. Furthermore, the deep learning features corresponding to the local images are input into a local interpretation network, which then outputs a local interpretation result corresponding to the local images. The training goal at this stage is to ensure that the local interpretation result corresponding to the local image and the local annotation result corresponding to the local image are as identical or close as possible.

[0083] Optionally, a first sub-loss is determined based on a second sample set, and the first sub-loss is used to measure the consistency between the local interpretation results corresponding to the two images in the same image pair; a second sub-loss is determined based on a third sample set, and the second sub-loss is used to measure the proximity of the local interpretation result corresponding to the labeled local image to the local annotation result; according to the first sub-loss and the second sub-loss, the preliminarily trained feature extraction network is trained to obtain a trained feature extraction network; wherein the local interpretation result is the result of task prediction based on the deep learning features obtained by the feature extraction network of the local interpretation network of the image interpretation model. The local interpretation network can also be a neural network. This application does not limit the structure of the local interpretation network, such as it can be a fully connected network.

[0084] In addition, the above-mentioned training of the preliminarily trained feature extraction network according to the first sub-loss and the second sub-loss to obtain the trained feature extraction network can include the following two possible implementation methods. In one possible implementation method, the second sample set and the third sample set are used alternately to train the feature extraction network. For example, the second sample set is first used to train the feature extraction network, and the parameters of the feature extraction network are adjusted according to the first sub-loss. Then, the third sample set is used to train the feature extraction network, and the parameters of the feature extraction network are adjusted according to the second sub-loss. Then, the second sample set is used again to train the feature extraction network, and the parameters of the feature extraction network are adjusted according to the first sub-loss. In this way, the feature extraction network is alternately iterated. In another possible implementation method, the second sample set and the third sample set can also be used to train the feature extraction network synchronously. For example, the total loss is calculated according to the first sub-loss and the second sub-loss, and then the parameters of the feature extraction network are adjusted based on the total loss. The above total loss can be calculated by weighted summation or other methods, which is not limited in this application.

[0085] In step 230 , the trained feature extraction network is used to extract deep learning features of the global image from the labeled global image.

[0086] After completing the training process for the feature extraction network, the trained feature extraction network can be used to extract deep learning features of the global image from the labeled global image.

[0087] Optionally, multiple local images are captured from a labeled global image; the multiple local images are respectively input into a trained feature extraction network to obtain deep learning features corresponding to the multiple local images; the deep learning features corresponding to the multiple local images are merged to obtain deep learning features of the global image.

[0088] In the embodiments of this application, there is no limit on the number of local images captured from the labeled global image; for example, the number can be 5, 10, 15, or 20, etc., which can be set based on actual needs. Furthermore, the deep learning features corresponding to multiple local images can be merged using a cascade or other method, which is also not limited in this application.

[0089] After multiple local images are captured from the same global image, since the multiple local images reflect the image information of different parts of the global image, the deep learning features of the multiple local images are merged to obtain the deep learning features of the global image.

[0090] In step 240 , a global interpretation result corresponding to the global image is obtained based on the fusion features of the global image through the global interpretation network; wherein the fusion features of the global image include: deep learning features of the global image and manual features of the global image.

[0091] Handcrafted features are in contrast to deep learning features. As mentioned above, deep learning features are extracted using neural networks. Deep learning features are deeper, more abstract features within an image. Handcrafted features refer to low-level and mid-level features extracted from an image using traditional image processing algorithms. Low-level features primarily refer to features that can be derived through simple operations on the image itself. Examples include color, texture, shape, and gradient, and they are also the foundation for extracting mid-level features. Mid-level features are derived by fusing multiple features based on low-level features, primarily the diverse fusion of color and texture. Handcrafted features can be described using traditional morphological image features. While they generalize and are interpretable, the limited number of handcrafted features makes them inadequate for image interpretation. Deep learning features, while less interpretable than handcrafted features, offer advantages in robustness and quantity.

[0092] In the embodiment of the present application, the deep learning features of the global image are fused with the manual features to obtain the fused features of the global image. The fused features of the global image are then input into the global interpretation network to output the global interpretation result corresponding to the global image. This approach takes into account both the interpretability and comprehensiveness of the features.

[0093] In the embodiments of the present application, there is no limitation on the fusion method of deep learning features and manual features. Optionally, the deep learning features of the global image and the manual features of the global image are spliced ​​to obtain the fusion features of the global image. For example, the deep learning features of the global image are a 512-dimensional vector, and the manual features of the global image are a 50-dimensional vector. By splicing the deep learning features and the manual features, a 562-dimensional vector can be obtained, and the 562-dimensional vector is the fusion feature of the global image. In the present application, the deep learning features and the manual features are fused in a simple splicing manner, which helps to improve the efficiency of feature fusion.

[0094] Step 250 : Based on the global interpretation results and global labeling results corresponding to the global image, the global interpretation network is trained to obtain a trained global interpretation network; wherein the trained image interpretation model includes a trained feature extraction network and a trained global interpretation network.

[0095] The global image is labeled, and the label of the global image is used to indicate the global annotation result of the global image. The global annotation result is task-dependent. For example, for a classification task, the global annotation result can be a certain labeled category.

[0096] Based on the global interpretation results and global annotation results corresponding to the global image, a loss function for the global interpretation network is constructed. This loss function can be used to measure the proximity between the global interpretation results and the global annotation results. With the goal of improving this proximity, the parameters of the global interpretation network are adjusted to obtain a trained global interpretation network. It should be noted that during the training of the global interpretation network, the parameters of the feature extraction network are fixed, i.e., the parameters of the feature extraction network are fixed after the completion of step 220. Ultimately, the trained image interpretation model includes the trained feature extraction network and the trained global interpretation network.

[0097] Optionally, the trained image interpretation model further includes a trained local interpretation network for outputting a local interpretation result corresponding to the local image.

[0098] In summary, the present application provides a method for image interpretation from local to global that mixes self-supervision and semi-supervision. First, in order to solve the problem of insufficient labeled data for deep learning feature training, the present application takes into account massive unlabeled local images, allowing the model to learn task-irrelevant features from massive unlabeled local images through self-supervised training, thereby having better generalization. Then, a small number of labeled local images are combined with massive unlabeled local images, and the model is fine-tuned through the semi-supervised training method to allow the model to further learn task-related features. Finally, the above-mentioned task-related deep learning features and manual features are combined, and through the model integration method, more representative global-level fusion features are obtained, thereby better interpreting the global image. Through the above process, while improving the accuracy of the final global interpretation result of the model, interpretability is also taken into account.

[0099] Below, we will introduce the technical solution of this application using the scenario of cytology assisted reading as an example. In this scenario, the global image is the whole cell slice image, and the local image is a partial area intercepted from the whole cell slice image. We call the local image the cell local image. Figure 3 This is a flowchart of a method for training an image interpretation model provided by another embodiment of the present application. The execution entity of each step of the method can be the model training device described above. The method can include the following steps (310-390):

[0100] Step 310 : extracting a plurality of label-free local cell images from the full-slice cell image.

[0101] Each full-slice image contains a large number of cells. Using a cell detection algorithm, we can identify multiple cells within that full-slice image. For example, assuming the number of full-slice images is in the thousands, and each contains tens of thousands of cells, the number of unlabeled localized cell images that can be extracted from these full-slice images could reach tens of millions. These tens of millions of unlabeled localized cell images contain a wealth of information that can be extracted.

[0102] Furthermore, in the embodiments of the present application, there is no limitation on the cell types included in the whole-cell image. For example, the whole-cell image can be a whole-cell image of any part of the human body, such as a whole-cell image of cervical cells, a whole-cell image of intestinal cells, a whole-cell image of liver cells, a whole-cell image of stomach cells, and so on. By performing both cellular and whole-cell interpretation of the whole-cell image, a negative or positive interpretation result for a particular disease can be output, assisting doctors in diagnosing the condition.

[0103] Step 320 : Generate a first sample set based on the unlabeled local cell image; wherein the first sample set includes at least one positive sample pair and at least one negative sample pair.

[0104] like Figure 4 As shown, the feature extraction network is initially trained using unlabeled local cell images 41 through self-supervised training to obtain a preliminarily trained feature extraction network. For example, by using a contrastive learning strategy to perform self-supervised training on the feature extraction network, the feature extraction network can learn task-independent feature information from a large number of unlabeled local cell images. This task-independent feature information can also be called domain-invariant feature information.

[0105] Optionally, two images with different attributes generated based on the same unlabeled cell localization image are used to construct a positive sample pair. For example, for any unlabeled cell localization image, the localization image is transformed to generate a transformed cell localization image that is different from the original cell localization image. The original cell localization image and the transformed cell localization image can constitute a positive sample pair. For another example, for any unlabeled cell localization image, two different transformed cell localization images are generated by transforming the cell localization image in two different ways. The two transformed cell localization images can constitute a positive sample pair.

[0106] Optionally, two different unlabeled cell local images are used to construct a negative sample pair.

[0107] Step 330: Use the first sample set to train the feature extraction network to obtain a preliminarily trained feature extraction network.

[0108] Regardless of whether the pair is positive or negative, the two images in the pair are fed into a feature extraction network, which then outputs the deep learning features corresponding to the two images. When constructing the loss function during the initial training of the feature extraction network, the distance between the deep learning features corresponding to the two images in the same pair can be measured. This loss function is constructed with the goal of narrowing the distance between the deep learning features corresponding to the two images in the positive pair and widening the distance between the deep learning features corresponding to the two images in the negative pair. Subsequently, with the goal of achieving convergence of this loss function, the parameters of the feature extraction network are adjusted using methods such as gradient descent to obtain a preliminarily trained feature extraction network.

[0109] For example, take the positive sample pair as an example, Figure 5As shown, for any unlabeled local cell image X, two different transformation methods are used to generate two different transformed local cell images, as shown in the figure, X1 and X2. X1 is input into feature extraction network 1, which outputs the deep learning features corresponding to X1. X2 is input into feature extraction network 2, which outputs the deep learning features corresponding to X2. A loss function is then calculated based on the distance between the deep learning features corresponding to X1 and X2. The processing of negative sample pairs is similar to that of positive sample pairs, except that the negative sample pairs contain two different unlabeled local cell images. Furthermore, the structures of feature extraction networks 1 and 2 are identical, but their parameters can be updated asynchronously. For example, the parameters of feature extraction network 1 can be updated using methods such as gradient descent based on the loss function, and the parameters of feature extraction network 2 can be updated using methods such as EMA (Exponential Moving Averaging) based on the parameters of feature extraction network 1.

[0110] Step 340 : Generate a second sample set based on the unlabeled cell local image; wherein the second sample set includes at least one image pair, and each image pair is two images with different attributes generated based on the same unlabeled cell local image.

[0111] like Figure 4 As shown, after completing the above preliminary training process, through semi-supervised training, unlabeled cell local images 41 and a small number of labeled cell local images 42 are used to fine-tune the parameters of the preliminary trained feature extraction network to obtain a feature extraction network that has been completed.

[0112] Alternatively, two images with different attributes generated based on the same unlabeled local cell image are used to construct an image pair. Figure 4 As shown, for any unlabeled cell local image 41, two different methods are used to transform the cell local image 41 to generate two different transformed cell local images x′. u and x′ u ′, these two transformed cell local images x′ u and x′ u' can constitute an image pair. The above-mentioned transformation processing may include weak transformation processing and strong transformation processing. Among them, weak transformation processing refers to that the difference between the two images before and after the transformation is small, for example, weak transformation processing includes but is not limited to flipping, rotation and other processing. Strong transformation processing refers to that the difference between the two images before and after the transformation is large, for example, strong transformation processing includes but is not limited to color transformation, random cropping, noise addition and other processing. In practical applications, a combination of weak transformation processing and strong transformation processing can also be adopted, and this application does not limit this.

[0113] Step 350 : Generate a third sample set based on the labeled cell local area image; wherein the third sample set includes at least one labeled cell local area image.

[0114] The label of the local cell image is used to indicate the local annotation result of the local cell image. For example, the local annotation result can be a negative or positive annotation result. Optionally, for the positive, if there are multiple different categories of positive, the category of the positive can be further labeled. Taking cervical cells as an example, the positive categories can include ASCUS (Atypical Squamous Cells Of Unknown Significance), LSIL (Low-grade Squamous Intraepithelial Lesion), HSIL (High-grade Squamous Intraepithelial Lesion), etc. If the local annotation result not only marks negative or positive, but also marks the positive category for the positive, then the local interpretation network that has been trained subsequently can have the ability to output the positive category.

[0115] Step 360: Use the second sample set and the third sample set to train the preliminarily trained feature extraction network to obtain a trained feature extraction network.

[0116] For semi-supervised training, there are two parts of data that can be used for learning. The first part is still the above-mentioned unlabeled cell local images (i.e., the second sample set), and the second part is a small number of labeled cell local images (i.e., the third sample set). Figure 4 As shown, the unsupervised consistency loss is determined based on the second sample set, and the unsupervised consistency loss is used to measure the two images x′ in the same image pair. u and x′ u ′ respectively correspond to the local interpretation results y′ u and y′ u ′; determine the supervision loss based on the third sample set, which is used to measure the labeled cell local image xl The corresponding local interpretation result y l Relative to the closeness of the local annotation result; according to the above-mentioned unsupervised consistency loss and supervised loss, the initially trained feature extraction network is trained to obtain a completed feature extraction network; wherein, the local interpretation result is the result of task prediction based on the deep learning features obtained by the feature extraction network of the local interpretation network of the image interpretation model.

[0117] For example, Figure 6 As shown, the unlabeled cell local image is transformed to generate two images x and Construct an image pair. Figure 6 The M in represents the model composed of the preliminarily trained feature extraction network and local interpretation network, and its parameters are represented by θ. After being processed by the model M, the local interpretation result p corresponding to the image x is obtained. θ (y|x) and images Corresponding local interpretation results Based on the above two local interpretation results, the unsupervised consistency loss is calculated. On the other hand, the labeled cell local image x* is processed by the model M to obtain the local interpretation result p corresponding to the image x*. θ (y|x * ), calculate the supervision loss based on the local interpretation result and the local annotation result y* corresponding to the image x*. The supervision loss can be the cross entropy loss, so in Figure 6 Finally, the unsupervised consistency loss and the supervised cross entropy loss are combined (e.g., direct summation or weighted summation) to calculate the total loss. The parameters θ of the model M are adjusted based on the total loss to obtain the trained model M. The trained model M includes the trained feature extraction network and the trained local interpretation network.

[0118] In step 370 , the trained feature extraction network is used to extract deep learning features of the whole-slice cell image from the labeled whole-slice cell image.

[0119] Alternatively, as Figure 7 As shown, multiple cell local images 51 are captured from a labeled cell full-slice image; the multiple cell local images 51 are respectively input into the trained feature extraction network 31 to obtain deep learning features corresponding to the multiple cell local images; the deep learning features corresponding to the multiple cell local images are merged to obtain the deep learning features of the cell full-slice image.

[0120] In step 380 , a global interpretation result corresponding to the whole-slice image is obtained based on the fusion features of the whole-slice image through a global interpretation network; wherein the fusion features of the whole-slice image include: deep learning features of the whole-slice image and manual features of the whole-slice image.

[0121] Alternatively, as Figure 7 As shown, the deep learning features of the whole cell slice image and the manual features of the whole cell slice image are fused to obtain the fused features of the whole cell slice image, and then the fused features of the whole cell slice image are input into the global interpretation network 32 to output the global interpretation result corresponding to the whole cell slice image.

[0122] Step 390 , based on the global interpretation results and global labeling results corresponding to the whole cell slice image, the global interpretation network is trained to obtain a trained global interpretation network; wherein the trained image interpretation model includes a trained feature extraction network and a trained global interpretation network.

[0123] The label of the whole cell slice image is used to indicate the global annotation result of the whole cell slice image. For example, the global annotation result can be a negative or positive annotation result. Optionally, for a positive, if there are multiple different categories of positives, the category of the positive can be further labeled. If the global annotation result, in addition to labeling negative or positive, also labels the positive category for the positive, then the global interpretation network that is subsequently trained can have the ability to output the positive category.

[0124] Alternatively, as Figure 8 As shown, while the global interpretation network 32 is being trained, the parameters of the feature extraction network 31 remain fixed. That is, the parameters of the feature extraction network 31 are fixed after step 360 is completed. Ultimately, the trained image interpretation model includes a trained feature extraction network and a trained global interpretation network. Optionally, the trained image interpretation model also includes a trained local interpretation network, which outputs local interpretation results corresponding to the local image.

[0125] In summary, this application provides a hybrid self-supervised and semi-supervised image interpretation method from the cellular level to the whole-film level for cytology-assisted image interpretation scenarios. This method improves the accuracy of the model's final global interpretation results while also taking into account interpretability, thereby providing doctors with more accurate and interpretable assisted interpretation results.

[0126] The above describes the training process of the image interpretation model. Below, we will explain the use of the image interpretation model through examples. It should be noted that the content related to the use of the model corresponds to the content related to the training process, and the two are interconnected. If there is a lack of detailed explanation on one side, please refer to the description on the other side.

[0127] Please refer to Figure 9 , which shows a flow chart of an image interpretation method provided by one embodiment of the present application. The execution subject of each step of the method can be the model using device described above. The method can include the following steps (910-950):

[0128] Step 910: intercept a plurality of local images from the global image to be interpreted.

[0129] In step 920 , the multiple local images are respectively input into the feature extraction network of the image interpretation model to obtain deep learning features corresponding to the multiple local images.

[0130] Step 930: Merge the deep learning features corresponding to the multiple local images to obtain the deep learning features of the global image.

[0131] Step 940: Fuse the deep learning features of the global image with the manual features of the global image to obtain fused features of the global image.

[0132] In step 950 , a global interpretation result corresponding to the global image is obtained based on the fusion features of the global image through the global interpretation network of the image interpretation model.

[0133] Optionally, the method may further include: obtaining, using a local interpretation network of the image interpretation model, a local interpretation result corresponding to the local image based on deep learning features corresponding to the local image. The local interpretation result corresponding to the local image may vary in different application scenarios. For example, in a cytology-assisted film reading scenario, the local interpretation result corresponding to the local image may include negative and positive. Optionally, the positive result may be further subdivided into multiple different categories.

[0134] Optionally, the method may further include: selecting a target local image whose local interpretation result satisfies a condition based on the local interpretation results corresponding to the multiple local images. The above-mentioned selection conditions for the target local image can be flexibly set in different application scenarios. Taking the cytology-assisted film reading scenario as an example, the above-mentioned condition may be that the local interpretation result is positive. In this case, the local image with a positive local interpretation result is selected from the multiple local images as the target local image, and the target local image can be provided to the doctor for reference.

[0135] For details not described in detail in the above steps 910 to 950, please refer to the above embodiments and will not be repeated here.

[0136] Still taking the application of the solution in the cytology assisted film reading scenario as an example, the global image is the cell full-film image, and the local image is the cell local image captured from the cell full-film image. In one example, the solution can be applied to cervical cytology assisted screening for negative results. When doctors actually read films, there are often a large number of negative films (about 80-90%). This method can screen out some negative films on the basis of ensuring positive recall, thereby better assisting doctors in reading films. In another example, the solution can be applied to positive film assisted diagnosis. This method combines suspicious positive cells and middle-level reference cells, and can output various categories of positive classifications (such as ASCUS, LSIL, HSIL, etc.) in accordance with the specifications of the structured report in the cytology TBS reporting system. In addition, the positive cells and middle-level reference cells of various categories selected by the model are displayed to the doctor during the assisted reading, so that the doctor can quickly find the middle-level reference cells and various positive cells for interpretation when looking at the positive films. This product can be used for cervical cytology films obtained by various preparation methods (sedimentation type or membrane type). It can be deployed locally in the hospital or in the cloud so that doctors can call it remotely.

[0137] In summary, this application provides a two-stage interpretation method from the local level to the global level, extracting deep learning features at the local level, then combining manual features to obtain global fusion features, and finally performing global-level interpretation based on the global fusion features. While improving the accuracy of the model's final global interpretation results, it also takes into account interpretability.

[0138] In addition, by outputting the local interpretation results through the local interpretation network, the model can provide more valuable and reference-worthy information.

[0139] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0140] Please refer to Figure 10 , which shows a block diagram of a training device for an image interpretation model provided in one embodiment of the present application. This device can be used to implement the aforementioned image interpretation model training method. The device 1000 may include: a first training module 1010, a second training module 1020, a feature extraction module 1030, a global interpretation module 1040, and a third training module 1050.

[0141] The first training module 1010 is used to train the feature extraction network using unlabeled local images to obtain a preliminarily trained feature extraction network.

[0142] The second training module 1020 is configured to train the preliminarily trained feature extraction network using the unlabeled local image and the labeled local image to obtain a trained feature extraction network.

[0143] The feature extraction module 1030 is configured to extract deep learning features of the global image from the labeled global image using the trained feature extraction network.

[0144] The global interpretation module 1040 is used to obtain a global interpretation result corresponding to the global image based on the fusion features of the global image through the global interpretation network; wherein the fusion features of the global image include: the deep learning features of the global image and the manual features of the global image.

[0145] The third training module 1050 is used to train the global interpretation network based on the global interpretation results and global labeling results corresponding to the global image to obtain a trained global interpretation network; wherein the trained image interpretation model includes the trained feature extraction network and the trained global interpretation network.

[0146] In some embodiments, the first training module 1010 is configured to:

[0147] Based on the unlabeled local image, generating a first sample set; wherein the first sample set includes at least one positive sample pair and at least one negative sample pair;

[0148] The feature extraction network is trained using the first sample set to obtain the preliminarily trained feature extraction network.

[0149] Optionally, the first training module 1010 is specifically configured to:

[0150] Constructing a positive sample pair by using two images with different attributes generated based on the same unlabeled local image;

[0151] Two different unlabeled local images are used to construct a negative sample pair.

[0152] Optionally, the first training module 1010 is specifically configured to:

[0153] For each sample pair in the first sample set, inputting the two images in the sample pair into the feature extraction network respectively, and outputting deep learning features corresponding to the two images respectively through the feature extraction network;

[0154] Determining a loss function value based on the distance between the deep learning features corresponding to the two images;

[0155] With the convergence of the loss function value as the goal, the parameters of the feature extraction network are adjusted to obtain the preliminarily trained feature extraction network.

[0156] In some embodiments, the second training module 1020 is configured to:

[0157] Based on the unlabeled local image, generating a second sample set; wherein the second sample set includes at least one image pair, each image pair being two images with different attributes generated based on the same unlabeled local image;

[0158] Generating a third sample set based on the labeled local image; wherein the third sample set includes at least one of the labeled local image;

[0159] The second sample set and the third sample set are used to train the initially trained feature extraction network to obtain the trained feature extraction network.

[0160] Optionally, the second training module 1020 is specifically configured to:

[0161] determining a first sub-loss based on the second sample set, where the first sub-loss is used to measure the consistency between the local interpretation results corresponding to the two images in the same image pair;

[0162] Determining a second sub-loss based on the third sample set, where the second sub-loss is used to measure the closeness of the local interpretation result corresponding to the labeled local image to the local annotation result;

[0163] Training the preliminarily trained feature extraction network according to the first sub-loss and the second sub-loss to obtain the trained feature extraction network;

[0164] Among them, the local interpretation result is the result obtained by performing task prediction based on the deep learning features obtained by the feature extraction network through the local interpretation network of the image interpretation model.

[0165] In some embodiments, the feature extraction module 1030 is configured to:

[0166] Extracting a plurality of local images from the labeled global image;

[0167] Inputting the multiple local area images into the trained feature extraction network respectively to obtain deep learning features corresponding to the multiple local area images respectively;

[0168] The deep learning features corresponding to the multiple local images are merged to obtain the deep learning features of the global image.

[0169] In some embodiments, the global interpretation module 1040 is further configured to combine the deep learning features of the global image with the manual features of the global image to obtain fused features of the global image.

[0170] In some embodiments, the global image is a whole-slice cell image, and the local image is a partial area intercepted from the whole-slice cell image.

[0171] In summary, the present application provides a method for image interpretation from local to global that mixes self-supervision and semi-supervision. First, in order to solve the problem of insufficient labeled data for deep learning feature training, the present application takes into account massive unlabeled local images, allowing the model to learn task-irrelevant features from massive unlabeled local images through self-supervised training, thereby having better generalization. Then, a small number of labeled local images are combined with massive unlabeled local images, and the model is fine-tuned through the semi-supervised training method to allow the model to further learn task-related features. Finally, the above-mentioned task-related deep learning features and manual features are combined, and through the model integration method, more representative global-level fusion features are obtained, thereby better interpreting the global image. Through the above process, while improving the accuracy of the final global interpretation result of the model, interpretability is also taken into account.

[0172] Please refer to Figure 11 , which shows a block diagram of an image interpretation device provided by one embodiment of the present application. This device can be used to implement the aforementioned image interpretation method. The device 1100 may include: an image capture module 1110, a feature extraction module 1120, a feature merging module 1130, a feature fusion module 1140, and a global interpretation module 1150.

[0173] The image capture module 1110 is configured to capture a plurality of local images from the global image to be interpreted.

[0174] The feature extraction module 1120 is used to input the multiple local images into the feature extraction network of the image interpretation model respectively to obtain deep learning features corresponding to the multiple local images respectively.

[0175] The feature merging module 1130 is configured to merge the deep learning features corresponding to the multiple local images to obtain the deep learning features of the global image.

[0176] The feature fusion module 1140 is used to fuse the deep learning features of the global image with the manual features of the global image to obtain the fused features of the global image.

[0177] The global interpretation module 1150 is configured to obtain a global interpretation result corresponding to the global image based on the fusion features of the global image through the global interpretation network of the image interpretation model.

[0178] In some embodiments, the device 1100 further includes a local interpretation module ( Figure 11 ), which is not shown in the figure, and is used to obtain a local interpretation result corresponding to the local image based on the deep learning features corresponding to the local image through the local interpretation network of the image interpretation model.

[0179] In some embodiments, the local area interpretation module is further configured to select a target local area image whose local area interpretation result satisfies a condition based on the local area interpretation results corresponding to the plurality of local area images.

[0180] In summary, this application provides a two-stage interpretation method from the local level to the global level, extracting deep learning features at the local level, then combining manual features to obtain global fusion features, and finally performing global-level interpretation based on the global fusion features. While improving the accuracy of the model's final global interpretation results, it also takes into account interpretability.

[0181] It should be noted that the apparatus provided in the above embodiments, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0182] Please refer to Figure 12 , which shows a schematic diagram of the structure of a computer device provided by an embodiment of the present application. The computer device can be any electronic device with data calculation, processing and storage functions, and the computer device can be realized as Figure 1 The model training device 10 and / or model using device 20 in the implementation environment of the scheme shown. Figure 1 When the model training device 10 in the embodiment shown in the embodiment is implemented, the computer device can be used to implement the training method of the image interpretation model provided in the above embodiment. Figure 1 When the model in the implementation environment of the illustrated solution uses the device 20, the computer device can be used to implement the image interpretation method provided in the above embodiment. Specifically:

[0183] The computer device 1200 includes a central processing unit (CPU, central processing unit), GPU (graphics processing unit), and FPGA (field programmable gate array) 1201, a system memory 1204 including RAM (random-access memory) 1202 and ROM (read-only memory) 1203, and a system bus 1205 connecting the system memory 1204 and the central processing unit 1201. The computer device 1200 also includes a basic input / output system (I / O system) 1206 for facilitating information transmission between various components within the server, and a mass storage device 1207 for storing an operating system 1213, application programs 1214, and other program modules 1215.

[0184] In some embodiments, the basic input / output system 1206 includes a display 1208 for displaying information and an input device 1209, such as a mouse or keyboard, for user input. Both the display 1208 and the input device 1209 are connected to the central processing unit 1201 via an input / output controller 1210 connected to the system bus 1205. The basic input / output system 1206 may also include an input / output controller 1210 for receiving and processing input from a variety of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1210 also provides output to a display screen, printer, or other types of output devices.

[0185] The mass storage device 1207 is connected to the central processing unit 1201 via a mass storage controller (not shown) connected to the system bus 1205. The mass storage device 1207 and its associated computer-readable media provide non-volatile storage for the computer device 1200. In other words, the mass storage device 1207 may include a computer-readable medium (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0186] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technology, CD-ROM, DVD (Digital Video Disc) or other optical storage, tape cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media is not limited to the above-mentioned ones. The above-mentioned system memory 1204 and mass storage device 1207 can be collectively referred to as memory.

[0187] According to an embodiment of the present application, the computer device 1200 can also be connected to a remote computer on a network such as the Internet for operation. That is, the computer device 1200 can be connected to the network 1212 via the network interface unit 1211 connected to the system bus 1205. Alternatively, the network interface unit 1211 can be used to connect to other types of networks or remote computer systems (not shown).

[0188] The memory also includes at least one instruction, at least one program, code set or instruction set, which is stored in the memory and configured to be executed by one or more processors to implement the above-mentioned image interpretation method or image interpretation model training method.

[0189] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein the storage medium stores at least one instruction, at least one program, a code set, or an instruction set, and when the at least one instruction, the at least one program, the code set, or the instruction set is executed by a processor of a computer device, the at least one instruction, the at least one program, the code set, or the instruction set implements the above-mentioned image interpretation method, or the image interpretation model training method.

[0190] Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random-Access Memory), SSD (Solid State Drives), or an optical disk, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0191] In an exemplary embodiment, a computer program product or computer program is also provided. The computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described image interpretation method or image interpretation model training method.

[0192] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application do not limit this.

[0193] The above description is merely an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A training method for an image interpretation model, characterized in that: The image interpretation model includes a feature extraction network and a global interpretation network, and the method includes: Using unlabeled local images to train the feature extraction network to obtain a preliminarily trained feature extraction network; Using the unlabeled local area image and the labeled local area image, training the preliminarily trained feature extraction network to obtain a trained feature extraction network; Extracting deep learning features of the global image from the labeled global image using the trained feature extraction network; Obtaining a global interpretation result corresponding to the global image through the global interpretation network according to the fusion features of the global image; wherein the fusion features of the global image include: deep learning features of the global image and manual features of the global image; Based on the global interpretation results and global labeling results corresponding to the global image, the global interpretation network is trained to obtain a trained global interpretation network; wherein the trained image interpretation model includes the trained feature extraction network and the trained global interpretation network.

2. The method according to claim 1, characterized in that The feature extraction network is trained using the unlabeled local image to obtain a preliminarily trained feature extraction network, including: Based on the unlabeled local image, generating a first sample set; wherein the first sample set includes at least one positive sample pair and at least one negative sample pair; The feature extraction network is trained using the first sample set to obtain the preliminarily trained feature extraction network.

3. The method according to claim 2, characterized in that Generating a first sample set based on the unlabeled local image includes: Using two images with different attributes generated based on the same unlabeled local image to construct a positive sample pair; Two different unlabeled local images are used to construct a negative sample pair.

4. The method according to claim 2, characterized in that The step of training the feature extraction network using the first sample set to obtain the preliminarily trained feature extraction network includes: For each sample pair in the first sample set, inputting the two images in the sample pair into the feature extraction network respectively, and outputting deep learning features corresponding to the two images respectively through the feature extraction network; Determining a loss function value based on the distance between the deep learning features corresponding to the two images; With the convergence of the loss function value as the goal, the parameters of the feature extraction network are adjusted to obtain the preliminarily trained feature extraction network.

5. The method according to claim 1, wherein The method of training the preliminarily trained feature extraction network using the unlabeled local image and the labeled local image to obtain a trained feature extraction network includes: Based on the unlabeled local image, generating a second sample set; wherein the second sample set includes at least one image pair, each image pair being two images with different attributes generated based on the same unlabeled local image; Generating a third sample set based on the labeled local image; wherein the third sample set includes at least one of the labeled local image; The second sample set and the third sample set are used to train the initially trained feature extraction network to obtain the trained feature extraction network.

6. The method according to claim 5, characterized in that The step of training the preliminarily trained feature extraction network using the second sample set and the third sample set to obtain the trained feature extraction network includes: determining a first sub-loss based on the second sample set, where the first sub-loss is used to measure the consistency between the local interpretation results corresponding to the two images in the same image pair; Determining a second sub-loss based on the third sample set, where the second sub-loss is used to measure the closeness of the local interpretation result corresponding to the labeled local image to the local annotation result; Training the preliminarily trained feature extraction network according to the first sub-loss and the second sub-loss to obtain the trained feature extraction network; Among them, the local interpretation result is the result obtained by performing task prediction based on the deep learning features obtained by the feature extraction network through the local interpretation network of the image interpretation model.

7. The method according to claim 1, characterized in that The step of extracting deep learning features of the global image from the labeled global image using the trained feature extraction network includes: Extracting a plurality of local images from the labeled global image; Inputting the multiple local area images into the trained feature extraction network respectively to obtain deep learning features corresponding to the multiple local area images respectively; The deep learning features corresponding to the multiple local images are merged to obtain the deep learning features of the global image.

8. The method according to claim 1, characterized in that Before obtaining the global interpretation result corresponding to the global image according to the fusion feature of the global image through the global interpretation network, the method further includes: The deep learning features of the global image and the manual features of the global image are spliced ​​to obtain the fusion features of the global image.

9. The method according to any one of claims 1 to 8, characterized in that The global image is a whole-cell slice image, and the local image is a partial area intercepted from the whole-cell slice image.

10. An image interpretation method, characterized in that: The method comprises: Extracting multiple local images from the global image to be interpreted; Inputting the multiple local images into a feature extraction network of an image interpretation model to obtain deep learning features corresponding to the multiple local images, respectively, wherein the feature extraction network is a neural network that is initially trained using unlabeled local images through self-supervised learning and then trained using the unlabeled local images and labeled local images through semi-supervised learning; Merging the deep learning features corresponding to the multiple local images to obtain the deep learning features of the global image; fusing the deep learning features of the global image with the manual features of the global image to obtain fused features of the global image; A global interpretation result corresponding to the global image is obtained through the global interpretation network of the image interpretation model based on the fusion features of the global image.

11. The method according to claim 10, characterized in that After inputting the plurality of local images into the feature extraction network of the image interpretation model to obtain the deep learning features corresponding to the plurality of local images, the method further includes: A local interpretation result corresponding to the local image is obtained through the local interpretation network of the image interpretation model based on the deep learning features corresponding to the local image.

12. The method according to claim 11, characterized in that After obtaining the local interpretation result corresponding to the local image based on the deep learning features corresponding to the local image through the local interpretation network of the image interpretation model, the method further includes: According to the local area interpretation results respectively corresponding to the plurality of local area images, a target local area image whose local area interpretation result meets a condition is selected.

13. A training device for an image interpretation model, characterized in that: The image interpretation model includes a feature extraction network and a global interpretation network, and the device includes: A first training module is used to train the feature extraction network using an unlabeled local image to obtain a preliminarily trained feature extraction network; A second training module is configured to train the preliminarily trained feature extraction network using the unlabeled local area images and the labeled local area images to obtain a trained feature extraction network; A feature extraction module, configured to extract deep learning features of the global image from the labeled global image using the trained feature extraction network; A global interpretation module, configured to obtain a global interpretation result corresponding to the global image based on the fusion features of the global image through the global interpretation network; wherein the fusion features of the global image include: deep learning features of the global image and manual features of the global image; The third training module is used to train the global interpretation network based on the global interpretation results and global labeling results corresponding to the global image to obtain a trained global interpretation network; wherein the trained image interpretation model includes the trained feature extraction network and the trained global interpretation network.

14. An image interpretation device, characterized in that: The device comprises: An image capture module, used to capture multiple local images from a global image to be interpreted; a feature extraction module, configured to input the plurality of local images into a feature extraction network of an image interpretation model to obtain deep learning features corresponding to the plurality of local images, wherein the feature extraction network is a neural network obtained by initially training the unlabeled local images through self-supervised learning and then training the unlabeled local images and labeled local images through semi-supervised learning; A feature merging module, configured to merge the deep learning features corresponding to the multiple local images to obtain the deep learning features of the global image; A feature fusion module, configured to fuse the deep learning features of the global image with the manual features of the global image to obtain fused features of the global image; The global interpretation module is used to obtain a global interpretation result corresponding to the global image based on the fusion features of the global image through the global interpretation network of the image interpretation model.

15. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the training method of the image interpretation model as described in any one of claims 1 to 9, or to implement the image interpretation method as described in any one of claims 10 to 12.

16. A computer-readable storage medium, characterized in that The storage medium stores at least one program, which is loaded and executed by the processor to implement the training method of the image interpretation model as described in any one of claims 1 to 9, or to implement the image interpretation method as described in any one of claims 10 to 12.

17. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor reads and executes the computer instructions from the computer-readable storage medium to implement the image interpretation model training method according to any one of claims 1 to 9, or the image interpretation method according to any one of claims 10 to 12.

Images