Image language fusion prediction learning method for unbalanced data of cervical cytology

Abstract

The invention relates to an image language fusion prediction learning method for unbalanced data of cervical cytology, and relates to the technical field of computers. The method comprises the following steps: acquiring a cervical cytology full-slice image, and cutting the cervical cytology full-slice image into a plurality of patch images; detecting the plurality of patch images to screen out a plurality of suspicious positive cell images from the plurality of patch images; acquiring a clinical text corresponding to the cervical cytology full slice image; extracting a plurality of second image features related to the clinical text from the first image features of the plurality of suspicious positive cell images; and fusing the plurality of second image features and the text features of the clinical text to obtain image language fusion features, and predicting a cervical lesion category to which the cervical cytology full-slice image belongs based on the image language fusion features. By adopting the method, the cervical lesion category prediction accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image language fusion prediction learning method for cervical cytology imbalanced data. Background Art

[0002] Thanks to the development of fully digital automatic scanners, computational cervical pathology classification algorithms based on whole slide images (WSI) of cervical cytology have made great progress, greatly reducing the burden of reading films for pathologists. However, these methods only use whole slide images, ignoring pathological information at different levels and scales provided by computed tomography (CT), magnetic resonance imaging (MRI), fluorescence imaging, clinical case information, diagnostic text, etc., which hinders the development of more accurate pathological diagnosis and more precise disease classification methods. In order to utilize information from different modalities, it is necessary to jointly model multiple modalities, reveal the potential associations and implicit knowledge between modalities, and extract more comprehensive, rich and discriminative features based on the complementarity between different modal data. One of the commonly used methods in traditional multimodal data joint modeling is feature fusion. This method extracts feature representations of each modality data and then combines or fuses these features to obtain a comprehensive representation. However, due to insufficient model capacity and simple fusion or splicing between modalities, it may not be possible to fully utilize the complementarity and mutual relationship between them, resulting in information loss or confusion, poor modeling effect, and low accuracy of cervical lesion category prediction. Summary of the invention

[0003] Based on this, it is necessary to provide an image language fusion prediction learning method for cervical cytology imbalanced data that can improve the prediction accuracy of cervical lesion categories in response to the above technical problems.

[0004] In a first aspect, the present application provides an image language fusion prediction learning method for cervical cytology imbalanced data, the method comprising:

[0005] Acquire a cervical cytology full-slice image, and cut the cervical cytology full-slice image into a plurality of patch images;

[0006] Detecting the plurality of patch images to screen out a plurality of suspicious positive cell images from the plurality of patch images;

[0007] Obtaining clinical text corresponding to the cervical cytology full slide image;

[0008] Extracting a plurality of second image features related to the clinical text from the first image features of the plurality of suspected positive cell images;

[0009] The multiple second image features are fused with the text features of the clinical text to obtain an image language fusion feature, and the cervical lesion category to which the cervical cytology full slide image belongs is predicted based on the image language fusion feature.

[0010] In a second aspect, the present application provides an image language fusion prediction learning device for cervical cytology imbalance data, the device comprising:

[0011] An acquisition module, used for acquiring a full-slice image of cervical cytology, and cutting the full-slice image of cervical cytology into a plurality of patch images;

[0012] A detection module, used for detecting the plurality of patch images to screen out a plurality of suspicious positive cell images from the plurality of patch images;

[0013] The acquisition module is also used to acquire the clinical text corresponding to the cervical cytology full-slice image;

[0014] An extraction module, configured to extract a plurality of second image features related to the clinical text from the first image features of the plurality of suspicious positive cell images;

[0015] A prediction module is used to fuse the multiple second image features with the text features of the clinical text to obtain an image language fusion feature, and predict the cervical lesion category to which the cervical cytology full slide image belongs based on the image language fusion feature.

[0016] In a third aspect, the present application provides a computer device including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps in the method embodiments of the present application are implemented.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the method embodiments of the present application.

[0018] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the steps in the method embodiments of the present application when the computer program is executed by a processor.

[0019] The above-mentioned image language fusion prediction learning method for cervical cytology imbalanced data obtains a full-slide image of cervical cytology and cuts the full-slide image of cervical cytology into multiple patch images; detects the multiple patch images to screen out multiple suspicious positive cell images from the multiple patch images; obtains the clinical text corresponding to the full-slide image of cervical cytology; extracts multiple second image features related to the clinical text from the first image features of the multiple suspicious positive cell images; fuses the multiple second image features with the text features of the clinical text to obtain image language fusion features, and predicts the cervical lesion category to which the full-slide image of cervical cytology belongs based on the image language fusion features. Compared with the traditional method of predicting cervical lesion categories based only on cervical cytology full-slide images, or simply fusing the image features of cervical cytology full-slide images with the text features of the corresponding text, the present application first cuts the cervical cytology full-slide image into multiple patch images, and screens out multiple suspicious positive cell images from the multiple patch images, and then extracts multiple second image features closely related to the clinical text from the first image features of the multiple suspicious positive cell images, and fuses the extracted multiple second image features with the text features of the clinical text, and then predicts the cervical lesion category based on the fused features. This can make full use of the complementarity and mutual relationship between images and texts, avoid information loss or confusion, and improve the accuracy of cervical lesion category prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flowchart of an image language fusion prediction learning method for cervical cytology imbalanced data in one embodiment;

[0021] Figure 2 A schematic diagram of a training process of a cervical lesion prediction model in one embodiment;

[0022] Figure 3 A schematic diagram of a network architecture of an image-text alignment network in one embodiment;

[0023] Figure 4 It is a structural block diagram of an image language fusion prediction learning device for cervical cytology imbalance data in one embodiment;

[0024] Figure 5 is an internal structure diagram of a computer device in one embodiment;

[0025] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0027] In one embodiment, Figure 1 As shown, an image language fusion prediction learning method for cervical cytology imbalance data is provided. The method can be applied to a computer device, which can be a terminal or a server. That is, the method can be executed by the terminal or the server alone, or can be implemented through the interaction between the terminal and the server. This embodiment takes the method applied to a computer device as an example for explanation, and includes the following steps:

[0028] Step 102: Acquire a whole-slice image of cervical cytology, and cut the whole-slice image of cervical cytology into a plurality of patch images.

[0029] It is understandable that in medical scenarios, the pixel size of the cervical cytology full slide image is very large. Before predicting the category of cervical lesions, the computer device can first cut the cervical cytology full slide image into several patch images. For example, the cervical cytology full slide image can be cut into several 1024×1024 patch images.

[0030] Step 104 , detecting the plurality of patch images to screen out a plurality of suspicious positive cell images from the plurality of patch images.

[0031] In one embodiment, the computer device may input multiple patch images into a trained cell detection model to screen multiple suspicious positive cell images from the multiple patch images through the cell detection model. It is understood that the number of suspicious positive cell images is less than the number of patch images.

[0032] Step 106, obtaining clinical text corresponding to the cervical cytology full slide image.

[0033] Among them, the clinical text corresponding to the cervical cytology full-slide image is the text used in medical clinical practice to describe the object to which the cervical cytology full-slide image belongs.

[0034] In one embodiment, the clinical text corresponding to the cervical cytology full slide image may specifically be at least one of the case, medical history or age of the subject to which the cervical cytology full slide image belongs.

[0035] Step 108 , extracting a plurality of second image features related to the clinical text from the first image features of the plurality of suspicious positive cell images.

[0036] In one embodiment, the computer device may input the first image features of the plurality of suspicious positive cell images into a trained image-text alignment network, so as to extract a plurality of second image features closely related to the clinical text from the first image features of the plurality of suspicious positive cell images through the image-text alignment network. It is understood that the number of the second image features is less than the number of the first image features.

[0037] Step 110, the plurality of second image features are fused with the text features of the clinical text to obtain an image language fusion feature, and the cervical lesion category to which the cervical cytology full slide image belongs is predicted based on the image language fusion feature.

[0038] In one embodiment, the computer device may concatenate multiple second image features and text features of the clinical text to obtain image language fusion features, and predict the cervical lesion category to which the cervical cytology full slide image belongs based on the image language fusion features.

[0039] In the above-mentioned image language fusion prediction learning method for cervical cytology imbalanced data, a full-slide image of cervical cytology is obtained and cut into multiple patch images; the multiple patch images are detected to screen out multiple suspicious positive cell images from the multiple patch images; the clinical text corresponding to the full-slide image of cervical cytology is obtained; from the first image features of the multiple suspicious positive cell images, multiple second image features related to the clinical text are extracted; the multiple second image features and the text features of the clinical text are fused to obtain image language fusion features, and the cervical lesion category to which the full-slide image of cervical cytology belongs is predicted based on the image language fusion features. Compared with the traditional method of predicting cervical lesion categories based only on cervical cytology full-slide images, or simply fusing the image features of cervical cytology full-slide images with the text features of the corresponding text, the present application first cuts the cervical cytology full-slide image into multiple patch images, and screens out multiple suspicious positive cell images from the multiple patch images, and then extracts multiple second image features closely related to the clinical text from the first image features of the multiple suspicious positive cell images, and fuses the extracted multiple second image features with the text features of the clinical text, and then predicts the cervical lesion category based on the fused features. This can make full use of the complementarity and mutual relationship between images and texts, avoid information loss or confusion, and improve the accuracy of cervical lesion category prediction.

[0040] In one embodiment, the cervical lesion category is predicted by a trained cervical lesion prediction model; the cervical lesion prediction model includes a trained visual network, a trained image-text alignment network, and a trained prediction network; the first image feature is extracted by the visual network in the cervical lesion prediction model; the second image feature is extracted by the image-text alignment network in the cervical lesion prediction model; the cervical lesion category is predicted by the prediction network in the cervical lesion prediction model.

[0041] In the above embodiment, the cervical lesion category to which the cervical cytology full-slide image belongs is predicted by a trained cervical lesion prediction model, and the cervical lesion prediction model is trained by a positive and negative sample balance method, which can further improve the accuracy of cervical lesion category prediction.

[0042] In one embodiment, the image-text alignment network includes an image processing unit and a text processing unit; the image processing unit and the text processing unit share feedforward layer parameters; the method also includes an image-text alignment network training step; the image-text alignment network training step includes: obtaining multiple first image-text sample pairs; each first image-text sample pair includes multiple first sample suspected positive cell images and corresponding first sample clinical texts; for each first image-text sample pair, the first sample image features and the first sample clinical text of the multiple first sample suspected positive cell images in the first image-text sample pair are input into the image-text alignment network to be trained, so as to extract multiple image features related to the first sample clinical text from the first sample image features of the multiple first sample suspected positive cell images through the image processing unit in the image-text alignment network to be trained, and obtain the second sample image features corresponding to the first image-text sample pair; extract the features of the first sample clinical text through the text processing unit in the image-text alignment network to be trained, and obtain the sample text features corresponding to the first image-text sample pair; determine the first loss value according to the second sample image features and sample text features corresponding to each of the multiple first image-text sample pairs; train the image-text alignment network to be trained according to the first loss value to obtain a trained image-text alignment network.

[0043] In the above embodiment, during the training of the image-text alignment network, the image features and text features are aligned by increasing the similarity of the same sample pairs and reducing the similarity of unpaired samples, and a joint representation space of image and text is constructed, so that the second image feature closely related to the clinical text can be accurately extracted from the first image feature of the suspicious positive cell image, thereby further improving the accuracy of cervical lesion category prediction.

[0044] In one embodiment, an image-text alignment network to be trained is trained according to a first loss value to obtain a trained image-text alignment network, including: obtaining category word-meta features; for each first image-text sample pair, interacting the category word-meta features with the first sample image features of the first sample suspected positive cell image in the first image-text sample pair and the sample text features of the corresponding first sample clinical text to obtain cervical lesion prediction task features, and inputting the cervical lesion prediction task features into a layer of fully connected classifiers to predict the probability of the first cervical lesion category corresponding to the first image-text sample pair; determining a second loss value according to the first cervical lesion category probabilities corresponding to each of the multiple first image-text sample pairs; and training the image-text alignment network to be trained according to the first loss value and the second loss value to obtain a trained image-text alignment network.

[0045] In one embodiment, the computer device may determine a target loss value according to the first loss value and the second loss value, and train the image-text alignment network to be trained based on the target loss value to obtain a trained image-text alignment network. The target loss value may be calculated by the following formula:

[0046] ;

[0047] ;

[0048] ;

[0049] Wherein, i represents the i-th first image-text sample pair among n first image-text sample pairs, represents the second sample image feature corresponding to the i-th first image-text sample pair, represents the sample text feature corresponding to the i-th first image-text sample pair, represents the other first image-text sample pairs except the i-th first image-text sample pair among the n first image-text sample pairs, t represents the preset temperature coefficient, represents the first loss value. represents the probability of the first cervical lesion category corresponding to the i-th first image-text sample pair among the n first image-text sample pairs, represents the second loss value, Represents the target loss value.

[0050] In the above embodiment, the target loss value is determined according to the first loss value and the second loss value, and the image-text alignment network to be trained is trained based on the target loss value, which can further improve the performance of the trained image-text alignment network, and can more accurately extract the second image feature closely related to the clinical text from the first image feature of the suspicious positive cell image, thereby further improving the accuracy of cervical lesion category prediction.

[0051] In one embodiment, the trained image-text alignment network is a preliminary trained image-text alignment network; the method further includes a cervical lesion prediction model training step; the cervical lesion prediction model training step includes: obtaining a plurality of second image-text sample pairs; each second image-text sample pair includes a plurality of second sample suspected positive cell images and corresponding second sample clinical texts; the second sample suspected positive cell image is obtained by covering the central area of ​​the first sample suspected positive cell image of the first image-text sample pair with a mask; the second sample clinical text is the first sample clinical text in the corresponding first image-text sample pair; for each second image-text sample pair, the second sample clinical text in the second image-text sample pair is A plurality of second sample suspicious positive cell images and a second sample clinical text are input into a cervical lesion prediction model to be trained, so as to extract third sample image features of the plurality of second sample suspicious positive cell images through a visual network to be trained in the cervical lesion prediction model to be trained, and a plurality of image features related to the second sample clinical text are extracted from the third sample image features of the plurality of second sample suspicious positive cell images through a preliminary trained image-text alignment network to be trained in the cervical lesion prediction model to be trained, so as to obtain a fourth sample image feature corresponding to the second image-text sample pair, and a fourth sample image feature corresponding to the second image-text sample pair is obtained through a prediction network in the cervical lesion prediction model to be trained based on the fourth sample image. The method comprises the following steps: using the first image-text sample pair and the sample text features of the second sample clinical text to predict the probability of the second cervical lesion category corresponding to the second image-text sample pair; determining a third loss value according to the probabilities of the cervical lesion categories corresponding to each of the plurality of second image-text sample pairs; for each first image-text sample pair, inputting the plurality of first sample suspected positive cell images and the first sample clinical text in the first image-text sample pair into the cervical lesion prediction model to be trained, so as to extract the fifth sample image features of the plurality of first sample suspected positive cell images through the visual network to be trained in the cervical lesion prediction model to be trained, and performing the image-text alignment network to be trained in the cervical lesion prediction model to be trained. , extract multiple image features related to the first sample clinical text from the fifth sample image features of multiple first sample suspicious positive cell images, obtain the sixth sample image feature corresponding to the first image-text sample pair, and predict the third cervical lesion category probability corresponding to the first image-text sample pair based on the sixth sample image feature and the sample text feature of the first sample clinical text through the prediction network in the cervical lesion prediction model to be trained; determine the fourth loss value according to the third cervical lesion category probabilities corresponding to each of the multiple first image-text sample pairs; train the second cervical lesion prediction model to be trained according to the third loss value and the fourth loss value to obtain a trained cervical lesion prediction model.

[0052] In one embodiment, the third loss value may be calculated by the following formula:

[0053] ;

[0054] Among them, here represents the probability of the second cervical lesion category corresponding to the i-th second image-text sample pair among the n second image-text sample pairs, Represents the third loss value.

[0055] In the above embodiment, in medical scenarios, data is often distributed in a long-tailed manner by category, that is, the categories are unbalanced. By covering the central area of ​​the suspected positive cell image of the first sample with a mask and adding the same number of negative samples to balance the training process, the problem of the model focusing too much on the majority class and causing poor performance of the minority class can be avoided, thereby further improving the accuracy of cervical lesion category prediction.

[0056] In one embodiment, multiple patch images are detected to screen out multiple suspicious positive cell images from the multiple patch images, including: inputting the multiple patch images into a trained cell detection model to determine the detection frames and confidence levels of the positive cells in the multiple patch images through the cell detection model; and selecting areas corresponding to the detection frames of a preset number of positive cells with higher confidence levels from the detection frames of the positive cells detected from the multiple patch images as multiple suspicious positive cell images.

[0057] In the above embodiment, a preset number of suspicious positive cell images with high confidence levels are detected from multiple patch images by a cell detection model as training data for a subsequent cervical lesion prediction model, which can further improve the efficiency and accuracy of the model training.

[0058] In one embodiment, Figure 2As shown, the cervical lesion prediction model includes a trained visual network (VisionTransformer), a trained image-text alignment network (Q-Former) and a prediction network (large language model). The computer device can obtain the whole slide image of cervical cytology and cut the whole slide image of cervical cytology into multiple patch images; input the multiple patch images into the trained cell detection model to determine the detection frame and confidence of the positive cells in the multiple patch images through the cell detection model; from the multiple detected cells, select a preset number of confidences as multiple suspicious positive cell images (i.e., Topk). The computer device can input multiple suspicious positive cell images into the visual network to obtain corresponding image features, input the corresponding image features of the multiple suspicious positive cell images and a set of pre-initialized image feature queries (Queries) into the image-text alignment network to extract a portion of image features related to clinical text from the corresponding image features of the multiple suspicious positive cell images, splice the extracted image features, and input the spliced ​​image features and the clinical text corresponding to the cervical cytology full slide image into the prediction network to predict the cervical lesion category of the cervical cytology full slide image (i.e., output the cervical lesion category in the form of text). It can be understood that in the training stage of the cervical lesion prediction model, the clinical text corresponding to the cervical cytology full slide image used for training needs to be input into the image-text alignment network for training, so that the image-text alignment network has the ability to extract a portion of image features related to clinical text. Therefore, in the application stage of the cervical lesion prediction model, it is not necessary to input the clinical text corresponding to the cervical cytology full slide image into the image-text alignment network.

[0059] In one embodiment, Figure 3 As shown in the figure, the image-text alignment network (Q-Former) contains two branches, an image processing unit (i.e., image Transformer) for processing learnable image feature queries, and a text processing unit (i.e., text Transformer) for processing clinical texts. Both are stacked with multiple blocks, where the image processing unit consists of self-attention layers, cross-attention layers, and feedforward layers, and each cross-attention layer interacts with the image features output by the Vision Transformer. The text processing unit lacks the cross-attention layer, and the image processing unit and the text processing unit share parameters.

[0060] It should be understood that, although each step in the flow chart of the above-mentioned embodiments is shown in order, these steps are not necessarily performed in order. Unless there is a clear explanation in this article, the execution of these steps does not have strict order restrictions, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above-mentioned embodiments may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in order, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0061] In one embodiment, Figure 4 As shown, an image language fusion prediction learning device 400 for cervical cytology imbalance data is provided, and the device specifically includes:

[0062] An acquisition module 402 is used to acquire a whole-slice image of cervical cytology and cut the whole-slice image of cervical cytology into a plurality of patch images;

[0063] A detection module 404 is used to detect the multiple patch images to screen out multiple suspicious positive cell images from the multiple patch images;

[0064] The acquisition module 402 is also used to acquire clinical text corresponding to the cervical cytology full slide image;

[0065] An extraction module 406, for extracting a plurality of second image features related to the clinical text from the first image features of the plurality of suspicious positive cell images;

[0066] The prediction module 408 is used to fuse the multiple second image features with the text features of the clinical text to obtain an image language fusion feature, and predict the cervical lesion category to which the cervical cytology full slide image belongs based on the image language fusion feature.

[0067] In one embodiment, the cervical lesion category is predicted by a trained cervical lesion prediction model; the cervical lesion prediction model includes a trained visual network, a trained image-text alignment network, and a trained prediction network; the first image feature is extracted by the visual network in the cervical lesion prediction model; the second image feature is extracted by the image-text alignment network in the cervical lesion prediction model; the cervical lesion category is predicted by the prediction network in the cervical lesion prediction model.

[0068] In one embodiment, the image-text alignment network comprises an image processing unit and a text processing unit; the image processing unit and the text processing unit share feed-forward layer parameters; the device further comprises:

[0069] A training module is used to obtain multiple first image-text sample pairs; each first image-text sample pair includes multiple first sample suspected positive cell images and corresponding first sample clinical texts; for each first image-text sample pair, the first sample image features of the multiple first sample suspected positive cell images and the first sample clinical texts in the first image-text sample pair are input into the image-text alignment network to be trained, so that the image processing unit in the image-text alignment network to be trained extracts multiple image features related to the first sample clinical text from the first sample image features of the multiple first sample suspected positive cell images to obtain the second sample image features corresponding to the first image-text sample pair, and the text processing unit in the image-text alignment network to be trained extracts the features of the first sample clinical text to obtain the sample text features corresponding to the first image-text sample pair; a first loss value is determined according to the second sample image features and sample text features corresponding to each of the multiple first image-text sample pairs; the image-text alignment network to be trained is trained according to the first loss value to obtain a trained image-text alignment network.

[0070] In one embodiment, the training module is also used to obtain category word-meta features; for each first image-text sample pair, the category word-meta features are interacted with the first sample image features of the first sample suspected positive cell image in the first image-text sample pair, and the sample text features of the corresponding first sample clinical text to obtain cervical lesion prediction task features, and the cervical lesion prediction task features are input into a layer of fully connected classifier to predict the first cervical lesion category probability corresponding to the first image-text sample pair; a second loss value is determined based on the first cervical lesion category probabilities corresponding to each of the multiple first image-text sample pairs; the image-text alignment network to be trained is trained based on the first loss value and the second loss value to obtain a trained image-text alignment network.

[0071] In one embodiment, the trained image-text alignment network is a preliminary trained image-text alignment network; the training module is also used to obtain multiple second image-text sample pairs; each second image-text sample pair contains multiple second sample suspected positive cell images and corresponding second sample clinical texts; the second sample suspected positive cell image is obtained by covering the central area of ​​the first sample suspected positive cell image of the first image-text sample pair with a mask; the second sample clinical text is the first sample clinical text in the corresponding first image-text sample pair; for each second image-text sample pair, the multiple second sample suspected positive cell images and the second sample clinical text in the second image-text sample pair are input into the cervical lesion prediction model to be trained, so as to extract the third sample image features of the multiple second sample suspected positive cell images through the visual network to be trained in the cervical lesion prediction model to be trained, and extract the multiple image features related to the second sample clinical text from the third sample image features of the multiple second sample suspected positive cell images through the preliminary trained image-text alignment network to be trained in the cervical lesion prediction model to be trained, so as to obtain the fourth sample image features corresponding to the second image-text sample pair, and obtain the fourth sample image features corresponding to the second image-text sample pair through the prediction network in the cervical lesion prediction model to be trained based on the fourth sample image features and the sample of the second sample clinical text. The present invention adopts text features to predict the probability of the second cervical lesion category corresponding to the second image-text sample pair; determines the third loss value according to the second cervical lesion category probabilities corresponding to each of the multiple second image-text sample pairs; for each first image-text sample pair, inputs the multiple first sample suspected positive cell images and the first sample clinical text in the first image-text sample pair into the cervical lesion prediction model to be trained, so as to extract the fifth sample image features of the multiple first sample suspected positive cell images through the visual network to be trained in the cervical lesion prediction model to be trained, and extracts the fifth sample image features of the multiple first sample suspected positive cell images through the preliminary trained image-text alignment network to be trained in the cervical lesion prediction model to be trained. From the fifth sample image feature of a sample suspicious positive cell image, multiple image features related to the first sample clinical text are extracted to obtain the sixth sample image feature corresponding to the first image-text sample pair; the probability of the third cervical lesion category corresponding to the first image-text sample pair is predicted based on the sixth sample image feature and the sample text feature of the first sample clinical text by the prediction network in the cervical lesion prediction model to be trained; the fourth loss value is determined according to the third cervical lesion category probabilities corresponding to each of the multiple first image-text sample pairs; the cervical lesion prediction model to be trained is trained according to the third loss value and the fourth loss value to obtain a trained cervical lesion prediction model.

[0072] In one embodiment, the detection module 404 is also used to input multiple patch images into a trained cell detection model to determine the detection frames of positive cells in the multiple patch images and their confidence levels through the cell detection model; from the detection frames of positive cells detected from the multiple patch images, areas corresponding to the detection frames of a preset number of positive cells with higher confidence levels are selected as multiple suspected positive cell images.

[0073] The above-mentioned image language fusion prediction learning device for cervical cytology imbalanced data obtains a cervical cytology full-slide image and cuts the cervical cytology full-slide image into multiple patch images; detects the multiple patch images to screen out multiple suspicious positive cell images from the multiple patch images; obtains the clinical text corresponding to the cervical cytology full-slide image; extracts multiple second image features related to the clinical text from the first image features of the multiple suspicious positive cell images; fuses the multiple second image features with the text features of the clinical text to obtain image language fusion features, and predicts the cervical lesion category to which the cervical cytology full-slide image belongs based on the image language fusion features. Compared with the traditional method of predicting cervical lesion categories based only on cervical cytology full-slide images, or simply fusing the image features of cervical cytology full-slide images with the text features of the corresponding text, the present application first cuts the cervical cytology full-slide image into multiple patch images, and screens out multiple suspicious positive cell images from the multiple patch images, and then extracts multiple second image features closely related to the clinical text from the first image features of the multiple suspicious positive cell images, and fuses the extracted multiple second image features with the text features of the clinical text, and then predicts the cervical lesion category based on the fused features. This can make full use of the complementarity and mutual relationship between images and texts, avoid information loss or confusion, and improve the accuracy of cervical lesion category prediction.

[0074] Each module in the above-mentioned image language fusion prediction learning device for cervical cytology imbalance data can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0075] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image language fusion prediction learning method for cervical cytology imbalance data is implemented.

[0076] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image language fusion prediction learning method for cervical cytology imbalance data is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0077] Those skilled in the art will understand that Figure 5 and Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0078] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0079] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0080] In one embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0081] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0082] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0083] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0084] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An image-language fusion prediction learning method for cervical cytology imbalanced data, characterized in that: The method comprises: Acquire a cervical cytology full-slice image, and cut the cervical cytology full-slice image into a plurality of patch images; Detecting the plurality of patch images to screen out a plurality of suspicious positive cell images from the plurality of patch images; Obtaining clinical text corresponding to the cervical cytology full slide image; Extracting a plurality of second image features related to the clinical text from the first image features of the plurality of suspected positive cell images; The multiple second image features and the text features of the clinical text are fused to obtain an image language fusion feature, and the cervical lesion category to which the cervical cytology full slide image belongs is predicted based on the image language fusion feature.

2. The method according to claim 1, characterized in that The cervical lesion category is predicted by a trained cervical lesion prediction model; the cervical lesion prediction model includes a trained visual network, a trained image-text alignment network and a trained prediction network; the first image feature is extracted by the visual network in the cervical lesion prediction model; the second image feature is extracted by the image-text alignment network in the cervical lesion prediction model; the cervical lesion category is predicted by the prediction network in the cervical lesion prediction model.

3. The method according to claim 2, characterized in that The image-text alignment network comprises an image processing unit and a text processing unit; the image processing unit and the text processing unit share feedforward layer parameters; the method further comprises an image-text alignment network training step; The image-text alignment network training step includes: Acquire a plurality of first image-text sample pairs; each of the first image-text sample pairs comprises a plurality of first sample suspected positive cell images and corresponding first sample clinical texts; For each of the first image-text sample pairs, the first sample image features of the multiple first sample suspected positive cell images and the first sample clinical text in the first image-text sample pair are input into the image-text alignment network to be trained, so as to extract multiple image features related to the first sample clinical text from the first sample image features of the multiple first sample suspected positive cell images through the image processing unit in the image-text alignment network to be trained, and obtain the second sample image features corresponding to the first image-text sample pair, and extract the features of the first sample clinical text through the text processing unit in the image-text alignment network to be trained, and obtain the sample text features corresponding to the first image-text sample pair; Determine a first loss value according to the second sample image features and the sample text features corresponding to each of the plurality of first image-text sample pairs; The image-text alignment network to be trained is trained according to the first loss value to obtain a trained image-text alignment network.

4. The method according to claim 3, characterized in that The step of training the image-text alignment network to be trained according to the first loss value to obtain a trained image-text alignment network includes: Get the category word features; For each of the first image-text sample pairs, the category word-element feature is interacted with the first sample image feature of the first sample suspected positive cell image in the first image-text sample pair and the sample text feature of the corresponding first sample clinical text to obtain a cervical lesion prediction task feature, and the cervical lesion prediction task feature is input into a one-layer fully connected classifier to predict the probability of the first cervical lesion category corresponding to the first image-text sample pair; Determine a second loss value according to the first cervical lesion category probabilities corresponding to each of the plurality of first image and text samples; The image-text alignment network to be trained is trained according to the first loss value and the second loss value to obtain a trained image-text alignment network.

5. The method according to claim 3, characterized in that: The trained image-text alignment network is a preliminarily trained image-text alignment network; the method further comprises a cervical lesion prediction model training step; the cervical lesion prediction model training step comprises: Acquire multiple second image-text sample pairs; each of the second image-text sample pairs includes multiple second sample suspected positive cell images and corresponding second sample clinical texts; the second sample suspected positive cell images are obtained by covering the central area of ​​the first sample suspected positive cell image of the first image-text sample pair with a mask; the second sample clinical text is the first sample clinical text in the corresponding first image-text sample pair; For each of the second image-text sample pairs, the multiple second sample suspicious positive cell images and the second sample clinical text in the second image-text sample pair are input into the cervical lesion prediction model to be trained, so as to extract the third sample image features of the multiple second sample suspicious positive cell images through the visual network to be trained in the cervical lesion prediction model to be trained, and extract multiple image features related to the second sample clinical text from the third sample image features of the multiple second sample suspicious positive cell images through the preliminary trained image-text alignment network to be trained in the cervical lesion prediction model to be trained, so as to obtain the fourth sample image features corresponding to the second image-text sample pair, and predict the second cervical lesion category probability corresponding to the second image-text sample pair based on the fourth sample image features and the sample text features of the second sample clinical text through the prediction network in the cervical lesion prediction model to be trained; Determine a third loss value according to the second cervical lesion category probabilities corresponding to each of the plurality of second image and text samples; For each of the first image-text sample pairs, the multiple first sample suspicious positive cell images and the first sample clinical text in the first image-text sample pair are input into the cervical lesion prediction model to be trained, so as to extract the fifth sample image feature of the multiple first sample suspicious positive cell images through the visual network to be trained in the cervical lesion prediction model to be trained, and extract multiple image features related to the first sample clinical text from the fifth sample image feature of the multiple first sample suspicious positive cell images through the preliminary trained image-text alignment network to be trained in the cervical lesion prediction model to be trained, so as to obtain the sixth sample image feature corresponding to the first image-text sample pair, and predict the probability of a third cervical lesion category corresponding to the first image-text sample pair based on the sixth sample image feature and the sample text feature of the first sample clinical text through the prediction network in the cervical lesion prediction model to be trained; Determine a fourth loss value according to the probabilities of the third cervical lesion category corresponding to each of the plurality of first image and text samples; The cervical lesion prediction model to be trained is trained according to the third loss value and the fourth loss value to obtain a trained cervical lesion prediction model.

6. The method according to any one of claims 1 to 5, characterized in that The detecting the plurality of patch images to screen out a plurality of suspicious positive cell images from the plurality of patch images includes: Inputting the plurality of patch images into a trained cell detection model to determine the detection frames and confidences of positive cells in the plurality of patch images through the cell detection model; From the detection frames of positive cells detected in the plurality of patch images, regions corresponding to a preset number of detection frames of positive cells with higher confidence levels are selected as a plurality of suspected positive cell images.

7. An image-language fusion prediction learning device for cervical cytology imbalanced data, characterized in that: The device comprises: An acquisition module, used for acquiring a whole-slice image of cervical cytology, and cutting the whole-slice image of cervical cytology into a plurality of patch images; A detection module, used for detecting the plurality of patch images to screen out a plurality of suspicious positive cell images from the plurality of patch images; The acquisition module is also used to acquire clinical text corresponding to the cervical cytology full-slice image; An extraction module, configured to extract a plurality of second image features related to the clinical text from the first image features of the plurality of suspicious positive cell images; A prediction module is used to fuse the multiple second image features with the text features of the clinical text to obtain an image language fusion feature, and predict the cervical lesion category to which the cervical cytology full slide image belongs based on the image language fusion feature.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images