Image segmentation method, device, computer equipment and storage medium
By extracting the timing characteristics of medical image sequences and combining image reports for model training, the accuracy of traditional medical image segmentation models under the lack of labeled data is solved, and the accuracy and robustness of segmentation results are improved, especially in multiple times Image changes in the case of scanning and follow-up examination.
Patent Information
- Application Number
- CN202311649793.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-04
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-12-04
AI Technical Summary
The traditional medical image segmentation model lacks accurate labeling data during training, resulting in low accuracy of segmentation results. Especially in the case of multiple scans and follow-up examinations, the report description of medical staff cannot accurately describe image changes, which affects the impact of Model understanding and segmentation effect.
By obtaining the timing image features of the medical image sequence, a sequence containing multiple medical images is constructed, the timing features are extracted and segmented, the model is trained in combination with the image report, and the pre-trained text encoder and feature extraction module are used for joint training to reduce Positive sample feature loss and increase negative sample feature loss to improve the accuracy of segmentation model.
The accuracy of medical image segmentation results is improved, the timing changes of the objects to be detected are taken into account, the model's ability to understand image changes is enhanced, and the accuracy and robustness of segmentation results are improved.
Smart Images

Figure CN117351197B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image segmentation method, apparatus, computer equipment, storage medium and computer program product. Background Art
[0002] In the field of medical image processing, the segmentation of medical images is generally achieved by training an image segmentation model through training samples, and then using the image segmentation model to perform intelligent recognition of medical images. In the traditional training method of image segmentation models, labeled supervised training is generally used. However, it is difficult to obtain annotations for medical images, resulting in fewer annotated medical images.
[0003] Although there are some methods that use imaging reports as labels for medical images to increase training samples, this method requires that the corresponding imaging reports must be able to accurately describe the corresponding medical images. In clinical practice, many test subjects have multiple scans and follow-up examinations. In this case, medical staff often only describe the changes in the test target compared to the last scan in the corresponding report, such as the disappearance of inflammation, the increase / decrease of lesions, etc. Therefore, if this kind of report is paired with medical images for model training, the trained image segmentation model will not be able to accurately understand the image and report, thus affecting the accuracy of the segmentation results. Summary of the invention
[0004] Based on this, it is necessary to provide an image segmentation method, apparatus, computer device, computer-readable storage medium and computer program product to address the technical problem that the segmentation results of the above-mentioned image segmentation method have low accuracy.
[0005] In a first aspect, the present application provides an image segmentation method. The method comprises:
[0006] Acquire medical image sequences;
[0007] Extracting temporal image features of the medical image sequence;
[0008] According to the time series image features, any medical image in the medical image sequence is segmented to obtain a region of interest in any medical image.
[0009] In one embodiment, extracting the temporal image features of the medical image sequence includes:
[0010] Extracting image features of each medical image in the medical image sequence;
[0011] According to the image features of the respective medical images, the temporal image features of the medical image sequence are determined.
[0012] In one embodiment, the image segmentation method is performed by an image segmentation model, and the image segmentation model is trained in the following manner:
[0013] Acquire a sample medical image sequence of a sample object and an image report corresponding to a target sample medical image in the sample medical image sequence; the sample medical image sequence includes a plurality of sample medical images, and the target sample medical image is a medical image acquired latest in the sample medical image sequence;
[0014] The image segmentation model is trained using the sample medical image sequence and the imaging report to obtain a trained image segmentation model.
[0015] In one embodiment, the image segmentation model is trained by the sample medical image sequence and the imaging report to obtain the trained image segmentation model, including:
[0016] Training the feature extraction module in the image segmentation model through the sample medical image sequence and the imaging report to obtain a trained feature extraction module;
[0017] Keeping the model parameters of the trained feature extraction module unchanged, the segmentation module in the image segmentation model is trained by using a labeled data set to obtain a trained segmentation module; the labeled data set includes a sample medical image and a real region of interest of the sample medical image;
[0018] Based on the trained feature extraction module and the trained segmentation module, a trained image segmentation model is obtained.
[0019] In one embodiment, the feature extraction module in the image segmentation model is trained by the sample medical image sequence and the imaging report to obtain the trained feature extraction module, including:
[0020] Training a text encoder using the image report to obtain a pre-trained text encoder;
[0021] The feature extraction module is trained by using the sample medical image sequence, the imaging report and the pre-trained text encoder to obtain a trained feature extraction module.
[0022] In one embodiment, the step of training a text encoder using the image report to obtain a pre-trained text encoder includes:
[0023] Performing random mask processing on the image report to obtain a mask report;
[0024] Taking the mask report as an input variable and the image report as supervisory information, the text encoder is trained in combination with a preset text decoder to obtain a pre-trained text encoder;
[0025] The input of the text decoder is the output of the text encoder.
[0026] In one embodiment, the feature extraction module is trained by using the sample medical image sequence, the imaging report and the pre-trained text encoder to obtain a trained feature extraction module, including:
[0027] Extracting sample time-series image features of the sample medical image sequence through the feature extraction module;
[0028] Extracting text features of the image report by using the pre-trained text encoder;
[0029] Based on the sample time-series image features and the text features, the feature extraction module and the pre-trained text encoder are jointly trained to obtain a trained feature extraction module.
[0030] In one embodiment, there are multiple sample medical image sequences, and correspondingly, there are also multiple sample time-series image features and text features;
[0031] Based on the sample time-series image features and the text features, the feature extraction module and the pre-trained text encoder are jointly trained to obtain a trained feature extraction module, including:
[0032] Based on multiple sample time-series image features and multiple text features, multiple feature pairs are obtained; the feature pairs include positive sample feature pairs and negative sample feature pairs;
[0033] Obtain the loss between the sample temporal image features and text features in each feature pair;
[0034] Under the condition of reducing the loss of positive sample feature pairs and increasing the loss of negative sample feature pairs, the feature extraction module and the pre-trained text encoder are jointly trained to obtain a trained feature extraction module.
[0035] In a second aspect, the present application also provides an image segmentation device. The device comprises:
[0036] An acquisition unit, used for acquiring a medical image sequence;
[0037] An extraction unit, used for extracting time-series image features of the medical image sequence;
[0038] The segmentation unit is used to segment any medical image in the medical image sequence according to the time series image features to obtain a region of interest in any medical image.
[0039] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any of the above methods are implemented.
[0040] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of any of the above methods when executed by a processor.
[0041] In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program, which implements the steps of any of the above methods when executed by a processor. The above-mentioned image segmentation method, apparatus, computer device, storage medium and computer program product, after acquiring a medical image sequence, first extract the time-series image features of the medical image sequence, and segment any medical image in the medical image sequence based on the time-series image features to obtain a region of interest of any medical image. When segmenting any medical image of an object to be detected, the method first constructs a medical image sequence containing the medical image, extracts the time-series image features of the medical image sequence, and segments the medical image based on the time-series image features, that is, the basis for image segmentation in this scheme takes into account the time-series changes of the object to be detected, and can improve the accuracy of the segmentation result compared to image segmentation by only extracting the image features of the medical image to be segmented. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 A diagram showing an application environment of an image segmentation method in an embodiment;
[0043] Figure 2 is a structural schematic diagram of an image segmentation model in one embodiment;
[0044] Figure 3 A schematic diagram of a process flow of a training process of an image segmentation model in one embodiment;
[0045] Figure 4 is a schematic diagram of a training principle of an image segmentation model in one embodiment;
[0046] Figure 5 is a flowchart of the training steps of an image segmentation model in another embodiment;
[0047] Figure 6 is a structural block diagram of an image segmentation device in one embodiment;
[0048] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0050] In one embodiment, Figure 1 As shown, an image segmentation method is provided. This embodiment uses the method applied to a server as an example for illustration. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be but is not limited to various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server can be implemented as an independent server or a server cluster composed of multiple servers. In this embodiment, the method includes the following steps:
[0051] Step S110, acquiring a medical image sequence.
[0052] Each medical image in the medical image sequence has a corresponding acquisition time, and each medical image in the medical image sequence may be the same or may be different.
[0053] The medical image is an image containing an object to be detected, and the medical image may be a two-dimensional image, a three-dimensional image, or the like.
[0054] The medical images in the medical image sequence may be medical images of the same object to be detected, or medical images of the same type of object but not belonging to the same object to be detected. For example, if the object to be detected is a breast, the medical image sequence may be medical images of the breast of the same user, or medical images of the breast of different users.
[0055] In a specific implementation, when performing abnormal segmentation on the object to be detected, the medical image of the object to be detected collected this time can be obtained as the target medical image. If there are medical images collected historically for the object to be detected, the medical images collected historically and the medical images collected this time constitute a medical image sequence. At this time, the medical images in the medical image sequence are different. If there are no medical images collected historically for the object to be detected, for example, it is the first time to collect the medical image of the object to be detected, or the medical image collected historically for the object to be detected is damaged or lost, only the medical image collected this time is repeated multiple times to constitute a medical image sequence. At this time, the medical images in the medical image sequence are the same.
[0056] It should be noted that the medical image of the object to be detected collected this time can be obtained by scanning the object to be detected with a medical scanning device, or can be obtained from a pre-stored medical image in a storage device such as a database or a server. The medical image collected historically can be obtained from a storage device such as a database or a server, or can be obtained by other methods, which is not specifically limited in this application.
[0057] After obtaining the medical image sequence, the boundary area of each medical image can be further cropped to remove redundant areas such as black edges or scanning platforms. The segmentation algorithm can adopt a threshold method, a random walk method, etc., which is not specifically limited in this application. Then, the cropped medical image is segmented.
[0058] Step S120, extracting temporal image features of the medical image sequence.
[0059] Among them, time-series image features refer to features extracted from a series of continuous medical images, which are used to describe the characteristics of image changes over time. The extracted features can include changes in the region of interest in the medical image. By extracting and analyzing time-series image features, it can help understand the dynamic change information in the medical image sequence.
[0060] In a specific implementation, the medical image sequence represents a time series obtained by sorting the medical images in order of acquisition time from earliest to latest. When extracting the time series image features of the medical image sequence, the image segmentation model can be used for extraction, and specifically, the time series image features of the medical image sequence can be extracted through the feature extraction module in the image segmentation model.
[0061] It can be understood that when the medical image sequence includes only one medical image, the extracted time-series image features are time-series image features of multiple identical medical images.
[0062] Step S130 , segmenting any medical image in the medical image sequence according to the time series image features to obtain a region of interest in any medical image.
[0063] The region of interest may be a lesion region.
[0064] In a specific implementation, the time series image features are obtained based on each medical image in the medical image sequence, and when segmenting, any medical image in the medical image sequence can be segmented. More specifically, any medical image in the medical image sequence can be segmented through an image segmentation model, specifically, through a segmentation module in the image segmentation model, any medical image in the medical image sequence is segmented according to the time series image features to obtain a region of interest in any medical image.
[0065] In practical applications, the segmentation module can use an image decoder to input the temporal image features into the image decoder, and the image decoder segments any medical image in the medical image sequence according to the temporal image features to obtain the region of interest in any medical image.
[0066] In the above-mentioned image segmentation method, after obtaining a medical image sequence, the temporal image features of the medical image sequence are first extracted, and any medical image in the medical image sequence is segmented based on the temporal image features to obtain a region of interest of any medical image. When segmenting any medical image of an object to be detected, this method first constructs a medical image sequence containing the medical image, extracts the temporal image features of the medical image sequence, and segments the medical image based on the temporal image features. That is, the basis for image segmentation in this scheme takes into account the temporal changes of the object to be detected, and can improve the accuracy of the segmentation result compared to performing image segmentation by only extracting the image features of the medical image to be segmented.
[0067] In an exemplary embodiment, extracting the time-series image features of the medical image sequence in step S120 includes: extracting image features of each medical image in the medical image sequence; and determining the time-series image features of the medical image sequence according to the image features of each medical image.
[0068] In a specific implementation, a feature extraction module for extracting temporal image features of a medical image sequence may include an image encoder and a temporal feature encoding module. When extracting temporal image features of a medical image sequence, each medical image may be input into an image encoder, and the image encoder performs encoding processing to obtain image features of each medical image. Afterwards, the image features of each medical image are input into a temporal feature encoding module, and the temporal feature encoding module performs encoding processing to obtain temporal image features of the medical image sequence. Among them, the image encoder may adopt a structure such as a convolutional neural network. And the number of image encoders may be one or more, and when the number of image encoders is more than one, the parameters of each image encoder are the same. When extracting image features of each medical image through an image encoder, if there are more than one image encoder, each medical image in the medical image sequence may be input into each image encoder respectively. Each image encoder performs encoding processing on each medical image respectively, that is, one image encoder processes one medical image, to obtain image features of each medical image. If there is only one image encoder, each medical image is sequentially input into the same image encoder, that is, one image encoder processes multiple medical images to extract image features of each medical image.
[0069] More specifically, when determining the temporal image features of the medical image sequence based on the image features of each medical image, the image features of each medical image may be first subjected to temporal coding processing to obtain the temporal coding features of each medical image. The temporal coding method used may be a coding method such as sinusoidal function position coding. The temporal coding features of each medical image are further spliced, and the temporal image features of the medical image sequence are obtained based on the spliced features. The temporal coding features of each medical image are still spliced according to the acquisition time of each medical image, specifically, the temporal coding features of each medical image are spliced in the order of acquisition time from early to late.
[0070] In addition, before performing feature splicing, the temporal coding features of each medical image may be expanded into a one-dimensional feature and then spliced to make the dimensions of the temporal coding features of each medical image consistent.
[0071] In this embodiment, when determining the time-series image features of a medical image sequence, the image features of each medical image are first extracted, and then the time-series image features of the medical image sequence are obtained through each image feature. This can improve the expressiveness of the time-series image features of the medical image sequence, thereby more accurately describing the features of the medical image sequence, thereby improving the accuracy of subsequent segmentation based on the time-series image features.
[0072] In one embodiment, in order to facilitate those skilled in the art to understand the above image segmentation method, the breast will be taken as the object to be detected and described in conjunction with specific examples in the accompanying drawings.
[0073] refer to Figure 2 , shows a schematic diagram of the structure of an image segmentation model, including multiple image encoders, a time series encoder, a feature concatenation module, a feature encoder and an image decoder. The structure of the feature encoder can adopt a transformer structure or an LSTM (Long short-term memory) structure to improve the feature extraction capability of medical images at different time points. Figure 2 The specific process of image segmentation by the image segmentation model shown is as follows:
[0074] (1) The breast image sequence (I t , I t-1 ,…,I t-i ) are input into each image encoder respectively to obtain the image feature F of each breast image t , F t-1 , …, F t-i .
[0075] (2) Inputting the image features of each breast image into a temporal encoder to obtain the temporal coding features of each breast image.
[0076] (3) After the temporal coding features of each breast image are dimensionally unified, they are spliced in the order of acquisition time to obtain the spliced features.
[0077] (4) Input the concatenated features into the feature encoder to obtain the temporal image features of the breast image sequence.
[0078] (5) The temporal image features of the breast image sequence are input into the image decoder to obtain the region of interest of the target breast image of the breast to be detected.
[0079] In this embodiment, when segmenting the target breast image collected for the breast to be detected, a breast image sequence including the target breast image is first constructed, the temporal image features of the breast image sequence are extracted through an image segmentation model, and the target breast image is segmented based on the temporal image features. That is, the basis for image segmentation in this scheme takes into account the temporal changes of the breast to be detected, and compared with image segmentation performed by only extracting the image features of the target breast image, the accuracy of the segmentation result can be improved.
[0080] In an exemplary embodiment, the above image segmentation method can be performed by an image segmentation model, such as Figure 3As shown, the training of the image segmentation model includes the following steps:
[0081] Step S310, obtaining a sample medical image sequence of a sample object and an imaging report corresponding to a target sample medical image in the sample medical image sequence; the sample medical image sequence includes multiple sample medical images, and the target sample medical image is the medical image with the latest acquisition time in the sample medical image sequence.
[0082] The sample object is the same as the object to be detected and belongs to the same type of object, for example, both are breasts.
[0083] In a specific implementation, the medical images collected by different users in the past when inspecting the object to be detected and the image report corresponding to the last collected medical image can be obtained. The medical images collected by each user in the past constitute a historical medical image sequence, and multiple sample medical image sequences can be obtained from the historical medical image sequences of different users. The last collected medical image of each user is used as the target sample medical image, and the image report corresponding to the last collected medical image is used as the image report corresponding to the target sample medical image in the sample medical image sequence of each user.
[0084] Step S320: training the image segmentation model using sample medical image sequences and imaging reports to obtain a trained image segmentation model.
[0085] The image segmentation model includes a feature extraction module and a segmentation module. The feature extraction module includes an image encoder and a temporal feature encoding module. The image encoder, the temporal feature encoding module and the segmentation module are connected in sequence, that is, the output of the image encoder is the input of the temporal feature encoding module, and the output of the temporal feature encoding module is the input of the segmentation module.
[0086] The feature extraction module is used to extract the time-series image features of the medical image sequence. Specifically, the image features of the medical image can be extracted by an image encoder, and the image features are input into the time-series feature encoding module to obtain the time-series image features of the medical image sequence.
[0087] The segmentation module is used to segment the target medical image in the medical image sequence based on the temporal image features to obtain a segmentation mask.
[0088] Specifically, the text encoder can be trained through the imaging report to obtain a pre-trained text encoder; the image segmentation model can be trained through the sample medical image sequence, the imaging report and the pre-trained text encoder to obtain a trained image segmentation model.
[0089] In this embodiment, the image segmentation model is trained by using a medical image sequence containing the target medical image to be segmented, so that the trained image segmentation model can extract the temporal changes of the detected object, and perform image segmentation based on the temporal changes, thereby improving the accuracy of the segmentation results of the trained image segmentation model.
[0090] In an exemplary embodiment, in the above step S320, the image segmentation model is trained by using sample medical image sequences and imaging reports to obtain a trained image segmentation model, which specifically includes:
[0091] Step S321, training the feature extraction module in the image segmentation model through the sample medical image sequence and the imaging report to obtain a trained feature extraction module;
[0092] Step S322, keeping the model parameters of the trained feature extraction module unchanged, training the segmentation module in the image segmentation model with a labeled data set to obtain a trained segmentation module; the labeled data set includes a sample medical image and a real region of interest of the sample medical image;
[0093] Step S323, obtaining a trained image segmentation model based on the trained feature extraction module and the trained segmentation module.
[0094] Specifically, the feature extraction module in the image segmentation model is used to extract the temporal image features of the medical image sequence, and the segmentation module is used to perform image segmentation. The segmentation module is only available in downstream tasks. Therefore, when training the image segmentation model, the training process can be divided into two stages. The first stage is to train the feature extraction module, and the second stage is to train the segmentation module. Finally, the trained feature extraction module and the trained segmentation module are combined to form a trained image segmentation model.
[0095] More specifically, in the first stage, the feature extraction module in the image segmentation model can be trained through sample medical image sequences and imaging reports to obtain a trained feature extraction module. Then, the segmentation module in the image segmentation model is trained through a labeled data set to obtain a trained segmentation module. Moreover, when performing the second stage of training, the model parameters of the feature extraction module trained in the first stage can be kept unchanged to train the segmentation module.
[0096] In one embodiment, the labeled data set can be divided into two parts, one part is used for the second stage of training, and the other part is used for fine-tuning the image segmentation model. Specifically, after completing the second stage of training with a part of the labeled data, some model parameters of the feature extraction module can be unfrozen, and the feature extraction module and the segmentation module can be jointly trained with the other part of the labeled data set to obtain a trained feature extraction module and a trained segmentation module to form a trained image segmentation model.
[0097] In this embodiment, by training the feature extraction module and the segmentation module in the image segmentation model in two stages, the training process can be simplified so that each stage can focus on its own task, reducing the complexity and computational complexity of the model and improving the generalization ability of the model.
[0098] In an exemplary embodiment, in the above step S321, the feature extraction module in the image segmentation model is trained by using the sample medical image sequence and the imaging report to obtain the trained feature extraction module, including:
[0099] Step S321a, training the text encoder through the image report to obtain a pre-trained text encoder.
[0100] In a specific implementation, one way to train the text encoder through the image report is to pre-label the text features of the image report as text feature labels. During training, the image report is used as an input variable and input into the text encoder to obtain text features. The text features and text feature labels are subjected to loss, and the text encoder is trained based on the loss.
[0101] Another way is to introduce an image decoder, take the image report as the input variable, and regenerate the image report by the text encoder and text decoder to obtain a regenerated report. Based on the loss between the regenerated report and the image report, the text encoder and the preset text decoder are jointly trained to obtain a pre-trained text encoder.
[0102] Step S321b, training the feature extraction module through sample medical image sequences, imaging reports and pre-trained text encoders to obtain a trained feature extraction module.
[0103] In this step, the pre-trained text encoder is used to assist the training of the image segmentation model.
[0104] In a specific implementation, the sample medical image sequence can be input into the feature extraction module to obtain the sample time-series image features of the sample medical image sequence. The image report can be input into the pre-trained text encoder to obtain the text features of the image report. Based on the sample time-series image features and text features, the feature extraction module and the pre-trained text encoder are jointly trained to obtain a trained feature extraction module.
[0105] In this embodiment, the text encoder is first pre-trained through the imaging report, and the pre-trained text encoder is used to assist in training the image segmentation model. This can further improve the ability of the image segmentation model to extract the temporal image features of the medical image sequence, thereby improving the accuracy of the segmentation results determined based on the temporal image features.
[0106] In an exemplary embodiment, in the above step S321a, the text encoder is trained through the image report to obtain a pre-trained text encoder, including: performing random masking on the image report to obtain a masked report; using the masked report as an input variable and the image report as supervision information, jointly training the text encoder and a preset text decoder to obtain a pre-trained text encoder; wherein the input of the text decoder is the output of the text encoder.
[0107] In this step, the text decoder is used to assist the training of the text encoder.
[0108] In a specific implementation, when training the text encoder through the image report, a text decoder can be introduced, and the text decoder can be used to assist in training the text encoder to improve the feature extraction ability of the pre-trained text encoder. At the same time, the image report used for training can be randomly masked. For example, the image report originally reads "Changes after chemotherapy of right breast cancer, irregular mass-like long T1 signal shadows can be seen in the lower quadrant of the right breast, and the lesion is obviously enhanced compared to the last scan, and burrs can be seen around it." The masked report after random masking is "Changes after chemotherapy of right breast cancer, irregular mass-like long [mask] signal shadows can be seen in the lower quadrant of the right breast, and the lesion is obviously enhanced compared to the last scan, and [mask] can be seen around it." With the mask report as the input variable and the image report as the supervision information, the mask report is regenerated by the text encoder and the text decoder to obtain a regenerated report. Based on the loss between the regenerated report and the image report, the text encoder and the preset text decoder are jointly trained to obtain a pre-trained text encoder.
[0109] More specifically, refer to Figure 4 , is a schematic diagram of the training principle of an image segmentation model shown in an embodiment, Figure 4In the training of the included text encoder, each time during training, the mask report is first input into the text encoder for feature extraction to obtain text features, and then the text features are input into the text decoder for report regeneration, and the regenerated report is output. Through the mask language learning loss function, the loss between the regenerated report and the input mask report is calculated. In order to reduce the loss, the text encoder and the preset text decoder are jointly trained to obtain a pre-trained text encoder.
[0110] In this embodiment, when pre-training the text encoder, a text decoder is introduced to assist the training of the text encoder, which can improve the feature extraction capability of the text encoder. At the same time, the image report used for training is randomly masked before training the text encoder, which can further improve the feature extraction capability of the trained text encoder.
[0111] In an exemplary embodiment, in the above step S321b, the feature extraction module in the image segmentation model is trained through the sample medical image sequence, the imaging report and the pre-trained text encoder to obtain a trained feature extraction module, including: extracting the sample time series image features of the sample medical image sequence through the feature extraction module; extracting the text features of the imaging report through the pre-trained text encoder; based on the sample time series image features and text features, the feature extraction module and the pre-trained text encoder are jointly trained to obtain a trained feature extraction module.
[0112] refer to Figure 4 The schematic diagram of the training principle of the image segmentation model shown in the figure shows that when the feature extraction module in the image segmentation model is trained through sample medical image sequences, imaging reports and pre-trained text encoders, the training is performed through image-text contrast learning. Specifically, in terms of images, the sample medical image sequence is processed through the feature extraction module to obtain sample time series image features. In terms of text, the imaging report is encoded through a pre-trained text encoder to obtain the text features of the imaging report. Finally, image-text contrast learning training is performed through sample time series image features and text features to obtain a trained image encoder and a trained time series feature encoding module.
[0113] More specifically, when processing a sample medical image sequence through a feature extraction module, the sample medical image sequence is first input into an image encoder to obtain image features of each sample medical image. Then, the image features of each sample medical image are input into a temporal feature encoding module to obtain sample temporal image features of the sample medical image sequence. When processing an image report through a pre-trained text encoder, the image report is first converted into standardized text information according to a standardized template. The standardized text information is input into a pre-trained text encoder to obtain text features.
[0114] In this embodiment, when training the feature extraction module in the image segmentation model, a pre-trained text encoder is introduced to assist in the training of the feature extraction module, which can improve the feature extraction capability of the feature extraction module.
[0115] Furthermore, in an exemplary embodiment, based on sample time-series image features and text features, a feature extraction module and a pre-trained text encoder are jointly trained to obtain a trained feature extraction module, specifically including: obtaining multiple feature pairs based on multiple sample time-series image features and multiple text features; the feature pairs include positive sample feature pairs and negative sample feature pairs; obtaining the loss between the sample time-series image features and text features in each feature pair; and jointly training the feature extraction module and the pre-trained text encoder on the condition of reducing the loss of the positive sample feature pairs and increasing the loss of the negative sample feature pairs to obtain a trained feature extraction module.
[0116] Specifically, there are multiple sample medical image sequences used for image segmentation model training. Correspondingly, there are also multiple sample time-series image features and text features obtained, for example, n. Then, the n sample time-series image features F i (1≤i≤n) and n text features P j (1≤j≤n) to form multiple feature pairs F i P j (1≤i≤n, 1≤j≤n). The matching image-text feature pair, that is, the feature pair with i=j, is a positive sample feature pair. The non-matching image-text feature pair, that is, the feature pair with i≠j, is a negative sample feature pair. n represents the number of sample medical image sequences, which is also the number of sample time-series image features and the number of text features. F i represents any feature of the sample time series image features, P j Represents any feature of text features.
[0117] After obtaining multiple feature pairs, the positive and negative sample feature pairs are trained through the contrastive loss function. Specifically, the loss of each feature pair is calculated through the contrastive loss function, with the purpose of reducing the loss of positive sample feature pairs and increasing the loss of negative sample feature pairs. The feature extraction module and the pre-trained text encoder are jointly trained to obtain a trained feature extraction module.
[0118] In this embodiment, by combining the sample time-series image features corresponding to the sample medical image sequence and the text features corresponding to the imaging report to form positive and negative samples, and performing comparative learning training on the feature extraction module, the model can learn more specific features and rules, better identify and eliminate noise, thereby improving the model's robustness and anti-interference ability, as well as its generalization ability for new samples. In addition, by constructing positive and negative sample pairs, the data imbalance problem can be solved to a certain extent, so that the model can better learn the characteristics of minority class samples.
[0119] In one embodiment, in order to facilitate those skilled in the art to understand the training method of the above image segmentation model, the breast is used as a sample object for specific description. Figure 5 , is a flow chart of the training process of an image segmentation model shown in another embodiment. In this embodiment, the training process specifically includes the following steps:
[0120] Step S510, obtaining a sample breast image sequence of the breast and an image report corresponding to a target sample breast image in the sample breast image sequence.
[0121] Step S520: training the text encoder through the image report to obtain a pre-trained text encoder.
[0122] Specifically include:
[0123] Step S520a, performing random mask processing on the image report to obtain a mask report.
[0124] Step S520b, using the mask report as an input variable and the image report as supervisory information, jointly train the text encoder and the preset text decoder to obtain a pre-trained text encoder; wherein the input of the text decoder is the output of the text encoder.
[0125] Step S530: Processing the sample breast image sequence through a feature extraction module to obtain sample time-series image features of the sample breast image sequence.
[0126] Step S540, encoding the image report through a pre-trained text encoder to obtain text features of the image report.
[0127] Step S550, based on the sample time-series image features and text features, jointly train the feature extraction module and the pre-trained text encoder to obtain a trained feature extraction module.
[0128] Step S560, keeping the model parameters of the trained feature extraction module unchanged, training the segmentation module in the image segmentation model with the labeled data set to obtain a trained segmentation module.
[0129] Step S570, obtaining a trained image segmentation model based on the trained feature extraction module and the trained segmentation module.
[0130] In one embodiment, the text encoder and the image segmentation model can also be trained simultaneously. Specifically, the text features of the image report of the target sample medical image are obtained through the text encoder, and the text features are input into the text decoder to obtain a regenerated report. And the sample medical image sequence is passed through the image encoder and the time series feature encoding module in the image segmentation model to obtain the sample time series image features. Based on the sample time series image features and the text features, a first loss is obtained, and based on the regenerated report and the image report, a second loss is obtained. Based on the first loss and the second loss, a total loss is obtained, and the image encoder and the time series feature encoding module, as well as the text encoder and the text decoder are jointly trained based on the total loss to obtain a trained image encoder, a trained time series feature encoding module, and a trained image decoder to form a trained image segmentation model.
[0131] In this embodiment, the image segmentation model is trained by using a sample breast image sequence containing a target breast image to be segmented, so that the trained image segmentation model can extract the temporal changes of the breast, and perform breast image segmentation based on the temporal changes, thereby improving the accuracy of the segmentation results of the trained image segmentation model. In addition, the text encoder is pre-trained by using the imaging report, and the image segmentation model is trained with the pre-trained text encoder, which can further improve the ability of the image segmentation model to extract temporal image features of the breast image sequence, thereby improving the accuracy of the segmentation results determined based on the temporal image features.
[0132] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0133] Based on the same inventive concept, the embodiment of the present application also provides an image segmentation device for implementing the above-mentioned image segmentation method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image segmentation device embodiments provided below can refer to the limitations on the image segmentation method above, and will not be repeated here.
[0134] In one embodiment, Figure 6 As shown, an image segmentation device is provided, comprising: an acquisition unit 610, an extraction unit 620 and a segmentation unit 630, wherein:
[0135] An acquisition unit 610, configured to acquire a medical image sequence;
[0136] An extraction unit 620, used to extract temporal image features of a medical image sequence;
[0137] The segmentation unit 630 is used to segment any medical image in the medical image sequence according to the time series image features to obtain a region of interest in any medical image.
[0138] In one embodiment, the extraction unit 620 is further configured to extract image features of each medical image in the medical image sequence; and determine the temporal image features of the medical image sequence according to the image features of each medical image.
[0139] In one of the embodiments, the image segmentation method is performed through an image segmentation model, and the device also includes a model training module, which is used to obtain a sample medical image sequence of a sample object, and an image report corresponding to a target sample medical image in the sample medical image sequence; the sample medical image sequence includes multiple sample medical images, and the target sample medical image is the medical image with the latest acquisition time in the sample medical image sequence; the image segmentation model is trained through the sample medical image sequence and the image report to obtain a trained image segmentation model.
[0140] In one of the embodiments, the model training module is also used to train the feature extraction module in the image segmentation model through sample medical image sequences and imaging reports to obtain a trained feature extraction module; keep the model parameters of the trained feature extraction module unchanged, and train the segmentation module in the image segmentation model through a labeled data set to obtain a trained segmentation module; the labeled data set includes sample medical images and real regions of interest in the sample medical images; based on the trained feature extraction module and the trained segmentation module, a trained image segmentation model is obtained.
[0141] In one of the embodiments, the model training module is also used to train the text encoder through the imaging report to obtain a pre-trained text encoder; and to train the feature extraction module through the sample medical image sequence, the imaging report and the pre-trained text encoder to obtain a trained feature extraction module.
[0142] In one of the embodiments, the model training module is also used to perform random mask processing on the image report to obtain a masked report; using the masked report as the input variable and the image report as the supervision information, combined with a preset text decoder, the text encoder is trained to obtain a pre-trained text encoder; wherein the input of the text decoder is the output of the text encoder.
[0143] In one of the embodiments, the model training module is also used to extract sample time-series image features of a sample medical image sequence through a feature extraction module; extract text features of an imaging report through a pre-trained text encoder; and jointly train the feature extraction module and the pre-trained text encoder based on the sample time-series image features and text features to obtain a trained feature extraction module.
[0144] In one of the embodiments, there are multiple sample medical image sequences, and correspondingly, there are also multiple sample time-series image features and text features; the model training module is also used to obtain multiple feature pairs based on the multiple sample time-series image features and the multiple text features; the feature pairs include positive sample feature pairs and negative sample feature pairs; the loss between the sample time-series image features and the text features in each feature pair is obtained; under the condition of reducing the loss of the positive sample feature pairs and increasing the loss of the negative sample feature pairs, the feature extraction module and the pre-trained text encoder are jointly trained to obtain a trained feature extraction module.
[0145] Each module in the above-mentioned image segmentation device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.
[0146] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data in the image segmentation process. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image segmentation method is implemented.
[0147] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0148] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.
[0149] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0150] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0151] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.
[0152] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0153] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0154] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. An image segmentation method, It is characterized in that The method comprises: Acquire a medical image sequence; the medical image sequence is a sequence of medical images acquired through different scans; Extracting image features of each medical image in the medical image sequence through a feature extraction module in the image segmentation model; performing temporal coding processing on the image features of each medical image to obtain temporal coding features of each medical image; performing splicing processing on the temporal coding features of each medical image to obtain temporal image features of the medical image sequence; Segmenting any medical image in the medical image sequence according to the time series image features through the segmentation module in the image segmentation model to obtain a region of interest in any medical image; The image segmentation model is trained in the following way: Acquire a sample medical image sequence of a sample object and an image report corresponding to a target sample medical image in the sample medical image sequence; The text encoder is trained by using the image report and a preset text decoder to obtain a pre-trained text encoder; the preset text decoder is used to assist the training of the text encoder; The feature extraction module in the image segmentation model is trained by using the sample medical image sequence, the imaging report and the pre-trained text encoder to obtain a trained feature extraction module; the pre-trained text encoder is used to assist the training of the feature extraction module; Keeping the model parameters of the trained feature extraction module unchanged, training the segmentation module in the image segmentation model using a labeled data set to obtain a trained segmentation module; The image segmentation model is obtained based on the trained feature extraction module and the trained segmentation module.
2. The method according to claim 1, It is characterized in that The step of training a text encoder through the image report and a preset text decoder to obtain a pre-trained text encoder includes: Performing random mask processing on the image report to obtain a mask report; The mask report is used as an input variable and the image report is used as supervisory information to jointly train the text encoder and the preset text decoder to obtain a pre-trained text encoder.
3. The method according to claim 1, It is characterized in that The feature extraction module in the image segmentation model is trained by using the sample medical image sequence, the imaging report and the pre-trained text encoder to obtain a trained feature extraction module, including: Extracting sample time-series image features of the sample medical image sequence through the feature extraction module; Extracting text features of the image report by using the pre-trained text encoder; Based on the sample time-series image features and the text features, the feature extraction module and the pre-trained text encoder are jointly trained to obtain a trained feature extraction module.
4. The method according to claim 3, It is characterized in that There are multiple sample medical image sequences, and correspondingly, there are also multiple sample time-series image features and text features; The method of jointly training the feature extraction module and the pre-trained text encoder based on the sample time-series image features and the text features to obtain a trained feature extraction module includes: Based on multiple sample time-series image features and multiple text features, multiple feature pairs are obtained; the feature pairs include positive sample feature pairs and negative sample feature pairs; Obtain the loss between the sample temporal image features and text features in each feature pair; Under the condition of reducing the loss of positive sample feature pairs and increasing the loss of negative sample feature pairs, the feature extraction module and the pre-trained text encoder are jointly trained to obtain a trained feature extraction module.
5. The method according to claim 1, It is characterized in that The step of training a text encoder through the image report and a preset text decoder to obtain a pre-trained text encoder includes: Taking the image report as an input variable, the text encoder and the preset text decoder regenerate the image report to obtain a regenerated report; Based on the loss between the regenerated report and the image report, the text encoder and the preset text decoder are jointly trained to obtain a pre-trained text encoder.
6. The method according to any one of claims 1 to 5, It is characterized in that The sample medical image sequence includes a plurality of sample medical images, and the target sample medical image is the medical image with the latest acquisition time in the sample medical image sequence.
7. An image segmentation device, It is characterized in that The device comprises: An acquisition unit, used for acquiring a medical image sequence; the medical image sequence is a sequence of medical images acquired through different scans; An extraction unit is used to extract image features of each medical image in the medical image sequence through a feature extraction module in an image segmentation model; perform time-series coding processing on the image features of each medical image to obtain time-series coding features of each medical image; and perform splicing processing on the time-series coding features of each medical image to obtain time-series image features of the medical image sequence; A segmentation unit, configured to segment any medical image in the medical image sequence according to the time series image features through a segmentation module in the image segmentation model, to obtain a region of interest in any medical image; A training unit is used to obtain a sample medical image sequence of a sample object and an image report corresponding to a target sample medical image in the sample medical image sequence; a text encoder is trained through the image report and a preset text decoder to obtain a pre-trained text encoder; the preset text decoder is used to assist in the training of the text encoder; a feature extraction module in the image segmentation model is trained through the sample medical image sequence, the image report and the pre-trained text encoder to obtain a trained feature extraction module; the pre-trained text encoder is used to assist in the training of the feature extraction module; the model parameters of the trained feature extraction module are kept unchanged, and the segmentation module in the image segmentation model is trained through a labeled data set to obtain a trained segmentation module; the image segmentation model is obtained based on the trained feature extraction module and the trained segmentation module.
8. The device according to claim 7, It is characterized in that The sample medical image sequence includes a plurality of sample medical images, and the target sample medical image is the medical image with the latest acquisition time in the sample medical image sequence.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the image segmentation method according to any one of claims 1 to 6 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the image segmentation method according to any one of claims 1 to 6 are implemented.
