Method for feature prediction of tomographic images and related device

Abstract

This application provides a feature prediction method and related equipment for tomographic scan images. The method involves acquiring a tomographic scan image, performing feature prediction segmentation on the tomographic scan image to obtain a feature prediction segmentation result, performing segmentation processing on the tomographic scan image based on the feature prediction segmentation result, enhancing the feature edges in the tomographic scan image corresponding to the feature prediction segmentation result during the segmentation process to obtain a feature edge enhancement result, and obtaining a feature prediction result based on the feature edge enhancement result and the feature prediction segmentation result. The method strengthens the feature edges by enhancing the feature edges corresponding to the feature prediction segmentation result in the tomographic scan image, and then obtaining a more accurate feature prediction result based on the enhanced feature edge result and the feature prediction segmentation result.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a feature prediction method and related equipment for tomographic scan images. Background Technology

[0002] Tomographic images can be used to assist in medical diagnosis and have a wide range of applications.

[0003] However, the inventors of this application have discovered that some features in tomographic images may exhibit blurred feature edges, which can lead to difficulties in feature segmentation and inaccurate feature segmentation. Summary of the Invention

[0004] In view of this, the purpose of this application is to propose a feature prediction method and related equipment for tomographic scan images, so as to solve or partially solve the above-mentioned technical problems.

[0005] To achieve the above objectives, a first aspect of this application provides a feature prediction method for tomographic scan images, comprising:

[0006] Acquire tomographic images;

[0007] The tomographic scan image is subjected to feature prediction segmentation to obtain the feature prediction segmentation result;

[0008] Based on the feature prediction segmentation result, the tomographic scan image is segmented, and during the segmentation process, the feature edges in the tomographic scan image corresponding to the feature prediction segmentation result are enhanced to obtain the feature edge enhancement result.

[0009] Based on the feature edge enhancement result and the feature prediction segmentation result, the feature prediction result is obtained.

[0010] Optionally, the step of performing feature prediction segmentation on the computed tomography image to obtain the feature prediction segmentation result includes:

[0011] The tomographic scan image is split in the first dimension of three-dimensional space according to a preset number of channels to obtain a split tomographic scan image.

[0012] The split tomographic image is downsampled in the second and third dimensions of three-dimensional space to obtain a downsampled tomographic image.

[0013] The downsampled tomographic image is subjected to feature prediction segmentation to obtain the downsampled feature prediction segmentation result;

[0014] The downsampled feature prediction segmentation result is upsampled to obtain the upsampled feature prediction segmentation result, and the upsampled feature prediction segmentation result is used as the feature prediction segmentation result.

[0015] Optionally, the step of performing feature prediction segmentation on the computed tomography image to obtain the feature prediction segmentation result includes:

[0016] The tomographic scan image is classified and predicted to obtain the classification prediction result;

[0017] Based on the classification prediction result, the features in the feature prediction segmentation result that do not correspond to the classification prediction result are set to zero.

[0018] Optionally, the feature prediction segmentation result is obtained using a trained feature prediction segmentation model. The feature prediction segmentation model is pre-trained based on a pre-acquired first training dataset. The first training dataset includes a first tomographic image, a second tomographic image, and a third tomographic image. The first tomographic image, the second tomographic image, and the third tomographic image are tomographic images of different types. The feature prediction segmentation model includes a classification branch layer for classification prediction.

[0019] The loss function used for training the feature prediction and segmentation model includes:

[0020] Loss=α1L cls +α2L seg

[0021] Where Loss represents the loss function of the feature prediction segmentation model, L cls L represents the classification loss of the classification branch layer. cls =β1loss ce +β2loss con loss ce Represents cross-entropy loss, loss con Let β1 represent the cross-entropy loss, where β represents the contrast loss. ce The weights, β2 represents the contrastive loss. con The weight, L seg Let α1 represent the segmentation loss of the feature prediction segmentation model, and L represent the classification loss. cls The weights are α2, which represents the weights of the segmentation loss.

[0022] Optionally, the step of enhancing the feature edges in the tomographic image corresponding to the feature prediction segmentation result during the segmentation process to obtain the feature edge enhancement result includes:

[0023] The tomographic scan image and the feature prediction segmentation result are fused to obtain a fused tomographic scan image.

[0024] The feature edges of the fused tomographic image are enhanced based on the fused tomographic image to obtain the feature edge enhancement result.

[0025] Optionally, the feature edge enhancement result is obtained using a trained feature edge enhancement model, which includes an edge enhancement network. The edge enhancement network includes a first convolutional layer and a second convolutional layer, wherein the first convolutional layer and the second convolutional layer are connected in series and are also connected in skip connections.

[0026] The enhancement of the feature edges of the tomographic image based on the fused tomographic image to obtain the feature edge enhancement result includes:

[0027] The fused tomographic image is segmented to obtain multiple tomographic image blocks;

[0028] The edge enhancement network is used to perform edge enhancement processing on the feature regions in the multiple tomographic image blocks that correspond to the feature prediction and segmentation results, so as to obtain the feature edge enhancement results of the multiple tomographic image blocks.

[0029] In the first dimension of three-dimensional space, the feature edge enhancement results of multiple tomographic scan image blocks will be stitched together.

[0030] Optionally, the step of segmenting the fused tomographic image to obtain multiple tomographic image blocks includes:

[0031] The fused tomographic image is cut in the second and third dimensions of three-dimensional space according to a preset overlap rate to obtain multiple tomographic image blocks.

[0032] Optionally, the loss function used to train the feature edge enhancement model includes:

[0033] Loss = γ1 loss Dice +γ2loss Bound

[0034] Where Loss represents the loss function of the feature edge enhancement model, loss Dice This represents Dice loss. Bound γ represents the edge fitting loss, and γ1 represents the Dice loss. Dice The weights, γ2 represents the edge fitting loss. Bound The weights;

[0035] Edge fitting loss Bound Specifically, it includes:

[0036]

[0037] Where, loss Bound The loss represents the edge fitting loss. Bxy ω1 represents the two-dimensional edge fitting loss in the second-dimensional direction x and the third-dimensional direction y in three-dimensional space. Bxy Weights, loss Bxz ω² represents the two-dimensional edge fitting loss along the second-dimensional direction x and the first-dimensional direction z in three-dimensional space. Bxz Weights, loss Byz ω3 represents the 2D edge fitting loss in the third dimension y and the first dimension z in 3D space. Byz The weight, Representation and loss Bxy The corresponding two-dimensional Euclidean distance, P′ xy This indicates that the predicted contour sampling points are taken from the first dimension direction z. This indicates that the true contour sampling points are taken from the first dimension direction z. Representation and loss Bxz The corresponding two-dimensional Euclidean distance, P′ xz This indicates that the predicted contour sampling points are taken from the third dimension direction y. This indicates that the true contour sampling points are taken from the third dimension direction y. Representation and loss Byz The corresponding two-dimensional Euclidean distance, P′ yz This indicates that the predicted value contour sampling points are taken from the second dimension direction x. This indicates that the true contour sampling point is taken from the second dimension direction x, n represents the number of training stitched tomographic image blocks, and i represents the i-th training stitched tomographic image block.

[0038] Optionally, the predicted value contour sampling points and the true value contour sampling points are matched by overlap calculation.

[0039] Optionally, the predicted value contour sampling points and the true value contour sampling points are matched by overlap calculation, including:

[0040] The watershed algorithm is used to extract the connected components of the predicted value contours and the connected components of the ground truth contours in the training stitched tomographic image blocks, respectively.

[0041] Based on the connected components of the predicted value profile, the sampling points of the predicted value profile are determined, and the sampling points of the predicted value profile are used to determine the first minimum bounding rectangle.

[0042] Based on the connected components of the truth contour, the sampling points of the truth contour are determined, and the sampling points of the truth contour are used to determine the second minimum bounding rectangle.

[0043] The overlap between the predicted value contour and the true value contour is calculated using the following formula based on the first minimum bounding rectangle and the second minimum bounding rectangle:

[0044]

[0045] Wherein, IoU represents the overlap between the predicted value contour and the true value contour, A represents the true value contour, Area(A) represents the area of ​​the second minimum bounding rectangle corresponding to the true value contour, B represents the predicted value contour, Area(B) represents the area of ​​the first minimum bounding rectangle corresponding to the predicted value contour, and Area(A∪B) represents the union of the area of ​​the second minimum bounding rectangle corresponding to the true value contour A and the area of ​​the first minimum bounding rectangle corresponding to the predicted value contour B.

[0046] The ground truth contour and the predicted contour when the overlap is greater than a preset threshold are matched to obtain a matching pair. The matching pair is then sampled according to a preset sampling interval to obtain the matched predicted contour sampling points and the ground truth contour sampling points.

[0047] Optionally, the feature edge enhancement model is obtained by training based on a pre-acquired first training dataset and a second training dataset. The first training dataset includes a first tomographic image, a second tomographic image, and a third tomographic image. The second training dataset includes a second tomographic image. The first tomographic image, the second tomographic image, and the third tomographic image are tomographic images of different types.

[0048] Optionally, the feature edge enhancement model includes a location index prediction model, which is obtained by transfer learning after pre-training the location index prediction model using the first training dataset and then training it using the second training dataset.

[0049] Based on the same inventive concept, a second aspect of this application provides a feature prediction device for tomographic images, comprising:

[0050] The tomographic image acquisition module is configured to acquire tomographic images;

[0051] The feature prediction and segmentation module is configured to perform feature prediction and segmentation on the tomographic scan image to obtain the feature prediction and segmentation result.

[0052] The feature edge enhancement network is configured to segment the tomographic image based on the feature prediction segmentation result, and enhance the feature edges in the tomographic image corresponding to the feature prediction segmentation result during the segmentation process to obtain the feature edge enhancement result.

[0053] The overlay processing module is configured to obtain a feature prediction result based on the feature edge enhancement result and the feature prediction segmentation result.

[0054] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the program to implement the method described in the first aspect.

[0055] A fourth aspect of this application provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method described in the first aspect. As can be seen from the above, the feature prediction method and related apparatus for computed tomography images provided by this application acquire a computed tomography image, perform feature prediction segmentation on the computed tomography image to obtain a feature prediction segmentation result, perform segmentation processing on the computed tomography image based on the feature prediction segmentation result, and enhance the feature edges in the computed tomography image corresponding to the feature prediction segmentation result during the segmentation process to obtain a feature edge enhancement result. Based on the feature edge enhancement result and the feature prediction segmentation result, a feature prediction result is obtained. By enhancing the feature edges in the computed tomography image corresponding to the feature prediction segmentation result, the feature edges are strengthened. Then, based on the enhanced feature edge result and the feature prediction segmentation result, a more accurate feature prediction result is obtained. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a flowchart of a feature prediction method for tomographic images according to an embodiment of this application;

[0058] Figure 2A This is a schematic diagram of the cascaded network architecture according to an embodiment of this application;

[0059] Figure 2B This is a schematic diagram of the model network structure used in the second stage of an embodiment of this application;

[0060] Figure 2C This is a schematic diagram of an edge enhancement network according to an embodiment of this application;

[0061] Figure 2D This is a schematic diagram of self-supervised pre-training in an embodiment of this application;

[0062] Figure 3 This is a schematic diagram of the feature prediction device for tomographic images according to an embodiment of this application;

[0063] Figure 4 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0065] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0066] It is understood that before using the technical solutions of the various embodiments in this disclosure, users will be informed of the personal information, scope of use, and usage scenarios involved in an appropriate manner, and user authorization will be obtained.

[0067] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose, based on the prompt message, whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media performing the operations of this disclosed technical solution.

[0068] As an optional but not limited implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0069] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0070] Related technologies such as tomography images can be used to assist in medical diagnosis and have a wide range of applications.

[0071] However, the inventors of this application have discovered that features located in tomographic images may exhibit blurred feature edges, which can lead to difficulties in feature segmentation and inaccurate feature segmentation.

[0072] Furthermore, the annotation of tomographic images is costly due to the need for specialized medical knowledge. Additionally, the small data size of tomographic images, caused by factors such as privacy protection, also contributes to the difficulty of feature segmentation.

[0073] The embodiments of this application provide a feature prediction method for tomographic scan images. By enhancing the feature edges corresponding to the feature prediction segmentation results in the tomographic scan, a more accurate feature prediction result is obtained based on the enhanced feature edge result and the feature prediction segmentation result. This avoids the problem of difficult feature segmentation caused by blurred feature edges in tomographic scan images.

[0074] like Figure 1 As shown, the method in this embodiment includes:

[0075] Step 101: Obtain the tomographic scan image.

[0076] In practice, computed tomography (CT) images can be obtained by taking pictures with a computed tomography camera or by scanning with an X-ray computed tomography scanner.

[0077] Step 102: Perform feature prediction segmentation on the tomographic scan image to obtain the feature prediction segmentation result.

[0078] In practice, the feature prediction segmentation results are used to represent the regions where features are located in the tomographic scan image.

[0079] The feature represents the target image to be segmented in the tomographic scan image.

[0080] By performing feature prediction segmentation on the tomographic scan image, the feature prediction segmentation result is obtained, and the region of the target image to be segmented in the tomographic scan image is represented by the feature prediction segmentation result.

[0081] For example, the feature prediction segmentation result can be a segmentation mask or a bounding box, preferably a segmentation mask. The features can include features of normal areas in the tomographic image as well as other features relative to the features of normal areas. For example, when the tomographic image is a lung image, the features can be common lung features (i.e., features inherent to the lung itself) as well as other features beyond the common features (not limited to lesion features). As an optional embodiment, the features can be lung lesion features.

[0082] Therefore, as an optional embodiment, step 102 may further include generating a segmentation mask in the tomographic image by performing feature prediction segmentation on the tomographic image.

[0083] Step 103: Based on the feature prediction segmentation result, the tomographic scan image is segmented, and during the segmentation process, the feature edges in the tomographic scan image corresponding to the feature prediction segmentation result are enhanced to obtain the feature edge enhancement result.

[0084] In practice, the feature edge enhancement result can represent a tomographic image with enhanced feature edges.

[0085] Based on the feature prediction segmentation results, the tomographic scan image is segmented, and during the segmentation process, the feature edges in the tomographic scan image corresponding to the feature prediction segmentation results are enhanced, so that the feature edges can be easily identified and distinguished, resulting in feature edge enhancement results.

[0086] For example, the feature edges in a tomographic image that correspond to the feature prediction segmentation result can be the edges of the target image to be segmented in the tomographic image. By enhancing the edges of the target image, a tomographic image containing the target image with more obvious edges can be obtained.

[0087] By enhancing the edges of the target image, the edges of the target image become more distinct, making it easier to identify and distinguish between regions of the target image and regions of non-target images, thus avoiding the problem of difficult segmentation caused by blurred edges.

[0088] Step 104: Based on the feature edge enhancement result and the feature prediction segmentation result, obtain the feature prediction result.

[0089] In practice, the feature prediction results represent tomographic images containing the identified features.

[0090] By superimposing the feature prediction segmentation result with the feature edge enhancement result, the feature prediction segmentation result can more clearly represent the feature edge enhancement result, and can distinguish the area in the tomographic scan image where the feature prediction segmentation result and the feature edge enhancement result are superimposed, as well as the area outside the superimposed area.

[0091] For example, the features could be unusual features of the lungs, and the feature prediction segmentation result could be a segmentation mask.

[0092] Feature edge enhancement results in tomographic images of non-typical features (or target images) with enhanced edges.

[0093] By overlaying a tomographic image with enhanced edges of unusual lung features onto a tomographic image with a segmentation mask, the regions containing unusual features are more clearly represented, thus distinguishing unusual feature regions from normal feature regions in the tomographic image.

[0094] The above scheme acquires a tomographic scan image, performs feature prediction segmentation on the tomographic scan image, obtains the feature prediction segmentation result, performs segmentation processing on the tomographic scan image based on the feature prediction segmentation result, and enhances the feature edges in the tomographic scan image corresponding to the feature prediction segmentation result during the segmentation process, obtains the feature edge enhancement result, and obtains the feature prediction result based on the feature edge enhancement result and the feature prediction segmentation result. By enhancing the feature edges in the tomographic scan image corresponding to the feature prediction segmentation result, the feature edges are strengthened. Then, based on the enhanced feature edge result and the feature prediction segmentation result, a more accurate feature prediction result is obtained, thereby avoiding the problem of difficult feature segmentation caused by blurred feature edges in the tomographic scan image.

[0095] In some embodiments, step 102 includes:

[0096] Step A1: The tomographic scan image is split in the first dimension of three-dimensional space according to the preset number of channels to obtain a split tomographic scan image.

[0097] Step A2: Downsample the split tomographic image in the second and third dimensions of three-dimensional space to obtain a downsampled tomographic image.

[0098] Step A3: Perform feature prediction segmentation on the downsampled tomographic image to obtain the downsampled feature prediction segmentation result.

[0099] Step A4: Upsample the downsampled feature prediction segmentation result to obtain the upsampled feature prediction segmentation result, and use the upsampled feature prediction segmentation result as the feature prediction segmentation result.

[0100] In practice, the split tomographic image is downsampled in the second and third dimensions of three-dimensional space to obtain a downsampled tomographic image. This reduces the size of the tomographic image, increasing the receptive field and facilitating the extraction of global information during feature prediction and segmentation. Feature prediction and segmentation are then performed on the downsampled tomographic image to obtain the downsampled feature prediction and segmentation result. Finally, the downsampled feature prediction and segmentation result is upsampled to restore its original size, resulting in the upsampled feature prediction and segmentation result.

[0101] In particular, downsampling of the split tomographic scan image in the second and third dimensions of three-dimensional space can be achieved by setting a preset downsampling ratio.

[0102] In some embodiments, such as Figure 2A As shown, a cascaded network architecture can be used to implement the feature prediction method for the tomographic scan image. Specifically, as... Figure 2A As shown, the cascaded network architecture can include the model used in the first stage (i.e., Stage 1) and the model used in the second stage (i.e., Stage 2). The first stage uses the 3DSwin-Unet (a semantic segmentation network based on self-attention mechanism) model based on Swin Transformer (a deep learning model based on self-attention mechanism). The 3DSwin-Unet model does not use Convolutional Neural Network (CNN) encoding. The second stage uses the 3DBR-Unet (a semantic segmentation model for medical image edge extraction) model.

[0103] In the first stage, a coarse segmentation can be performed on the low-resolution 3D tomographic image. This coarse segmentation step can be used to implement step 102. First, the 3D tomographic image can be split along the first dimension (e.g., the z-direction) according to a preset number of channels (e.g., 128), resulting in split data (i.e., split tomographic images). The split data is then downsampled along the second and third dimensions (e.g., the xy-direction) to reduce the size of the 3D tomographic image according to a preset downsampling ratio, resulting in a downsampled tomographic image. The downsampled tomographic image is then subjected to feature prediction segmentation using 3D Swin-Unet to obtain a downsampled feature prediction segmentation result. This downsampled feature prediction segmentation result is then upsampled to restore the original size, resulting in an upsampled feature prediction segmentation result. This upsampled feature prediction segmentation result is then used as the feature prediction segmentation result.

[0104] In some embodiments, step 102 includes:

[0105] Step B1: Classify and predict the tomographic scan image to obtain the classification prediction result.

[0106] Step B2: Based on the classification prediction result, set the features in the feature prediction segmentation result that are not corresponding to the classification prediction result to zero.

[0107] In practice, the tomographic scan image is classified and predicted to obtain the classification prediction result. The feature prediction segmentation result is constrained by the classification prediction result. The features of the non-classification prediction result in the feature prediction segmentation result are set to zero, so that the features of the classification prediction result in the feature prediction segmentation result can be distinguished.

[0108] In some embodiments, the feature prediction segmentation result is obtained using a trained feature prediction segmentation model, which is pre-trained based on a pre-acquired first training dataset. The first training dataset includes a first tomographic image, a second tomographic image, and a third tomographic image. The first tomographic image, the second tomographic image, and the third tomographic image are tomographic images of different types. The feature prediction segmentation model includes a classification branch layer for classification prediction.

[0109] The loss function used for training the feature prediction and segmentation model includes:

[0110] LoSs=α1L cls +α2L seg

[0111] Where Loss represents the loss function of the feature prediction segmentation model, L cls L represents the classification loss of the classification branch layer. cls =β1loss ce +β2loss con loss ce Represents cross-entropy loss, loss con Let β1 represent the cross-entropy loss, where β represents the contrast loss. ce The weights, β2 represents the contrastive loss. con The weight, L seg Let α1 represent the segmentation loss of the feature prediction segmentation model, and L represent the classification loss. cls The weights are α2, which represents the weights of the segmentation loss.

[0112] In practice, the feature prediction segmentation results are obtained using a trained feature prediction segmentation model.

[0113] Among them, such as Figure 2A As shown, the feature prediction segmentation model can be the 3DSwin-Unet model of the first stage mentioned above.

[0114] The feature prediction segmentation model is pre-trained based on a first training dataset, which includes a first tomographic scan image, a second tomographic scan image, and a third tomographic scan image, in order to address the problem of insufficient labeled data for tomographic scan images.

[0115] For example, normal lung tomographic images (i.e., the first tomographic image), patient lung tomographic images (i.e., the second tomographic image), and tomographic images of other lung diseases (such as lung tumors, pleural effusions, etc.) (i.e., the third tomographic image) are used to participate in training, learn the common features of lung diseases, and assist in the segmentation of specific lesions under a small number of labeled samples.

[0116] The first stage of training the 3D Swin-Unet model uses normal lung tomographic images (i.e., the first tomographic image), patient lung tomographic images (i.e., the second tomographic image), and tomographic images of other lung diseases (i.e., the third tomographic image). The segmentation labels include both normal and abnormal categories, meaning they do not distinguish between different disease lesions, thus enhancing the 3D Swin-Unet model's sensitivity to abnormal lung regions. To address the issue of confusion between different lesions, a classification branch layer (which can be a linear layer) is added to classify normal samples and samples with different diseases (disease slices without lesions are labeled as normal; the number of predicted categories is related to the number of categories in the other lung disease data used). Simultaneously, because the representations of different lung diseases in tomographic images have a certain similarity, making differentiation difficult, a contrastive loss is added to reduce intra-class distance and increase inter-class distance. During 3D Swin-Unet model prediction, the final feature prediction segmentation result output from the first stage is constrained by the classification result, setting lesions in other lung disease tomographic image categories to 0.

[0117] Optionally, during the training of the first stage of the 3D Swin-Unet model:

[0118] The same segmentation label can be used for all different lesions, keeping the segmentation head output unchanged. An additional classification branch layer is connected after the bottleneck module of 3DSwin-Unet to classify the disease categories.

[0119] For the first stage, the loss function is:

[0120] Looss=α1L cls +α2L seg

[0121] Where Loss represents the loss function of the first-stage 3D Swin-Unet model (i.e., the feature prediction and segmentation model), L cls L represents the classification loss. cls =β1loss ce +β2loss con loss ce Represents cross-entropy loss, loss con Let β1 represent the cross-entropy loss, where β represents the contrast loss. ce The weights, β2 represents the contrastive loss. con The weight, L seg Let α1 represent the segmentation loss and α2 represent the classification loss. cls The weights are α2, which represents the weights of the segmentation loss.

[0122] In some embodiments, step 103 involves enhancing the feature edges in the tomographic scan image corresponding to the feature prediction segmentation result during the segmentation process to obtain a feature edge enhancement result, including:

[0123] Step 1031: The tomographic scan image and the feature prediction segmentation result are fused to obtain a fused tomographic scan image.

[0124] Step 1032: Enhance the feature edges of the tomographic image based on the fused tomographic image to obtain the feature edge enhancement result.

[0125] In practice, the fused tomographic image represents a tomographic image that includes the feature prediction segmentation results.

[0126] The tomographic scan image is fused with the feature prediction segmentation result to obtain a tomographic scan image containing the feature prediction segmentation result, so that the feature prediction segmentation result can be displayed in the tomographic scan image. Then, the feature edges of the tomographic scan image are enhanced based on the fused tomographic scan image to obtain the feature edge enhancement result.

[0127] For example, the feature prediction segmentation result uses a segmentation mask to represent the region where the target image is located in the tomographic scan image. The tomographic scan image and the segmentation mask are then fused, so that the segmentation mask represents the region where the target image is located in the tomographic scan image. Then, based on the fused tomographic scan image, the feature edges of the target image in the tomographic scan image are enhanced, resulting in a tomographic scan image with enhanced edges of the target image.

[0128] In some embodiments, the feature edge enhancement result is achieved using a trained feature edge enhancement model, which includes an edge enhancement network. The edge enhancement network includes a first convolutional layer and a second convolutional layer, wherein the first convolutional layer and the second convolutional layer are connected in series and are also connected in skip connections.

[0129] Step 1032 includes:

[0130] Step 10321: The fused tomographic image is cut to obtain multiple tomographic image blocks.

[0131] Step 10322: The edge enhancement network is used to perform edge enhancement processing on the feature regions in the multiple tomographic image blocks that correspond to the feature prediction and segmentation results, and the feature edge enhancement results corresponding to the multiple tomographic image blocks are obtained.

[0132] Step 10323: In the first dimension of three-dimensional space, the feature edge enhancement results corresponding to multiple tomographic scan image blocks are stitched together to obtain the feature edge enhancement result.

[0133] In practice, to achieve more accurate and refined segmentation, the fused tomographic image needs to be segmented into multiple tomographic image blocks. For each tomographic image block, a prediction is made, and an edge enhancement network is used to enhance the feature regions in all tomographic image blocks that correspond to the feature prediction segmentation results, thereby strengthening the feature edges. The prediction results of all tomographic image blocks are then fused (the prediction probabilities of the overlapping parts are averaged) to obtain the feature edge enhancement result.

[0134] In some embodiments, step 10321 includes:

[0135] The fused tomographic image is cut in the second and third dimensions of three-dimensional space according to a preset overlap rate to obtain multiple tomographic image blocks.

[0136] In practice, to ensure that the integrity of the fused tomographic image is not compromised after cutting, a preset overlap rate is set for each tomographic image block. The fused tomographic image is then cut in the second and third dimensions of three-dimensional space according to the preset overlap rate to obtain multiple tomographic image blocks.

[0137] For example, such as Figure 2A As shown, in the second stage (Stage 2), the fused tomographic image is cropped in the xy direction (i.e., the second and third dimension directions) with an overlap ratio of 0.5, resulting in multiple tomographic image blocks.

[0138] In some embodiments, the loss function used to train the feature edge enhancement model includes:

[0139] Loss=γ1loss Dice +γ2loss Bound

[0140] Where Loss represents the loss function of the feature edge enhancement model, loss Dice This represents Dice loss. Bound γ represents the edge fitting loss, and γ1 represents the Dice loss. Dice The weights, γ2 represents the edge fitting loss. Bound The weights;

[0141] Edge fitting loss Bound Specifically, it includes:

[0142]

[0143] Where, loss Bound The loss represents the edge fitting loss. Bxy ω1 represents the two-dimensional edge fitting loss in the second-dimensional direction x and the third-dimensional direction y in three-dimensional space. Bxy Weights, loss Bxz ω² represents the two-dimensional edge fitting loss along the second-dimensional direction x and the first-dimensional direction z in three-dimensional space. Bxz Weights, loss Byz ω3 represents the 2D edge fitting loss in the third dimension y and the first dimension z in 3D space. Byz The weight, Representation and loss Bxy The corresponding two-dimensional Euclidean distance, P′ xy This indicates that the predicted contour sampling points are taken from the first dimension direction z. This indicates that the true contour sampling points are taken from the first dimension direction z. Representation and loss Bxz The corresponding two-dimensional Euclidean distance, P′ xz This indicates that the predicted contour sampling points are taken from the third dimension direction y. This indicates that the true contour sampling points are taken from the third dimension direction y. Representation and loss Byz The corresponding two-dimensional Euclidean distance, P′ yz This indicates that the predicted value contour sampling points are taken from the second dimension direction x. This indicates that the true contour sampling point is taken from the second dimension direction x, n represents the number of training stitched tomographic image blocks, and i represents the i-th training stitched tomographic image block.

[0144] In practice, the edge enhancement network is used to perform edge enhancement processing on the feature regions corresponding to the feature prediction and segmentation results in multiple tomographic image blocks. This results in feature edge enhancement results where the feature edges of the feature regions have high activation values ​​and the feature regions other than the feature edges have low activation values. The activation value represents the degree of difficulty in being recognized. A high activation value makes it easier to be recognized, thus increasing the recognition difference between the two to achieve the effect of edge enhancement processing.

[0145] For example, such as Figure 2A As shown, in the second stage (stage 2), the lateral connection portion of the 3D Unet network is inserted with an edge reinforcement network (BR) to form a 3D BR-Unet model. The network structure of the 3D BR-Unet model is as follows: Figure 2B As shown.

[0146] Among them, edge enhancement network BR such Figure 2C As shown, the network consists of two 3×3 convolutional layers (Conv) and a skip connection. It essentially introduces residual information during the network decoding stage. The two convolutional layers in the edge enhancement network (BR) learn the residual between the input and output features. Feature regions other than feature edges, corresponding to the feature prediction segmentation result, have high activation values, while feature edges have low activation values ​​relative to feature regions other than feature edges.

[0147] By using two convolutional layers to learn and enhance edges, we obtain the feature edge enhancement result where the feature edge of the feature region has a high activation value and the feature region other than the feature edge has a low activation value. The activation value represents the difficulty of being recognized. A high activation value makes it easier to be recognized. By increasing the recognition difference between the two, we can achieve the effect of edge enhancement processing.

[0148] Finally, the feature edge enhancement results are superimposed with the feature prediction segmentation results through the edge enhancement network BR (i.e., Sum) to obtain the feature prediction results.

[0149] Among them, such as Figure 2A As shown, the feature edge enhancement model can be a second-stage (i.e., stage 2) 3D BR-Unet model.

[0150] To improve edge segmentation capabilities, an edge fitting loss is designed into the loss function of the edge enhancement network to calculate the distance between the predicted edge and the ground truth edge.

[0151] In the edge fitting loss calculation, to facilitate contour point sampling, the three-dimensional contour is reduced to two dimensions, and the edge loss is calculated separately in the x, y, and z directions. The edge fitting loss formula is as follows:

[0152]

[0153] Where, loss Bound The loss represents the edge fitting loss. Bxy ω1 represents the two-dimensional edge fitting loss in the second-dimensional direction x and the third-dimensional direction y in three-dimensional space. Bxy Weights, loss Bxz ω² represents the two-dimensional edge fitting loss along the second-dimensional direction x and the first-dimensional direction z in three-dimensional space. Bxz Weights, loss Byz ω3 represents the 2D edge fitting loss in the third dimension y and the first dimension z in 3D space. Byz The weight, Representation and loss BxyThe corresponding two-dimensional Euclidean distance, P′ xy This indicates that the predicted contour sampling points are taken from the first dimension direction z. This indicates that the true contour sampling points are taken from the first dimension direction z. Representation and loss Bxz The corresponding two-dimensional Euclidean distance, P′ xz This indicates that the predicted contour sampling points are taken from the third dimension direction y. This indicates that the true contour sampling points are taken from the third dimension direction y. Representation and loss Byz The corresponding two-dimensional Euclidean distance, P′ yz This indicates that the predicted value contour sampling points are taken from the second dimension direction x. This indicates that the true contour sampling point is taken from the second dimension direction x, n represents the number of training stitched tomographic image blocks, and i represents the i-th training stitched tomographic image block.

[0154] Where, loss Bxy The weight ω1 can be set to 1, and the loss Bxz The weight ω2 can be set to 0.4, and the loss... Byz The weight ω3 can be set to 0.4.

[0155] In some embodiments, the predicted contour sampling points and the true contour sampling points are matched by overlap calculation.

[0156] In practice, the ground truth contour sampling points are matched with the corresponding predicted contour sampling points using the overlap calculation formula. This allows the edge fitting loss function to be calculated using the matched ground truth contour sampling points and the corresponding predicted contour sampling points. This enables the training of the feature edge enhancement model to adjust the activation values ​​of the feature regions corresponding to the feature prediction segmentation results in multiple tomographic image blocks.

[0157] In some embodiments, the predicted contour sampling points and the ground truth contour sampling points are matched by overlap calculation, including:

[0158] Step C1: Use the watershed algorithm to extract the connected components of the predicted value contours and the connected components of the ground truth contours in the training stitched tomographic image blocks.

[0159] Step C2: Determine the sampling points of the predicted value profile based on the connected components of the predicted value profile, and use the sampling points of the predicted value profile to determine the first minimum bounding rectangle.

[0160] Step C3: Determine the sampling points of the truth contour based on the connected components of the truth contour, and use the sampling points of the truth contour to determine the second minimum bounding rectangle.

[0161] Step C4: Calculate the overlap between the predicted value contour and the true value contour based on the first minimum bounding rectangle and the second minimum bounding rectangle using the following formula.

[0162]

[0163] Wherein, IoU represents the overlap between the predicted value contour and the true value contour, A represents the true value contour, Area(A) represents the area of ​​the second minimum bounding rectangle corresponding to the true value contour, B represents the predicted value contour, Area(B) represents the area of ​​the first minimum bounding rectangle corresponding to the predicted value contour, and Area(A∪B) represents the union of the area of ​​the second minimum bounding rectangle corresponding to the true value contour A and the area of ​​the first minimum bounding rectangle corresponding to the predicted value contour B.

[0164] Step A5: Match the ground truth contour and the predicted contour when the overlap is greater than a preset threshold to obtain a matching pair, and sample the matching pair according to a preset sampling interval to obtain the matched predicted contour sampling points and the ground truth contour sampling points.

[0165] In practice, for each dimension of the training stitched tomographic image block in three-dimensional space, the watershed algorithm is used to extract the connected components of the predicted contour and the ground truth contour respectively.

[0166] Based on the connected components of the predicted value contour, the sampling points of the predicted value contour on the connected components are extracted, and the corresponding minimum bounding rectangle (i.e., the first minimum bounding rectangle) is determined using the sampling points of the predicted value contour.

[0167] Based on the connected components of the truth contour, the sampling points of the truth contour on the connected components of the truth contour are extracted, and the corresponding minimum bounding rectangle (i.e. the second minimum bounding rectangle) is determined using the sampling points of the truth contour.

[0168] The overlap ratio (IoU) is calculated using the areas of the first and second minimum bounding rectangles, as shown in the following formula:

[0169]

[0170] Wherein, IoU represents the overlap between the predicted value contour and the true value contour, A represents the true value contour, Area(A) represents the area of ​​the second minimum bounding rectangle corresponding to the true value contour, B represents the predicted value contour, Area(B) represents the area of ​​the first minimum bounding rectangle corresponding to the predicted value contour, and Area(A∪B) represents the union of the area of ​​the second minimum bounding rectangle corresponding to the true value contour A and the area of ​​the first minimum bounding rectangle corresponding to the predicted value contour B.

[0171] For ground truth contours and predicted contours where the overlap IoU is greater than a preset threshold, they are paired. Here, the preset threshold is preferably 0.8, that is, ground truth contours and predicted contours are matched when the overlap IoU is greater than 0.8.

[0172] The formula for calculating the Intersection over Union (IoU) uses the minimum area of ​​the first and second minimum bounding rectangles to accommodate cases of inaccurate segmentation. For each ground truth contour A, if the number of matches in the predicted contour B is greater than 1, the union of multiple predicted contours is taken.

[0173] The matching pairs are sampled according to a preset sampling interval to obtain the matched predicted value contour sampling points and the true value contour sampling points.

[0174] For each pair of matched ground truth contours and predicted contours, the matched pair is sampled according to a preset sampling interval to obtain the matched predicted contour sampling points and the ground truth contour sampling points.

[0175] For example, the sampling points are the centroid of the contour region and m contour sampling points, for a total of m+1. The contour sampling points are uniformly sampled at angles with the centroid as the center, and the sampling angle interval is 360 / m.

[0176] In some embodiments, the feature edge enhancement model is obtained by training on a pre-acquired first training dataset and a second training dataset. The first training dataset includes a first tomographic image, a second tomographic image, and a third tomographic image. The second training dataset includes a second tomographic image. The first tomographic image, the second tomographic image, and the third tomographic image are tomographic images of different types.

[0177] In practice, a feature edge enhancement model is used. Based on the feature prediction segmentation results, the feature edges in the tomographic scan image corresponding to the feature prediction segmentation results are enhanced to obtain the feature edge enhancement results.

[0178] The feature edge enhancement model is obtained by pre-training on a first training dataset and a second training dataset. The first training dataset includes a first tomographic image, a second tomographic image, and a third tomographic image, while the second training dataset includes a second tomographic image.

[0179] For example, such as Figure 2A As shown, the feature edge enhancement model can be a second-stage (i.e., stage 2) 3D BR-Unet model.

[0180] First, the 3D BR-Unet model is self-supervised pre-trained using the first training dataset, which includes: normal lung tomographic images (i.e., the first tomographic image), patient lung tomographic images (i.e., the second tomographic image), and tomographic images of other lung diseases (i.e., the third tomographic image). Based on the image blocks of the normal lung tomographic images, patient lung tomographic images, and tomographic images of other lung diseases, corresponding sequences are predicted. Based on the texture structure of the image blocks, common features of different lung diseases are learned.

[0181] Then, transfer learning was performed on the pre-trained 3D BR-Unet model using the second training dataset, using only the patient's lung tomography images (i.e., the second tomography images).

[0182] In some embodiments, the feature edge enhancement model includes a location index prediction model, which is obtained by transfer learning after pre-training the location index prediction model using the first training dataset and then training it using the second training dataset.

[0183] In practice, the feature edge enhancement model includes a location index prediction model, which is based on an encoder structure.

[0184] The feature edge enhancement model is obtained by transfer learning after the encoder of the location index prediction model is pre-trained using the first training dataset.

[0185] For example, such as Figure 2A As shown, the feature edge enhancement model is the second stage (i.e., stage 2) of the 3DBR-Unet model.

[0186] First, the 3D BR-Unet model is self-supervised pre-trained using the first training dataset, which includes: normal lung tomographic images (i.e., the first tomographic image), patient lung tomographic images (i.e., the second tomographic image), and tomographic images of other lung diseases (i.e., the third tomographic image). Based on the image blocks of the normal lung tomographic images, patient lung tomographic images, and tomographic images of other lung diseases, corresponding sequences are predicted. Based on the texture structure of the image blocks, common features of different lung diseases are learned.

[0187] Then, transfer learning was performed on the pre-trained 3D BR-Unet model using the second training dataset, using only the patient's lung tomography images (i.e., the second tomography images).

[0188] For self-supervised pre-training of the 3D BR-Unet model using the first training dataset, such as Figure 2D As shown, the location index prediction model has an encoder structure as its backbone. The input is an image sub-block, and the output is a probability vector of the sub-block location index with a size of (9×1). It is transformed into the predicted sub-block location index by a normalized exponential function (softmax).

[0189] First, the computed tomography (CT) images are preprocessed, as follows:

[0190] ① Resampling normalizes voxels of different sizes to the same size (different devices and different acquisition protocols will produce tomographic images with different voxel spacing);

[0191] ② Adjust window width and window level (window width: CT value range and window level: center position of the window);

[0192] ③Remove the last few slices of tissue from the tomographic image sequence;

[0193] ④ Remove the black borders (zero-crop) from the tomographic scan image.

[0194] The preprocessed tomographic images are randomly cropped. The crop size in the xy direction is randomly set to 0.7 to 0.9 times the original image size, and the crop size in the z direction is set to 1 / 3 of the crop size in the xy direction (to avoid cropping in the z direction, which would increase the difficulty of sequence prediction). The cropped data is then sliced ​​into a 3×3 grid to obtain 9 sub-blocks, each with the same resolution. The order of the sub-blocks is shuffled, and they are sequentially input into nine networks. The position index prediction model processes each image sub-block independently with shared weights, and the 9 prediction indices are concatenated to obtain the sub-block sequence arrangement (since lung data has a relatively fixed content structure, the sequence prediction difficulty is lower than that of natural scene images, so the permutation set is not predefined).

[0195] The loss is calculated using edit distance, which is the minimum number of operations required to recover the true index from the predicted sequence index.

[0196] When performing transfer learning on the pre-trained 3D BR-Unet model using the second training dataset, 3D BR-Unet learns to segment specific lesions, using only data from COVID-19 lung infection patients and non-COVID-19 data based on negative sample mining, no longer directly using data from other lung diseases. Specifically, the non-COVID-19 data obtained through negative sample mining is obtained by pre-setting negative sample update phases [n1, n2...n]. When training reaches the corresponding epoch (training times), lesion segmentation is performed on normal tomographic images and other lung disease tomographic images, extracting data that missegmented specific lesions as negative samples and adding them to the training.

[0197] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0198] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0199] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a feature prediction device for tomographic scan images.

[0200] refer to Figure 3 The feature prediction device for the tomographic scan image includes:

[0201] The tomographic scan image acquisition module 301 is configured to acquire tomographic scan images;

[0202] The feature prediction and segmentation module 302 is configured to perform feature prediction and segmentation on the tomographic scan image to obtain the feature prediction and segmentation result.

[0203] The feature edge enhancement module 303 is configured to perform segmentation processing on the tomographic scan image based on the feature prediction segmentation result, and enhance the feature edges in the tomographic scan image corresponding to the feature prediction segmentation result during the segmentation process to obtain the feature edge enhancement result.

[0204] The overlay processing module 304 is configured to obtain a feature prediction result based on the feature edge enhancement result and the feature prediction segmentation result.

[0205] In some embodiments, the feature prediction and segmentation module 302 is specifically configured as follows:

[0206] The tomographic scan image is split in the first dimension of three-dimensional space according to a preset number of channels to obtain a split tomographic scan image.

[0207] The split tomographic image is downsampled in the second and third dimensions of three-dimensional space to obtain a downsampled tomographic image.

[0208] The downsampled tomographic image is subjected to feature prediction segmentation to obtain the downsampled feature prediction segmentation result;

[0209] The downsampled feature prediction segmentation result is upsampled to obtain the upsampled feature prediction segmentation result, and the upsampled feature prediction segmentation result is used as the feature prediction segmentation result.

[0210] In some embodiments, the feature prediction and segmentation module 302 is specifically configured as follows:

[0211] The tomographic scan image is classified and predicted to obtain the classification prediction result;

[0212] Based on the classification prediction result, the features in the feature prediction segmentation result that do not correspond to the classification prediction result are set to zero.

[0213] In some embodiments, the feature prediction segmentation result is obtained using a trained feature prediction segmentation model, which is pre-trained based on a pre-acquired first training dataset. The first training dataset includes a first tomographic image, a second tomographic image, and a third tomographic image. The first tomographic image, the second tomographic image, and the third tomographic image are tomographic images of different types. The feature prediction segmentation model includes a classification branch layer for classification prediction.

[0214] The loss function used for training the feature prediction and segmentation model includes:

[0215] Loss=α1L cls+α2L seg

[0216] Where Loss represents the loss function of the feature prediction segmentation model, L cls L represents the classification loss of the classification branch layer. cls =β1loss ce +β2loss con loss ce Represents cross-entropy loss, loss con Let β1 represent the cross-entropy loss, where β represents the contrast loss. ce The weights, β2 represents the contrastive loss. con The weight, L seg Let α1 represent the segmentation loss of the feature prediction segmentation model, and L represent the classification loss. cls The weights are α2, which represents the weights of the segmentation loss.

[0217] In some embodiments, the feature edge enhancement module 303 includes:

[0218] The fusion unit is configured to fuse the tomographic image and the feature prediction segmentation result to obtain a fused tomographic image.

[0219] The enhancement processing unit is configured to enhance the feature edges of the tomographic image based on the fused tomographic image to obtain a feature edge enhancement result.

[0220] In some embodiments, the feature edge enhancement result is performed using a trained feature edge enhancement model, the feature edge enhancement model including an edge enhancement network, the edge enhancement network including a first convolutional layer and a second convolutional layer, the first convolutional layer and the second convolutional layer being connected in series and skipped between the first convolutional layer and the second convolutional layer;

[0221] Enhanced processing unit, including:

[0222] The cutting subunit is configured to cut the fused tomographic image to obtain multiple tomographic image blocks;

[0223] The edge enhancement processing subunit is configured to perform edge enhancement processing on the feature regions in the plurality of tomographic image blocks that correspond to the feature prediction segmentation results through the edge enhancement network, so as to obtain feature edge enhancement results corresponding to the plurality of tomographic image blocks;

[0224] The stitching unit is configured to stitch together the feature edge enhancement results corresponding to multiple tomographic scan image blocks in the first dimension of three-dimensional space to obtain the feature edge enhancement result.

[0225] In some embodiments, the cutting subunit is specifically configured as follows:

[0226] The fused tomographic image is cut in the second and third dimensions of three-dimensional space according to a preset overlap rate to obtain multiple tomographic image blocks.

[0227] In some embodiments, the loss function used to train the feature edge enhancement model includes:

[0228] Loss=γ1loss Dice +γ2loss Bound

[0229] Where Loss represents the loss function of the feature edge enhancement model, loss Dice This represents Dice loss. Bound γ represents the edge fitting loss, and γ1 represents the Dice loss. Dice The weights, γ2 represents the edge fitting loss. Bound The weights;

[0230] edge fitting loss Bound Specifically, it includes:

[0231]

[0232] Where, loss Bound The loss represents the edge fitting loss. Bxy ω1 represents the two-dimensional edge fitting loss in the second-dimensional direction x and the third-dimensional direction y in three-dimensional space. Bxy Weights, loss Bxz ω² represents the 2D edge fitting loss along the second-dimensional direction x and the first-dimensional direction z in 3D space, where ω² represents the loss. Bxz Weights, loss Byz ω3 represents the 2D edge fitting loss in the third dimension y and the first dimension z in 3D space. Byz The weight, Representation and loss Bxy The corresponding two-dimensional Euclidean distance, P′ xy This indicates that the predicted contour sampling points are taken from the first dimension direction z. This indicates that the true contour sampling points are taken from the first dimension direction z. Representation and loss Bxz The corresponding two-dimensional Euclidean distance, P′ xz This indicates that the predicted contour sampling points are taken from the third dimension direction y. This indicates that the true contour sampling points are taken from the third dimension direction y. Representation and loss Byz The corresponding two-dimensional Euclidean distance, P′ yz This indicates that the predicted value contour sampling points are taken from the second dimension direction x. This indicates that the true contour sampling point is taken from the second dimension direction x, n represents the number of training stitched tomographic image blocks, and i represents the i-th training stitched tomographic image block.

[0233] In some embodiments, the feature prediction device for the tomographic scan image further includes a matching module, the matching module comprising:

[0234] The matching unit is configured to match the predicted value contour sampling points with the true value contour sampling points by calculating the overlap.

[0235] In some embodiments, the matching unit is specifically configured as follows:

[0236] The watershed algorithm is used to extract the connected components of the predicted value contours and the connected components of the ground truth contours in the training stitched tomographic image blocks, respectively.

[0237] Based on the connected components of the predicted value profile, the sampling points of the predicted value profile are determined, and the sampling points of the predicted value profile are used to determine the first minimum bounding rectangle.

[0238] Based on the connected components of the truth contour, the sampling points of the truth contour are determined, and the sampling points of the truth contour are used to determine the second minimum bounding rectangle.

[0239] The overlap between the predicted value contour and the true value contour is calculated using the following formula based on the first minimum bounding rectangle and the second minimum bounding rectangle:

[0240]

[0241] Wherein, IoU represents the overlap between the predicted value contour and the true value contour, A represents the true value contour, Area(A) represents the area of ​​the second minimum bounding rectangle corresponding to the true value contour, B represents the predicted value contour, Area(B) represents the area of ​​the first minimum bounding rectangle corresponding to the predicted value contour, and Area(A∪B) represents the union of the area of ​​the second minimum bounding rectangle corresponding to the true value contour A and the area of ​​the first minimum bounding rectangle corresponding to the predicted value contour B.

[0242] The ground truth contour and the predicted contour when the overlap is greater than a preset threshold are matched to obtain a matching pair. The matching pair is then sampled according to a preset sampling interval to obtain the matched predicted contour sampling points and the ground truth contour sampling points.

[0243] In some embodiments, the feature edge enhancement model is obtained by training on a pre-acquired first training dataset and a second training dataset. The first training dataset includes a first tomographic image, a second tomographic image, and a third tomographic image. The second training dataset includes a second tomographic image. The first tomographic image, the second tomographic image, and the third tomographic image are tomographic images of different types.

[0244] In some embodiments, the feature edge enhancement model includes a location index prediction model, which is obtained by transfer learning after pre-training the location index prediction model using the first training dataset and then training it using the second training dataset.

[0245] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0246] The apparatus of the above embodiments is used to implement the feature prediction method of the corresponding tomographic scan image in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0247] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the feature prediction method for tomographic scan images described in any of the above embodiments.

[0248] Figure 4 This illustration shows a more specific hardware structure diagram of an electronic device provided in this embodiment. The device may include: a processor 401, a memory 402, an input / output interface 403, a communication interface 404, and a bus 405. The processor 401, memory 402, input / output interface 403, and communication interface 404 are interconnected internally via the bus 405.

[0249] The processor 401 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0250] The memory 402 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 402 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 402 and is called and executed by the processor 401.

[0251] Input / output interface 403 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0252] Communication interface 404 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0253] Bus 405 includes a pathway for transmitting information between various components of the device (e.g., processor 401, memory 402, input / output interface 403, and communication interface 404).

[0254] It should be noted that although the above-described device only shows the processor 401, memory 402, input / output interface 403, communication interface 404, and bus 405, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0255] The electronic devices described above are used to implement the feature prediction method for corresponding tomographic scan images in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0256] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the feature prediction method for computed tomography images as described in any of the above embodiments.

[0257] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0258] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the feature prediction method of the tomographic scan image as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0259] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0260] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0261] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0262] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A feature prediction method for tomographic scan images, characterized in that, include: Acquire tomographic images; The tomographic scan image is subjected to feature prediction segmentation to obtain the feature prediction segmentation result; Based on the feature prediction segmentation result, the tomographic scan image is segmented, and during the segmentation process, the feature edges in the tomographic scan image corresponding to the feature prediction segmentation result are enhanced to obtain the feature edge enhancement result. Based on the feature edge enhancement result and the feature prediction segmentation result, the feature prediction result is obtained; wherein, The step of enhancing the feature edges in the tomographic scan image corresponding to the feature prediction segmentation result during the segmentation process to obtain the feature edge enhancement result includes: The tomographic scan image and the feature prediction segmentation result are fused to obtain a fused tomographic scan image. Based on the fused tomographic image, the feature edges of the tomographic image are enhanced to obtain the feature edge enhancement result; The feature edge enhancement result is obtained using a trained feature edge enhancement model. The loss function used to train the feature edge enhancement model includes: in, This represents the loss function of the feature edge enhancement model. This indicates Dice's loss. This represents the edge fitting loss. Indicates Dice loss The weight, Represents edge fitting loss The weights; Edge fitting loss Specifically, it includes: in, This represents the edge fitting loss. Represents the second dimension direction in three-dimensional space. and the third dimension direction Two-dimensional edge fitting loss on the surface, express The weight, Represents the second dimension direction in three-dimensional space. and the first dimension direction Two-dimensional edge fitting loss on the surface, express The weight, Represents the third dimension direction in three-dimensional space. and the first dimension direction Two-dimensional edge fitting loss on the surface, express The weight, Indicates and The corresponding two-dimensional Euclidean distance, Indicates direction from the first dimension Take the predicted value contour sampling points, Indicates direction from the first dimension Take the true contour sampling points. Indicates and The corresponding two-dimensional Euclidean distance, Indicates direction from the third dimension Take the predicted value contour sampling points, Indicates direction from the third dimension Take the true contour sampling points. Indicates and The corresponding two-dimensional Euclidean distance, Indicates direction from the second dimension Take the predicted value contour sampling points, Indicates direction from the second dimension Take the true contour sampling points. This indicates the number of stitched tomographic image patches used in the training process. Indicates the first A training patch of stitched tomographic images.

2. The method according to claim 1, characterized in that, The step of performing feature prediction segmentation on the tomographic scan image to obtain the feature prediction segmentation result includes: The tomographic scan image is split in the first dimension of three-dimensional space according to a preset number of channels to obtain a split tomographic scan image. The split tomographic image is downsampled in the second and third dimensions of three-dimensional space to obtain a downsampled tomographic image. The downsampled tomographic image is subjected to feature prediction segmentation to obtain the downsampled feature prediction segmentation result; The downsampled feature prediction segmentation result is upsampled to obtain the upsampled feature prediction segmentation result, and the upsampled feature prediction segmentation result is used as the feature prediction segmentation result.

3. The method according to claim 2, characterized in that, The step of performing feature prediction segmentation on the tomographic scan image to obtain the feature prediction segmentation result includes: The tomographic scan image is classified and predicted to obtain the classification prediction result; Based on the classification prediction result, the features in the feature prediction segmentation result that do not correspond to the classification prediction result are set to zero.

4. The method according to claim 2, characterized in that, The feature prediction segmentation result is obtained using a trained feature prediction segmentation model. The feature prediction segmentation model is trained based on a pre-acquired first training dataset, which includes a first tomographic image, a second tomographic image, and a third tomographic image. The first tomographic image, the second tomographic image, and the third tomographic image are tomographic images of different types. The feature prediction segmentation model includes a classification branch layer for classification prediction. The loss function used for training the feature prediction and segmentation model includes: in, This represents the loss function of the feature prediction and segmentation model. This represents the classification loss of the classification branch layer. , Represents cross-entropy loss, Indicates comparative loss, Represents cross-entropy loss The weight, Indicates comparative loss The weight, This represents the segmentation loss of the feature prediction segmentation model. Represents classification loss The weight, This represents the weight of the segmentation loss.

5. The method according to claim 1, characterized in that, The feature edge enhancement model includes an edge enhancement network, which includes a first convolutional layer and a second convolutional layer. The first convolutional layer and the second convolutional layer are connected in series, and the first convolutional layer and the second convolutional layer are connected in skip connections. The enhancement of the feature edges of the tomographic image based on the fused tomographic image to obtain the feature edge enhancement result includes: The fused tomographic image is segmented to obtain multiple tomographic image blocks; The edge enhancement network is used to perform edge enhancement processing on the feature regions in the multiple tomographic image blocks that correspond to the feature prediction and segmentation results, so as to obtain the feature edge enhancement results corresponding to the multiple tomographic image blocks. In the first dimension of three-dimensional space, the feature edge enhancement results corresponding to multiple tomographic scan image blocks are stitched together to obtain the feature edge enhancement result.

6. The method according to claim 5, characterized in that, The process of segmenting the fused tomographic image to obtain multiple tomographic image blocks includes: The fused tomographic image is cut in the second and third dimensions of three-dimensional space according to a preset overlap rate to obtain multiple tomographic image blocks.

7. The method according to claim 6, characterized in that, The predicted value contour sampling points and the true value contour sampling points are matched by overlap calculation.

8. The method according to claim 7, characterized in that, The predicted value contour sampling points and the true value contour sampling points are matched by overlap calculation, including: The watershed algorithm is used to extract the connected components of the predicted value contours and the connected components of the ground truth contours in the training stitched tomographic image blocks, respectively. Based on the connected components of the predicted value profile, the sampling points of the predicted value profile are determined, and the sampling points of the predicted value profile are used to determine the first minimum bounding rectangle. Based on the connected components of the truth contour, the sampling points of the truth contour are determined, and the sampling points of the truth contour are used to determine the second minimum bounding rectangle. The overlap between the predicted value contour and the true value contour is calculated using the following formula based on the first minimum bounding rectangle and the second minimum bounding rectangle: in, This indicates the degree of overlap between the predicted value profile and the true value profile. Represents the truth profile. Let represent the area of ​​the second smallest bounding rectangle corresponding to the truth contour. Represents the profile of the predicted values. This represents the area of ​​the first minimum bounding rectangle corresponding to the predicted value profile. Representing the truth profile The area of ​​the corresponding second minimum bounding rectangle and the predicted profile The union of the areas of the corresponding first minimum bounding rectangles; The ground truth contour and the predicted contour when the overlap is greater than a preset threshold are matched to obtain a matching pair. The matching pair is then sampled according to a preset sampling interval to obtain the matched predicted contour sampling points and the ground truth contour sampling points.

9. The method according to claim 1, characterized in that, The feature edge enhancement model is obtained by training on a first training dataset and a second training dataset obtained in advance. The first training dataset includes a first tomographic image, a second tomographic image, and a third tomographic image. The second training dataset includes a second tomographic image. The first tomographic image, the second tomographic image, and the third tomographic image are tomographic images of different types.

10. The method according to claim 9, characterized in that, The feature edge enhancement model includes a location index prediction model, which is obtained by training the location index prediction model using the second training dataset after pre-training it using the first training dataset.

11. A feature prediction device for tomographic images, characterized in that, include: The tomographic image acquisition module is configured to acquire tomographic images; The feature prediction and segmentation module is configured to perform feature prediction and segmentation on the tomographic scan image to obtain the feature prediction and segmentation result. The feature edge enhancement network is configured to segment the tomographic image based on the feature prediction segmentation result, and enhance the feature edges in the tomographic image corresponding to the feature prediction segmentation result during the segmentation process to obtain the feature edge enhancement result. The overlay processing module is configured to obtain a feature prediction result based on the feature edge enhancement result and the feature prediction segmentation result; wherein, The feature edge enhancement network is further configured to fuse the tomographic image and the feature prediction segmentation result to obtain a fused tomographic image; Based on the fused tomographic image, the feature edges of the tomographic image are enhanced to obtain the feature edge enhancement result; The feature edge enhancement result is obtained using a trained feature edge enhancement model. The loss function used to train the feature edge enhancement model includes: in, This represents the loss function of the feature edge enhancement model. This indicates Dice's loss. This represents the edge fitting loss. Indicates Dice loss The weight, Represents edge fitting loss The weights; Edge fitting loss Specifically, it includes: in, This represents the edge fitting loss. Represents the second dimension direction in three-dimensional space. and the third dimension direction Two-dimensional edge fitting loss on the surface, express The weight, Represents the second dimension direction in three-dimensional space. and the first dimension direction Two-dimensional edge fitting loss on the surface, express The weight, Represents the third dimension direction in three-dimensional space. and the first dimension direction Two-dimensional edge fitting loss on the surface, express The weight, Indicates and The corresponding two-dimensional Euclidean distance, Indicates direction from the first dimension Take the predicted value contour sampling points, Indicates direction from the first dimension Take the true contour sampling points. Indicates and The corresponding two-dimensional Euclidean distance, Indicates direction from the third dimension Take the predicted value contour sampling points, Indicates direction from the third dimension Take the true contour sampling points. Indicates and The corresponding two-dimensional Euclidean distance, Indicates direction from the second dimension Take the predicted value contour sampling points, Indicates direction from the second dimension Take the true contour sampling points. This indicates the number of stitched tomographic image patches used in the training process. Indicates the first A training patch of stitched tomographic images.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method as described in any one of claims 1 to 10.

Images