Region-of-interest segmentation method and device and computer equipment

Abstract

The invention relates to a region-of-interest segmentation method and device and computer equipment. The method comprises the following steps: acquiring a first modal image and a second modal image of a target part; extracting a first image feature in the first modal image through a first modal feature extraction network, and extracting a second image feature in the second modal image through a second modal feature extraction network; the second modal feature extraction network is obtained by updating the network structure of a network having the same framework as the first modal feature extraction network; and obtaining a segmentation result of the region of interest in the target part according to the first image feature and the second image feature. By adopting the method, the region of interest can be accurately segmented.

Description

Technical Field

[0001] The present application relates to the technical field of image segmentation, and in particular to a method, device and computer equipment for segmenting a region of interest. Background Art

[0002] With the continuous development of image segmentation technology, image segmentation is increasingly used in the segmentation process of medical images.

[0003] In the related art, the region of interest in the target area is segmented using a multimodal image, and the region of interest obtained is more accurate. Taking multimodal images including positron emission tomography (PET) and computed tomography (CT) as an example, the PET sample image and the CT sample image are combined into a tensor training image segmentation model, and the trained image segmentation model is used to segment the region of interest in the target area to obtain the segmentation result of the region of interest.

[0004] However, the related art has the problem that the segmentation results of the region of interest are inaccurate. Summary of the invention

[0005] Based on this, it is necessary to provide a method, an apparatus and a computer device for segmenting a region of interest in order to solve the above technical problems, so as to accurately segment the region of interest.

[0006] In a first aspect, the present application provides a method for segmenting a region of interest, the method comprising:

[0007] Acquiring a first modality image and a second modality image of a target part;

[0008] A first image feature in a first modality image is extracted by a first modality feature extraction network, and a second image feature in a second modality image is extracted by a second modality feature extraction network; the second modality feature extraction network is obtained by updating a network structure having the same framework as the first modality feature extraction network;

[0009] A segmentation result of a region of interest in the target part is obtained according to the first image feature and the second image feature.

[0010] In one embodiment, the process of constructing the first modality feature extraction network includes:

[0011] Acquire a first modality image sample of the target part;

[0012] The first modality image samples are used to train the first modality feature extraction initial network to obtain the first modality feature extraction network.

[0013] In one embodiment, the process of constructing the second modality feature extraction network includes:

[0014] Acquire a second modality image sample of the target area;

[0015] Adjusting the network structure of the first modal feature extraction network to obtain a second modal feature extraction initial network;

[0016] Based on the second modality image samples, the network structure of the second modality feature extraction initial network is updated until the updated second modality feature extraction initial network does not meet the network structure update condition, thereby obtaining the second modality feature extraction network.

[0017] In one embodiment, the network structure of the first modal feature extraction network is adjusted to obtain a second modal feature extraction initial network, including:

[0018] Reducing the size of each feature extraction layer of the first modal feature extraction network; the size of each feature extraction layer includes the number of neurons or the number of channels;

[0019] The reduced first modality feature extraction network is pre-trained using the second modality image samples until the pre-training conditions are met to obtain the second modality feature extraction initial network.

[0020] In one embodiment, based on the second modality image sample, updating the network structure of the second modality feature extraction initial network includes:

[0021] When the initial network for extracting the second modal features satisfies the network structure update condition, the network scale of the initial network for extracting the second modal features is expanded;

[0022] The enlarged second modality feature extraction initial network is trained to convergence using second modality image samples.

[0023] In one embodiment, the method further comprises:

[0024] Extracting features between pixels in the first modality image sample using the first modality feature extraction network to obtain a plurality of first modality image features;

[0025] Using a second modality feature extraction network to extract features between pixels in the second modality image sample, to obtain a plurality of second modality image features;

[0026] Based on each first modality image feature and each second modality image feature, it is determined whether the updated second modality feature extraction initial network meets the network structure update condition.

[0027] In one embodiment, based on each first modality image feature and each second modality image feature, determining whether the updated second modality feature extraction initial network meets the network structure update condition includes:

[0028] Determine a feature contribution quantization value of the second modality image sample based on each first modality image feature and each second modality image feature; the feature contribution quantization value indicates the contribution degree of the second modality image to the segmentation task;

[0029] When the difference between two adjacent updated feature contribution quantization values ​​is less than or equal to a preset threshold, it is determined that the updated second modality feature extraction initial network meets the network structure update condition.

[0030] In one embodiment, determining a feature contribution quantization value of the second modality image based on each first modality image feature and each second modality image feature includes:

[0031] Obtaining the total feature weight of each first modality image feature and each second modality image feature in the segmentation task;

[0032] The ratio of the sum of the feature weights of each second modality image feature in the segmentation task to the total feature weight is used as the feature contribution quantification value of the second modality image sample.

[0033] In a second aspect, the present application further provides a device for segmenting a region of interest, the device comprising:

[0034] An acquisition module, used for acquiring a first modality image and a second modality image of a target part;

[0035] An extraction module, used for extracting a first image feature in a first modality image through a first modality feature extraction network, and extracting a second image feature in a second modality image through a second modality feature extraction network; the second modality feature extraction network is obtained by updating a network structure having the same framework as the first modality feature extraction network;

[0036] The segmentation result determination module is used to obtain the segmentation result of the region of interest in the target part according to the first image feature and the second image feature.

[0037] In a third aspect, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the content of any one of the embodiments of the method for segmenting a region of interest in the first aspect is implemented.

[0038] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the content of any one of the embodiments of the method for segmenting a region of interest in the first aspect.

[0039] In a fifth aspect, the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the content of any one of the embodiments of the method for segmenting a region of interest in the first aspect.

[0040] The above-mentioned method, device and computer equipment for segmenting the region of interest obtain the first modality image and the second modality image of the target part, extract the first image feature in the first modality image through the first modality feature extraction network, and extract the second image feature in the second modality image through the second modality feature extraction network; the second modality feature extraction network is obtained by updating the network structure of the network with the same framework as the first modality feature extraction network; according to the first image feature and the second image feature, the segmentation result of the region of interest in the target part is obtained. The method can accurately obtain the image features of the first modality image through the first modality feature extraction network, and by updating the network structure of the network with the same framework as the first modality feature extraction network, the obtained second modality feature extraction network is more suitable for extracting the image features of the second modality, avoiding insufficient extraction of the image features of the second modality or overfitting, and improving the performance and prediction accuracy of the second modality feature extraction network. Then, the image features of the second modality image can be more accurately extracted through the second modality feature extraction network. According to the image features of different modalities, the region of interest of the target part can be more accurately determined from different dimensions, so that the region of interest can be accurately segmented. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A diagram showing an application environment of a method for segmenting a region of interest in one embodiment;

[0042] Figure 2 is a schematic flow chart of a method for segmenting a region of interest in one embodiment;

[0043] Figure 3 is a schematic flow chart of a method for segmenting a region of interest in one embodiment;

[0044] Figure 4 is a schematic flow chart of a method for segmenting a region of interest in one embodiment;

[0045] Figure 5 is a schematic flow chart of a method for segmenting a region of interest in one embodiment;

[0046] Figure 6 is a schematic flow chart of a method for segmenting a region of interest in one embodiment;

[0047] Figure 7 Schematic diagram of updating the Lth layer of neurons or channels in one embodiment;

[0048] Figure 8 is a schematic flow chart of a method for segmenting a region of interest in one embodiment;

[0049] Fig. 9 is a schematic flow chart of a method for segmenting a region of interest in one embodiment;

[0050] Fig.10 is a schematic flow chart of a method for segmenting a region of interest in one embodiment;

[0051] Fig.11 is a schematic flow chart of a method for segmenting a region of interest in one embodiment;

[0052] Fig.12 is a schematic flow chart of a method for segmenting a region of interest in one embodiment;

[0053] Fig.13 is a schematic flow chart of a method for segmenting a region of interest in one embodiment;

[0054] Fig.14 is a schematic flow chart of a method for segmenting a region of interest in one embodiment;

[0055] Fig.15 FIG. 4 is a structural block diagram of a device for segmenting a region of interest in an embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] Before introducing the technical solution of the present application in detail, a brief introduction to the background technology of the present application is first given.

[0058] With the continuous development of image segmentation technology, image segmentation technology is increasingly used in the segmentation process of medical images. Generally, the medical image of a single modality of the target part is segmented by an image segmentation model to determine the region of interest in the target part.

[0059] Since a single medical image contains limited image information, when segmenting the region of interest of a target part, feature information of medical images of the target part under multiple modalities can be extracted, and multiple feature information can be fused to accurately segment the region of interest in the target part.

[0060] Taking medical images of different modalities including PET images and CT images as examples, in one method, the respective feature networks are trained and features are extracted through PET and CT images respectively. However, this method greatly increases the difficulty of adjusting network parameters, which often leads to insufficient feature extraction or overfitting problems. In another method, the PET sample image and the CT sample image are combined into a tensor training image segmentation model, and the trained image segmentation model is used to segment the region of interest in the target area to obtain the segmentation result of the region of interest. In the process of combining the PET sample image and the CT sample image, this method generally unifies the pixel value of the CT sample image to the same order of magnitude as the pixel value of the PET sample image, which will cause some information in the CT sample image to be lost, that is, some features of the CT are lost.

[0061] In view of the above problems, the present application provides a method for segmenting a region of interest. The method obtains different feature extraction networks by automatically updating the network structure, so that medical images under different modalities can be extracted more comprehensively and fully, and the region of interest can be accurately segmented based on the extracted image features. The specific content of the method for segmenting a region of interest provided by the present application is introduced below.

[0062] The region of interest segmentation method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. For example, the computer device may be a server, a personal computer, a laptop, a smart phone, a tablet computer, a smart mobile phone, etc. The computer device may include a processor, a memory and a network interface connected by a system bus or wirelessly. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data in the process of segmenting the region of interest. The network interface of the computer device is used to communicate with an external terminal through a network connection, and the computer program is executed by the processor to implement a method for segmenting the region of interest. Among them, the computer device can be implemented by an independent computer device or a computer device cluster composed of multiple computer devices. It should be noted that the memory of the computer device is not limited to the above-mentioned memory, and may also include a high-speed random access memory, a volatile solid-state memory, etc. In addition, the composition architecture of the computer device is not limited to the above-mentioned situation, and some components may be added or omitted.

[0063] In one embodiment, Figure 2 As shown, a method for segmenting a region of interest is provided, which is applied to Figure 1 The computer device in the example is used to illustrate, including the following steps:

[0064] S201, acquiring a first modality image and a second modality image of a target part.

[0065] The target part refers to any part on the simulated body or the prosthesis, for example, the target part can be the brain, stomach or intestines. The first modality image and the second modality image refer to images of the same part in different modalities. The first modality image can be a PET image, and the second modality image can be a CT image or a Magnetic Resonance Imaging (MRI) image.

[0066] Optionally, the computer device may obtain the first modality image and the second modality image of the target part in different modalities from a Picture Archiving and Communication System (PACS). Alternatively, the computer device may also directly obtain the first modality image and the second modality image of the target part in different modalities from different medical imaging devices. For example, the computer device may obtain the first modality image from a PET imaging device, and obtain the second modality image from a CT imaging device or an MRI imaging device.

[0067] S202, extracting a first image feature in a first modality image through a first modality feature extraction network, and extracting a second image feature in a second modality image through a second modality feature extraction network; the second modality feature extraction network is obtained by updating the network structure of a network having the same framework as the first modality feature extraction network.

[0068] The first modality feature extraction network is obtained by training and adjusting parameters through the image samples of the first modality. The second modality feature extraction network is obtained by updating the network structure of the network with the same network framework as the first modality feature extraction network through the image samples of the second modality. The network model of the first modality feature extraction network can be a fully convolutional network (Fully Convolutional Networks), such as a V-Net network, a U-Net network, etc., or a generative adversarial network (Generative Adversarial Nets), such as a pix2pix network, a WGAN network, etc.

[0069] In this embodiment, when the first modality image is a PET image, a relatively mature feature extraction network has already appeared for the feature extraction of PET images. Therefore, one of these relatively mature networks can be directly selected as the first modality feature extraction initial network. Afterwards, the first modality feature extraction initial network is trained and adjusted using the image samples of the first modality to obtain the first modality feature extraction network. The computer device can input the first modality image into the first modality feature extraction network, and use the first modality feature extraction network to extract the image features in the first modality image to obtain the first modality image features. Among them, when the first modality image is a PET image, the first image feature is the image feature of the PET image.

[0070] In addition, the computer device can also input the second modality image into the second modality feature extraction network, and use the second modality feature extraction network to extract the image features of the second modality image to obtain the second modality image features. Wherein, when the second modality image is a CT image, the second image features are the image features of the CT image; when the second modality image is an MRI image, the second image features are the image features of the MRI image.

[0071] S203: Obtain a segmentation result of a region of interest in the target part according to the first image feature and the second image feature.

[0072] Optionally, the computer device may fuse the information of the first image feature and the second image feature into one image feature. For example, the first image feature may be fused into the second image feature to determine the probability that each pixel on the second modality image is a region of interest. Alternatively, the second image feature may be fused into the first image feature to determine the probability that each pixel on the first modality image is a region of interest. Based on the probability value and the preset probability threshold, the region of interest in the target part may be determined. Optionally, the computer device may also analyze the first modality image based on the first image feature to determine the region of interest in the first modality image; and, based on the second image feature, analyze the second modality image to determine the region of interest in the second modality image. Afterwards, the region of interest in the first modality image and the region of interest in the second modality image are fused, and the fused region of interest is determined as the region of interest in the target part. Afterwards, the region of interest in the target part may be segmented by a segmentation algorithm to obtain a segmentation result of the region of interest in the target part. This embodiment does not limit the method of determining the region of interest in the target part according to the first image feature and the second image feature.

[0073] In the above-mentioned region of interest segmentation method, the first modality image and the second modality image of the target part are obtained, the first image feature in the first modality image is extracted by the first modality feature extraction network, and the second image feature in the second modality image is extracted by the second modality feature extraction network; the second modality feature extraction network is obtained by updating the network structure of the network with the same framework as the first modality feature extraction network; according to the first image feature and the second image feature, the segmentation result of the region of interest in the target part is obtained. The method can accurately obtain the image features of the first modality image through the first modality feature extraction network, and by updating the network structure of the network with the same framework as the first modality feature extraction network, the obtained second modality feature extraction network is more suitable for extracting the image features of the second modality, avoiding insufficient extraction of the image features of the second modality or overfitting, and improving the performance and prediction accuracy of the second modality feature extraction network. Then, the image features of the second modality image can be more accurately extracted through the second modality feature extraction network. According to the image features of different modalities, the region of interest of the target part can be more accurately determined from different dimensions, so that the region of interest can be accurately segmented.

[0074] Based on the above embodiment, this embodiment introduces and explains the relevant contents of the construction process of the above first modality feature extraction network. Figure 3 As shown, as a non-limiting example, the above method may also include the following content:

[0075] S301, obtaining a first modality image sample of a target part.

[0076] In this embodiment, the computer device can obtain the first modality images of the target parts of different users from the PACS, and use multiple first modality images as the first modality image samples of the target parts. Alternatively, the computer device can also obtain the public first modality images of the target parts of different users, and use a large number of public first modality images and a large number of historical first modality images as the first modality image samples of the target parts. Among them, various image features in the first modality image samples are annotated by doctors, providing a "gold standard" for the first modality feature extraction network.

[0077] S302: Use the first modality image samples to train the first modality feature extraction initial network to obtain the first modality feature extraction network.

[0078] Among them, the first modality feature extraction initial network is a relatively mature network model that is publicly available for feature extraction of first modality images.

[0079] In this embodiment, the computer device can use the first modality image samples as training samples, and use the first modality image samples to train the first modality feature extraction initial network until the prediction result output by the initial first modality feature extraction network meets the preset conditions, thereby obtaining the first modality feature extraction network. It should be noted that the training process is to adjust the training parameters of the first modality feature extraction initial network through the first modality image samples. For example, the training parameters may be a learning rate, a loss function, etc.

[0080] In the above-mentioned region of interest segmentation method, a first modality image sample of the target part is obtained, and the first modality image sample is used to train the first modality feature extraction initial network to obtain the first modality feature extraction network. This method trains the first modality feature extraction initial network through the first modality image sample, so that the obtained first modality feature extraction network is more accurate, so that the first modality feature extraction network can be used to more accurately extract image features from the first modality image.

[0081] Based on the above embodiment, this embodiment introduces and explains the relevant contents of the construction process of the above second modality feature extraction network. Figure 4 As shown, as a non-limiting example, the above method may also include the following content:

[0082] S401, obtaining a second modality image sample of a target part.

[0083] In this embodiment, the computer device can obtain the second modality images of the target part of different users from the PACS in the same manner as the aforementioned first modality image samples, and use the plurality of second modality images as the second modality image samples of the target part. Alternatively, the computer device can also obtain the public second modality images of the target part, and use the public second modality images and the second modality images in the PACS as the second modality image samples of the target part.

[0084] S402, adjusting the network structure of the first modal feature extraction network to obtain a second modal feature extraction initial network.

[0085] In this embodiment, the second modal feature extraction initial network is obtained by adjusting the network structure of the first modal feature extraction network. The computer device can adjust the size of the network structure of the trained first modal feature extraction network, or can also adjust the network structure of any layer in the trained first modal feature extraction network to obtain the second modal feature extraction initial network after adjustment. This embodiment does not limit the method of extracting the second modal feature initial network.

[0086] S403, based on the second modality image sample, updating the network structure of the second modality feature extraction initial network until the updated second modality feature extraction initial network does not meet the network structure update condition, thereby obtaining the second modality feature extraction network.

[0087] The update condition may be that the size of the network structure is smaller than a preset threshold, that the network structure of any layer satisfies the preset condition, or that the learning speed of each layer of the network structure is greater than a preset speed threshold.

[0088] In this embodiment, after the network structure adjustment of the first modality feature extraction network is completed, the computer device can use the second modality image to update the network structure of the second modality feature extraction initial network until the feature extraction result of the second modality feature extraction initial network meets the update condition, and then stop updating the second modality feature extraction initial network to obtain the second modality feature extraction network.

[0089] In the above-mentioned region of interest segmentation method, a second modality image sample of the target part is obtained, and the network structure of the first modality feature extraction network is adjusted to obtain a second modality feature extraction initial network; based on the second modality image sample, the network structure of the second modality feature extraction initial network is updated until the updated second modality feature extraction initial network does not meet the network structure update condition, and the second modality feature extraction network is obtained. This method makes the obtained second modality feature extraction initial network more suitable for the training process of the second modality image sample by adjusting the network structure of the first modality feature extraction network. Afterwards, the second modality image sample is used to update the second modality feature extraction initial network, so that the obtained second modality feature extraction network is more accurate, so that the features of the second modality image can be accurately extracted.

[0090] Based on the above embodiments, this embodiment is to Figure 4 The relevant contents of step S402 "adjusting the network structure of the first modal feature extraction network to obtain the second modal feature extraction initial network" are introduced and explained. Figure 5 As shown, as a non-limiting example, the above step S402 may include the following content:

[0091] S501, reducing the size of each feature extraction layer of the first modal feature extraction network; the size of each feature extraction layer includes the number of neurons or the number of channels.

[0092] In this embodiment, the computer device can determine the reduction coefficient of the network scale of each feature extraction layer of the first modal feature extraction network based on historical experience. And based on multiple reduction coefficients, the network scale of the corresponding feature extraction layer in the first modal feature extraction network is reduced to obtain a reduced first modal feature extraction network. The reduced first modal feature extraction network is a shallower network, but its network framework is consistent with the network framework of the first modal feature extraction network. For example, the reduction coefficient of a feature extraction layer can be 60%, that is, the number of neurons or channel data in the feature extraction layer is reduced by 60%. The reduction coefficients of different feature extraction layers can be the same or different.

[0093] S502, pre-training the reduced first modality feature extraction network using the second modality image samples until the pre-training conditions are met to obtain the second modality feature extraction initial network.

[0094] In this embodiment, after obtaining the reduced first modality feature extraction network, the computer device can use the second modality image samples to pre-train the reduced first modality feature extraction network. If the feature extraction result of the reduced first modality feature extraction network meets the preset conditions, it is determined that the pre-training process is completed and the second modality feature extraction initial network is obtained.

[0095] In the above-mentioned region of interest segmentation method, the scale of each feature extraction layer of the first modality feature extraction network is reduced; the scale of each feature extraction layer includes the number of neurons or the number of channels; the reduced first modality feature extraction network is pre-trained using the second modality image sample until the pre-training conditions are met to obtain the second modality feature extraction initial network. This method can reduce the influence of the feature extraction of the first modality image sample on the feature extraction of the second modality image sample by reducing the scale of each feature extraction layer of the first modality feature extraction network. The reduced first modality feature extraction network is pre-trained using the second modality image sample to ensure that the trained second modality feature extraction initial network can be applied to the feature extraction process of the second modality image sample.

[0096] Based on the above embodiments, this embodiment is to Figure 4 The relevant contents of step S403 "updating the network structure of the initial network for extracting the second modality features based on the second modality image sample" are introduced and explained. Figure 6 As shown, as a non-limiting example, the above step S403 may include the following content:

[0097] S601, when the second modal feature extraction initial network meets the network structure update condition, the network scale of the second modal feature extraction initial network is expanded.

[0098] In this embodiment, since the adjustment process of the first modal feature extraction network includes a network scale reduction and a pre-training process, after the adjustment is completed, each update process of the second modal feature extraction initial network includes a network scale expansion process and a training process. For the network scale expansion process, the computer device can determine the expansion coefficient of each update process based on historical experience. The expansion coefficients of multiple update processes can be the same or different. Afterwards, the computer device expands the network scale of the second modal feature extraction initial network based on the expansion coefficient to obtain an expanded second modal feature extraction initial network. For example, the expansion coefficient of the second modal feature extraction initial network can be 20% or 30%.

[0099] S602: Train the expanded second modality feature extraction initial network through the second modality image samples until convergence.

[0100] In this embodiment, after obtaining the enlarged second modality feature extraction initial network, the computer device can use the second modality image sample to train the enlarged second modality feature extraction initial network until the feature extraction result of the enlarged second modality feature extraction initial network meets the preset conditions, and then determine that one update is completed. After that, the computer device once again expands the network scale and uses the second modality image sample to train the enlarged second modality feature extraction initial network to complete the secondary update process. In the above manner, the enlarged second modality feature extraction initial network is updated multiple times to obtain the second modality feature extraction network.

[0101] Figure 7 This is a schematic diagram of updating the L-th layer of neurons or channels. The left side of the figure shows the fixed neurons or channels of the L-th layer of the first modal feature extraction network, and the right side shows the neurons or channels of the L-th layer of the second modal feature extraction network. Compared with the first modal feature extraction network, the second modal feature extraction network has added some neurons or channels.

[0102] In the above-mentioned region of interest segmentation method, when the second modality feature extraction initial network meets the network structure update conditions, the network scale of the second modality feature extraction initial network is expanded; and the expanded second modality feature extraction initial network is trained until convergence through the second modality image samples. In each update process of this method, it is necessary to expand the network scale and then train the expanded second modality feature extraction initial network with the second modality image samples, that is, in the training process, the depth of the second modality feature extraction initial network is continuously increased, which can improve the feature extraction capability of the second modality feature extraction network.

[0103] Based on the above embodiment, this embodiment introduces and explains the relevant contents of judging whether the second modal feature extraction network meets the update condition. Figure 8 As shown, as a non-limiting example, the above method may also include the following content:

[0104] S701, using a first modality feature extraction network to extract features between pixels in a first modality image sample to obtain a plurality of first modality image features.

[0105] In this embodiment, the computer device can input the first modality image sample into the first modality feature extraction network, use the first modality feature extraction network to extract features between each pixel in the first modality image sample, and use the features between each pixel as the first modality image feature. The features between each pixel include shape features, texture features, contrast features, and the like.

[0106] S702, using a second modality feature extraction network to extract features between pixels in a second modality image sample to obtain a plurality of second modality image features.

[0107] In this embodiment, the computer device can input the second modality image sample into the second modality feature extraction network, use the second modality feature extraction network to extract the features between each pixel in the second modality image sample, and use the features between each pixel as the second modality image features. Similar to the features in the first modality image sample, the features between each pixel in the second modality image sample also include shape features, texture features, contrast features, and the like.

[0108] S703: Determine whether the second modality feature extraction network meets the update condition based on each first modality image feature and each second modality image feature.

[0109] In this embodiment, the computer device can score each second modality image feature based on each first modality image feature to obtain the number of scores higher than a preset score threshold. If the number is greater than the preset number threshold, it is determined that the second modality feature extraction network meets the update condition; if the number is less than or equal to the preset number threshold, it is determined that the second modality feature extraction network does not meet the update condition.

[0110] In the above-mentioned region of interest segmentation method, the first modality feature extraction network is used to extract features between each pixel in the first modality image sample to obtain multiple first modality image features; the second modality feature extraction network is used to extract features between each pixel in the second modality image sample to obtain multiple second modality image features; based on each first modality image feature and each second modality image feature, it is determined whether the updated second modality feature extraction initial network meets the network structure update condition. This method can accurately evaluate the second modality feature extraction network through multiple first modality image features corresponding to the first modality image sample and multiple second modality image features corresponding to the second modality image sample to accurately determine whether it meets the update condition.

[0111] Based on the above embodiments, this embodiment is to Figure 8 The relevant contents of step S703 "determining whether the updated second modality feature extraction initial network meets the network structure update condition based on each first modality image feature and each second modality image feature" are introduced and explained. Fig. 9 As shown, as a non-limiting example, the above step S703 may include the following content:

[0112] S801, determining a feature contribution quantization value of a second modality image sample based on each first modality image feature and each second modality image feature; the feature contribution quantization value represents the contribution degree of the second modality image to the segmentation task.

[0113] In this embodiment, after obtaining each first modality image feature and each second modality image feature, the computer device can obtain the feature contribution value of each first modality image feature and the total feature contribution value of each second modality image feature. The ratio between the feature contribution value of each second modality image feature and the total feature contribution value is determined as the feature contribution quantization value of the second modality image sample.

[0114] S802: When the difference between two adjacent updated feature contribution quantization values ​​is less than or equal to a preset threshold, determine that the updated second modality feature extraction initial network meets the network structure update condition.

[0115] In this embodiment, the computer device can obtain the feature contribution quantization value of the second modality image sample in two adjacent update processes, calculate the difference between the two feature contribution quantization values, and compare the difference with a preset threshold. If the difference is less than or equal to the preset threshold, it can be determined that the current second modality feature extraction network meets the update condition. The preset threshold can be set according to historical experience.

[0116] In the above-mentioned region of interest segmentation method, based on each first modality image feature and each second modality image feature, the feature contribution quantization value of the second modality image sample is determined; the feature contribution quantization value represents the contribution degree of the second modality image to the segmentation task; when the difference between the feature contribution quantization values ​​updated twice adjacently is less than or equal to the preset threshold, it is determined that the updated second modality feature extraction initial network meets the network structure update condition. This method can accurately determine the feature contribution quantization value of the second modality image sample through each first modality image feature and each second modality image feature. Based on the relationship between the difference between the feature contribution quantization values ​​updated twice adjacently and the preset threshold, it can be accurately judged whether the second modality feature extraction network meets the update condition, thereby improving the accuracy during the training process.

[0117] Based on the above embodiments, this embodiment is to Fig. 9 The relevant contents of step S801 "determining the feature contribution quantization value of the second modality image based on each first modality image feature and each second modality image feature" are introduced and explained. Fig.10 As shown, as a non-limiting example, the above step S801 may include the following content:

[0118] S901, obtaining the total feature weight of each first modality image feature and each second modality image feature in the segmentation task.

[0119] In this embodiment, the computer device may calculate the sum of all first feature weights in each first modality image feature, and calculate the sum of all second feature weights in each second modality image feature. The sum of the first feature weights and the sum of the second feature weights are added together, and the result of the addition is used as the total feature weight of each first modality image feature and each second modality image feature.

[0120] S902: Taking the ratio of the sum of the feature weights of each second modality image feature in the segmentation task to the total feature weight as the feature contribution quantization value of the second modality image sample.

[0121] Exemplarily, when the first modality image sample is a PET image, the feature weight of each first modality image feature can be expressed as W PET_n When the first modality image sample is a CT image, the feature weight of each first modality image feature can be expressed as W CT_m , then the quantized value of the feature contribution of the second modality image sample can be expressed as:

[0122]

[0123] in, represents the quantized value of the feature contribution of each second modality image feature updated for the kth time; N represents the number of first modality image features; and M represents the number of second modality image features.

[0124] Fig.11 A schematic diagram of obtaining the feature contribution quantization value of the k-th second modality image sample, the process includes: S1001: extracting the features of the first modality image sample and the features of the second modality image sample respectively; S1002: calculating the feature weight corresponding to each feature; S1003: calculating the feature contribution quantization value of the second modality image sample based on the feature weight and each feature.

[0125] In the above-mentioned region of interest segmentation method, the total feature weight of each first modality image feature and each second modality image feature in the segmentation task is obtained; the ratio of the sum of each feature weight of each second modality image feature in the segmentation task to the total feature weight is used as the feature contribution quantification value of the second modality image sample. This method can accurately obtain the ratio of the feature weight of the second modality image feature to the total feature weight by calculating the total feature weight of all sample image features, thereby accurately obtaining the feature contribution quantification value of the second modality image sample.

[0126] In one embodiment, the construction process of the second modality feature extraction network is described in detail. Fig.12 As shown, the method may include:

[0127] S1101: Acquire a first modality image sample of a target part;

[0128] S1102: training a first modality feature extraction initial network using first modality image samples to obtain a first modality feature extraction network;

[0129] S1103: Acquire a second modality image sample of the target part;

[0130] S1104: reducing the size of each feature extraction layer of the first modal feature extraction network;

[0131] S1105: Pre-training the reduced first modality feature extraction network using the second modality image samples until a pre-training condition is met to obtain a second modality feature extraction initial network;

[0132] S1106: When the initial network for extracting the second modal features satisfies the network structure update condition, the network scale of the initial network for extracting the second modal features is expanded;

[0133] S1107: training the expanded second modality feature extraction initial network through the second modality image samples until convergence;

[0134] S1108: extracting features between pixels in the first modality image sample using the first modality feature extraction network to obtain a plurality of first modality image features;

[0135] S1109: extracting features between pixels in the second modality image sample using a second modality feature extraction network to obtain a plurality of second modality image features;

[0136] S1110: Obtaining total feature weights of each first modality image feature and each second modality image feature in the segmentation task;

[0137] S1111: taking the ratio of the sum of the feature weights of each second modality image feature in the segmentation task to the total feature weight as the feature contribution quantization value of the second modality image sample;

[0138] S1112: When the difference between two adjacent updated feature contribution quantization values ​​is less than or equal to a preset threshold, determine that the updated second modality feature extraction initial network meets the network structure update condition.

[0139] Fig.13The present invention is a flow chart of a method for segmenting a region of interest, and the method comprises: S1201: using a first modality image sample to train a first modality feature extraction initial network to obtain a first modality feature extraction network; S1202: using the first modality feature extraction network to extract first modality image features of the first modality image sample; S1203: shrinking the first modality feature extraction network to obtain a shrunken feature extraction network; S1204: using the shrunken feature extraction network to extract second modality image features of the second modality image sample; S1205: based on each first modality image feature and each second modality image feature, calculating a feature contribution quantization value of the second modality image sample; S1206: updating the shrunken feature extraction network based on the feature contribution quantization value of the second modality image sample, until the difference between two adjacent updated feature contribution quantization values ​​is less than or equal to a preset threshold, determining that the updating process is completed, and obtaining the second modality feature extraction network.

[0140] Fig.14 The present invention is a flow chart of a method for segmenting a region of interest, and the method comprises: S1301: updating an initial network for extracting second modal features using a second modal image sample; S1302: determining whether the loss function of the updated initial network for extracting second modal features meets the convergence criteria, if not, continuing to execute step S1301; if satisfied, executing step S1303; S1303: calculating a quantized value of a feature contribution of the current second modal image sample; S1304: obtaining a difference between the quantized values ​​of the feature contributions of the current and previous second modal image samples; S1305: determining whether the difference is greater than a preset threshold, if greater than the preset threshold, executing step S1306; if less than or equal to the preset threshold, terminating the updating process; S1306: expanding the network scale of the updated initial network for extracting second modal features, and then executing the operation in step S1301.

[0141] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0142] Based on the same inventive concept, the embodiment of the present application also provides a region of interest segmentation device for implementing the region of interest segmentation method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more region of interest segmentation device embodiments provided below can refer to the limitations of the region of interest segmentation method above, and will not be repeated here.

[0143] In one embodiment, Fig.15 As shown, a device for segmenting a region of interest is provided, comprising: an acquisition module 11, an extraction module 12 and a determination module 13, wherein:

[0144] An acquisition module 11 is used to acquire a first modality image and a second modality image of a target part;

[0145] An extraction module 12 is used to extract a first image feature in a first modality image through a first modality feature extraction network, and to extract a second image feature in a second modality image through a second modality feature extraction network; the second modality feature extraction network is obtained by updating a network structure having the same framework as the first modality feature extraction network;

[0146] The segmentation result determination module 13 is used to obtain the segmentation result of the region of interest in the target part according to the first image feature and the second image feature.

[0147] In one embodiment, the above-mentioned region of interest segmentation device further includes: a first sample acquisition module and a training module, wherein:

[0148] A sample acquisition module, used to acquire a first modality image sample of a target part;

[0149] The training module is used to train the first modality feature extraction initial network using the first modality image samples to obtain the first modality feature extraction network.

[0150] In one embodiment, the above-mentioned region of interest segmentation device further includes: a second sample acquisition module, an adjustment module and an update module, wherein:

[0151] A second sample acquisition module, used to acquire a second modality image sample of the target part;

[0152] An adjustment module, used to adjust the network structure of the first modal feature extraction network to obtain a second modal feature extraction initial network;

[0153] The updating module is used to update the network structure of the second modality feature extraction initial network based on the second modality image sample until the updated second modality feature extraction initial network does not meet the network structure update condition, thereby obtaining the second modality feature extraction network.

[0154] In one embodiment, the adjustment module includes: a scaling unit and a pre-training unit, wherein:

[0155] A scaling unit, configured to reduce the size of each feature extraction layer of the first modal feature extraction network; the size of each feature extraction layer includes the number of neurons or the number of channels;

[0156] The pre-training unit is used to pre-train the reduced first modality feature extraction network using the second modality image samples until the pre-training conditions are met to obtain the second modality feature extraction initial network.

[0157] In one embodiment, the updating module comprises: an expansion unit and a training unit, wherein:

[0158] An expansion unit, used for expanding the network scale of the second modal feature extraction initial network when the second modal feature extraction initial network meets the network structure update condition;

[0159] The training unit is used to train the expanded second modality feature extraction initial network until convergence through the second modality image samples.

[0160] In one embodiment, the above-mentioned region of interest segmentation device further includes: a first extraction module, a second extraction module and a determination module, wherein:

[0161] A first extraction module, configured to extract features between pixels in a first modality image sample using a first modality feature extraction network to obtain a plurality of first modality image features;

[0162] A second extraction module, used to extract features between pixels in the second modality image sample using a second modality feature extraction network to obtain a plurality of second modality image features;

[0163] The determination module is used to determine whether the updated second modality feature extraction initial network meets the network structure update condition based on each first modality image feature and each second modality image feature.

[0164] In one embodiment, the above-mentioned determination module is also used to determine the feature contribution quantization value of the second modality image sample based on each first modality image feature and each second modality image feature; the feature contribution quantization value represents the contribution degree of the second modality image to the segmentation task; when the difference between two adjacent updated feature contribution quantization values ​​is less than or equal to a preset threshold, it is determined that the updated second modality feature extraction initial network meets the network structure update condition.

[0165] In one embodiment, the above-mentioned determination module is also used to obtain the total feature weight of each first modality image feature and each second modality image feature in the segmentation task; and the ratio of the sum of each feature weight of each second modality image feature in the segmentation task to the total feature weight is used as the feature contribution quantification value of the second modality image sample.

[0166] Each module in the above-mentioned region of interest segmentation device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0167] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, any one of the contents of the above-mentioned region of interest segmentation method is implemented.

[0168] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any one of the contents of the above-mentioned region of interest segmentation method is implemented.

[0169] In one embodiment, a computer program product is provided, including a computer program, which implements any one of the above-mentioned methods for segmenting a region of interest when executed by a processor.

[0170] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0171] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0172] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0173] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for segmenting a region of interest, characterized in that: The method comprises: Acquiring a first modality image and a second modality image of a target part; Extracting a first image feature in the first modality image through a first modality feature extraction network, and extracting a second image feature in the second modality image through a second modality feature extraction network; the second modality feature extraction network is obtained by updating the network structure of a network having the same framework as the first modality feature extraction network; A segmentation result of a region of interest in the target part is obtained according to the first image feature and the second image feature.

2. The method according to claim 1, characterized in that The construction process of the first modality feature extraction network includes: Acquire a first modality image sample of the target part; The first modality image samples are used to train the first modality feature extraction initial network to obtain the first modality feature extraction network.

3. The method according to claim 2, characterized in that The construction process of the second modality feature extraction network includes: Acquiring a second modality image sample of the target part; Adjusting the network structure of the first modal feature extraction network to obtain a second modal feature extraction initial network; Based on the second modality image samples, the network structure of the second modality feature extraction initial network is updated until the updated second modality feature extraction initial network does not meet the network structure update condition, thereby obtaining the second modality feature extraction network.

4. The method according to claim 3, characterized in that The adjusting the network structure of the first modal feature extraction network to obtain a second modal feature extraction initial network includes: Reducing the size of each feature extraction layer of the first modal feature extraction network; the size of each feature extraction layer includes the number of neurons or the number of channels; The reduced first modality feature extraction network is pre-trained using the second modality image samples until a pre-training condition is met to obtain the second modality feature extraction initial network.

5. The method according to claim 3, characterized in that: The updating of the network structure of the initial network for extracting features of the second modality based on the image sample of the second modality includes: When the second modal feature extraction initial network meets the network structure update condition, the network scale of the second modal feature extraction initial network is expanded; The enlarged second modality feature extraction initial network is trained until convergence using the second modality image samples.

6. The method according to any one of claims 3 to 5, characterized in that: The method further comprises: Extracting features between pixels in the first modality image sample using the first modality feature extraction network to obtain a plurality of first modality image features; Extracting features between pixels in the second modality image sample using the second modality feature extraction network to obtain a plurality of second modality image features; Based on each of the first modality image features and each of the second modality image features, it is determined whether the updated second modality feature extraction initial network meets the network structure update condition.

7. The method according to claim 6, characterized in that The determining, based on each of the first modality image features and each of the second modality image features, whether the updated second modality feature extraction initial network meets the network structure update condition includes: Determine a feature contribution quantization value of the second modality image sample based on each of the first modality image features and each of the second modality image features; the feature contribution quantization value indicates the contribution degree of the second modality image to the segmentation task; When the difference between two adjacent updated feature contribution quantization values ​​is less than or equal to a preset threshold, it is determined that the updated second modality feature extraction initial network meets the network structure update condition.

8. The method according to claim 7, characterized in that The determining, based on each of the first modality image features and each of the second modality image features, a feature contribution quantization value of the second modality image includes: Obtaining a total feature weight of each of the first modality image features and each of the second modality image features in the segmentation task; The ratio of the sum of the feature weights of each of the second modality image features in the segmentation task to the total feature weight is used as the feature contribution quantization value of the second modality image sample.

9. A device for segmenting a region of interest, characterized in that: The device comprises: An acquisition module, used for acquiring a first modality image and a second modality image of a target part; An extraction module, configured to extract a first image feature in the first modality image through a first modality feature extraction network, and to extract a second image feature in the second modality image through a second modality feature extraction network; the second modality feature extraction network is obtained by updating a network structure having the same framework as the first modality feature extraction network; The segmentation result determination module is used to obtain the segmentation result of the region of interest in the target part according to the first image feature and the second image feature.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images