An image segmentation method, device, apparatus and storage medium

Abstract

This application relates to an image segmentation method, apparatus, device, and storage medium, specifically in the field of image detection technology. The method includes: acquiring sample image data; extracting features from the sample image data using a downsampling module of an object detection model to obtain a target sample feature map; upsampling the target sample feature map using an upsampling module in the object detection model to obtain at least two levels of upsampled feature maps; obtaining a first loss function value based on the top-level upsampled feature map and sample annotations; performing channel compression on the at least two levels of upsampled feature maps to obtain at least two layers of compressed feature maps; obtaining a second loss function value based on the at least two layers of compressed feature maps and sample annotations; and training the object detection model using the first and second loss function values ​​to obtain a trained object detection model. Based on the above scheme, the image recognition accuracy of the object detection model is improved.

Description

Technical Field

[0001] This application relates to the field of image detection, specifically to an image segmentation method, apparatus, device, and storage medium. Background Technology

[0002] CT images are obtained by scanning different tissues and organs of the human body to detect the different absorption capabilities of X-rays, and are composed of many axial slices to form a three-dimensional image.

[0003] In recent years, deep learning has been increasingly applied to image segmentation. The U-shaped network structure Unet is a semantic segmentation algorithm based on deep learning, following the principles of FCN and with corresponding improvements, to adapt to simple segmentation with few samples. Deep supervision is widely used in both classification and segmentation. Its central idea is to provide direct supervision to the hidden layers, allowing gradient information to be injected deeper into the network, effectively addressing gradient vanishing and promoting the training of intermediate layers. By combining deep supervision with the U-shaped network structure, the problems of gradient vanishing and slow convergence speed in the U-shaped network structure can be effectively solved.

[0004] However, because the effective information in the CT images only remains in the shallow layer of the network during the training process, the recognition accuracy of the network model is poor. Summary of the Invention

[0005] This application provides an image segmentation method, apparatus, device, and storage medium, which achieves high recognition accuracy when performing image segmentation. The technical solution is as follows.

[0006] On the one hand, an image segmentation method is provided, the method comprising:

[0007] Acquire sample image data; the sample image data includes sample annotations;

[0008] The target sample feature map is obtained by extracting features from the sample image data through the downsampling module of the target detection model.

[0009] The target sample feature map is upsampled by the upsampling module in the target detection model to obtain at least two levels of upsampled feature maps; the upsampling module includes at least two upsampling layers;

[0010] The first loss function value is obtained based on the top-level upsampled feature map and the sample annotations;

[0011] Channel compression is performed on the at least two layers of upsampled feature maps respectively to obtain at least two layers of compressed feature maps;

[0012] The second loss function value is obtained based on at least two layers of compressed feature maps and the sample annotations;

[0013] The target detection model is trained based on the first loss function value and the second loss function value to obtain a trained target detection model; the trained target detection model is used to process target image data to obtain target segmentation results.

[0014] In another aspect, an image segmentation apparatus is provided, the apparatus comprising:

[0015] A data acquisition unit is used to acquire sample image data; the sample image data includes sample annotations.

[0016] The feature extraction unit is used to extract features from the sample image data through the downsampling module of the target detection model to obtain the target sample feature map;

[0017] An upsampling unit is used to upsample the feature map of a target sample through the upsampling module in the target detection model to obtain at least two levels of upsampled feature maps; the upsampling module includes at least two upsampling layers;

[0018] The first loss function acquisition unit is used to obtain the first loss function value based on the top-level upsampled feature map and the sample annotation;

[0019] The channel compression unit is used to perform channel compression on the at least two layers of upsampled feature maps respectively to obtain at least two layers of compressed feature maps;

[0020] The second loss function acquisition unit is used to acquire the value of the second loss function based on at least two layers of compressed feature maps and the sample annotations.

[0021] The training unit is used to train the target detection model based on the first loss function value and the second loss function value to obtain the trained target detection model; the trained target detection model is used to process target image data to obtain target segmentation results.

[0022] In one possible implementation, obtaining the second loss function value based on at least two layers of compressed feature maps and the sample annotations includes:

[0023] Based on the number of layers corresponding to each compressed feature map, each compressed feature map is enlarged to a specified resolution to obtain each first sample feature map.

[0024] The feature maps of the first samples are concatenated according to channels and processed through convolutional layers and fully connected layers to obtain the first recognition result;

[0025] Based on the first identification result and the sample labeling, the value of the second loss function is determined.

[0026] In one possible implementation, obtaining the first loss function value based on the top-level upsampled feature map and the sample annotations includes:

[0027] The top-level upsampled feature map is upsampled to obtain a second sample feature map;

[0028] The first loss function value is obtained based on the second sample feature map and the sample annotation.

[0029] In one possible implementation, the outer ring boundary is determined in the sample image based on pixels located outside the sample-labeled area and at a distance of a first threshold from the boundary of the sample label.

[0030] In the sample image, the inner ring boundary is determined based on the pixels located within the sample annotation area and at a distance of a second threshold from the sample annotation boundary;

[0031] The region between the outer ring boundary and the boundary of the sample annotation is defined as the outer ring region;

[0032] The area between the inner ring boundary and the boundary of the sample annotation is defined as the inner ring area;

[0033] The value of the third loss function is determined based on the inner ring region, the outer ring region, and the upsampled feature map;

[0034] The step of training the target detection model based on the first loss function value and the second loss function value includes:

[0035] The target detection model is trained based on the first loss function value, the second loss function value, and the third loss function value.

[0036] In one possible implementation, determining the third loss function value based on the inner ring region, the outer ring region, and the upsampled feature map includes:

[0037] Based on the upsampled feature map, the predicted probability of each pixel in the inner ring region of the sample image is determined;

[0038] Based on the upsampled feature map, the predicted probability of each pixel in the outer ring region of the sample image is determined;

[0039] The third loss function value is obtained based on the sum of the predicted probabilities of each pixel in the inner ring region and the sum of the predicted probabilities of each pixel in the outer ring region.

[0040] In one possible implementation, the downsampling module includes at least two feature extraction layers;

[0041] The step of extracting features from the sample image data using the downsampling module of the target detection model to obtain the target sample feature map includes:

[0042] The sample image data is processed by performing feature extraction on the at least two feature extraction layers in sequence to obtain at least two downsampled feature maps; the target sample feature map is the bottom downsampled feature map.

[0043] In one possible implementation, the feature extraction layer includes a convolutional layer and a GC block layer.

[0044] In another aspect, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, the at least one instruction being loaded and executed by the processor to implement the above-described image segmentation method.

[0045] In another aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the above-described image segmentation method.

[0046] In another aspect, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the image segmentation method described above.

[0047] The technical solution provided in this application may include the following beneficial effects:

[0048] First, sample image data is acquired, including sample annotations. Features are extracted from the sample image data using the downsampling module of the target detection model to obtain target sample feature maps. Then, the target sample feature maps are upsampled using the upsampling module of the target detection model to obtain at least two levels of upsampled feature maps. The upsampling module includes at least two upsampling layers. A first loss function value is obtained based on the top-level upsampled feature map and sample annotations. Channel compression is performed on the at least two levels of upsampled feature maps to obtain at least two compressed feature maps. A second loss function value is obtained based on the at least two compressed feature maps and sample annotations. The target detection model is trained using the first and second loss function values ​​to obtain a trained target detection model. The trained target detection model is then used to process target image data to obtain target segmentation results. This scheme improves the image recognition accuracy of the target detection model by comparing the top-level upsampled feature map and sample annotations to obtain the first loss function value, comparing the at least two compressed feature maps and sample annotations to obtain the second loss function value, and then optimizing the target detection model based on the first and second loss function values. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram illustrating the structure of an image segmentation system according to an exemplary embodiment.

[0051] Figure 2 This is a flowchart illustrating an image segmentation method according to an exemplary embodiment.

[0052] Figure 3 This is a flowchart illustrating an image segmentation method according to an exemplary embodiment.

[0053] Figure 4 This is a schematic diagram of the structure of a target detection model involved in an embodiment of this application.

[0054] Figure 5 This is a schematic diagram of the tumor margin region involved in the embodiments of this application.

[0055] Figure 6 This is a structural block diagram of an image segmentation apparatus according to an exemplary embodiment.

[0056] Figure 7 This is a structural block diagram of a computer device according to an exemplary embodiment. Detailed Implementation

[0057] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0058] It should be understood that the term "instruction" mentioned in the embodiments of this application can be a direct instruction, an indirect instruction, or an indication of a relationship. For example, A instructing B can mean that A directly instructs B, such as B being able to obtain information through A; it can also mean that A indirectly instructs B, such as A instructing C, so B can obtain information through C; or it can mean that there is a relationship between A and B.

[0059] In the description of the embodiments of this application, the term "correspondence" may indicate that there is a direct or indirect correspondence between two things, or that there is an association between two things, or that there is a relationship of instruction and being instructed, configuration and being configured, etc.

[0060] In the embodiments of this application, "predefined" can be achieved by pre-storing corresponding codes, tables or other means that can be used to indicate relevant information in the device (e.g., including terminal devices and network devices). This application does not limit the specific implementation method.

[0061] Figure 1 This is a schematic diagram illustrating the structure of an image segmentation system according to an exemplary embodiment. The image segmentation system includes a server 110 and a terminal device 120. The terminal device 120 may include a data processing device and a data storage module.

[0062] Optionally, the terminal device 120 can communicate with the server 110 through a transmission network (such as a wireless communication network). The terminal device 120 can upload various data (such as image data) stored in the data storage module to the server 110 through the wireless communication network so that the server 110 can process the acquired image data, for example, by training a convolutional neural network model for image segmentation and other applications using the uploaded image data.

[0063] Optionally, the terminal device 120 may also include command input components, such as a mouse, keyboard, touch screen, etc. Figure 1(Not shown), after receiving a specified command from the user, the command input component can input corresponding data on the terminal device. For example, when image segmentation software is installed on the terminal device, the user can input corresponding commands to the terminal device through the command input component to control the image segmentation software to output the corresponding image segmentation results.

[0064] Optionally, the terminal device can upload the image file to the server 110 so that the server 110 can train convolutional neural network models for applications such as image segmentation.

[0065] Optionally, the terminal device also includes a data processing device, which can segment the image file using a convolutional neural network model sent by the server when the terminal device 120 opens the image file.

[0066] Optionally, the server 110 can obtain image files uploaded by various terminal devices and annotate the image files manually or otherwise, so as to train the convolutional neural network model using the uploaded image files and annotation information. After training is completed, the trained convolutional neural network model can be transmitted to the terminal device so that the terminal device can perform image segmentation on the image files.

[0067] Optionally, after the terminal device receives and opens the image file, it can upload the image file to the server 110 so that the pre-trained convolutional neural network model in the server 110 can segment the image file, obtain the segmentation result, and return it to the terminal device, thereby realizing online segmentation of the image file.

[0068] Optionally, the aforementioned server can be a server cluster or a distributed system consisting of multiple physical servers, or it can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other technology computing services.

[0069] Optionally, the system may also include a management device for managing the system (such as managing the connection status between each module and the server), and the management device is connected to the server via a communication network. Optionally, the communication network may be a wired network or a wireless network.

[0070] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can also be any other network, including but not limited to any combination of local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), mobile, wired or wireless networks, private networks, or virtual private networks (VPNs). In some embodiments, technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), VPNs, and Internet Protocol (IP) security can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0071] Figure 2 This is a flowchart illustrating an image segmentation method according to an exemplary embodiment. The method is performed by a computer device, which may be, for example... Figure 1 One of the terminal devices and the server shown. Figure 2 As shown, the image segmentation method may include the following steps:

[0072] Step 201: Obtain sample image data; the sample image data includes sample annotations.

[0073] Image segmentation refers to dividing an image into several non-overlapping regions based on features such as grayscale, color, spatial texture, and geometric shape. This ensures that these features exhibit consistency or similarity within the same region, while showing significant differences between different regions. Simply put, it is about separating the target from the background in an image.

[0074] To improve the accuracy of the object detection model for image segmentation, it is necessary to train the model. First, it is necessary to obtain sample image data for training, which includes sample annotations obtained by experts who perform contour annotations on the target objects in the sample images.

[0075] Step 202: The target image data is used to extract features through the downsampling module of the target detection model to obtain the target sample feature map.

[0076] Downsampling can be simply understood as reducing the size of an image and decreasing the number of sampling points in a matrix. Feature extraction converts arbitrary data (such as text or images) into numerical features that can be used for machine learning. By using a downsampling module to extract features from sample image data, a thumbnail of the corresponding image (i.e., the target sample feature map) can be generated, thus reducing the amount of computation.

[0077] Step 203: Upsample the target sample feature map using the upsampling module in the target detection model to obtain at least two levels of upsampled feature maps.

[0078] The main purpose of upsampling is to enlarge the original image to the target resolution. Optionally, upsampling can be performed on the feature map of the target sample through interpolation. Interpolation involves inserting new pixels between the pixels of the original image using an appropriate interpolation algorithm.

[0079] This upsampling module includes at least two upsampling layers. The number of upsampling layers can be set according to actual needs.

[0080] Step 204: Obtain the first loss function value based on the top-level upsampled feature map and the sample label.

[0081] The loss function reflects the difference between the forward computation result and the true value in each iteration of the neural network, thus guiding the next step of training in the right direction. By obtaining the loss function value, the prediction accuracy of the object detection model can be measured, and the object detection model can be optimized to make the loss function value smaller, thereby improving the prediction accuracy of the object detection model.

[0082] First, the first loss function value is obtained based on the difference between the top-level upsampled feature map and the sample annotation.

[0083] Step 205: Perform channel compression on the at least two layers of upsampled feature maps respectively to obtain at least two layers of compressed feature maps.

[0084] Generally speaking, increasing the depth of a neural network can improve its representational ability to some extent. However, as the depth increases, the neural network becomes increasingly difficult to train, leading to phenomena such as vanishing and exploding gradients. Deep supervision involves adding an auxiliary classifier as a branch to some intermediate hidden layers of a deep neural network to supervise the main network. This addresses problems such as vanishing gradients and slow convergence speed during deep neural network training.

[0085] The object detection model can be optimized by introducing deep supervision. However, the large size differences between different objects in the target image mean that during downsampling, effective information about small-sized objects only exists in the shallow layers of the object detection model. Supervision signals applied to lower-resolution layers will suffer from a lack of effective information, leading to disordered feature learning. Therefore, channel compression can be performed on at least two layers of upsampled feature maps during the upsampling process.

[0086] Step 206: Obtain the second loss function value based on at least two layers of compressed feature maps and the sample label.

[0087] After channel compression of at least two layers of upsampled feature maps to obtain at least two compressed feature maps, these at least two compressed feature maps can be stitched together. The stitched feature map contains effective information about the target object at multiple levels, thereby solving the problem of feature learning disorder caused by the lack of effective information in the supervision signal applied to the lower resolution layer.

[0088] Furthermore, the stitched feature map is compared with the sample annotations, and the difference between the stitched feature map and the sample annotations is used to obtain the second loss function value.

[0089] Step 207: Based on the first loss function value and the second loss function value, train the target detection model to obtain the trained target detection model.

[0090] The trained object detection model is used to process object image data to obtain object segmentation results.

[0091] In summary, this method first acquires sample image data, which includes sample annotations. Features are extracted from the sample image data using the downsampling module of the target detection model to obtain target sample feature maps. The target sample feature maps are then upsampled using the upsampling module of the target detection model to obtain at least two levels of upsampled feature maps. Each upsampling module includes at least two upsampling layers. A first loss function value is obtained based on the top-level upsampled feature map and the sample annotations. Channel compression is performed on the at least two levels of upsampled feature maps to obtain at least two compressed feature maps. A second loss function value is obtained based on the at least two compressed feature maps and the sample annotations. The target detection model is then trained using the first and second loss function values ​​to obtain a trained target detection model. This trained target detection model is used to process target image data to obtain target segmentation results. The above scheme obtains the first loss function value by comparing the top-level upsampled feature map and the sample annotation, obtains the second loss function value by comparing at least two layers of compressed feature maps and the sample annotation, and then optimizes the target detection model based on the first and second loss function values, thereby improving the image recognition accuracy of the target detection model.

[0092] Figure 3 This is a flowchart illustrating an image segmentation method according to an exemplary embodiment. The method is performed by a computer device, which may be, for example... Figure 1 One of the terminal devices and the server shown. Figure 3 As shown, the image segmentation method may include the following steps:

[0093] Step 301: Obtain sample image data.

[0094] The sample image data includes sample annotations.

[0095] Optionally, the sample image data can be obtained by directly downloading an existing sample image dataset, or you can take sample images yourself and have them labeled by experts to create a sample image dataset.

[0096] Optionally, the sample images can be labeled using the data labeling tool LabelMe to obtain a sample image dataset.

[0097] Optionally, the sample image is a CT image containing tumor information.

[0098] Step 302: Perform feature extraction processing on the sample image data sequentially through at least two feature extraction layers to obtain at least two downsampled feature maps.

[0099] Optionally, a U-shaped network, Unet, can be used as a model for CT image segmentation (object detection model). Figure 4 This is a schematic diagram of the structure of an object detection model according to an embodiment of this application. Unet is a symmetrical convolutional neural network structure, with downsampling on the left and upsampling on the right. Figure 4 In this context, Roi (region of interest) represents the region of interest, crop means cropping (directly extracting a portion of the image while preserving the original image's true size ratio), downsample means downsampling, upsample means upsampling, conv3×3×3 means 3×3×3 convolution, softmax means normalized exponential function, non-local operation means non-local operation, and L... dice The dice coefficient is a set similarity metric used to calculate the similarity between two samples.

[0100] Optionally, the downsampling module includes at least two feature extraction layers. These feature extraction layers include convolutional layers and GC block layers. The number of feature extraction layers can be set according to actual needs; for example, in... Figure 4 The model contains 6 feature extraction layers.

[0101] In CT image segmentation, capturing long-range dependencies can provide contextual information about the target region, improving the model's global understanding of the visual scene and facilitating target region localization. However, convolution is only a local operation. Convolutional neural networks (CNNs) capture long-range dependencies by stacking convolutional layers. Small tumor CT images (i.e., CT images containing information about small tumors) lose information during downsampling, with effective information remaining only in shallow layers. However, the number of shallow layers is limited, so existing CNNs cannot capture effective long-range dependencies, resulting in poor segmentation performance for small tumor CT images. This necessitates improvements to the convolutional layers, the fundamental component of CNNs, to enable them to effectively capture long-range dependencies of the target even without a sufficient number of shallow convolutional layers.

[0102] To improve the segmentation performance of small targets (i.e., small tumor CT images), it was decided to use a GC block (Global Context Block) to structurally improve the shallow convolutional layers of the encoder (i.e., feature extraction layers). A GC block (i.e., global context block) was added at the end of the shallow convolutional layers of the encoder. Figure 4 Part ③ of the code helps the shallow layers of the network capture long-range dependencies of the target, improving the segmentation effect on CT images of small tumors. This GC block can be implemented using existing structures. The GC block simplifies the query calculation between pixels, saving significant computational costs and achieving module lightweighting, making it possible to apply the module to the shallow convolutional layers of convolutional neural networks. Furthermore, it is integrated with the channel attention module SEblock, enabling the module to effectively establish not only long-range spatial dependencies but also channel-level dependencies.

[0103] Step 303: Upsample the target sample feature map through the upsampling module in the target detection model to obtain at least two levels of upsampled feature maps.

[0104] This upsampling module includes at least two upsampling layers. The target sample feature map is the bottom-level downsampling feature map. The number of upsampling layers can be set according to actual needs, for example, in... Figure 4 The model contains 5 upsampling layers.

[0105] Step 304: Obtain the first loss function value based on the top-level upsampled feature map and the sample label.

[0106] In training an object detection model, it is necessary to first obtain the target loss function of the model, and then optimize the model based on this target loss function. To obtain the target loss function, the first loss function value must first be obtained.

[0107] Optionally, the top-level upsampled feature map can be upsampled to obtain a second sample feature map.

[0108] Furthermore, the first loss function value is obtained based on the second sample feature map and the sample label.

[0109] Step 305: Perform channel compression on the at least two layers of upsampled feature maps respectively to obtain at least two layers of compressed feature maps.

[0110] Deep supervision is widely used in classification and segmentation. Its central idea is to provide direct supervision to the hidden layers, allowing gradient information to be injected deeper into the network, effectively addressing gradient vanishing and promoting the training of intermediate layers. Optionally, deep supervision can be introduced into this object detection model to optimize it.

[0111] However, in practical applications, deep supervision involves applying the same masked supervision signal to some layers of the decoder as to the output layer. This can negatively impact the segmentation training of small lesions (e.g., small tumors) when dealing with lesion segmentation with large size differences (e.g., tumors). Due to encoder downsampling, the effective information of small tumors only remains in the shallow layers of the network. The supervision signal applied to the lower resolution layers will lead to disordered feature learning due to the lack of effective information.

[0112] Optionally, to address the contradiction between injecting deeper supervisory signals into the target detection model and training for segmentation of small tumor CT images, the deep supervision structure was modified (i.e. Figure 4 (Part ②) uses a 1×1×1 convolution to compress the number of channels in the upsampled feature map of the deep-supervised upsampled layer to 1 / 4 of the number of channels in the upsampled feature map of the second layer.

[0113] Step 306: Obtain the second loss function value based on at least two layers of compressed feature maps and the sample label.

[0114] After channel compression of the upsampled feature maps of the upsampled layers with deep supervision, they are then stitched together. After stitching, channel compression and prediction are performed again to obtain the second loss function to optimize the target detection model. This can effectively avoid the feature learning disorder caused by the same supervision signal contained in all upsampled layers with deep supervision during the segmentation training of small tumor CT images.

[0115] First, based on the number of layers corresponding to each compressed feature map, each compressed feature map is enlarged to a specified resolution to obtain each first sample feature map.

[0116] Furthermore, the feature maps of each first sample are spliced ​​together according to channels and processed through convolutional layers and fully connected layers to obtain the first recognition result.

[0117] Furthermore, based on the first identification result and the sample labeling, the value of the second loss function is determined.

[0118] Step 307: Obtain the third loss function value based on the sum of the predicted probabilities of each pixel in the inner ring region and the sum of the predicted probabilities of each pixel in the outer ring region.

[0119] To improve the segmentation accuracy of uncertain targets (such as tumors) in CT images, edge loss (i.e., Figure 4 (See section ① in the text). Because the grayscale of the tumor and surrounding tissues is similar in CT images, the segmentation prediction of the tumor edge region often results in severe oversegmentation and undersegmentation. Furthermore, the region at a certain distance from the tumor edge (e.g., 5 mm) contains rich spatial information, and identifying this information is very helpful for tumor differentiation and localization. Figure 5 This is a schematic diagram of the tumor margin region involved in an embodiment of this application. Figure 5 As shown, the tumor edge can be marked by experts first (i.e., sample annotation), and then the inner and outer ring regions of the tumor can be defined based on the sample annotation. Then, an edge loss function can be designed for the inner and outer ring regions of the tumor to improve over-segmentation and under-segmentation caused by edge blurring and improve the edge segmentation accuracy. Figure 5 In this context, the segmentation result refers to the recognition result of the target detection model on the sample image (e.g., the first recognition result), the under-segmented region refers to the region that is under-segmented after comparing the segmentation result with the sample label, and the over-segmented region refers to the region that is over-segmented after comparing the segmentation result with the sample label.

[0120] Optionally, in the sample image, the outer ring boundary is determined based on pixels located outside the sample-labeled area and at a distance of a first threshold from the boundary of the sample label.

[0121] In the sample image, the inner ring boundary is determined based on the pixels located within the sample annotation area and at a distance of a second threshold from the sample annotation boundary.

[0122] The area between the outer ring boundary and the boundary marked on the sample is defined as the outer ring region;

[0123] The area between the inner ring boundary and the boundary marked on the sample is defined as the inner ring region;

[0124] Furthermore, based on the inner ring region, the outer ring region, and the upsampled feature map, the value of the third loss function is determined.

[0125] First, based on the upsampled feature map, determine the sum of the predicted probabilities of each pixel in the inner ring region of the sample image; then, based on the upsampled feature map, determine the sum of the predicted probabilities of each pixel in the outer ring region of the sample image.

[0126] Furthermore, an edge loss function is designed to determine the value of the third loss function L. boundary Optionally, the third loss function value L can be calculated using the following formula. boundary :

[0127]

[0128] In a CT image, the inner ring region of a tumor has M pixels. The target detection model predicts the foreground probability of the m-th pixel in the inner ring region as P. m The outer ring region of the tumor has N pixels. The probability of the target detection model predicting the foreground of the nth pixel in the outer ring region is P. n ε is an infinitesimal.

[0129] This edge loss function improves the network's ability to distinguish between the foreground and background of the tumor edge region by increasing the difference in the average foreground prediction probability of all pixels in the inner and outer ring regions, thereby reducing oversegmentation and undersegmentation and improving the edge segmentation effect.

[0130] Step 308: Based on the first loss function value, the second loss function value, and the third loss function value, train the target detection model to obtain the trained target detection model.

[0131] By training the target detection model, the accuracy of the target detection model in identifying tumors can be improved, making the prediction results (prediction mask) of the target detection model closer to the sample annotation (gold standard).

[0132] It should be noted that, Figure 4 The left side of the image shows the downsampling module, and the right side shows the upsampling module.

[0133] It should be noted that the trained object detection model is used to process object image data to obtain object segmentation results.

[0134] In summary, this method first acquires sample image data, which includes sample annotations. Features are extracted from the sample image data using the downsampling module of the target detection model to obtain target sample feature maps. The target sample feature maps are then upsampled using the upsampling module of the target detection model to obtain at least two levels of upsampled feature maps. Each upsampling module includes at least two upsampling layers. A first loss function value is obtained based on the top-level upsampled feature map and the sample annotations. Channel compression is performed on the at least two levels of upsampled feature maps to obtain at least two compressed feature maps. A second loss function value is obtained based on the at least two compressed feature maps and the sample annotations. The target detection model is then trained using the first and second loss function values ​​to obtain a trained target detection model. This trained target detection model is used to process target image data to obtain target segmentation results. The above scheme obtains the first loss function value by comparing the top-level upsampled feature map and the sample annotation, obtains the second loss function value by comparing at least two layers of compressed feature maps and the sample annotation, and then optimizes the target detection model based on the first and second loss function values, thereby improving the image recognition accuracy of the target detection model.

[0135] Figure 6 This is a structural block diagram illustrating an image segmentation apparatus according to an exemplary embodiment. The image segmentation apparatus includes:

[0136] The data acquisition unit 601 is used to acquire sample image data, which includes sample annotations.

[0137] The feature extraction unit 602 is used to extract features from the sample image data through the downsampling module of the target detection model to obtain the target sample feature map;

[0138] The upsampling unit 603 is used to upsample the feature map of the target sample through the upsampling module in the target detection model to obtain at least two levels of upsampled feature maps; the upsampling module includes at least two upsampling layers;

[0139] The first loss function acquisition unit 604 is used to obtain the first loss function value based on the top-level upsampled feature map and the sample label;

[0140] The channel compression unit 605 is used to perform channel compression on the at least two layers of upsampled feature maps respectively to obtain at least two layers of compressed feature maps;

[0141] The second loss function acquisition unit 606 is used to acquire the value of the second loss function based on at least two layers of compressed feature maps and the sample label;

[0142] The training unit 607 is used to train the object detection model based on the first loss function value and the second loss function value to obtain the trained object detection model; the trained object detection model is used to process the object image data to obtain the object segmentation result.

[0143] In one possible implementation, the second loss function value is obtained based on at least two layers of compressed feature maps and the sample annotation, including:

[0144] Based on the number of layers corresponding to each compressed feature map, each compressed feature map is enlarged to a specified resolution to obtain each first sample feature map.

[0145] The feature maps of the first samples are concatenated according to channels and processed through convolutional and fully connected layers to obtain the first recognition result;

[0146] Based on the first identification result and the sample labeling, the value of the second loss function is determined.

[0147] In one possible implementation, the first loss function value is obtained based on the top-level upsampled feature map and the sample label, including:

[0148] Upsample the top-level upsampled feature map to obtain the second sample feature map;

[0149] The first loss function value is obtained based on the feature map of the second sample and the sample label.

[0150] In one possible implementation, the outer ring boundary is determined in the sample image based on pixels located outside the region labeled in the sample and at a distance of a first threshold from the boundary of the sample label.

[0151] In the sample image, the inner ring boundary is determined based on the pixels located within the sample annotation area and at a distance of a second threshold from the sample annotation boundary.

[0152] The area between the outer ring boundary and the boundary marked on the sample is defined as the outer ring region;

[0153] The area between the inner ring boundary and the boundary marked on the sample is defined as the inner ring region;

[0154] The value of the third loss function is determined based on the inner ring region, the outer ring region, and the upsampled feature map;

[0155] The target detection model is trained based on the first loss function value and the second loss function value, including:

[0156] The target detection model is trained based on the first loss function value, the second loss function value, and the third loss function value.

[0157] In one possible implementation, determining the third loss function value based on the inner ring region, the outer ring region, and the upsampled feature map includes:

[0158] Based on the upsampled feature map, determine the predicted probability of each pixel in the inner ring region of the sample image;

[0159] Based on the upsampled feature map, determine the predicted probability of each pixel in the outer ring region of the sample image;

[0160] The third loss function value is obtained based on the sum of the predicted probabilities of each pixel in the inner ring region and the sum of the predicted probabilities of each pixel in the outer ring region.

[0161] In one possible implementation, the downsampling module includes at least two feature extraction layers;

[0162] The downsampling module of the target detection model extracts features from the sample image data to obtain a target sample feature map, including:

[0163] The sample image data is processed sequentially by the at least two feature extraction layers to obtain at least two downsampled feature maps; the target sample feature map is the bottom downsampled feature map.

[0164] In one possible implementation, the feature extraction layer includes a convolutional layer and a GC block layer.

[0165] In summary, this method first acquires sample image data, which includes sample annotations. Features are extracted from the sample image data using the downsampling module of the target detection model to obtain target sample feature maps. The target sample feature maps are then upsampled using the upsampling module of the target detection model to obtain at least two levels of upsampled feature maps. Each upsampling module includes at least two upsampling layers. A first loss function value is obtained based on the top-level upsampled feature map and the sample annotations. Channel compression is performed on the at least two levels of upsampled feature maps to obtain at least two compressed feature maps. A second loss function value is obtained based on the at least two compressed feature maps and the sample annotations. The target detection model is then trained using the first and second loss function values ​​to obtain a trained target detection model. This trained target detection model is used to process target image data to obtain target segmentation results. The above scheme obtains the first loss function value by comparing the top-level upsampled feature map and the sample annotation, obtains the second loss function value by comparing at least two layers of compressed feature maps and the sample annotation, and then optimizes the target detection model based on the first and second loss function values, thereby improving the image recognition accuracy of the target detection model.

[0166] Figure 7 A structural block diagram of a computer device 700 illustrated in an exemplary embodiment of this application is shown. This computer device can be implemented as a server as described above in this application. The computer device 700 includes a Central Processing Unit (CPU) 701, a system memory 704 including Random Access Memory (RAM) 702 and Read-Only Memory (ROM) 703, and a system bus 705 connecting the system memory 704 and the CPU 701. The computer device 700 also includes a mass storage device 706 for storing an operating system 709, application programs 710, and other program modules 711.

[0167] The mass storage device 706 is connected to the central processing unit 701 via a mass storage controller (not shown) connected to the system bus 705. The mass storage device 706 and its associated computer-readable media provide non-volatile storage for the computer device 700. That is, the mass storage device 706 may include computer-readable media (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0168] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 704 and mass storage device 706 described above can be collectively referred to as memory.

[0169] According to various embodiments of this disclosure, the computer device 700 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 700 can be connected to a network 708 via a network interface unit 707 connected to the system bus 705, or it can use the network interface unit 707 to connect to other types of networks or remote computer systems (not shown).

[0170] The memory also includes at least one computer program stored in the memory, and the central processing unit 701 executes the at least one computer program to implement all or part of the steps in the methods shown in the above embodiments.

[0171] In one exemplary embodiment, a computer-readable storage medium is also provided for storing at least one computer program, which is loaded and executed by a processor to implement all or part of the steps in the above-described method. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, or optical data storage device, etc.

[0172] In one exemplary embodiment, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned actions. Figure 2 or Figure 3 All or part of the steps of the method shown in any embodiment.

[0173] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0174] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. An image segmentation method, characterized in that, The method includes: Acquire sample image data; the sample image data includes sample annotations; The target image data is subjected to feature extraction by the downsampling module of the target detection model to obtain the target sample feature map; the downsampling module includes at least two feature extraction layers; the feature extraction layer includes a convolutional layer and a GC block layer; the GC block layer is at the end of the convolutional layer; The target sample feature map is upsampled by the upsampling module in the target detection model to obtain at least two levels of upsampled feature maps; the upsampling module includes at least two upsampling layers; The first loss function value is obtained based on the top-level upsampled feature map and the sample annotations; Channel compression is performed on the at least two layers of upsampled feature maps to obtain at least two layers of compressed feature maps; wherein, the number of channels of the upsampled feature map of the upsampled layer with deep supervision is compressed to 1 / 4 of the number of channels of the upsampled feature map of the second layer. The second loss function value is obtained based on at least two layers of compressed feature maps and the sample annotations; The target detection model is trained based on the first loss function value and the second loss function value to obtain a trained target detection model; the trained target detection model is used to process target image data to obtain target segmentation results. The step of obtaining the second loss function value based on at least two layers of compressed feature maps and the sample annotations includes: Based on the number of layers corresponding to each compressed feature map, each compressed feature map is enlarged to a specified resolution to obtain each first sample feature map. The feature maps of the first samples are concatenated according to channels and processed through convolutional layers and fully connected layers to obtain the first recognition result; Based on the first identification result and the sample labeling, determine the value of the second loss function; The method further includes: In the sample image, the outer ring boundary is determined based on the pixels located outside the sample annotation area and at a distance of a first threshold from the sample annotation boundary; In the sample image, the inner ring boundary is determined based on the pixels located within the sample annotation area and at a distance of a second threshold from the sample annotation boundary; The region between the outer ring boundary and the boundary of the sample annotation is defined as the outer ring region; The area between the inner ring boundary and the boundary of the sample annotation is defined as the inner ring area; The third loss function value is determined based on the inner ring region, the outer ring region, and the upsampled feature map; The step of training the target detection model based on the first loss function value and the second loss function value includes: The target detection model is trained based on the first loss function value, the second loss function value, and the third loss function value.

2. The method according to claim 1, characterized in that, The first loss function value is obtained based on the top-level upsampled feature map and the sample annotations, including: The top-level upsampled feature map is upsampled to obtain a second sample feature map; The first loss function value is obtained based on the second sample feature map and the sample annotation.

3. The method according to claim 1 or 2, characterized in that, The step of determining the third loss function value based on the inner ring region, the outer ring region, and the upsampled feature map includes: Based on the upsampled feature map, the predicted probability of each pixel in the inner ring region of the sample image is determined; Based on the upsampled feature map, the predicted probability of each pixel in the outer ring region of the sample image is determined; The third loss function value is obtained based on the sum of the predicted probabilities of each pixel in the inner ring region and the sum of the predicted probabilities of each pixel in the outer ring region.

4. The method according to claim 1 or 2, characterized in that, The step of extracting features from the sample image data using the downsampling module of the target detection model to obtain the target sample feature map includes: The sample image data is processed by performing feature extraction on the at least two feature extraction layers in sequence to obtain at least two downsampled feature maps; the target sample feature map is the bottom downsampled feature map.

5. An image segmentation apparatus, characterized in that, The device includes: A data acquisition unit is used to acquire sample image data; the sample image data includes sample annotations. The feature extraction unit is used to extract features from the sample image data through the downsampling module of the target detection model to obtain a target sample feature map; the downsampling module includes at least two feature extraction layers; the feature extraction layers include convolutional layers and GC block layers; the GC block layer is at the end of the convolutional layers; An upsampling unit is used to upsample the feature map of a target sample through the upsampling module in the target detection model to obtain at least two levels of upsampled feature maps; the upsampling module includes at least two upsampling layers; The first loss function acquisition unit is used to obtain the first loss function value based on the top-level upsampled feature map and the sample annotation; The channel compression unit is used to compress the channels of the at least two layers of upsampled feature maps respectively to obtain at least two layers of compressed feature maps; wherein, the number of channels of the upsampled feature map of the upsampled layer with deep supervision is compressed to 1 / 4 of the number of channels of the upsampled feature map of the second layer. The second loss function acquisition unit is used to acquire the value of the second loss function based on at least two layers of compressed feature maps and the sample annotations. The training unit is used to train the target detection model based on the first loss function value and the second loss function value to obtain the trained target detection model; the trained target detection model is used to process target image data to obtain target segmentation results. The second loss function acquisition unit is also used for: Based on the number of layers corresponding to each compressed feature map, each compressed feature map is enlarged to a specified resolution to obtain each first sample feature map. The feature maps of the first samples are concatenated according to channels and processed through convolutional layers and fully connected layers to obtain the first recognition result; Based on the first identification result and the sample labeling, determine the value of the second loss function; The device further includes a region determination unit for: In the sample image, the outer ring boundary is determined based on the pixels located outside the sample annotation area and at a distance of a first threshold from the sample annotation boundary; In the sample image, the inner ring boundary is determined based on the pixels located within the sample annotation area and at a distance of a second threshold from the sample annotation boundary; The region between the outer ring boundary and the boundary of the sample annotation is defined as the outer ring region; The area between the inner ring boundary and the boundary of the sample annotation is defined as the inner ring area; The third loss function value is determined based on the inner ring region, the outer ring region, and the upsampled feature map; The training unit is also used for: The target detection model is trained based on the first loss function value, the second loss function value, and the third loss function value.

6. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, which is loaded and executed by the processor to implement the image segmentation method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to implement the image segmentation method as described in any one of claims 1 to 4.

Images