A human-computer collaborative tumor image interactive segmentation and labeling system and method thereof

Abstract

The application relates to the technical field of image processing, in particular to a human-computer collaborative tumor image interactive segmentation and labeling system and a method thereof, which comprises the following steps: S1, obtaining a display image by manually setting the display range size and display reference value of an original medical image; S2, for each original medical image, randomly scrambling the display range size while keeping the display reference value, or randomly scrambling the display reference value while keeping the display range size, or randomly scrambling both the display range size and the display reference value, so as to obtain a plurality of display images; S3, manually preliminarily marking each display image to obtain training images, jointly training a pixel marking main model and a pixel marking auxiliary model by using all the training images, and processing a newly obtained display image by using the trained pixel marking main model. The application solves the problems of insufficient training images and high image labeling cost.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a human-computer collaborative interactive segmentation and annotation system and method for tumor images. Background Technology

[0002] Tumor image segmentation and annotation refers to the process of delineating precise tumor boundaries. The mainstream method in current technology is to train a tumor image segmentation network through supervised learning. For example, patent application CN112348769A discloses a U-based... A method and device for intelligent segmentation of renal tumors in CT images using a Net deep network model, specifically including the following steps: reading patient CT images and corresponding segmentation mask maps; preprocessing CT image data to generate a training sample set; and constructing a U... A Net network model is used to segment kidneys in CT images. Background images and kidney masking images from training samples are input into the segmentation model for supervised learning. After training and convergence, the model is used to segment kidneys in CT images. An improved U-shaped segmentation model is constructed by incorporating an attention mechanism. The Net network model is used to segment kidney tumors. The kidney mask map and tumor mask map of the training sample are input into the segmentation model for supervised learning. After training and convergence, the model is used to segment kidney tumors in CT images.

[0003] However, training an accurate tumor image segmentation network through supervised learning requires not only acquiring a large number of tumor images, but also setting accurate labels for each of these images, resulting in high labor costs. Summary of the Invention

[0004] To address the technical challenges of training tumor image segmentation networks, which require a large number of tumor images and incur high costs for manual annotation, this application generates more display images based on original medical images with manually set display range size and display baseline values. Initial manual labeling is performed only on the display images. A pixel-labeled master model and a pixel-labeled auxiliary model are jointly trained using the labeled training images. The trained pixel-labeled master model is then used to perform semantic segmentation on the newly acquired display images.

[0005] This application provides a human-computer collaborative interactive segmentation and annotation method for tumor images, comprising the following steps: S1. Acquire several original medical images, and for each original medical image, manually set the display range size and display reference value to obtain the display image; S2. For each original medical image, random scrambling is performed on the display area size while retaining the display reference value, or random scrambling is performed on the display reference value while retaining the display area size, or random scrambling is performed on both the display area size and the display reference value, in order to obtain several display images; S3. Manually label each display image to obtain training images, and use all the training images to jointly train a pixel labeling master model and a pixel labeling auxiliary model. After training is completed, use the trained pixel labeling master model to process newly acquired display images.

[0006] As a preferred technical solution of this application, obtaining a display image based on the display range size and the display reference value includes: calculating the difference between the display reference value and half of the display range size as a lower limit value, calculating the sum of the display reference value and half of the display range size as an upper limit value, and for each pixel in the original medical image, adjusting the pixel value of the higher bit depth to the lower limit value when the pixel value of the higher bit depth is less than the lower limit value, adjusting the pixel value of the higher bit depth to the upper limit value when the pixel value of the higher bit depth is greater than the upper limit value, and converting the pixel value of the higher bit depth to the pixel value of the lower bit depth.

[0007] As a preferred technical solution of this application, random scrambling processing is performed on the display range size, including the following steps: S211. Obtain all manually set display range sizes, and determine the display range size control interval based on the minimum and maximum display range sizes, and also calculate the standard deviation of all display range sizes; S212. Generate several random values ​​that conform to a normal distribution, wherein the mean of the normal distribution is zero, and the standard deviation of the normal distribution is the same as the calculated standard deviation. S213. For each random value, add the display range size to the random value, and if the result of the addition is not within the display range size control range, adjust the result of the addition to a boundary value of the display range size control range to obtain the randomly scrambled display range size.

[0008] As a preferred technical solution of this application, random scrambling processing is performed on the display reference value, including the following steps: S221. Obtain all manually set display reference values, and determine the display reference value control range based on the minimum and maximum display reference values, and also calculate the standard deviation of all display reference values; S222. Generate several random values ​​that conform to a normal distribution, wherein the mean of the normal distribution is zero, and the standard deviation of the normal distribution is the same as the calculated standard deviation. S223. For each random value, add the display reference value to the random value, and if the result of the addition is not within the control range of the display reference value, adjust the result of the addition to a boundary value of the control range of the display reference value to obtain a randomly scrambled display reference value.

[0009] As a preferred technical solution of this application, each displayed image is manually pre-marked, including: discrete point-based or local area-based location marking within the tumor region of the displayed image.

[0010] As a preferred technical solution of this application, a pixel-labeled main model and a pixel-labeled auxiliary model are jointly trained using all training images, including the following steps: S311. Input the training image into the pixel labeling master model to obtain the output labeling result, calculate the first boundary information based on the output labeling result, and simultaneously input the training image into the pixel labeling auxiliary model to obtain the output second boundary information. S312. Calculate the labeling loss based on the output labeling result and the preliminary labeling result corresponding to the training image, and calculate the boundary loss based on the calculated first boundary information and the output second boundary information. S313. Regarding the main pixel labeling model, the weights of the main pixel labeling model are updated through backpropagation based on the weighted sum of the labeling loss and the boundary loss. Regarding the auxiliary pixel labeling model, the weights of the auxiliary pixel labeling model are updated through backpropagation based on the boundary loss. S314. Determine whether the training termination condition is met. If not, jump to S311. If yes, stop the execution of the step.

[0011] As a preferred technical solution of this application, the method of jointly training a pixel-labeled main model and a pixel-labeled auxiliary model using all training images further includes the following steps: S321. Input the training image into the pixel labeling master model to obtain the output labeling result, calculate the first boundary information based on the output labeling result, and simultaneously input the training image into the pixel labeling auxiliary model to obtain the output second boundary information. S322. Directly perform boundary extraction processing on the training image to obtain third boundary information, and use the third boundary information to correct the first boundary information. S323. Calculate the labeling loss based on the output labeling result and the preliminary labeling result corresponding to the training image, and calculate the boundary loss based on the corrected first boundary information and the output second boundary information. S324. Regarding the main pixel labeling model, update the weights of the main pixel labeling model through backpropagation based on the weighted sum of the labeling loss and the boundary loss. Regarding the auxiliary pixel labeling model, update the weights of the auxiliary pixel labeling model through backpropagation based on the boundary loss. S325. Determine whether the training termination condition is met. If not, jump to S321. If yes, stop the execution of the step.

[0012] This application also provides a human-computer collaborative interactive segmentation and annotation system for tumor images, including the following modules: The interactive module is used to acquire several raw medical images, and for each raw medical image, the display range size and display reference value are manually set to obtain the display image; The expansion module is used to perform random scrambling on each original medical image, either by scrambling the display area size while retaining the display reference value, or by randomly scrambling the display reference value while retaining the display area size, or by randomly scrambling both the display area size and the display reference value, to obtain several display images. The segmentation module is used to manually perform preliminary labeling on each display image to obtain training images. It also uses all the training images to jointly train a pixel-labeled main model and a pixel-labeled auxiliary model. After training is completed, the trained pixel-labeled main model is used to process newly acquired display images.

[0013] Compared with the prior art, the beneficial effects of this application are at least as follows: In the technical solution provided in this application, firstly, several original medical images are acquired, and for each original medical image, the display range size and display reference value are manually set to obtain a display image. Secondly, for each original medical image, random scrambling is applied to the display range size while retaining the display reference value, or random scrambling is applied to the display reference value while retaining the display range size, or random scrambling is applied to both the display range size and the display reference value, resulting in several display images. Finally, each display image is manually pre-labeled to obtain training images, and all training images are used to jointly train a pixel-labeled master model and a pixel-labeled auxiliary model. After training, the trained pixel-labeled master model is used to process newly acquired display images. This application can automatically acquire a large number of display images, solving the problem of insufficient training images for the pixel-labeled master model. Furthermore, this application can train an accurate pixel-labeled master model without requiring accurate manual labeling of display images, solving the problem of high manual costs associated with labeling display images. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart of a human-computer collaborative interactive segmentation and annotation method for tumor images, as described in an embodiment of this application. Figure 2 This is a schematic diagram of a human-computer collaborative interactive segmentation and annotation system for tumor images, as described in an embodiment of this application. Detailed Implementation

[0016] This application provides a human-computer collaborative interactive tumor image segmentation and annotation system and method. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0017] For ease of understanding, the specific process of the embodiments of this application is described below. Please refer to [link / reference]. Figure 1 The human-computer collaborative interactive segmentation and annotation method for tumor images in this application includes the following main steps: S1. Acquire several original medical images, and for each original medical image, manually set the display range size and display reference value to obtain the display image; S2. For each original medical image, random scrambling is performed on the display area size while retaining the display reference value, or random scrambling is performed on the display reference value while retaining the display area size, or random scrambling is performed on both the display area size and the display reference value, in order to obtain several display images; S3. Manually label each display image to obtain training images, and use all the training images to jointly train a pixel labeling master model and a pixel labeling auxiliary model. After training is completed, use the trained pixel labeling master model to process newly acquired display images.

[0018] Specifically, the method proposed in this application is described using a brain tumor as an example. In step S1, multiple original medical images are first obtained from the hospital. The original medical images refer to T2-weighted brain images in DICOM format, which store high-bit-depth pixel values, such as 12 bits. It should be noted that the brain tumor types corresponding to the multiple original medical images are the same. Then, for each original medical image, the display range size and display reference value are manually set to obtain a display image. For example, the doctor sets the display range size (i.e., window width) and display reference value (i.e., window level) that they believe are most suitable for making medical judgments. The obtained display image stores low-bit-depth pixel values, such as 8 bits. In step S2, for each original medical image, multiple display images are obtained through three methods: Method 1: random scrambling is applied to the manually set display area size while retaining the manually set display reference value; Method 2: random scrambling is applied to the manually set display reference value while retaining the manually set display area size; or Method 3: random scrambling is applied to both the manually set display area size and the manually set display reference value. This step can obtain a large number of display images, alleviating the problem of insufficient training data for the semantic segmentation model. In step S3, each display image obtained from all the display images obtained in steps S1 and S2 is manually pre-labeled to obtain training images. It should be noted that the manual pre-labeling does not need to accurately delineate the boundaries of the brain tumor. Then, all the training images are used to jointly train a pixel-labeled main model and a pixel-labeled auxiliary model. After training, the trained pixel-labeled main model is used to process newly acquired display images. Newly acquired display images refer to those obtained after setting the display area size and display reference value for the newly acquired original medical images.

[0019] Furthermore, obtaining the display image based on the display range size and the display reference value includes: calculating the difference between the display reference value and half of the display range size as the lower limit value, calculating the sum of the display reference value and half of the display range size as the upper limit value, and for each pixel in the original medical image, adjusting the pixel value of the higher bit depth to the lower limit value when the pixel value of the higher bit depth is less than the lower limit value, adjusting the pixel value of the higher bit depth to the upper limit value when the pixel value of the higher bit depth is greater than the upper limit value, and converting the pixel value of the higher bit depth to the pixel value of the lower bit depth.

[0020] Specifically, this explains how to obtain the display image based on the display area size and the display reference value. First, calculate the difference between the display reference value and half of the display area size as the lower limit, and calculate the sum of the display reference value and half of the display area size as the upper limit. Second, for each pixel in the original medical image, if the pixel value at a higher bit depth is less than the lower limit, then adjust the pixel value at the higher bit depth to the lower limit; if the pixel value at a higher bit depth is greater than the upper limit, then adjust the pixel value at the higher bit depth to the upper limit. Finally, for each pixel in the original medical image, use the formula: Low bit depth pixel value = [(High bit depth pixel value - Lower limit) / Display area size] × 255 to convert the high bit depth pixel value to the low bit depth pixel value. For example, if the high bit depth pixel value is 1000, the display baseline value is 800, and the display range size is 400, then the lower limit value is 800 - 400 / 2 = 600, the upper limit value is 800 + 400 / 2 = 1000, and the low bit depth pixel value is [(1000 - 600) / 400] × 255 = 255.

[0021] Furthermore, random scrambling is applied to the display area size, including the following steps: S211. Obtain all manually set display range sizes, and determine the display range size control interval based on the minimum and maximum display range sizes, and also calculate the standard deviation of all display range sizes; S212. Generate several random values ​​that conform to a normal distribution, wherein the mean of the normal distribution is zero, and the standard deviation of the normal distribution is the same as the calculated standard deviation. S213. For each random value, add the display range size to the random value, and if the result of the addition is not within the display range size control range, adjust the result of the addition to a boundary value of the display range size control range to obtain the randomly scrambled display range size.

[0022] Specifically, this section describes how to perform random scrambling on the display range size. In step S211, all manually set display range sizes are obtained, i.e., all display range sizes set by the doctor for all original medical images. A display range size control interval is determined based on the minimum and maximum display range sizes. This interval can be defined as [minimum display range size, maximum display range size], or it can be appropriately relaxed, for example, defined as [minimum display range size * (1-10%), maximum display range size * (1+10%)]. This allows for display range sizes that slightly exceed the doctor's set range but remain within clinical limits. The standard deviation of all manually set display range sizes is also calculated. In step S212, multiple random values ​​conforming to a normal distribution are generated, where the mean of the normal distribution is zero, and the standard deviation of the normal distribution is the same as the standard deviation calculated in step S211. In step S213, for each generated random value, the random value is added to the display range size set by the doctor. If the addition result is not within the display range size control interval, the addition result is adjusted to a boundary value of the display range size control interval. That is, if the addition result is less than the low boundary value of the display range size control interval, the addition result is adjusted to the low boundary value; if the addition result is greater than the high boundary value of the display range size control interval, the addition result is adjusted to the high boundary value, thereby obtaining the randomly scrambled display range size.

[0023] Random scrambling is applied to the display reference value, including the following steps: S221. Obtain all manually set display reference values, and determine the display reference value control range based on the minimum and maximum display reference values, and also calculate the standard deviation of all display reference values; S222. Generate several random values ​​that conform to a normal distribution, wherein the mean of the normal distribution is zero, and the standard deviation of the normal distribution is the same as the calculated standard deviation. S223. For each random value, add the display reference value to the random value, and if the result of the addition is not within the control range of the display reference value, adjust the result of the addition to a boundary value of the control range of the display reference value to obtain a randomly scrambled display reference value.

[0024] Specifically, the method for random scrambling of the display reference values ​​is explained as follows: In step S221, all manually set display reference values ​​are obtained, that is, all display reference values ​​set by the doctor for all original medical images. Later, a display reference value control interval is determined based on the minimum and maximum display reference values. Similarly, the display reference value control interval can also be determined as [minimum display reference value, maximum display reference value]. However, since the display reference value is more sensitive than the display range size, further relaxation is not necessary. Then, the standard deviation of all manually set display reference values ​​is calculated. In step S222, multiple random values ​​conforming to a normal distribution are generated, where the mean of the normal distribution is zero, and the standard deviation of the normal distribution is the same as the standard deviation calculated in step S221. In step S223, for each generated random value, the display reference value set by the doctor is added to the random value. If the addition result is not within the display reference value control range, the addition result is adjusted to a boundary value of the display reference value control range. That is, if the addition result is less than the low boundary value of the display reference value control range, the addition result is adjusted to the low boundary value; if the addition result is greater than the high boundary value of the display reference value control range, the addition result is adjusted to the high boundary value, thereby obtaining a randomly scrambled display reference value.

[0025] The preceding text has introduced how to perform random scrambling on the display area size and display reference value. Based on this, multiple display images can be obtained from a single original medical image using either Method 1, Method 2, or Method 3. Further explaining Method 3, in this method, both the display area size and display reference value set by the doctor are randomly scrambled on the original medical image. This allows for the acquisition of multiple randomly scrambled display reference values ​​and multiple randomly scrambled display area sizes. Combining all the randomly scrambled display reference values ​​and all the randomly scrambled display area sizes—for example, if the number of randomly scrambled display reference values ​​is n and the number of randomly scrambled display area sizes is m—results in a total of n * ... There are m combinations, each consisting of a randomly scrambled display reference value and a randomly scrambled display range size. For each combination, the difference between the randomly scrambled display reference value and half of the randomly scrambled display range size is calculated as the lower limit of scrambling, and the sum of the randomly scrambled display reference value and half of the randomly scrambled display range size is calculated as the upper limit of scrambling. Simultaneously, the difference between the doctor-set display reference value and half of the doctor-set display range size is calculated as the lower limit, and the sum of the doctor-set display reference value and half of the doctor-set display range size is calculated as the upper limit. Furthermore, the overlap ratio between the interval [lower limit of scrambling, upper limit of scrambling] and the interval [lower limit, upper limit] is calculated. Specifically, the length of the overlapping part is calculated as a proportion of the doctor-set display range size. If the calculated proportion is greater than a preset proportion threshold, the combination is retained and used to generate the display image; otherwise, the combination is discarded and not used to generate the display image. The proportion threshold is set according to the actual application scenario, for example, 70%.

[0026] Furthermore, each displayed image is manually pre-marked, including: discrete point-based or local area-based location marking within the tumor region of the displayed image.

[0027] Specifically, this section explains how to perform preliminary manual marking for each displayed image. The specific method is as follows: For the displayed image obtained by manually setting the display range size and display reference value, discrete point-based or localized area-covering positional markings are performed within the tumor region of the displayed image. In other words, it is not necessary to precisely delineate the tumor boundary. This significantly reduces the manual cost of marking the displayed image. For example, multiple dense points can be marked within the tumor region of the displayed image, or multiple dense lines can be drawn within the tumor region of the displayed image, or a portion of the tumor can be enclosed in a closed geometric shape within the tumor region of the displayed image. For displayed images obtained through Method 1, Method 2, or Method 3, it is not necessary to repeat the preliminary manual marking; the marking results of the corresponding displayed image obtained by manually setting the display range size and display reference value can be directly copied.

[0028] Furthermore, a pixel-labeled main model and a pixel-labeled auxiliary model are jointly trained using all the training images, including the following steps: S311. Input the training image into the pixel labeling master model to obtain the output labeling result, calculate the first boundary information based on the output labeling result, and simultaneously input the training image into the pixel labeling auxiliary model to obtain the output second boundary information. S312. Calculate the labeling loss based on the output labeling result and the preliminary labeling result corresponding to the training image, and calculate the boundary loss based on the calculated first boundary information and the output second boundary information. S313. Regarding the main pixel labeling model, the weights of the main pixel labeling model are updated through backpropagation based on the weighted sum of the labeling loss and the boundary loss. Regarding the auxiliary pixel labeling model, the weights of the auxiliary pixel labeling model are updated through backpropagation based on the boundary loss. S314. Determine whether the training termination condition is met. If not, jump to S311. If yes, stop the execution of the step.

[0029] Specifically, this describes a first method for jointly training a pixel labeling master model and a pixel labeling auxiliary model using all training images. In step S311, the training images are input into the pixel labeling master model for processing to obtain the output labeling results. The output labeling results include the probability that each pixel in the training image belongs to different categories, including the probability of belonging to the tumor category and the background category. The pixel labeling master model can be a U-Net model, whose encoder sequentially performs multiple stages of processing. Each stage first performs two 3×3 convolutions, followed by 2×2 max pooling downsampling, with the aim of halving the feature map size and doubling the number of channels to gradually extract semantic features. Its bottleneck layer performs two convolutions to obtain the deepest semantic features. Its decoder also sequentially performs multiple stages of processing. Each stage first performs 2×2 upsampling to double the size and halve the number of channels. Skip connections are used to concatenate the feature map with the feature map of the corresponding stage in the encoder to fuse semantic and detail information. Then, two 3×3 convolutions are performed. After multiple stages, a 1×1 convolution and a Softmax activation function are used to obtain the probability that each pixel in the training image belongs to different categories. The first boundary information, i.e., the probability that each pixel in the training image belongs to the tumor edge, is calculated based on the labeling results output by the main pixel labeling model. This can be specifically calculated using the Sobel algorithm. Simultaneously, the training image is also input into the auxiliary pixel labeling model for processing, thereby obtaining the output second boundary information, i.e., the probability that each pixel in the training image belongs to an edge. Here, edges include not only tumor edges but also the edges of other objects. The auxiliary pixel labeling model can be the DexiNed model, which mainly consists of multiple densely connected Inception modules. In each Inception module, convolutional kernels of different scales are used for parallel processing. The different feature maps obtained from the parallel processing are concatenated to obtain the output feature map of each Inception module. The concatenation result of the output feature maps of all the preceding Inception modules is used as the input of the next Inception module. An edge probability map is generated after each Inception module. All edge probability maps are concatenated and fused using a 1×1 convolution to obtain the final edge probability map. The final edge probability map records the probability that each pixel belongs to an edge.

[0030] In step S312, the labeling loss is calculated based on the output labeling results and the preliminary labeling results corresponding to the training image. Specifically, the cross-entropy loss is calculated only on labeled pixels, such as pixels corresponding to manually labeled points, pixels on manually labeled lines, and pixels within manually labeled closed regions. Furthermore, the boundary loss is calculated based on the calculated first boundary information and the output second boundary information, specifically calculating the L1 norm loss or L2 norm loss across the entire image. In step S313, for the pixel labeling master model, the weights of the pixel labeling master model are updated through backpropagation based on the weighted sum of the labeling loss and the boundary loss. The weighted sum is specifically labeling loss + λ * boundary loss. In the early stages of training, λ can be set to a small value, such as 0.01, allowing the pixel labeling master model to focus on learning pixel classification tasks. In the later stages of training, after the pixel labeling master model has a certain classification ability, λ can be gradually increased to allow the boundary loss to gradually take effect, thus refining the tumor boundary. Furthermore, for the pixel labeling auxiliary model, based on boundary loss, the weights of the labeling auxiliary model are updated through backpropagation. As training progresses, the pixel labeling auxiliary model will be gradually trained into an edge detection model that is more sensitive to tumor boundaries. In step S314, it is determined whether the training termination condition is met. If not, the process jumps to step S311 to continue execution; if it is met, the execution of the step is stopped. The training termination condition is, for example, reaching a preset maximum epoch, or, before reaching the maximum epoch, the performance metrics of the pixel labeling main model on the validation set, such as the Dice coefficient, do not improve after several consecutive epochs.

[0031] Furthermore, jointly training a pixel-labeled main model and a pixel-labeled auxiliary model using all the training images also includes the following steps: S321. Input the training image into the pixel labeling master model to obtain the output labeling result, calculate the first boundary information based on the output labeling result, and simultaneously input the training image into the pixel labeling auxiliary model to obtain the output second boundary information. S322. Directly perform boundary extraction processing on the training image to obtain third boundary information, and use the third boundary information to correct the first boundary information. S323. Calculate the labeling loss based on the output labeling result and the preliminary labeling result corresponding to the training image, and calculate the boundary loss based on the corrected first boundary information and the output second boundary information. S324. Regarding the main pixel labeling model, update the weights of the main pixel labeling model through backpropagation based on the weighted sum of the labeling loss and the boundary loss. Regarding the auxiliary pixel labeling model, update the weights of the auxiliary pixel labeling model through backpropagation based on the boundary loss. S325. Determine whether the training termination condition is met. If not, jump to S321. If yes, stop the execution of the step.

[0032] The second method for jointly training a primary pixel labeling model and an auxiliary pixel labeling model using all training images is described below: In step S321, the training images are input into the primary pixel labeling model for processing. The primary pixel labeling model can be a U-Net model, thereby obtaining the output labeling results. The output labeling results include the probability of each pixel in the training image belonging to different categories, including the probability of belonging to the tumor category and the background category. Based on the output labeling results, the first boundary information is calculated, which includes the probability that each pixel in the training image belongs to the tumor edge. Specifically, this can be calculated using the Sobel algorithm. At the same time, the training images are also input into the auxiliary pixel labeling model for processing. The auxiliary pixel labeling model can be a DexiNed model, thereby obtaining the output second boundary information. The second boundary information includes the probability that each pixel in the training image belongs to an edge. Here, the edge includes not only the edge of the tumor but also the edge of other objects. In step S322, boundary extraction processing is directly performed on the training images to obtain the third boundary information. The third boundary information includes the probability that each pixel in the training image belongs to an edge. Here, the edge also includes not only the edge of the tumor but also the edge of other objects. Specifically, the third boundary information can also be obtained using the Sobel algorithm. Then, the first boundary information is corrected based on the third boundary information. Specifically, this includes: locating the tumor contour in the first boundary information by setting a probability threshold; similarly, locating multiple contours in the third boundary information; aligning the third boundary information with the first boundary information in pixel space; identifying the contour closest to the tumor contour among the located contours and determining it as the tumor contour; for each pixel on the tumor contour in the third boundary information, identifying pixels with the same position in the first boundary information; if the edge probability value of the former is greater than that of the latter, then retaining the edge probability value of the former; if the edge probability value of the former is less than or equal to that of the latter, then retaining the edge probability value of the latter, thus obtaining the corrected first boundary information. In step S323, the labeling loss is calculated based on the output labeling result and the preliminary labeling result corresponding to the training image. Specifically, the cross-entropy loss is calculated only on the labeled pixels. Unlike step S312, the boundary loss is calculated based on the corrected first boundary information and the output second boundary information. Specifically, the L1 norm loss or L2 norm loss is calculated across the entire image range. In step S324, similar to step S313, for the pixel labeling master model, the weights of the pixel labeling master model are updated through backpropagation based on the weighted sum of the labeling loss and the boundary loss. For the pixel labeling auxiliary model, the weights of the labeling auxiliary model are updated through backpropagation based on the boundary loss. In step S325, it is checked whether the training termination condition is met. If not, the process jumps to step S321 to continue execution; if it is met, the execution of this step is stopped. The training termination condition here is the same as the training termination condition in step S314.The corrected first boundary information obtained through the second method described above is more accurate, giving the pixel labeling auxiliary model a more accurate learning target and enabling the pixel labeling main model to learn a more precise boundary.

[0033] In summary, the method provided in this application first acquires several original medical images, and for each original medical image, the display range size and display reference value are manually set to obtain a display image. Secondly, for each original medical image, the display range size is randomly scrambled while retaining the display reference value, or the display reference value is randomly scrambled while retaining the display range size, or both the display range size and display reference value are randomly scrambled, resulting in several display images. Finally, each display image is manually pre-labeled to obtain training images, and all training images are used to jointly train a pixel-labeling master model and a pixel-labeling auxiliary model. After training, the trained pixel-labeling master model is used to process newly acquired display images. This application can automatically acquire a large number of display images, solving the problem of insufficient training images for the pixel-labeling master model. Furthermore, this application can train an accurate pixel-labeling master model without requiring accurate manual labeling of display images, solving the problem of high manual costs associated with display image labeling.

[0034] According to another aspect of the embodiments of this application, reference is made to... Figure 2 As shown, this application also provides a human-computer collaborative interactive segmentation and annotation system for tumor images, including an interaction module, an expansion module, and a segmentation module, to realize the human-computer collaborative interactive segmentation and annotation method for tumor images described above.

[0035] The functions of each module are as follows: The interactive module is used to acquire several raw medical images, and for each raw medical image, the display range size and display reference value are manually set to obtain the display image; The expansion module is used to perform random scrambling on each original medical image, either by scrambling the display area size while retaining the display reference value, or by randomly scrambling the display reference value while retaining the display area size, or by randomly scrambling both the display area size and the display reference value, to obtain several display images. The segmentation module is used to manually perform preliminary labeling on each display image to obtain training images. It also uses all the training images to jointly train a pixel-labeled main model and a pixel-labeled auxiliary model. After training is completed, the trained pixel-labeled main model is used to process newly acquired display images.

[0036] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0037] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0038] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A human-computer collaborative interactive segmentation and annotation method for tumor images, characterized in that, The method includes the following steps: S1. Acquire several original medical images, and for each original medical image, manually set the display range size and display reference value to obtain the display image; S2. For each original medical image, random scrambling is performed on the display area size while retaining the display reference value, or random scrambling is performed on the display reference value while retaining the display area size, or random scrambling is performed on both the display area size and the display reference value, in order to obtain several display images; S3. Manually label each display image to obtain training images, and use all the training images to jointly train a pixel labeling master model and a pixel labeling auxiliary model. After training is completed, use the trained pixel labeling master model to process newly acquired display images.

2. The method according to claim 1, characterized in that, The process of obtaining a display image based on the display range size and the display reference value includes: calculating the difference between the display reference value and half of the display range size as the lower limit value; calculating the sum of the display reference value and half of the display range size as the upper limit value; for each pixel in the original medical image, if the pixel value at a higher bit depth is less than the lower limit value, adjusting the pixel value at a higher bit depth to the lower limit value; if the pixel value at a higher bit depth is greater than the upper limit value, adjusting the pixel value at a higher bit depth to the upper limit value; and converting the pixel value at a higher bit depth to a pixel value at a lower bit depth.

3. The method according to claim 1, characterized in that, Random scrambling is applied to the display area size, including the following steps: S211. Obtain all manually set display range sizes, and determine the display range size control interval based on the minimum and maximum display range sizes, and also calculate the standard deviation of all display range sizes; S212. Generate several random values ​​that conform to a normal distribution, wherein the mean of the normal distribution is zero, and the standard deviation of the normal distribution is the same as the calculated standard deviation. S213. For each random value, add the display range size to the random value, and if the result of the addition is not within the display range size control range, adjust the result of the addition to a boundary value of the display range size control range to obtain the randomly scrambled display range size.

4. The method according to claim 1, characterized in that, Random scrambling is applied to the display reference value, including the following steps: S221. Obtain all manually set display reference values, and determine the display reference value control range based on the minimum and maximum display reference values, and also calculate the standard deviation of all display reference values; S222. Generate several random values ​​that conform to a normal distribution, wherein the mean of the normal distribution is zero, and the standard deviation of the normal distribution is the same as the calculated standard deviation. S223. For each random value, add the display reference value to the random value, and if the result of the addition is not within the control range of the display reference value, adjust the result of the addition to a boundary value of the control range of the display reference value to obtain a randomly scrambled display reference value.

5. The method according to claim 1, characterized in that, For each displayed image, preliminary manual marking is performed, including: discrete point marking or local area coverage marking within the tumor region of the displayed image.

6. The method according to claim 5, characterized in that, Jointly train a pixel-labeled main model and a pixel-labeled auxiliary model using all training images, including the following steps: S311. Input the training image into the pixel labeling master model to obtain the output labeling result, calculate the first boundary information based on the output labeling result, and simultaneously input the training image into the pixel labeling auxiliary model to obtain the output second boundary information. S312. Calculate the labeling loss based on the output labeling result and the preliminary labeling result corresponding to the training image, and calculate the boundary loss based on the calculated first boundary information and the output second boundary information. S313. Regarding the main pixel labeling model, the weights of the main pixel labeling model are updated through backpropagation based on the weighted sum of the labeling loss and the boundary loss. Regarding the auxiliary pixel labeling model, the weights of the auxiliary pixel labeling model are updated through backpropagation based on the boundary loss. S314. Determine whether the training termination condition is met. If not, jump to S311. If yes, stop the execution of the step.

7. The method according to claim 5, characterized in that, Jointly training a pixel-labeled main model and a pixel-labeled auxiliary model using all training images also includes the following steps: S321. Input the training image into the pixel labeling master model to obtain the output labeling result, calculate the first boundary information based on the output labeling result, and simultaneously input the training image into the pixel labeling auxiliary model to obtain the output second boundary information. S322. Directly perform boundary extraction processing on the training image to obtain third boundary information, and use the third boundary information to correct the first boundary information. S323. Calculate the labeling loss based on the output labeling result and the preliminary labeling result corresponding to the training image, and calculate the boundary loss based on the corrected first boundary information and the output second boundary information. S324. Regarding the main pixel labeling model, update the weights of the main pixel labeling model through backpropagation based on the weighted sum of the labeling loss and the boundary loss. Regarding the auxiliary pixel labeling model, update the weights of the auxiliary pixel labeling model through backpropagation based on the boundary loss. S325. Determine whether the training termination condition is met. If not, jump to S321. If yes, stop the execution of the step.

8. A human-computer collaborative interactive tumor image segmentation and annotation system, used to implement the method as described in any one of claims 1 to 7, characterized in that, Includes the following modules: The interactive module is used to acquire several raw medical images, and for each raw medical image, the display range size and display reference value are manually set to obtain the display image; The expansion module is used to perform random scrambling on each original medical image, either by scrambling the display area size while retaining the display reference value, or by randomly scrambling the display reference value while retaining the display area size, or by randomly scrambling both the display area size and the display reference value, to obtain several display images. The segmentation module is used to manually perform preliminary labeling on each display image to obtain training images. It also uses all the training images to jointly train a pixel-labeled main model and a pixel-labeled auxiliary model. After training is completed, the trained pixel-labeled main model is used to process newly acquired display images.

Images