An image segmentation method, device, medium and equipment according to a reference image

Abstract

The application discloses an image segmentation method and device based on a reference image, a medium and equipment. The method first inputs a segmented image and a reference image into a deep neural network to obtain a plurality of target feature maps and reference feature maps, wherein the target feature maps are also input into a threshold derivation unit to obtain a threshold; then, the channel support degree corresponding to each reference feature map and the segmented object mask of the reference image is calculated to obtain a support vector, the preliminary mask is calculated according to the support vector, the target feature map and the threshold, the confidence is calculated according to the preliminary mask of each target feature map, the fusion weight of each target feature map is calculated according to the confidence, and finally, the preliminary mask of each target feature map is fused according to the fusion weight to obtain the final segmentation mask. Through adaptive fusion of hierarchical features, the application overcomes the defect of insufficient single feature representation, effectively extracts the features of local details and global semantics, and improves the segmentation accuracy.

Description

Technical Field

[0001] This invention relates to the field of image segmentation. Background Technology

[0002] Deep learning has achieved remarkable success in medical image segmentation, but its performance heavily relies on large amounts of data with pixel-level annotations. However, annotating medical images requires specialized medical knowledge and is a time-consuming and costly task. Furthermore, when dealing with new disease categories, new imaging equipment, or new medical institutions, the challenge of "few samples"—where labeled data is extremely scarce—is frequently encountered, making it exceptionally difficult to train a high-performance, fully supervised model.

[0003] On the other hand, existing technologies typically use a single feature to generate prototypes, which cannot effectively capture the information of local texture and global context across layers in medical images, reducing segmentation accuracy and making them prone to overfitting when model training data is scarce. Summary of the Invention

[0004] The problem to be solved by this invention: 1. Improve segmentation accuracy; 2. Model training under limited sample conditions.

[0005] To solve the above problems, the present invention adopts the following solution: An image segmentation method based on a reference image according to the present invention includes an object segmentation step; the object segmentation step includes the following steps: Step SA1: Receive the segmented image and the reference image; the reference image is a standard-sized image; the reference image includes a mask of the segmented object; Step SA2: Perform image size scaling and alignment processing on the segmented image so that the aligned segmented image is of standard size; Step SA3: Input the aligned segmented image and the reference image into the ResNet deep neural network, respectively, and extract the output from several specified intermediate layers of the ResNet deep neural network. After scaling and aligning to the standard size, the corresponding feature maps at different levels are obtained, thus forming the set {fq(i)|i∈[1..K]} and the set {fs(i)|i∈[1..K]}; where, K is the specified number of intermediate layers; fq(i) represents the i-th feature map of the segmented image; fs(i) represents the i-th feature map referenced from the image; Step SA4: Input each feature map fq(i) into the threshold deduction unit to obtain the corresponding threshold h(i); where, h(i) represents the threshold of the i-th feature map of the segmented image; The threshold deduction unit includes a global average pooling layer and a fully connected layer connected in sequence; In the threshold deduction unit, the feature map fq(i) is input to the fully connected layer after being globally averaged by the global average pooling layer, and the fully connected layer outputs the threshold h(i). Step SA5: Calculate the channel support based on the feature map fs(i) and the mask of the segmented object in the reference image, thus forming the support vector P(i) = {P(i,j)|j∈[1..C(i)]}; where, C(i) is the number of channels in the i-th feature map; P(i) represents the support vectors referenced to the i-th feature map; P(i,j) represents the channel support of the j-th channel of the i-th feature map of the reference image. P(i,j)=sum_pixel(fs(i,j,x,y)*Ms(x,y)) / (sum_pixel(Ms(x,y))+e); where, sum_pixel represents the sum of pixels in the image; fs(i,j,x,y) represents the pixel value of the j-th channel of the i-th feature map of the reference image, with the pixel position (x,y). Ms(x,y) represents the pixel mask value at the pixel position (x,y) of the object being segmented from the reference image. e is used to avoid small quantities where the divisor is 0; Step SA6: Calculate the pixel mask values ​​based on the feature map fq(i), threshold h(i), and support vector P(i), thus forming the preliminary mask Mq1(i) = {Mq1(i,x,y)|x∈[1..W],y∈[1..H]}; where, Mq1(i) represents the initial mask of the i-th feature map of the segmented image; W represents the standard width, and H represents the standard height. Mq1(i,x,y) represents the initial mask value of the pixel at position (x,y) of the i-th feature map of the segmented image. Mq1(i,x,y)=1-Sigmoid(-A*sim(P(i),fq(i,x,y))-h(i)); where, Sigmoid represents the Sigmoid activation function; A is a pre-defined scaling factor; sim represents vector similarity; fq(i,x,y) represents a vector composed of the pixel values ​​of each channel at the position (x,y) of the i-th feature map pixel in the segmented image. Step SA7: Calculate the confidence level Ctrust(i) based on each Mq1(i); where, Ctrust(i) represents the confidence level of the i-th feature map of the segmented image. Ctrust(i)=sqrt(Cintra(i)*Cinter(i)); where, sqrt represents the square root; Cintra(i)=sum_pixel(abs(2*Mq1(i,x,y)-1)) / H / W; Cinter(i)=1-sum_j_1_to_K_exp_i(sum_pixel(abs(Mq1(j,x,y)-Mq1(i,x,y))) / H / W) / (K-1); where, abs represents absolute value; sum_j_1_to_K_exp_i means to accumulate the values ​​of sum_pixel(abs(Mq1(j,x,y)-Mq1(i,x,y))) / H / W within the parentheses from j=1 to j=K, but excluding the case where j=i; Mq1(j,x,y) represents the pixel mask value at position (x,y) of the initial mask pixel of the j-th feature map of the segmented image. Step SA8: Based on each confidence level Ctrust(i), the mask fusion weight weight(i) is obtained after normalization using the Softmax function; where weight(i) represents the mask fusion weight of the i-th feature map of the segmented image. Step SA9: Output the segmentation mask Mqs={Mqs(x,y)|x∈[1..W],y∈[1..H]}; where, Mqs(x,y) represents the pixel mask value at the segmentation mask pixel position (x,y); Mqs(x,y)=sum_i_1_to_K(Mq1(i,x,y)*weight(i)); where, sum_i_1_to_K means to accumulate the value of Mq1(i,x,y)*weight(i) within the parentheses from i=1 to i=K.

[0006] Furthermore, the image segmentation method based on the reference image according to the present invention further includes a model training step; the model training step includes: Step ST1: Receive a training sample set; the training samples include a first image and a second image; both the first image and the second image include a segmented object mask, and the objects segmented by the segmented object masks in the first image and the second image are the same; Step ST2: Using the first image of the training samples as the image to be segmented and the second image as the reference image, perform the object segmentation step and collect the preliminary mask M12b(k,i) output in step SA6 and the segmentation mask M12(k) output in step SA9 in the object segmentation step. Step ST3: Using the second image of the training sample as the image to be segmented and the first image as the reference image, perform the object segmentation step and collect the preliminary mask M21b(k,i) output in step SA6 and the segmentation mask M21(k) output in step SA9 in the object segmentation step. Step ST4: Calculate the total loss based on the BCE loss: L(k)=wa*(BCE(M12(k),Ms2(k))+BCE(M21(k),Ms1(k)))+ wb*sum_i_1_to_K(BCE(M12b(k,i),Ms2(k)+BCE(M21b(k,i),Ms1(k)));wherein L(k) represents the overall loss of the k-th training sample; wa and wb are pre-defined weighting coefficients; BCE represents the BCE loss calculation function; M12(k) represents the segmentation mask obtained by using the first image of the kth training sample as the image to be segmented and the second image as the reference image; M21(k) represents the segmentation mask obtained by using the second image of the kth training sample as the image to be segmented and the first image as the reference image; M12b(k,i) represents the initial mask obtained by using the first image of the k-th training sample as the segmented image and the second image as the reference image to obtain the i-th feature map; M21b(k,i) represents the initial mask with the second image of the kth training sample as the segmented image and the first image as the i-th feature map obtained by referring to the image; Ms1(k) represents the mask of the segmented object in the first image of the kth training sample; Ms2(k) represents the mask of the segmented object in the second image of the kth training sample; sum_i_1_to_K means to accumulate the value of BCE(M12b(k,i),Ms2(k)+BCE(M21b(k,i),Ms1(k)) from i=1 to i=K; Step ST5: Update the model parameters based on the loss L(k).

[0007] According to an image segmentation apparatus based on a reference image, the present invention includes an object segmentation module; the object segmentation module includes the following modules: Module MA1: Receives the segmented image and a reference image; the reference image is a standard-sized image; the reference image includes a mask of the segmented object; Module MA2: Performs image size scaling and alignment processing on the segmented image so that the aligned segmented image is of standard size; Module MA3: Inputs the aligned segmented image and the reference image into the ResNet deep neural network, respectively, and extracts the output from several specified intermediate layers of the ResNet deep neural network. After scaling and aligning to the standard size, it obtains the feature maps corresponding to different layers, thus forming the set {fq(i)|i∈[1..K]} and the set {fs(i)|i∈[1..K]}; where, K is the specified number of intermediate layers; fq(i) represents the i-th feature map of the segmented image; fs(i) represents the i-th feature map referenced from the image; Module MA4: Inputs each feature map fq(i) into the threshold deduction unit to obtain the corresponding threshold h(i); where, h(i) represents the threshold of the i-th feature map of the segmented image; The threshold deduction unit includes a global average pooling layer and a fully connected layer connected in sequence; In the threshold deduction unit, the feature map fq(i) is input to the fully connected layer after being globally averaged by the global average pooling layer, and the fully connected layer outputs the threshold h(i). Module MA5: Calculates channel support based on feature map fs(i) and the mask of the segmented object in the reference image, thus forming support vector P(i) = {P(i,j)|j∈[1..C(i)]}; where, C(i) is the number of channels in the i-th feature map; P(i) represents the support vectors referenced to the i-th feature map; P(i,j) represents the channel support of the j-th channel of the i-th feature map of the reference image. P(i,j)=sum_pixel(fs(i,j,x,y)*Ms(x,y)) / (sum_pixel(Ms(x,y))+e); where, sum_pixel represents the sum of pixels in the image; fs(i,j,x,y) represents the pixel value of the j-th channel of the i-th feature map of the reference image, with the pixel position (x,y). Ms(x,y) represents the pixel mask value at the pixel position (x,y) of the object being segmented from the reference image. e is used to avoid small quantities where the divisor is 0; Module MA6: Calculates pixel mask values ​​based on feature map fq(i), threshold h(i), and support vector P(i), thus forming a preliminary mask Mq1(i) = {Mq1(i,x,y)|x∈[1..W],y∈[1..H]}; where, Mq1(i) represents the initial mask of the i-th feature map of the segmented image; W represents the standard width, and H represents the standard height. Mq1(i,x,y) represents the initial mask value of the pixel at position (x,y) of the i-th feature map of the segmented image. Mq1(i,x,y)=1-Sigmoid(-A*sim(P(i),fq(i,x,y))-h(i)); where, Sigmoid represents the Sigmoid activation function; A is a pre-defined scaling factor; sim represents vector similarity; fq(i,x,y) represents a vector composed of the pixel values ​​of each channel at the position (x,y) of the i-th feature map pixel in the segmented image. Module MA7: Calculates the confidence level Ctrust(i) based on each Mq1(i); where, Ctrust(i) represents the confidence level of the i-th feature map of the segmented image. Ctrust(i)=sqrt(Cintra(i)*Cinter(i)); where, sqrt represents the square root; Cintra(i)=sum_pixel(abs(2*Mq1(i,x,y)-1)) / H / W; Cinter(i)=1-sum_j_1_to_K_exp_i(sum_pixel(abs(Mq1(j,x,y)-Mq1(i,x,y))) / H / W) / (K-1); where, abs represents absolute value; sum_j_1_to_K_exp_i means to accumulate the values ​​of sum_pixel(abs(Mq1(j,x,y)-Mq1(i,x,y))) / H / W within the parentheses from j=1 to j=K, but excluding the case where j=i; Mq1(j,x,y) represents the pixel mask value at position (x,y) of the initial mask pixel of the j-th feature map of the segmented image. Module MA8: Based on each confidence level Ctrust(i), the mask fusion weight weight(i) is obtained after normalization by the Softmax function; where weight(i) represents the mask fusion weight of the i-th feature map of the segmented image; Module MA9: Outputs the segmentation mask Mqs={Mqs(x,y)|x∈[1..W],y∈[1..H]}; where, Mqs(x,y) represents the pixel mask value at the segmentation mask pixel position (x,y); Mqs(x,y)=sum_i_1_to_K(Mq1(i,x,y)*weight(i)); where, sum_i_1_to_K means to accumulate the value of Mq1(i,x,y)*weight(i) within the parentheses from i=1 to i=K.

[0008] Furthermore, the image segmentation apparatus based on the reference image according to the present invention further includes a model training module; the model training module includes: Module MT1: Receives a training sample set; the training samples include a first image and a second image; both the first image and the second image include a segmented object mask, and the objects segmented by the segmented object masks in the first image and the second image are the same; Module MT2: The object segmentation module is executed with the first image of the training sample as the image to be segmented and the second image as the reference image. The module collects the preliminary mask M12b(k,i) output by module MA6 and the segmentation mask M12(k) output by module MA9 in the object segmentation module. Module MT3: The object segmentation module is executed with the second image of the training sample as the image to be segmented and the first image as the reference image. The module collects the preliminary mask M21b(k,i) output by module MA6 and the segmentation mask M21(k) output by module MA9 in the object segmentation module. Module MT4: Calculating total loss based on BCE loss: L(k)=wa*(BCE(M12(k),Ms2(k))+BCE(M21(k),Ms1(k)))+ wb*sum_i_1_to_K(BCE(M12b(k,i),Ms2(k)+BCE(M21b(k,i),Ms1(k)));wherein L(k) represents the overall loss of the k-th training sample; wa and wb are pre-defined weighting coefficients; BCE represents the BCE loss calculation function; M12(k) represents the segmentation mask obtained by using the first image of the kth training sample as the image to be segmented and the second image as the reference image; M21(k) represents the segmentation mask obtained by using the second image of the kth training sample as the image to be segmented and the first image as the reference image; M12b(k,i) represents the initial mask obtained by using the first image of the k-th training sample as the segmented image and the second image as the reference image to obtain the i-th feature map; M21b(k,i) represents the initial mask with the second image of the kth training sample as the segmented image and the first image as the i-th feature map obtained by referring to the image; Ms1(k) represents the mask of the segmented object in the first image of the kth training sample; Ms2(k) represents the mask of the segmented object in the second image of the kth training sample; sum_i_1_to_K means to accumulate the value of BCE(M12b(k,i),Ms2(k)+BCE(M21b(k,i),Ms1(k)) from i=1 to i=K; Module MT5: Updates model parameters based on the loss L(k).

[0009] According to a machine-readable medium of the present invention, the medium stores a set of program instructions that can be loaded and executed by a machine; when the set of program instructions stored in the medium is loaded and executed by a machine, the above-described image segmentation method based on a reference image can be implemented.

[0010] According to an electronic device of the present invention, the device includes a processor and a memory connected together; the memory stores a set of program instructions; characterized in that, when the set of program instructions stored in the memory is executed by the processor, the device is capable of implementing the above-described image segmentation method based on a reference image.

[0011] The technical effects of this invention are as follows: 1. This invention overcomes the shortcomings of insufficient representation by a single feature by extracting features at different levels and then adaptively fusing them, effectively extracting features of local details and global semantics, and improving segmentation accuracy; 2. This invention optimizes model parameters by introducing loss fine-tuning through bidirectional model training of sample pairs and iteratively mining the correspondence between samples. Attached Figure Description

[0012] Figure 1 This is a flowchart of the method according to an embodiment of the present invention.

[0013] Figure 2 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention.

[0014] Figure 3 This is a schematic diagram of the segmentation model in an embodiment of the present invention. Detailed Implementation

[0015] The present invention will now be described in further detail with reference to the accompanying drawings.

[0016] Figure 2 An example of an electronic device is provided, which is a general-purpose computer device in the von Neumann architecture, comprising at least a processor 101 and a memory 102 connected together. The memory 102 is used to store computer program instruction sets and data. The processor 101 implements the image segmentation method based on a reference image as described in this invention by loading and executing the computer program instruction set stored in the memory 102. The memory 102 is also referred to as a machine-readable medium in this invention, typically a persistent storage device, including but not limited to disks, magnetic tapes, solid-state drives, etc. It should be noted that the processor 101 here is a component capable of executing computer program instructions in a broad sense, not limited to a general-purpose processor, or even a single component. For example, in this embodiment, the processor 101 is a combination of a CPU and a GPU, meaning that the functions implemented by the processor 101 are jointly implemented by the CPU and the GPU.

[0017] The image segmentation method based on a reference image of the present invention includes an object segmentation step and a model training step. The object segmentation step, based on a reference image... Figure 1 This includes model execution steps and adaptive fusion steps.

[0018] The model used in the model execution step is the segmentation model referred to in this invention. (See reference...) Figure 3 The segmentation model includes residual network units and several threshold inference units. The residual network unit is the deep neural network ResNet referred to in this invention. Those skilled in the art will understand that there are many different versions of ResNet. Specifically, in this embodiment, the ResNet101 version is used. The threshold inference unit includes a globally average pooling layer and a fully connected layer connected in sequence.

[0019] The model execution steps involve inputting the image to be segmented and the reference image into the segmentation model to obtain several target feature maps, reference feature maps, and feature map thresholds.

[0020] The input to the object segmentation step is the image to be segmented and a reference image. The aforementioned step SA1, receiving the image to be segmented and the reference image, indicates that the image to be segmented and the reference image are the inputs to the object segmentation step. The segmentation model has a fixed image size. Therefore, for the image to be segmented, image size scaling and alignment processing is required to make the aligned image to a standard size. Adjusting the image to the standard size is the aforementioned step SA2. Here, the standard size is the image size input to the segmentation model. Specifically, in this embodiment, the ResNet101 input image size uses the default 224×224, meaning the standard size is 224×224. For the reference image, the image size itself is the standard image, and the reference image also includes a mask for the object to be segmented. For the reference image, the mask for the object to be segmented can be seen as a channel data of the reference image; that is, in addition to the RGB three-color channels, the reference image also includes a mask channel. On the other hand, the mask can be seen as image segmentation; the image of the object to be segmented is obtained by filtering the RGB three-color channels of the reference image through the mask channel.

[0021] It should be noted that the specific method of obtaining the reference image is not within the scope of this invention, which uses the reference image as input for the object segmentation step. In one embodiment, the reference image can be submitted and uploaded by the user. In another optional embodiment, the user can specify an image as the reference image through a user interface. In yet another more complex embodiment, the user inputs the text content of the object to be segmented through a user interface, and then, by parsing the text content, several reference images of the segmented object that match the semantics of the text content are found from the database. Finally, the user selects one of these optional reference images as input.

[0022] The model execution steps can be divided into two steps, namely the aforementioned steps SA3 and SA4.

[0023] Step SA3, specifically, involves inputting the aligned segmented image and the reference image into a ResNet deep neural network, respectively. Outputs are extracted from several specified intermediate layers of the ResNet, scaled and aligned to a standard size, resulting in feature maps corresponding to different layers. These feature maps form sets {fq(i)|i∈[1..K]} and {fs(i)|i∈[1..K]}. Here, K is the specified number of intermediate layers; fq(i) is the target feature map mentioned earlier, obtained by inputting the segmented image into the ResNet, representing the i-th feature map of the segmented image; and fs(i) is the reference feature map mentioned earlier, obtained by inputting the reference image into the ResNet, representing the i-th feature map of the reference image.

[0024] Those skilled in the art will understand that the ResNet101 used in this embodiment is a trainable layer with 101 layers, as described above. Figure 1 The ResNet101 consists of an initial convolutional layer (first layer), a classification layer (last layer), and 99 intermediate layers arranged in 33 residual blocks between the first and last layers. Each residual block includes 3 intermediate layers. The aforementioned "extracting output from several specified intermediate layers of the ResNet" refers to extracting output from several specified intermediate layers among these 99 intermediate layers. Specifically, in this embodiment, ResNet101 is divided into 5 stages. Besides the initial convolutional layer of the first layer being the first stage, the 33 residual blocks are divided into four stages: conv2_x, conv3_x, conv4_x, and conv5_x. In this embodiment, the last convolutional layer of these four stages is used as the aforementioned specified intermediate layer, meaning the number of specified intermediate layers K is 4. The feature maps of the four different levels are as follows: The first feature map, namely fq(1) and fs(1), comes from the output of the last convolutional layer of the residual block 3, that is, the output of the 10th trainable layer. The feature map output of the 10th trainable layer has a size of 56×56 and 256 channels. After size normalization, the final image size of fq(1) and fs(1) is 224×224 and 256 channels. The second feature map, namely fq(2) and fs(2), comes from the output of the last convolutional layer of the residual block 7, that is, the output of the 22nd trainable layer. The feature map output of the 22nd trainable layer has a size of 28×28 and 512 channels. After size normalization, the final image size of fq(2) and fs(2) is 224×224 and 512 channels. The third feature map, namely fq(3) and fs(3), comes from the output of the last convolutional layer of residual block 30, that is, the output of the 91st trainable layer. The feature map output of the 91st trainable layer has a size of 14×14 and 1024 channels. After size normalization, the final image size of fq(3) and fs(3) is 224×224 and the number of channels is 1024. The fourth feature map, namely fq(4) and fs(4), comes from the output of the last convolutional layer of the residual block 3, that is, the output of the 100th trainable layer. The feature map output of the 100th trainable layer has a size of 7×7 and 2048 channels. After size normalization, the final image size of fq(3) and fs(3) is 224×224 and the number of channels is 2048.

[0025] The aforementioned residual blocks 3, 7, 30, and 33 are sequential numbers from the 33 residual blocks of ResNet101. These 33 residual blocks can be sequentially numbered as: Residual Block 1, Residual Block 2, ..., Residual Block 33. In this embodiment, bilinear interpolation is used for image magnification and filling when the feature map size is normalized and enlarged.

[0026] Step SA4, specifically, involves inputting each feature map fq(i) into its corresponding threshold deduction unit to obtain the corresponding threshold h(i). Here, h(i) is the aforementioned feature map threshold, or simply the threshold, representing the threshold of the i-th feature map of the segmented image. "Corresponding to each" means that different levels of feature maps correspond to different threshold deduction units. Specifically, in this embodiment, the four levels of feature maps correspond to four threshold deduction units. More specifically, the first, second, third, and fourth feature maps correspond to the first, second, third, and fourth threshold deduction units, respectively. It should be noted that the feature maps input to the threshold deduction unit for processing are limited to the target feature map; reference feature maps do not need to be input to the threshold deduction unit.

[0027] The threshold estimation unit includes a globally average pooling layer and a fully connected layer connected in sequence. The feature map fq(i) is input to the fully connected layer after being globally average pooled by the globally average pooling layer, and the fully connected layer outputs the threshold h(i). In this embodiment, more specifically: In the first threshold deduction unit, the first feature map with a size of 224×224 and 256 channels is formed into a 256-dimensional vector after global average pooling, and then outputs the threshold h(1) through a 256-dimensional fully connected layer. In the second threshold deduction unit, the second feature map with a size of 224×224 and 512 channels is formed into a 512-dimensional vector after global average pooling, and then outputs the threshold h(2) through a 512-dimensional fully connected layer. In the third threshold deduction unit, the second feature map with a size of 224×224 and 1024 channels is formed into a 1024-dimensional vector after global average pooling, and then outputs the threshold h(3) through a 1024-dimensional fully connected layer. In the fourth threshold derivation unit, the second feature map with a size of 224×224 and 2048 channels is transformed into a 2048-dimensional vector after global average pooling, and then outputs the threshold h (4) through a 2048-dimensional fully connected layer.

[0028] The adaptive fusion process can be divided into four steps, namely the aforementioned steps SA5, SA6, SA7, SA8 and SA9.

[0029] Step SA5: Calculate the channel support based on the feature map fs(i) and the mask of the segmented object in the reference image, thereby forming the support vector P(i)={P(i,j)|j∈[1..C(i)]}; Step SA6: Calculate the pixel mask value based on the feature map fq(i), threshold h(i) and support vector P(i) to form the preliminary mask Mq1(i)={Mq1(i,x,y)|x∈[1..W],y∈[1..H]}; Step SA7: Calculate the confidence level Ctrust(i) based on each Mq1(i); Step SA8: Based on each confidence level Ctrust(i), the mask fusion weight weight(i) is obtained after normalization using the Softmax function; Step SA9: Output the segmentation mask of the segmented image Mqs={Mqs(x,y)|x∈[1..W],y∈[1..H]}.

[0030] In step SA5 above, C(i) is the number of channels in the i-th feature map. Specifically, in this embodiment, the number of channels in the four-level feature maps are 256, 512, 1024, and 2048, respectively. Correspondingly, C(1) = 256, C(2) = 512, C(3) = 1024, and C(4) = 2048. P(i) represents the support vector of the i-th feature map. Specifically, in this embodiment, the support vectors of the four-level feature maps are P(1), P(2), P(3), and P(4), with dimensions of 256, 512, 1024, and 2048, respectively. P(i,j) represents the channel support of the j-th channel of the i-th feature map of the image. The channel support P(i,j) is calculated using the following formula: P(i,j)=sum_pixel(fs(i,j,x,y)*Ms(x,y)) / (sum_pixel(Ms(x,y))+e); where, sum_pixel represents the sum of pixels in the image; fs(i,j,x,y) represents the pixel value of the j-th channel of the i-th feature map of the reference image, with the pixel position (x,y). Ms(x,y) represents the pixel mask value at the pixel position (x,y) of the object being segmented from the reference image. e is used to avoid small quantities where the divisor is 0.

[0031] In step SA6 above, W is the width of the standard size and H is the height of the standard size. Specifically, in this embodiment, W=224 and H=224. Mq1(i) represents the preliminary mask of the i-th feature map of the segmented image, and Mq1(i,x,y) represents the pixel mask value of the pixel at position (x,y) of the preliminary mask of the i-th feature map of the segmented image. The pixel mask value Mq1(i,x,y) is calculated according to the following formula: Mq1(i,x,y)=1-Sigmoid(-A*sim(P(i),fq(i,x,y))-h(i)); where, Sigmoid represents the Sigmoid activation function; A is a pre-set scaling factor, and in this embodiment, A is preferably 20; sim represents vector similarity, and in this embodiment, the cosine similarity calculation formula is preferably used; fq(i,x,y) represents a vector composed of the pixel values ​​of each channel of the i-th feature map pixel at position (x,y) in the segmented image.

[0032] In step SA7 above, Ctrust(i) represents the confidence level of the i-th feature map of the segmented image. The confidence level Ctrust(i) is calculated using the following formula: Ctrust(i)=sqrt(Cintra(i)*Cinter(i)); where, sqrt represents the square root; Cintra(i)=sum_pixel(abs(2*Mq1(i,x,y)-1)) / H / W; Cinter(i)=1-sum_j_1_to_K_exp_i(sum_pixel(abs(Mq1(j,x,y)-Mq1(i,x,y))) / H / W) / (K-1); where, abs represents absolute value; sum_j_1_to_K_exp_i means to accumulate the values ​​of sum_pixel(abs(Mq1(j,x,y)-Mq1(i,x,y))) / H / W within the parentheses from j=1 to j=K, but excluding the case where j=i; Mq1(j,x,y) represents the pixel mask value at position (x,y) of the initial mask pixel of the j-th feature map of the segmented image. sum_pixel represents the sum of pixels in the image.

[0033] In step SA9 above, weight(i) represents the mask fusion weight of the i-th feature map of the segmented image. Mqs(x,y) represents the pixel mask value at pixel position (x,y). The pixel mask value Mqs(x,y) is calculated according to the following formula: Mqs(x,y)=sum_i_1_to_K(Mq1(i,x,y)*weight(i)); where, sum_i_1_to_K means to accumulate the value of Mq1(i,x,y)*weight(i) within the parentheses from i=1 to i=K.

[0034] In this embodiment, the pixel mask value is represented as a floating-point number between 0.0 and 1.0. In another optional implementation, the pixel mask value is 0 or 1. In this case, Mqs(x,y) can round the floating-point number between 0.0 and 1.0 to either 0 or 1.

[0035] In this embodiment, initially, the parameters in the ResNet deep neural network are directly taken from the parameters of the pre-trained model, while the parameters in the threshold inference unit are randomly generated. Finally, the parameters in the segmentation model are obtained through model training.

[0036] Model training, or the aforementioned model training steps, specifically includes the following steps: Step ST1: Receive the training sample set. Here, "receive" means the training sample set is the input for the model training step. The training samples in the training sample set are collected manually. The training samples include a first image and a second image. Both the first and second images include a mask for the segmented object, and the objects segmented by the mask in both images are the same. In this embodiment, the masks for the segmented object in the first and second images are manually annotated, and the pairing relationship between the first and second images is manually specified.

[0037] Step ST2: Using the first image of the training sample as the image to be segmented and the second image as the reference image, perform the aforementioned object segmentation step, and collect the preliminary mask M12b(k,i) output by step SA6 and the segmentation mask M12(k) output by step SA9 in the object segmentation step.

[0038] Step ST3: Using the second image of the training sample as the image to be segmented and the first image as the reference image, perform the object segmentation step and collect the preliminary mask M21b(k,i) output by step SA6 and the segmentation mask M21(k) output by step SA9 in the object segmentation step.

[0039] Step ST4: Calculate the total loss L(k) based on the BCE loss.

[0040] Step ST5: Update the model parameters based on the loss L(k).

[0041] In the model training step, k represents the k-th training sample, that is: L(k) represents the overall loss of the k-th training sample; M12(k) represents the segmentation mask obtained by using the first image of the kth training sample as the image to be segmented and the second image as the reference image; M21(k) represents the segmentation mask obtained by using the second image of the kth training sample as the image to be segmented and the first image as the reference image; M12b(k,i) represents the initial mask obtained by using the first image of the k-th training sample as the segmented image and the second image as the reference image to obtain the i-th feature map; M21b(k,i) represents the initial mask with the second image of the kth training sample as the segmented image and the first image as the i-th feature map obtained by referring to the image; Ms1(k) represents the mask of the segmented object in the first image of the kth training sample; Ms2(k) represents the mask of the segmented object in the second image of the k-th training sample.

[0042] In step ST4, the total loss L(k) is calculated using the following formula: L(k)=wa*(BCE(M12(k),Ms2(k))+BCE(M21(k),Ms1(k)))+ wb*sum_i_1_to_K(BCE(M12b(k,i),Ms2(k)+BCE(M21b(k,i),Ms1(k))). Wherein... wa and wb are pre-defined weighting coefficients; sum_i_1_to_K means to accumulate the value of BCE(M12b(k,i),Ms2(k)+BCE(M21b(k,i),Ms1(k)) from i=1 to i=K; BCE represents the BCE loss calculation function.

[0043] The above steps indicate that the model training employs a bidirectional strategy, and the loss calculation considers feature maps at different levels. In actual model training, training samples can be divided into batches of different sizes according to different stages of model training. When updating model parameters, the overall loss for the entire batch is calculated according to the loss L(k), and then the model parameters are updated based on the overall loss.

[0044] Furthermore, it should be noted that the image segmentation device based on the reference image mentioned above is a virtual device implemented by executing computer program instructions. Its modules correspond one-to-one with the steps in the image segmentation method based on the reference image, and need not be elaborated further.

Claims

1. An image segmentation method based on a reference image, characterized in that, The process includes an object segmentation step; the object segmentation step includes the following steps: Step SA1: Receive the segmented image and the reference image; the reference image is a standard-sized image; the reference image includes a mask of the segmented object; Step SA2: Perform image size scaling and alignment processing on the segmented image so that the aligned segmented image is of standard size; Step SA3: Input the aligned segmented image and the reference image into the ResNet deep neural network, respectively, and extract the output from several specified intermediate layers of the ResNet deep neural network. After scaling and aligning to the standard size, the corresponding feature maps at different levels are obtained, thus forming the set {fq(i)|i∈[1..K]} and the set {fs(i)|i∈[1..K]}; where, K is the specified number of intermediate layers; fq(i) represents the i-th feature map of the segmented image; fs(i) represents the i-th feature map referenced from the image; Step SA4: Input each feature map fq(i) into its corresponding threshold deduction unit to obtain the corresponding threshold h(i); where, h(i) represents the threshold of the i-th feature map of the segmented image; The threshold deduction unit includes a global average pooling layer and a fully connected layer connected in sequence; In the threshold deduction unit, the feature map fq(i) is input to the fully connected layer after being globally averaged by the global average pooling layer, and the fully connected layer outputs the threshold h(i). Step SA5: Calculate the channel support based on the feature map fs(i) and the mask of the segmented object in the reference image, thus forming the support vector P(i) = {P(i,j)|j∈[1..C(i)]}; where, C(i) is the number of channels in the i-th feature map; P(i) represents the support vectors referenced to the i-th feature map; P(i,j) represents the channel support of the j-th channel of the i-th feature map of the reference image. P(i,j)=sum_pixel(fs(i,j,x,y)*Ms(x,y)) / (sum_pixel(Ms(x,y))+e); where, sum_pixel represents the sum of pixels in the image; fs(i,j,x,y) represents the pixel value of the j-th channel of the i-th feature map of the reference image, with the pixel position (x,y). Ms(x,y) represents the pixel mask value at the position (x,y) of the object mask of the reference image being segmented; e is used to avoid small quantities where the divisor is 0; Step SA6: Calculate the pixel mask values ​​based on the feature map fq(i), threshold h(i), and support vector P(i), thus forming the preliminary mask Mq1(i) = {Mq1(i,x,y)|x∈[1..W],y∈[1..H]}; where, Mq1(i) represents the initial mask of the i-th feature map of the segmented image; W represents the standard width, and H represents the standard height. Mq1(i,x,y) represents the initial mask value of the pixel at position (x,y) of the i-th feature map of the segmented image. Mq1(i,x,y)=1-Sigmoid(-A*sim(P(i),fq(i,x,y))-h(i)); where, Sigmoid represents the Sigmoid activation function; A is a pre-defined scaling factor; sim represents vector similarity; fq(i,x,y) represents a vector composed of the pixel values ​​of each channel at the position (x,y) of the i-th feature map pixel in the segmented image. Step SA7: Calculate the confidence level Ctrust(i) based on each Mq1(i); where, Ctrust(i) represents the confidence level of the i-th feature map of the segmented image. Ctrust(i)=sqrt(Cintra(i)*Cinter(i)); where, sqrt represents the square root; Cintra(i)=sum_pixel(abs(2*Mq1(i,x,y)-1)) / H / W; Cinter(i)=1-sum_j_1_to_K_exp_i(sum_pixel(abs(Mq1(j,x,y)-Mq1(i,x,y))) / H / W) / (K-1); where, abs represents absolute value; sum_j_1_to_K_exp_i means to accumulate the values ​​of sum_pixel(abs(Mq1(j,x,y)-Mq1(i,x,y))) / H / W within the parentheses from j=1 to j=K, but excluding the case where j=i; Mq1(j,x,y) represents the pixel mask value at position (x,y) of the initial mask pixel of the j-th feature map of the segmented image. Step SA8: Based on each confidence level Ctrust(i), the mask fusion weight weight(i) is obtained after normalization using the Softmax function; where weight(i) represents the mask fusion weight of the i-th feature map of the segmented image. Step SA9: Output the segmentation mask Mqs={Mqs(x,y)|x∈[1..W],y∈[1..H]}; where, Mqs(x,y) represents the pixel mask value at the segmentation mask pixel position (x,y); Mqs(x,y)=sum_i_1_to_K(Mq1(i,x,y)*weight(i)); where, sum_i_1_to_K means to accumulate the value of Mq1(i,x,y)*weight(i) within the parentheses from i=1 to i=K.

2. The image segmentation method based on a reference image according to claim 1, characterized in that, It also includes a model training step; the model training step includes: Step ST1: Receive a training sample set; the training samples include a first image and a second image; both the first image and the second image include a segmented object mask, and the objects segmented by the segmented object masks in the first image and the second image are the same; Step ST2: Using the first image of the training samples as the image to be segmented and the second image as the reference image, perform the object segmentation step and collect the preliminary mask M12b(k,i) output by step SA6 and the segmentation mask M12(k) output by step SA9 in the object segmentation step. Step ST3: Using the second image of the training sample as the image to be segmented and the first image as the reference image, perform the object segmentation step and collect the preliminary mask M21b(k,i) output in step SA6 and the segmentation mask M21(k) output in step SA9 in the object segmentation step. Step ST4: Calculate the total loss based on the BCE loss: L(k)=wa*(BCE(M12(k),Ms2(k))+BCE(M21(k),Ms1(k)))+ wb*sum_i_1_to_K(BCE(M12b(k,i),Ms2(k)+BCE(M21b(k,i),Ms1(k)));wherein L(k) represents the overall loss of the k-th training sample; wa and wb are pre-defined weighting coefficients; BCE represents the BCE loss calculation function; M12(k) represents the segmentation mask obtained by using the first image of the kth training sample as the image to be segmented and the second image as the reference image; M21(k) represents the segmentation mask obtained by using the second image of the kth training sample as the image to be segmented and the first image as the reference image; M12b(k,i) represents the initial mask obtained by using the first image of the k-th training sample as the segmented image and the second image as the reference image to obtain the i-th feature map; M21b(k,i) represents the initial mask with the second image of the kth training sample as the segmented image and the first image as the i-th feature map obtained by referring to the image; Ms1(k) represents the mask of the segmented object in the first image of the kth training sample; Ms2(k) represents the mask of the segmented object in the second image of the kth training sample; sum_i_1_to_K means to accumulate the value of BCE(M12b(k,i),Ms2(k)+BCE(M21b(k,i),Ms1(k)) from i=1 to i=K; Step ST5: Update the model parameters based on the loss L(k).

3. An image segmentation apparatus based on a reference image, characterized in that, Includes an object segmentation module; the object segmentation module includes the following modules: Module MA1: Receives the segmented image and a reference image; the reference image is a standard-sized image; the reference image includes a mask of the segmented object; Module MA2: Performs image size scaling and alignment processing on the segmented image so that the aligned segmented image is of standard size; Module MA3: Inputs the aligned segmented image and the reference image into the ResNet deep neural network, respectively, and extracts the output from several specified intermediate layers of the ResNet deep neural network. After scaling and aligning to the standard size, it obtains the feature maps corresponding to different layers, thus forming the set {fq(i)|i∈[1..K]} and the set {fs(i)|i∈[1..K]}; where, K is the specified number of intermediate layers; fq(i) represents the i-th feature map of the segmented image; fs(i) represents the i-th feature map referenced from the image; Module MA4: Inputs each feature map fq(i) into its corresponding threshold deduction unit to obtain the corresponding threshold h(i); where, h(i) represents the threshold of the i-th feature map of the segmented image; The threshold deduction unit includes a global average pooling layer and a fully connected layer connected in sequence; In the threshold deduction unit, the feature map fq(i) is input to the fully connected layer after being globally averaged by the global average pooling layer, and the fully connected layer outputs the threshold h(i). Module MA5: Calculates channel support based on feature map fs(i) and the mask of the segmented object in the reference image, thus forming support vector P(i) = {P(i,j)|j∈[1..C(i)]}; where, C(i) is the number of channels in the i-th feature map; P(i) represents the support vectors referenced to the i-th feature map; P(i,j) represents the channel support of the j-th channel of the i-th feature map of the reference image. P(i,j)=sum_pixel(fs(i,j,x,y)*Ms(x,y)) / (sum_pixel(Ms(x,y))+e); where, sum_pixel represents the sum of pixels in the image; fs(i,j,x,y) represents the pixel value of the j-th channel of the i-th feature map of the reference image, with the pixel position (x,y). Ms(x,y) represents the pixel mask value at the position (x,y) of the object mask of the reference image being segmented; e is used to avoid small quantities where the divisor is 0; Module MA6: Calculates pixel mask values ​​based on feature map fq(i), threshold h(i), and support vector P(i), thus forming a preliminary mask Mq1(i) = {Mq1(i,x,y)|x∈[1..W],y∈[1..H]}; where, Mq1(i) represents the initial mask of the i-th feature map of the segmented image; W represents the standard width, and H represents the standard height. Mq1(i,x,y) represents the initial mask value of the pixel at position (x,y) of the i-th feature map of the segmented image. Mq1(i,x,y)=1-Sigmoid(-A*sim(P(i),fq(i,x,y))-h(i)); where, Sigmoid represents the Sigmoid activation function; A is a pre-defined scaling factor; sim represents vector similarity; fq(i,x,y) represents a vector composed of the pixel values ​​of each channel at the position (x,y) of the i-th feature map pixel in the segmented image. Module MA7: Calculates the confidence level Ctrust(i) based on each Mq1(i); where, Ctrust(i) represents the confidence level of the i-th feature map of the segmented image. Ctrust(i)=sqrt(Cintra(i)*Cinter(i)); where, sqrt represents the square root; Cintra(i)=sum_pixel(abs(2*Mq1(i,x,y)-1)) / H / W; Cinter(i)=1-sum_j_1_to_K_exp_i(sum_pixel(abs(Mq1(j,x,y)-Mq1(i,x,y))) / H / W) / (K-1); where, abs represents absolute value; sum_j_1_to_K_exp_i means to accumulate the values ​​of sum_pixel(abs(Mq1(j,x,y)-Mq1(i,x,y))) / H / W within the parentheses from j=1 to j=K, but excluding the case where j=i; Mq1(j,x,y) represents the pixel mask value at position (x,y) of the initial mask pixel of the j-th feature map of the segmented image. Module MA8: Based on each confidence level Ctrust(i), the mask fusion weight weight(i) is obtained after normalization by the Softmax function; where weight(i) represents the mask fusion weight of the i-th feature map of the segmented image; Module MA9: Outputs the segmentation mask Mqs={Mqs(x,y)|x∈[1..W],y∈[1..H]}; where, Mqs(x,y) represents the pixel mask value at the segmentation mask pixel position (x,y); Mqs(x,y)=sum_i_1_to_K(Mq1(i,x,y)*weight(i)); where, sum_i_1_to_K means to accumulate the value of Mq1(i,x,y)*weight(i) within the parentheses from i=1 to i=K.

4. The image segmentation apparatus based on a reference image according to claim 3, characterized in that, It also includes a model training module; the model training module includes: Module MT1: Receives a training sample set; the training samples include a first image and a second image; both the first image and the second image include a segmented object mask, and the objects segmented by the segmented object masks in the first image and the second image are the same; Module MT2: The object segmentation module is executed with the first image of the training sample as the image to be segmented and the second image as the reference image. The module collects the preliminary mask M12b(k,i) output by module MA6 and the segmentation mask M12(k) output by module MA9 in the object segmentation module. Module MT3: The object segmentation module is executed with the second image of the training sample as the image to be segmented and the first image as the reference image. The module collects the preliminary mask M21b(k,i) output by module MA6 and the segmentation mask M21(k) output by module MA9 in the object segmentation module. Module MT4: Calculating total loss based on BCE loss: L(k)=wa*(BCE(M12(k),Ms2(k))+BCE(M21(k),Ms1(k)))+ wb*sum_i_1_to_K(BCE(M12b(k,i),Ms2(k)+BCE(M21b(k,i),Ms1(k)));wherein L(k) represents the overall loss of the k-th training sample; wa and wb are pre-defined weighting coefficients; BCE represents the BCE loss calculation function; M12(k) represents the segmentation mask obtained by using the first image of the kth training sample as the image to be segmented and the second image as the reference image; M21(k) represents the segmentation mask obtained by using the second image of the kth training sample as the image to be segmented and the first image as the reference image; M12b(k,i) represents the initial mask obtained by using the first image of the k-th training sample as the segmented image and the second image as the reference image to obtain the i-th feature map; M21b(k,i) represents the initial mask with the second image of the kth training sample as the segmented image and the first image as the i-th feature map obtained by referring to the image; Ms1(k) represents the mask of the segmented object in the first image of the kth training sample; Ms2(k) represents the mask of the segmented object in the second image of the kth training sample; sum_i_1_to_K means to accumulate the value of BCE(M12b(k,i),Ms2(k)+BCE(M21b(k,i),Ms1(k)) from i=1 to i=K; Module MT5: Updates model parameters based on the loss L(k).

5. A machine-readable medium, characterized in that, The medium stores a set of program instructions that can be loaded and executed by a machine; when the set of program instructions stored in the medium is loaded and executed by a machine, the image segmentation method based on the reference image as described in claim 1 or 2 can be implemented.

6. An electronic device comprising a processor and a memory connected together; the memory storing a program instruction set; characterized in that, When the program instruction set stored in the memory is executed by the processor, the device is able to implement the image segmentation method based on a reference image as described in claim 1 or 2.

Images