Image segmentation method, system and storage medium based on full convolutional network

Abstract

The present invention provides an image segmentation method, system and storage medium based on a full convolutional network, the method comprising: performing a first full convolutional processing on an input image, the first full convolutional processing comprising sequentially compressing, decompressing, image segmenting and reconstructing the input image, and obtaining a first reconstructed image after the first full convolutional processing; obtaining a reconstructed error image between the first reconstructed image and the input image, and fusing the reconstructed error image with the first reconstructed image to obtain a predicted image; performing a second full convolutional processing on the predicted image, the first full convolutional processing comprising sequentially compressing, decompressing and image segmenting the predicted image, and obtaining a foreground segmented image after the second full convolutional processing. In the scheme of the present invention, the full convolutional network model is simple to implement and the image segmentation result obtained has high accuracy.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image segmentation method, system and storage medium based on a fully convolutional network. Background Art

[0002] With the rapid development of computer vision technology, image segmentation algorithms have become a key step in the image processing process. Image segmentation is the process of separating the foreground and background of an image. An appropriate image segmentation algorithm needs to meet the requirements of accurate and fast segmentation results. Traditional image segmentation methods include region growing method, clustering method, segmentation method based on graph matching, and segmentation method based on anomaly detection. Traditional segmentation methods still require human intervention and do not achieve complete automatic segmentation. They have poor robustness and still have large errors in segmentation results.

[0003] In recent years, image segmentation methods based on deep learning have achieved rapid development. However, the current deep learning models are large in size and difficult to implement. In order to meet the processing requirements of deep learning models, some schemes need to significantly reduce the resolution of the input image, resulting in a large amount of information loss in the input image, and the final segmentation results are not accurate. Summary of the invention

[0004] The present invention aims to solve the technical problem that the existing image segmentation model is difficult to implement or the segmentation result is of low accuracy. To this end, the present invention proposes an image segmentation method, system and storage medium based on a fully convolutional network.

[0005] In view of the above technical problems, some embodiments of the present invention provide an image segmentation method based on a fully convolutional network, comprising the following steps:

[0006] Performing a first full convolution process on the input image, the first full convolution process comprising sequentially compressing, decompressing, segmenting and reconstructing the input image, and obtaining a first reconstructed image after the first full convolution process;

[0007] Obtaining a reconstruction error image between the first reconstructed image and the input image, and fusing the reconstruction error image with the first reconstructed image to obtain a predicted image;

[0008] A second full convolution process is performed on the predicted image. The first full convolution process includes sequentially compressing, decompressing and segmenting the predicted image. After the second full convolution process, a foreground segmentation image is obtained.

[0009] In the image segmentation method based on the full convolution network provided by some embodiments of the present invention, in the first full convolution processing and the second full convolution processing, compression, decompression and image segmentation are achieved through the following steps:

[0010] Compression step: The input image or predicted image is compressed in N stages to obtain the compressed image Y N The compression process of the nth stage includes: obtaining the image Y to be compressed obtained in the (n-1)th stage n-1 , reduce the Y of the image to be compressed n-1 The resolution of the low-resolution image YL to be compressed is obtained n-1 ; The image to be compressed Y n-1 After downsampling, the image Y to be compressed in the nth stage is obtained. n ; Wherein, 1≤n≤N, and when n=1, the input image or the predicted image is used as the image to be compressed obtained in the previous stage;

[0011] Decompression step: Decompression of the compressed image Y N The decompression process of N stages is performed in sequence to obtain a decompressed image. The decompression process of the mth stage includes: obtaining the image to be decompressed J obtained in the (m-1)th stage m-1 , get the image Y to be compressed (N-m+1) , treat the decompressed image J m-1 and the image Y to be compressed (N-m+1) The resolution enhancement process is performed together to obtain a high-resolution image to be decompressed JG m-1 ; The image to be decompressed J m-1 After upsampling, the image J to be decompressed in the mth stage is obtained. m ; Where 1≤m≤N, and when m=1, the compressed image Y obtained by the Nth stage compression process N As the image to be decompressed obtained in the previous stage;

[0012] Image segmentation step: Segment the decompressed image to obtain foreground segmentation image and background segmentation image.

[0013] In the image segmentation method based on the full convolution network provided by some embodiments of the present invention, in the first full convolution process and the second full convolution process, N=5 in the compression step and the decompression step, wherein:

[0014] Except for the first-stage compression process and the fifth-stage decompression process in the first full convolution processing and the first-stage compression process and the fifth-stage decompression process in the second full convolution processing, the compression process and decompression process in the remaining stages are implemented using depthwise separable convolutional layers.

[0015] Some embodiments of the present invention provide an image segmentation method based on a fully convolutional network. In the image segmentation steps of the first full convolution processing and the second full convolution processing, the decompressed image is processed by a convolution layer with a kernel size of 1×1 and a normalization processing function to segment the decompressed image into a foreground segmentation image and a background segmentation image.

[0016] The image segmentation method based on the fully convolutional network provided by some embodiments of the present invention further includes:

[0017] The final reconstructed image is obtained by reconstructing the foreground segmentation image and the background segmentation image obtained by the second full convolution process;

[0018] The similarity between the final reconstructed image and the input image is obtained by determining the model through the reconstruction loss completed in advance;

[0019] The variable parameters in the first full convolution process and the second full convolution process are adjusted according to the similarity result until the similarity result reaches a maximum value.

[0020] Some embodiments of the present invention provide an image segmentation system based on a fully convolutional network, comprising:

[0021] First full convolution model: performing a first full convolution process on the input image, the first full convolution process includes sequentially compressing, decompressing, image segmenting and image reconstruction on the input image, obtaining a first reconstructed image after the first full convolution process; obtaining a reconstructed error image between the first reconstructed image and the input image, and fusing the reconstructed error image with the first reconstructed image to obtain a predicted image;

[0022] Second full convolution model: performing a second full convolution processing on the predicted image, the first full convolution processing includes compressing, decompressing and image segmenting the predicted image in sequence, and obtaining a foreground segmentation image after the second full convolution processing.

[0023] In a fully convolutional network-based image segmentation system provided by some embodiments of the present invention, both the first fully convolutional model and the second fully convolutional model include:

[0024] Compression module: The input image or predicted image is compressed in N stages to obtain the compressed image Y N The compression process of the nth stage includes: obtaining the image Y to be compressed obtained in the (n-1)th stage n-1 , reduce the Y of the image to be compressed n-1 The resolution of the low-resolution image YL to be compressed is obtained n-1 ; The image to be compressed Y n-1 After downsampling, the image Y to be compressed in the nth stage is obtained. n ; Wherein, 1≤n≤N, and when n=1, the input image or the predicted image is used as the image to be compressed obtained in the previous stage;

[0025] Decompression module: Decompression of the compressed image Y NThe decompression process of N stages is performed in sequence to obtain a decompressed image. The decompression process of the mth stage includes: obtaining the image to be decompressed J obtained in the (m-1)th stage m-1 , get the image Y to be compressed (N-m+1) , treat the decompressed image J m-1 and the image Y to be compressed (N-m+1) The resolution enhancement process is performed together to obtain a high-resolution image to be decompressed JG m-1 ; The image to be decompressed J m-1 After upsampling, the image J to be decompressed in the mth stage is obtained. m ; Where 1≤m≤N, and when m=1, the compressed image Y obtained by the Nth stage compression process N As the image to be decompressed obtained in the previous stage;

[0026] Image segmentation module: Segment the decompressed image to obtain foreground segmentation image and background segmentation image.

[0027] In the image segmentation system based on the fully convolutional network provided by some embodiments of the present invention, in the image segmentation steps of the first full convolution processing and the second full convolution processing, the decompressed image is processed by a convolution layer with a kernel size of 1×1 and a normalization processing function, and the decompressed image is segmented into a foreground segmentation image and a background segmentation image.

[0028] The image segmentation system based on the fully convolutional network provided by some embodiments of the present invention further includes:

[0029] A reconstruction loss determination model is provided, wherein a final reconstructed image is obtained after reconstructing the foreground segmentation image and the background segmentation image obtained according to the second full convolution processing model; the similarity between the final reconstructed image and the input image is obtained; and variable parameters in the first full convolution processing and the second full convolution processing are adjusted according to the similarity result until the similarity result reaches a maximum value.

[0030] In some embodiments of the present invention, a storage medium is provided, in which program information is stored. After a computer reads the program information, it executes any of the above-mentioned image segmentation methods based on a fully convolutional network.

[0031] The technical solution of the present invention has the following technical effects compared with the prior art: by performing two full convolution processes on the input image, and the deviation brought by the first full convolution process is also used as the input of the second full convolution process, thereby reducing the image loss of the input image during the processing process and ensuring the accuracy of the image segmentation result. Moreover, this solution can be implemented after improvement based on the existing U-Net network, reducing the difficulty of model implementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, which will help to understand the objects and advantages of the present invention, wherein:

[0033] Figure 1 A flowchart of an image segmentation method based on a fully convolutional network according to an embodiment of the present invention;

[0034] Figure 2 This is a schematic diagram of a fully convolutional network architecture of an image segmentation method based on a fully convolutional network according to an embodiment of the present invention;

[0035] Figure 3 A schematic diagram of an image segmentation result obtained by using the image segmentation method based on a fully convolutional network according to an embodiment of the present invention;

[0036] Figure 4 This is a structural block diagram of an image segmentation system based on a fully convolutional network according to an embodiment of the present invention;

[0037] Figure 5 A schematic diagram of the hardware connection relationship of a device for executing an image segmentation method based on a fully convolutional network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0040] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0041] This embodiment provides an image segmentation method based on a fully convolutional network, which is applied to an image processing system, such as Figure 1 As shown, the following steps are included:

[0042] S101: performing a first full convolution process on an input image, wherein the first full convolution process includes sequentially compressing, decompressing, segmenting and reconstructing the input image, and obtaining a first reconstructed image after the first full convolution process.

[0043] S102: Obtain a reconstructed error image between the first reconstructed image and the input image, and fuse the reconstructed error image with the first reconstructed image to obtain a predicted image.

[0044] S103: performing a second full convolution process on the predicted image. The first full convolution process includes sequentially compressing, decompressing and segmenting the predicted image. After the second full convolution process, a foreground segmentation image is obtained.

[0045] refer to Figure 2 Given the image processing network architecture and Figure 3 The image segmentation result shown in FIG. 1 is a result of the image segmentation shown in FIG. 1 . The key to this solution is to provide two fully convolutional processing networks connected together to automatically segment the input image. The steps performed in step S101 and step S102 correspond to Figure 2 The full convolution process provided by UEnc on the left side of the figure, step S103 corresponds to Figure 2 The full convolution process provided by UDec on the right. In UEnc on the left, the input image is segmented in sequence using the first full convolution process. There is a high probability that there is some deviation between the image reconstructed by the segmentation result and the input image. After that, the deviation between the image reconstructed by the first segmentation result and the first segmentation result is input to UDec on the right, and the second full convolution process is performed on the right to obtain the final segmentation result. Figure 2 It can be seen that the network architecture adopted in this embodiment is equivalent to a double U-shaped architecture obtained by expanding the typical U-shaped architecture of the U-Net network. The existing U-Net network model is used to process gray images to process single-channel input images. This solution improves and expands it to more accurately reconstruct the original input image and predict the segmentation map.

[0046] The image fusion proposed in the embodiment of the present invention can be implemented by using the existing image fusion method, for example: superposition of pixel values ​​at the same position in two images.

[0047] The above solution provided in this embodiment performs two full convolution processes on the input image, and the deviation brought by the first full convolution process is also used as the input of the second full convolution process, thereby reducing the image loss of the input image during the processing and ensuring the accuracy of the image segmentation result. Moreover, this solution can be implemented after improvement based on the existing U-Net network, reducing the difficulty of model implementation.

[0048] Preferably, in the first full convolution process and the second full convolution process, compression, decompression and image segmentation are achieved by the following steps:

[0049] Compression step: The input image or predicted image is compressed in N stages to obtain the compressed image Y N The compression process of the nth stage includes: obtaining the image Y to be compressed obtained in the (n-1)th stage n-1 , reduce the Y of the image to be compressed n-1 The resolution of the low-resolution image YL to be compressed is obtained n-1 ; The image to be compressed Y n-1 After nonlinear and normalized processing, the captured information S is obtained n-1 The image to be compressed Y n-1 After downsampling, the image Y to be compressed in the nth stage is obtained. n ; Wherein, 1≤n≤N, and when n=1, the input image or the predicted image is used as the image to be compressed obtained in the previous stage.

[0050] Decompression step: Decompression of the compressed image Y N The decompression process of N stages is performed in sequence to obtain a decompressed image. The decompression process of the mth stage includes: obtaining the image to be decompressed J obtained in the (m-1)th stage m-1 , get the image Y to be compressed (N-m+1) Capture information S (N-m+1) , treat the decompressed image J m-1 With the capture information S (N-m+1) The resolution enhancement process is performed together to obtain a high-resolution image to be decompressed JG m-1 ; The image to be decompressed J m-1 After upsampling, the image J to be decompressed in the mth stage is obtained. m ; Where 1≤m≤N, and when m=1, the compressed image Y obtained by the Nth stage compression process N As the image to be decompressed obtained in the previous stage.

[0051] Image segmentation step: Segment the decompressed image to obtain foreground segmentation image and background segmentation image.

[0052] like Figure 2As shown, the network architecture provided by the present application includes a total of 18 convolution modules, each module contains more than 2 convolution layers, preferably each convolution module can be implemented by selecting 3×3 convolution layers, and in specific implementation, each convolution layer in each convolution module performs a nonlinear (can be implemented by ReLU function) and batch normalization processing (can be implemented by Softmax function) to reduce the resolution of the image, and the image to be compressed obtained at each stage is not only input to the next convolution module that performs compression, but also directly jumps to the corresponding decompression stage in the decompression process (corresponding to the skip layer arrow in the figure). Take the input image resolution of 224×224 as shown in the figure as an example:

[0053] On the left side UEnc of the network structure, in two adjacent compression stages, the image to be compressed output by the convolution module used in the previous compression stage is connected to the convolution module of the next compression stage through a 2×2 maximum pooling layer downsampling, that is, the number of feature channels in the convolution module of the next stage can be doubled in each downsampling process. Based on a similar principle, in the decompression process, the convolution module in the decompression process is implemented by a transposed convolution algorithm corresponding to the convolution algorithm in the convolution module of the compression stage. The decompression process is the process of expanding the compression result. The image to be decompressed output by the convolution module of the previous decompression stage is connected to the convolution module of the next decompression stage through a transposed 2×2 convolution pooling layer upsampling, so the number of feature channels is halved in each upsampling step. In this scheme, in the compression path, the data obtained in each compression process can be used as the decompression step of the corresponding stage, and the spatial information lost due to downsampling in the compression process can be restored in the decompression stage.

[0054] In some schemes, in the image segmentation step of the first full convolution process and the second convolution process, the decompressed image is processed by a convolution layer with a kernel size of 1×1 and a normalization processing function to segment the decompressed image into a foreground segmentation image and a background segmentation image. Theoretically, two feature maps can be calculated through one convolution layer, and the two feature maps have the same volume size. Finally, Softmax (normalization processing function) is applied to perform probability segmentation of the foreground and background areas on the pixels to obtain the segmented image. The so-called probability segmentation means: for a pixel, if the probability that it belongs to the foreground part is large, it is considered that it should be segmented into the segmented image of the foreground part. If the probability that it belongs to the background part is large, it is considered that it should be segmented into the segmented image of the background part. Figure 2As shown, the final output image resolution of the left UEnc of the full convolutional network architecture is 224×224. In the last convolution module of the left UEnc, the convolution layer selects a 1×1 convolution connected to a Softmax layer to achieve image segmentation. The 1x1 convolution maps each 64-component feature vector to the required number of classes K, and then the Softmax layer rescales them so that the elements of the K-dimensional output are in the range (0,1) and the sum is 1. The architecture of the right UDec is similar to the left UEnc, except that it reads the output of the left UEnc of size 224×224×K. The last convolution module of the right UDec also selects a 1×1 convolution to map the 64-component feature vector back to the reconstruction of the original input image.

[0055] Figure 2 The figure shows the preferred network architecture of the present application, that is, in the first full convolution processing and the second full convolution processing, N=5 in the compression step and the decompression step, wherein: except for the first stage compression process and the fifth stage decompression process in the first full convolution processing, and the first stage compression process and the fifth stage decompression process in the second full convolution processing, the compression process and decompression process of the remaining stages are all implemented using a depth-separable convolution layer. Figure 2 The fully convolutional network architecture given in the embodiment includes 18 convolutional modules. An important improvement of this solution is that except for convolutional module 1, convolutional module 9, convolutional module 10 and convolutional module 18, all modules use depth-separable convolutional layers. The depth-separable convolution operation consists of depth-wise convolution and point-wise convolution. The depth-separable convolutional layer can realize independent inspection of spatial correlation and cross-channel correlation, that is, the depth-wise convolution performs spatial convolution independently on each channel, and then the point-by-point convolution projects the feature channel to a new channel space through depth-wise convolution. Therefore, the network architecture provided by this embodiment can obtain more effective image segmentation information, so that the final image segmentation effect is better. In the fully convolutional network architecture proposed in the present application, it can be seen that no fully connected layers are included, which allows it to learn arbitrarily large images and perform segmentation predictions on the corresponding sizes.

[0056] from Figure 2It can be seen that the reconstructed error image between the first reconstructed image output by the left UEnc of the fully convolutional network architecture and the input image is obtained by the soft normalized cut loss function (Soft Normalized Cut Loss: Soft N-Cut Loss). The output of the left UEnc is a K-class 224×224 resolution image after normalization. By summing the K-class images, the K-class prediction results for each pixel in the decompressed image are obtained. The normalized cut loss function is to perform normalization calculation on the K-class prediction results, and the normalized calculation results are used as the global standard for image segmentation. The above scheme is essentially to add the loss functions in the K-class prediction results, but the sum function such as the "argmax function" cannot be differentiated, so the corresponding gradient cannot be calculated during the back propagation process. On this basis, the normalized cut loss function can be smoothed to obtain its corresponding soft version, making it differentiable, specifically as the processing process from formula (1) to formula (2).

[0057]

[0058] Ncut k (V) is the normalized loss function, Ak is the set of pixels in the k-th predicted image, V is the set of all predicted image pixels, and w is the weight between the two pixels u and v or u and t being measured;

[0059]

[0060] Where p(u=A k ) is used to measure the pixel u belongs to A k The probability function of the class is directly calculated by the corresponding encoder in the prior art. In practical application, by using J soft-N cut loss is minimized as the left U Enc The standard for completing training.

[0061] The calculation result J of the soft normalized loss function in this step is soft-N cut Minimizing the loss can make the reconstructed image obtained in the first full convolution process as similar to the input image as possible.

[0062] Based on similar principles, Figure 2As shown, the above scheme may also include: obtaining a final reconstructed image after reconstructing the foreground segmentation image and the background segmentation image obtained by the second full convolution process; obtaining the similarity between the final reconstructed image and the input image (i.e. obtaining the reconstruction error Reconstruction Loss); adjusting the variable parameters in the first full convolution process and the second full convolution process according to the similarity result until the similarity result reaches the maximum value. The error function selected in this step may also be a soft normalized loss function, and its calculation process is similar to that in the first full convolution process, and may also be determined by the following loss function:

[0063]

[0064] Where W Enc represents the parameters in the convolution module during the first full convolution process, W Dec represents the parameters in the convolution module during the second full convolution process, and X is the input image. By minimizing the J between the reconstructed image and the original input reconstr The fully convolutional network model in the present application is trained so that the fully convolutional network model in the present application can eventually reconstruct a reconstructed image with the highest similarity to the input image.

[0065] Further preferably, the above solution provided by the present application may also include the following steps: processing the final reconstructed image obtained in the second convolution process by a random field model to restore the edge information of the foreground part of the final reconstructed image to obtain an edge-restored image; hierarchically segmenting the edge-restored image output by the random field model to obtain a foreground segmentation image and a background segmentation image. Wherein: the random field model is implemented using an energy function:

[0066] E(X)=∑ u Φ(u)+∑ u,v Ψ(u,v);

[0067] where u, v are pixels on the input image X. Unary potential Φ(u) = -log p(u), where p(u) is U Enc The softmax layer in ρ(u,v) computes the label annotation probabilities. The pairwise potential Ψ(u,v) measures the weighted penalty when two pixels are assigned different labels by using two Gaussian kernels in different feature spaces.

[0068] Hierarchical segmentation involves obtaining over-segmented partitions of the input image after summing the outputs of the fully connected random field model, and then appropriately merging the segments to form the final image segments. In this scheme, the over-segmented partitions are first converted into weighted boundary maps, and then the most similar regions are merged iteratively. The "importance" of each pixel on the boundary of the initial over-segmented partition is measured by calculating a weighted combination of multi-scale local cues based on spectral clustering and global boundary measurements, and then a hierarchical segmentation is constructed from the weighted boundaries. This step is equivalent to using a greedy graph-based region merging algorithm to obtain a directional watershed transform and ultrametric contour map of the over-segmented region.

[0069] In some embodiments of the present invention, an image segmentation system based on a fully convolutional network is also provided. Figure 4 As shown, including:

[0070] The first full convolution model 401: performs a first full convolution process on the input image, the first full convolution process includes sequentially compressing, decompressing, image segmenting and image reconstruction on the input image, and obtains a first reconstructed image after the first full convolution process; obtains a reconstructed error image between the first reconstructed image and the input image, and obtains a predicted image after fusing the reconstructed error image with the first reconstructed image;

[0071] The second full convolution model 402: performs a second full convolution processing on the predicted image. The first full convolution processing includes compressing, decompressing and segmenting the predicted image in sequence. After the second full convolution processing, a foreground segmentation image is obtained.

[0072] The above solution provided in this embodiment performs two full convolution processes on the input image, and the deviation brought by the first full convolution process is also used as the input of the second full convolution process, thereby reducing the image loss of the input image during the processing and ensuring the accuracy of the image segmentation result. Moreover, this solution can be implemented based on the existing U-Net network after improvement, which reduces the difficulty of model implementation.

[0073] In the above scheme, the image segmentation system based on the fully convolutional network includes the following:

[0074] Compression module: The input image or predicted image is compressed in N stages to obtain the compressed image Y N The compression process of the nth stage includes: obtaining the image Y to be compressed obtained in the (n-1)th stage n-1 , reduce the Y of the image to be compressed n-1 The resolution of the low-resolution image YL to be compressed is obtained n-1 ; The image to be compressed Y n-1After downsampling, the image Y to be compressed in the nth stage is obtained. n ; Where 1≤n≤N, and when n=1, the input image or predicted image is used as the image to be compressed obtained in the previous stage. Decompression module: N The decompression process of N stages is performed in sequence to obtain a decompressed image. The decompression process of the mth stage includes: obtaining the image to be decompressed J obtained in the (m-1)th stage m-1 , get the image Y to be compressed (N-m+1) , treat the decompressed image J m-1 and the image Y to be compressed (N-m+1) The resolution enhancement process is performed together to obtain a high-resolution image to be decompressed JG m-1 ; The image to be decompressed J m-1 After upsampling, the image J to be decompressed in the mth stage is obtained. m ; Where 1≤m≤N, and when m=1, the compressed image Y obtained by the Nth stage compression process N As the image to be decompressed obtained in the previous stage. Image segmentation module: Segment the decompressed image to obtain the foreground segmentation image and the background segmentation image.

[0075] In two adjacent compression stages, the image to be compressed output by the convolution module used in the previous compression stage is connected to the convolution module of the next compression stage through downsampling, that is, the number of feature channels in the convolution module of the next stage can be doubled in each downsampling process. Based on a similar principle, in the decompression process, the convolution module in the decompression process is realized by a transposed convolution algorithm corresponding to the convolution algorithm in the convolution module of the compression stage. The decompression process is the process of expanding the compression result. The image to be decompressed output by the convolution module of the previous decompression stage is connected to the convolution module of the next decompression stage through upsampling of the transposed convolution pooling layer, so the number of feature channels is halved in each upsampling step. In this scheme, in the compression path, the data obtained in each compression process can be used as the decompression step of the corresponding stage, and the spatial information lost due to downsampling in the compression process can be restored in the decompression stage. In addition, the image to be compressed obtained in each stage is not only input into the next convolution module that performs compression, but also directly jumps to the corresponding decompression stage in the decompression process, so as to avoid the adverse effect of image information loss in the compression process on image reconstruction.

[0076] Preferably, in the first full convolution processing model 401 of the above scheme, the decompressed image is processed by a convolution layer with a kernel size of 1×1 and a normalization processing function to segment the decompressed image into a foreground segmentation image and a background segmentation image; in the second full convolution processing model 402, the decompressed image obtained by decompression is processed by a random field model to restore the edge information of the foreground part of the decompressed image to obtain an edge-restored image; the edge-restored image output by the random field model is hierarchically segmented to obtain a foreground segmentation image and a background segmentation image. Two feature maps can be calculated through a convolution layer, and the two feature maps have the same size volume. Finally, Softmax (normalization processing function) is applied to perform probability segmentation of the foreground and background areas on the pixels to obtain the segmented image.

[0077] The image segmentation system based on the full convolution network in some schemes also includes a reconstruction loss determination model, and the final reconstructed image is obtained after reconstructing the foreground segmentation image and the background segmentation image obtained by the second full convolution processing model; the similarity between the final reconstructed image and the input image is obtained; and the variable parameters in the first full convolution processing and the second full convolution processing are adjusted according to the similarity result until the similarity result reaches a maximum value. The full convolution network model in this application is trained by minimizing the loss between the reconstructed image and the original input, so that the full convolution network model in this application can finally reconstruct a reconstructed image with the highest similarity to the input image.

[0078] Some embodiments of the present invention provide a storage medium having program instructions stored therein. After a computer reads the program instructions, the computer executes the image segmentation method based on a fully convolutional network as described in any of the method embodiments above.

[0079] Figure 5 5 is a schematic diagram of the hardware structure of a device for executing an image segmentation method based on a fully convolutional network provided in this embodiment. The device includes one or more processors 501 and a memory 502. Figure 5 A processor 501 is taken as an example. The device may further include: an input device 503 and an output device 504. The processor 501, the memory 502, the input device 503 and the output device 504 may be connected via a bus or other means.

[0080] The memory 502 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The processor 501 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 502, that is, the image segmentation method based on the full convolutional network of the above method embodiment is implemented. The above-mentioned device can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0081] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0082] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0083] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. An image segmentation method based on a fully convolutional network, characterized in that: The steps include: Performing a first full convolution process on the input image, the first full convolution process comprising sequentially compressing, decompressing, segmenting and reconstructing the input image, and obtaining a first reconstructed image after the first full convolution process; Obtaining a reconstruction error image between the first reconstructed image and the input image, and fusing the reconstruction error image with the first reconstructed image to obtain a predicted image; A second full convolution process is performed on the predicted image, the second full convolution process includes sequentially compressing, decompressing and segmenting the predicted image, and obtaining a foreground segmentation image after the second full convolution process.

2. The image segmentation method based on a fully convolutional network according to claim 1, characterized in that: In the first full convolution process and the second full convolution process, compression, decompression and image segmentation are achieved through the following steps: Compression step: The input image or predicted image is compressed in N stages to obtain the compressed image Y N The compression process of the nth stage includes: obtaining the image Y to be compressed obtained in the (n-1)th stage n-1 , reduce the Y of the image to be compressed n-1 The resolution of the low-resolution image YL to be compressed is obtained n-1 ; The image to be compressed Y n-1 After downsampling, the image Y to be compressed in the nth stage is obtained. n ; Wherein, 1≤n≤N, and when n=1, the input image or the predicted image is used as the image to be compressed obtained in the previous stage; Decompression step: Decompression of the compressed image Y N The decompression process of N stages is performed in sequence to obtain a decompressed image. The decompression process of the mth stage includes: obtaining the image to be decompressed J obtained in the (m-1)th stage m-1 , get the image Y to be compressed (N-m+1) , treat the decompressed image J n-1 and the image Y to be compressed (N-m+1) The resolution enhancement process is performed together to obtain a high-resolution image to be decompressed JG m-1 ; The image to be decompressed J m-1 After upsampling, the image J to be decompressed in the mth stage is obtained. m ; Where 1≤m≤N, and when m=1, the compressed image Y obtained by the Nth stage compression process N As the image to be decompressed obtained in the previous stage; Image segmentation step: Segment the decompressed image to obtain foreground segmentation image and background segmentation image.

3. The image segmentation method based on a fully convolutional network according to claim 2, characterized in that: In the first full convolution process and the second full convolution process, N=5 in the compression step and the decompression step, where: Except for the first-stage compression process and the fifth-stage decompression process in the first full convolution processing and the first-stage compression process and the fifth-stage decompression process in the second full convolution processing, the compression process and decompression process in the remaining stages are implemented using depthwise separable convolutional layers.

4. The image segmentation method based on a fully convolutional network according to claim 3, characterized in that: In the image segmentation steps of the first full convolution processing and the second full convolution processing, the decompressed image is processed by a convolution layer with a kernel size of 1×1 and a normalization processing function to segment the decompressed image into a foreground segmentation image and a background segmentation image.

5. The image segmentation method based on a fully convolutional network according to any one of claims 1 to 4, characterized in that: Also includes: The final reconstructed image is obtained by reconstructing the foreground segmentation image and the background segmentation image obtained by the second full convolution process; The similarity between the final reconstructed image and the input image is obtained by determining the model through the reconstruction loss completed in advance; The variable parameters in the first full convolution process and the second full convolution process are adjusted according to the similarity result until the similarity result reaches a maximum value.

6. An image segmentation system based on a fully convolutional network, characterized in that: include: First full convolution model: performing a first full convolution process on the input image, the first full convolution process includes sequentially compressing, decompressing, image segmenting and image reconstruction on the input image, obtaining a first reconstructed image after the first full convolution process; obtaining a reconstructed error image between the first reconstructed image and the input image, and fusing the reconstructed error image with the first reconstructed image to obtain a predicted image; Second full convolution model: performing a second full convolution processing on the predicted image, the second full convolution processing includes compressing, decompressing and image segmenting the predicted image in sequence, and obtaining a foreground segmentation image after the second full convolution processing.

7. The image segmentation system based on a fully convolutional network according to claim 6, characterized in that: Both the first fully convolutional model and the second fully convolutional model include: Compression module: The input image or predicted image is compressed in N stages to obtain the compressed image Y N The compression process of the nth stage includes: obtaining the image Y to be compressed obtained in the (n-1)th stage n-1 , reduce the Y of the image to be compressed n-1 The resolution of the low-resolution image YL to be compressed is obtained n-1 ; The image to be compressed Y n-1 After downsampling, the image Y to be compressed in the nth stage is obtained. n ; Wherein, 1≤n≤N, and when n=1, the input image or the predicted image is used as the image to be compressed obtained in the previous stage; Decompression module: Decompression of the compressed image Y N The decompression process of N stages is performed in sequence to obtain a decompressed image. The decompression process of the mth stage includes: obtaining the image to be decompressed J obtained in the (m-1)th stage m-1 , get the image Y to be compressed (N-m+1) , treat the decompressed image J n-1 and the image Y to be compressed (N-m+1) The resolution enhancement process is performed together to obtain a high-resolution image to be decompressed JG m-1 ; The image to be decompressed J m-1 After upsampling, the image J to be decompressed in the mth stage is obtained. m ; Where 1≤m≤N, and when m=1, the compressed image Y obtained by the Nth stage compression process N As the image to be decompressed obtained in the previous stage; Image segmentation module: Segment the decompressed image to obtain foreground segmentation image and background segmentation image.

8. The image segmentation system based on a fully convolutional network according to claim 7, characterized in that: In the image segmentation steps of the first full convolution processing and the second full convolution processing, the decompressed image is processed by a convolution layer with a kernel size of 1×1 and a normalization processing function to segment the decompressed image into a foreground segmentation image and a background segmentation image.

9. The image segmentation system based on a fully convolutional network according to any one of claims 6 to 8, characterized in that: Also includes: A reconstruction loss determination model is used to reconstruct the foreground segmentation image and the background segmentation image obtained by the second full convolution processing model to obtain a final reconstructed image; The similarity between the final reconstructed image and the input image is obtained; and the variable parameters in the first full convolution process and the second full convolution process are adjusted according to the similarity result until the similarity result reaches a maximum value.

10. A storage medium, characterized in that: The storage medium stores program information, and after the computer reads the program information, it executes the image segmentation method based on the fully convolutional network described in any one of claims 1 to 5.

Images