Image processing method and device

Abstract

Embodiments of the present invention provide an image processing method and apparatus, wherein the method comprises: obtaining an image to be processed; inputting the image to be processed into a first image processing model for image segmentation to obtain an initial segmentation result, a contour probability mask, and hidden layer feature parameters; using a second image processing model to intercept a target image from the image to be processed based on the contour probability mask, and refining the target image based on the initial segmentation result and hidden layer feature parameters to obtain a target segmentation result. The first image processing model generates a contour probability mask, which provides a benchmark for the subsequent refinement process, so that the image segmentation network structure only corrects the uncertain regions in the segmentation result, reducing a large amount of redundant calculations, improving the speed of image segmentation, and achieving a very precise segmentation effect.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image processing method and an image processing device. Background Art

[0002] In some video call scenarios, it is necessary to block the background of the characters in the video, or to replace the background arbitrarily for entertainment purposes. For example, when a user is video chatting with a stranger through the video call function, some privacy-conscious users sometimes do not want to expose too much information about their home environment. After all, safety is the top priority when facing strangers. Therefore, it is necessary to block or replace the background in the video screen.

[0003] In the existing technology, the background of the video image is segmented and then masked or replaced. The difficulty in the field of video image segmentation is how to achieve both speed and accuracy, which is often a contradiction. In recent years, deep learning-based image segmentation algorithms such as Mask RCNN and DeepLABV3+ are still good choices. However, although these algorithms can achieve good results on low-resolution images, they cannot achieve accurate segmentation on high-resolution images. Figure 1 The diagram below shows a schematic diagram of an image after segmentation. Although these algorithms can roughly segment the characters, the details are still relatively rough when you look closely. In addition, in terms of segmentation speed, it is difficult for the above two image segmentation networks to achieve real-time segmentation on 4K high-definition video.

[0004] Therefore, it is necessary to solve the problem that the image segmentation algorithm has rough edge contour segmentation in high-definition images, and the problem that it cannot perform real-time segmentation in high-definition videos due to the limitation of segmentation speed. Summary of the Invention

[0005] In view of the problem that the above-mentioned image segmentation algorithm has rough edge contour segmentation in high-definition images, and is unable to perform real-time segmentation in high-definition videos due to the limitation of segmentation speed, an embodiment of the present invention is proposed to provide an image processing method and a corresponding image processing device that overcome the above-mentioned problems or at least partially solve the above-mentioned problems.

[0006] An embodiment of the present invention discloses an image processing method, comprising:

[0007] Get the image to be processed;

[0008] Inputting the image to be processed into a first image processing model for image segmentation to obtain an initial segmentation result, a contour probability mask and hidden layer feature parameters;

[0009] A second image processing model is used to intercept a target image from the image to be processed according to the contour probability mask, and the target image is refined according to the initial segmentation result and hidden layer feature parameters to obtain a target segmentation result.

[0010] Optionally, inputting the image to be processed into a first image processing model for image segmentation to obtain an initial segmentation result, a contour probability mask, and hidden layer feature parameters includes:

[0011] reducing the image to be processed according to a first preset coefficient;

[0012] The reduced image to be processed is input into the first image processing model, and the Encoder module, ASPP module and Decoder module of the first image processing model are used to perform image segmentation to obtain an initial segmentation result, a contour probability mask and hidden layer feature parameters.

[0013] Optionally, the adopting of the second image processing model, intercepting a target image from the image to be processed according to the contour probability mask, and performing a refinement process on the target image according to the initial segmentation result and hidden layer feature parameters to obtain a target segmentation result, includes:

[0014] reducing the image to be processed according to a second preset coefficient;

[0015] Combining the initial segmentation result, the hidden layer feature parameters and the reduced image to be processed to obtain refined data;

[0016] The refined data is input into the second image processing model, and the second image processing model is used to intercept the target image from the reduced image to be processed according to the contour probability mask, and the target image is refined according to the initial segmentation result and the hidden layer feature parameters to obtain the target segmentation result.

[0017] Optionally, the first image processing model is trained and generated in the following manner:

[0018] Get a sample image;

[0019] The sample image is used to train an original image segmentation model; wherein the original image segmentation model includes a segmentation result network and a contour probability mask network, the segmentation result network is trained using a first loss function, and the contour probability mask network is trained using a second loss function;

[0020] When it is determined through the first loss function and the second loss function that the original image segmentation module meets the requirements, the training of the original image segmentation module is stopped, and the original image segmentation module is determined to be the first image processing model.

[0021] Optionally, the first loss function is expressed as follows:

[0022]

[0023] Wherein, F represents the segmentation result, F* represents the label of the image, ▽F represents the Sobel gradient operator of the segmentation result, and ▽F* represents the Sobel gradient operator of the image label.

[0024] Optionally, the second loss function is expressed as follows:

[0025] LossE=||EE * ||2;

[0026] Wherein, E represents the contour probability mask, E* represents the label of E, and E*=|FF*|.

[0027] Optionally, the second image processing model is trained and generated in the following manner:

[0028] According to the initial segmentation result, the contour probability mask and the hidden layer feature parameters, the original refined network module is trained by the first loss function to obtain a second image processing model.

[0029] The embodiment of the present invention further discloses an image processing device, comprising:

[0030] An image acquisition module, used for acquiring an image to be processed;

[0031] An image segmentation module is used to input the image to be processed into a first image processing model for image segmentation to obtain an initial segmentation result, a contour probability mask and hidden layer feature parameters;

[0032] An image refinement module is used to use a second image processing model to intercept a target image from the image to be processed according to the contour probability mask, and to refine the target image according to the initial segmentation result and hidden layer feature parameters to obtain a target segmentation result.

[0033] An embodiment of the present invention further discloses an electronic device, including:

[0034] one or more processors; and

[0035] One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, enable the electronic device to perform one or more steps of the method as described in the embodiments of the present invention.

[0036] An embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, causes the processors to perform one or more steps of the method described in the embodiment of the present invention.

[0037] The embodiments of the present invention include the following advantages:

[0038] In an embodiment of the present invention, an image to be processed is input into a first image processing model for image segmentation to obtain an initial segmentation result, a contour probability mask, and hidden layer feature parameters. A second image processing model is then used to extract a target image from the image to be processed based on the contour probability mask, and the target image is refined based on the initial segmentation result and hidden layer feature parameters to obtain a target segmentation result. By designing the first image processing model to generate a contour probability mask, which provides a benchmark for subsequent refinement, the image segmentation network structure only corrects the uncertain regions in the segmentation result, reducing a large amount of redundant computation, increasing the speed of image segmentation, and achieving a very precise segmentation effect. This allows the background in the video image to be masked while accurately preserving the outline of the person (down to the hairline) without affecting the smoothness of the video call. Furthermore, the background replacement function can make video calls more entertaining. Furthermore, other entertainment features such as emoticons can be added, providing a good user experience in terms of both security and entertainment. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the description of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 It is a schematic diagram of an image segmentation result in the prior art;

[0041] Figure 2 is a flowchart of steps of an embodiment of an image processing method of the present invention;

[0042] Figure 3 is a schematic diagram of an image segmentation result of the present invention;

[0043] Figure 4 is a schematic diagram of a model network architecture of the present invention;

[0044] Figure 5A is a schematic diagram of a segmentation result of the present invention;

[0045] Figure 5B is a schematic diagram of a contour probability mask of the present invention;

[0046] Figure 6 is a schematic diagram of a sample image of the present invention;

[0047] Figure 7 It is a structural block diagram of an embodiment of an image processing device of the present invention. DETAILED DESCRIPTION

[0048] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are also within the scope of protection of the present invention.

[0049] To address the problems of rough edge contour segmentation in high-definition images and the inability to perform real-time segmentation in high-definition videos due to segmentation speed limitations in existing image segmentation algorithms, an image segmentation method is provided in an embodiment of the present invention. The method comprises inputting an image to be processed into a first image processing model for image segmentation to obtain an initial segmentation result, a contour probability mask, and hidden layer feature parameters. A second image processing model is used to extract a target image from the image to be processed based on the contour probability mask, and the target image is refined based on the initial segmentation result and hidden layer feature parameters to obtain a target segmentation result. The locations with lower probabilities in the contour probability mask represent uncertain regions. The first image processing model performs rough segmentation on the image to be processed, and the second image processing model refines the image in the uncertain region, thereby significantly enhancing the accuracy of image segmentation. Furthermore, since the refinement is performed only on a small portion of the image determined by the contour probability mask, the speed of image segmentation can be increased. This allows you to block out the background in the video image while accurately retaining the outline of the person (down to the hair) without affecting the smoothness of the video call. The background replacement function can make video calls more entertaining, and other entertainment functions such as emoticons can be added on this basis, providing a good user experience in terms of security and entertainment.

[0050] Reference Figure 2 , shows a flowchart of an image processing method embodiment of the present invention, which may specifically include the following steps:

[0051] Step 201, obtaining an image to be processed;

[0052] The image to be processed refers to the image that needs to be segmented. For example, the image to be processed can be each frame of a video or an image uploaded by a user. The image to be processed can be a high-definition image with any resolution, such as 1080*900px, which is not limited in this embodiment of the present invention.

[0053] Step 202: Input the image to be processed into a first image processing model for image segmentation to obtain an initial segmentation result, a contour probability mask, and hidden layer feature parameters;

[0054] The first image processing model is a pre-trained convolutional neural network model used for basic image segmentation. In an embodiment of the present invention, the image to be processed can be input into the first image processing model for image segmentation to obtain an initial segmentation result, a contour probability mask, and hidden layer feature parameters. Among them, the initial segmentation result is a rough image segmentation result, which is usually not detailed enough for the processing of the target contour edge; the contour probability mask is used to determine the position in the image that needs to be refined. Among the pixels with a probability greater than 0 in the contour probability mask, the pixel representation with a higher probability (greater than a preset threshold, which is set according to the accuracy requirements) is better aligned with the true contour, and the pixel representation with a lower probability (less than or equal to the preset threshold) is an uncertain area that needs to be refined; the hidden layer feature parameters are the image feature parameters extracted by the convolution layer in the network model.

[0055] Step 203 : Using a second image processing model, a target image is intercepted from the image to be processed according to the contour probability mask, and the target image is refined according to the initial segmentation result and hidden layer feature parameters to obtain a target segmentation result.

[0056] The second image processing model is used to refine the uncertain areas in the image after rough segmentation. The target image refers to the image corresponding to the uncertain area, and the target image of the uncertain area needs to be refined. In an embodiment of the present invention, the second image processing model can intercept the target image from the image to be processed according to the contour probability mask, and refine the target image according to the initial segmentation result and the hidden layer feature parameters to obtain the target segmentation result. After obtaining the target segmentation result, the image to be processed is segmented according to the target segmentation result, and the foreground of the image can be separated, so that the background of the image can be shielded or replaced. As an example, Figure 3 As shown in the figure, when segmenting an image, the accuracy of image segmentation is greatly improved by refining the contour edge of the target in the image. Figure 1 The segmented image in , the segmentation effect is better.

[0057] In a specific implementation, the overall framework of the first image processing model and the second image processing model is as follows: Figure 4 As shown in the figure, the basic network corresponds to the first image processing model, and the refined network corresponds to the second image processing model. The image to be processed is first reduced to 1 / 4 of the original image size and input into the first image processing model (i.e., the basic network) for image segmentation, and the segmentation result, contour probability mask, and hidden layer feature parameters are output. The segmentation result, contour probability mask, and hidden layer feature parameters are input as input data into the second image processing model (i.e., the refined network). The second image processing model intercepts the patch that needs to be refined from the original image according to the contour probability mask, and refines the intercepted patch to obtain the final segmentation result.

[0058] In a preferred embodiment of the present invention, step 202 may specifically include the following sub-steps:

[0059] The image to be processed is reduced according to a first preset coefficient; the reduced image to be processed is input into the first image processing model, and the Encoder module, ASPP module and Decoder module of the first image processing model are used to perform image segmentation to obtain an initial segmentation result, a contour probability mask and hidden layer feature parameters.

[0060] The first image processing model can be called a basic network, which is used to roughly segment the image. Specifically, the first image processing model can be a fully convolutional Encoder-Decoder network. The structure of this network can refer to DeepLabV3+, because DeepLabV3+ is still the network with the best semantic segmentation performance. Of course, the network structure of the first image processing model can also be set to other network structures, such as Deeplabv1, etc., and the embodiment of the present invention does not limit this. The first image processing model can include an Encoder module, an ASPP module, and a Decoder module, wherein the Backbone of the Encoder (the backbone network is a network used for feature extraction, representing a part of the network, extracting image information through the front end to generate a feature map Feature Map for use by the subsequent network) selects Resnet50, and can also be replaced with Resnet101 or MobilenetV2 based on the priority of segmentation accuracy and speed. The embodiment of the present invention does not limit this. Then, the ASPP module (Atrous Spatial Pyramid Pooling) is connected to the back of the Encoder module. Its function is to increase the receptive field while capturing multi-scale information. The decoder module uses bilinear interpolation for upsampling and fuses the output results of the corresponding layer in the encoder process.

[0061] To increase the speed of image segmentation, the image to be processed can be reduced by a first preset coefficient. The reduced image to be processed is input into the first image processing model, and image segmentation is performed using the encoder module, ASPP module, and decoder module of the first image processing model to obtain an initial segmentation result, a contour probability mask, and hidden layer feature parameters. The first preset coefficient is a pre-set reduction coefficient, such as 1 / 4. The first preset coefficient can be set as needed and is not limited in this embodiment of the present invention.

[0062] The specific image segmentation process of the first image processing model is as follows:

[0063] (1) The high-definition original image to be segmented (i.e., the image to be processed) is scaled to 1 / 2 size and input into the network of the first image processing model. Assuming that the length and width of the original image are W and H respectively, the size of the input to the network is 3*H / 4*W / 4.

[0064] (2) Feature extraction and downsampling are performed through a series of convolutional layers in the Encoder module. Each layer uses a 3*3 convolution kernel, followed by a BN layer (Batch Normalization) and a Relu (Rectified Linear Unit) activation layer. Finally, a 1*1 convolution layer is used to expand the number of channels to 34 dimensions.

[0065] (3) The above results are input into the ASPP module to extract multi-scale information. In an embodiment of the present invention, the ASPP module can use three dilated convolution kernels with different dilation rates for feature extraction, and the dilation rates are 3, 6, and 9 respectively.

[0066] (4) The above results are input into the Decoder module for upsampling, using the bilinear interpolation algorithm, and fused with the corresponding downsampling layer. The fusion method is superposition. The output of the Decoder module is 34*H / 4*W / 4, which includes a 1-dimensional segmentation result F, a 1-dimensional contour probability mask E, and a 32-dimensional hidden layer feature H. Figures 5A-5B As shown, Figure 5A is the original segmentation result, Figure 5B The contour probability mask. The original segmentation result is based on pixels with a probability greater than 0 in the contour probability mask and is a rough segmentation result. Among pixels with a probability greater than 0 in the contour probability mask, pixels with a higher probability indicate a closer fit to the true contour, while pixels with a lower probability indicate uncertain areas that require subsequent refinement.

[0067] In a preferred embodiment of the present invention, step 203 may specifically include the following sub-steps:

[0068] The image to be processed is reduced according to a second preset coefficient; the initial segmentation result, the hidden layer feature parameters and the reduced image to be processed are merged to obtain refined data; the refined data is input into the second image processing model, and the second image processing model is used to intercept the target image from the reduced image to be processed according to the contour probability mask, and the target image is refined according to the initial segmentation result and the hidden layer feature parameters to obtain a target segmentation result.

[0069] The second preset coefficient is a preset reduction coefficient, such as 1 / 2. The second preset coefficient can be set as needed, and the embodiment of the present invention does not limit this.

[0070] The second image processing model can be called a refinement network. The purpose of refinement through the refinement network is to reduce redundant calculations and repair the segmentation details of the image to be processed. The above-mentioned first image processing model calculates the entire image, while the refinement network only calculates some small patches. These small patches are selected through the contour probability mask. By setting a probability threshold, the pixels in the contour probability mask that are less than the probability threshold are selected as the patches that need to be refined. The pixels that are greater than the probability threshold indicate that they are well aligned with the true contour and do not need to be refined. Specifically, the contour probability mask is resized to 1 / 4 the size of the original image to be processed, which is recorded as E4. In this way, each pixel on the contour probability mask represents a 4*4 area on the original image. The topk pixels with the lowest probability are selected to determine the k 4*4 areas to be refined on the original image, totaling 16k pixels.

[0071] The specific refinement process of the second image processing model is divided into two stages:

[0072] (1) In the first stage, the original segmentation result F, the hidden layer feature H, and the image to be processed I are scaled to 1 / 2 of the original resolution by bilinear interpolation upsampling, and these data are merged according to the channel and recorded as M. According to the above description, the size of the merged image is [(1+32+3)*H / 2*W / 2]. Then, according to the pixel position selected on E4, K 8*8 patches are intercepted on M, with a size of [k*36*8*8], as shown in the following example: Figure 4 As shown, it then passes through two 3*3 convolutional layers, each of which has BN and ReLU, so that the 8*8 patch is reduced to a 4*4 patch, and the size becomes [k*16*4*4].

[0073] (2) In the second stage, the 4*4 patch is upsampled back to 8*8 through nearest neighbor interpolation, and it is fused with the 8*8 patch cut from the corresponding position on the image to be processed I according to the channel, with a size of [k*(3+16)*8*8]. Then two 3*3 convolutional layers are connected to change the size to [k*1*4*4]. This is the result after refinement. Finally, we enlarge E4 to its original size and replace the 4*4 patch at the corresponding position to form the final segmentation result.

[0074] In a preferred embodiment of the present invention, the first image processing model is trained and generated in the following manner:

[0075] Acquire a sample image; use the sample image to train an original image segmentation model; wherein the original image segmentation model includes a segmentation result network and a contour probability mask network; train the segmentation result network through a first loss function, and train the contour probability mask network through a second loss function; when it is judged that the original image segmentation module meets the requirements through the first loss function and the second loss function, stop training the original image segmentation module, and determine that the original image segmentation module is the first image processing model.

[0076] Specifically, the sample image datasets used can include the AIM dataset, which contains 269 training sets and 11 test sets, with an average size of 1000x1000 high-definition images, and the Distinctions-646 subset that only contains people, which has 362 training sets and 11 test sets, with an average size of 1700x2000. In addition, there is VideoMatte240K, which includes 484 high-resolution videos, including 100 HD videos and 384 4K high-definition videos, which can be divided into 240,709 frames in total. These datasets cover a large number of human images, postures, and clothing. Sample images such as Figure 6 As shown, since all of them are green screen videos, they all come with labels.

[0077] In an embodiment of the present invention, a sample image can be used to train the original image segmentation model, wherein the original image segmentation model includes a segmentation result network and a contour probability mask network. The segmentation result network is trained by a first loss function, and the contour probability mask network is trained by a second loss function. When the original image segmentation module is judged to meet the requirements by the first loss function and the second loss function, the training of the original image segmentation module is stopped, and the original image segmentation module is determined to be the first image processing model.

[0078] In a preferred embodiment of the present invention, the first loss function is expressed as follows:

[0079]

[0080] Wherein, F represents the segmentation result, F* represents the label of the image, ▽F represents the Sobel gradient operator of the segmentation result, and ▽F* represents the Sobel gradient operator of the image label.

[0081] First, in order to train the segmentation result network, the L1 loss function can be used. The first loss function consists of two parts: the L1 loss of F and F*, and the L1 loss of the Sobel gradient operator of F and the Sobel gradient operator of F*. The purpose of adding the L1 loss of the Sobel gradient operator is to allow the network to learn the boundary effect of the target contour.

[0082] In a preferred embodiment of the present invention, the second loss function is expressed as follows:

[0083] LossE=||EE * ||2;

[0084] Wherein, E represents the contour probability mask, E* represents the label of E, and E*=|FF*|.

[0085] Secondly, in order to train the contour probability mask network, the label E* of E is defined as |FF*|, and then the mean square error loss between E and E* is used as the loss function of E.

[0086] The loss function of the entire first image processing model can be expressed as LossF+LossE. When LossF+LossE is minimized after training, it is determined that the original image segmentation module meets the requirements, the training of the original image segmentation module is stopped, and the various parameters in the original image segmentation module are fixed, and the original image segmentation module is determined as the first image processing model.

[0087] In a preferred embodiment of the present invention, the second image processing model is trained and generated in the following manner:

[0088] According to the initial segmentation result, the contour probability mask and the hidden layer feature parameters, the original refined network module is trained by the first loss function to obtain a second image processing model.

[0089] Specifically, the original refined network module can be trained through the first loss function according to the initial segmentation result, contour probability mask and hidden layer feature parameters output from the basic network to obtain the second image processing model. The Loss loss function of the second image processing model only contains one item LossF. The entire training process is to first train the first image processing model, and then train the second image processing model after the basic network converges to form the final robust contour probability mask. In some examples, in order to increase the speed of image segmentation, the graphics are scaled differently in the basic network and the refined network. Therefore, the size of the segmentation result F in the refined network may be different from the size of F in the basic network. For example, the size of the segmentation result F in the refined network may be the size of the original image, while the size of F in the basic network is 1 / 4 of the original image.

[0090] The image segmentation method of the present invention can be applied to the video call function in smart home robots. When users are talking to each other, they can choose whether to block or replace the background to protect their privacy. The image segmentation function can also be added to camera apps, allowing users to select various special effects backgrounds to replace the current background and record their favorite entertainment short videos. By generating a contour probability mask through the loss design in the basic network, it provides a benchmark for the subsequent refinement process. The image segmentation network structure only corrects the uncertain areas of the segmentation result, reducing a large amount of redundant calculations. The design of processing on low-resolution images and then upsampling to high resolution enables the network to process large high-definition images. In practical applications, HD and even 4K high-definition videos can be processed in real time if the hardware supports it, and the segmentation effect is very precise. In addition, by designing a new contour refinement network structure, based on the contour probability mask generated by the basic network, only the patches corresponding to low-probability pixels are refined. The input of the processing process is all low-resolution images, and the processed patches are simply replaced with the corresponding positions in the high-resolution image, improving the performance of the overall network.

[0091] It should be noted that for the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0092] Reference Figure 7 , shows a structural block diagram of an embodiment of an image processing device of the present invention, which may specifically include the following modules:

[0093] Image acquisition module 701, used to acquire the image to be processed;

[0094] An image segmentation module 702 is configured to input the image to be processed into a first image processing model for image segmentation to obtain an initial segmentation result, a contour probability mask, and hidden layer feature parameters;

[0095] The image refinement module 703 is used to use the second image processing model to intercept the target image from the image to be processed according to the contour probability mask, and refine the target image according to the initial segmentation result and the hidden layer feature parameters to obtain a target segmentation result.

[0096] In a preferred embodiment of the present invention, the image segmentation module 702 includes:

[0097] An image processing submodule, configured to reduce the image to be processed according to a first preset coefficient;

[0098] The image segmentation submodule is used to input the reduced image to be processed into the first image processing model, and use the Encoder module, ASPP module and Decoder module of the first image processing model to perform image segmentation to obtain the initial segmentation result, contour probability mask and hidden layer feature parameters.

[0099] In a preferred embodiment of the present invention, the image retouching module 703 includes:

[0100] An image processing submodule, configured to reduce the image to be processed according to a second preset coefficient;

[0101] A data merging submodule, configured to merge the initial segmentation result, the hidden layer feature parameters, and the reduced image to be processed to obtain refined data;

[0102] The image refinement submodule is used to input the refined data into the second image processing model, adopt the second image processing model, intercept the target image from the reduced image to be processed according to the contour probability mask, and refine the target image according to the initial segmentation result and hidden layer feature parameters to obtain a target segmentation result.

[0103] In a preferred embodiment of the present invention, the first image processing model is generated by training the following modules:

[0104] A sample acquisition module, used to acquire sample images;

[0105] a first model training module, configured to train an original image segmentation model using the sample image; wherein the original image segmentation model includes a segmentation result network and a contour probability mask network, the segmentation result network is trained using a first loss function, and the contour probability mask network is trained using a second loss function;

[0106] A training completion module is used to stop training the original image segmentation module and determine that the original image segmentation module is the first image processing model when it is judged that the original image segmentation module meets the requirements through the first loss function and the second loss function.

[0107] In a preferred embodiment of the present invention, the first loss function is expressed as follows:

[0108]

[0109] Wherein, F represents the segmentation result, F* represents the label of the image, ▽F represents the Sobel gradient operator of the segmentation result, and ▽F* represents the Sobel gradient operator of the image label.

[0110] In a preferred embodiment of the present invention, the second loss function is expressed as follows:

[0111] LossE=||EE * ||2;

[0112] Wherein, E represents the contour probability mask, E* represents the label of E, and E*=|FF*|.

[0113] In a preferred embodiment of the present invention, the second image processing model is generated by training the following modules:

[0114] The second model training module is used to train the original refined network module through the first loss function according to the initial segmentation result, the contour probability mask and the hidden layer feature parameters to obtain a second image processing model.

[0115] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0116] An embodiment of the present invention further provides an electronic device, including:

[0117] one or more processors; and

[0118] One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, enable the electronic device to perform the steps of the method described in the embodiment of the present invention.

[0119] An embodiment of the present invention further provides a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, causes the processors to perform the steps of the method described in the embodiment of the present invention.

[0120] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0121] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, embodiments of the present invention may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware. Furthermore, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the process in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0123] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1A step that specifies a function in one or more boxes.

[0125] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0126] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0127] The above is a detailed introduction to an image processing method and an image processing device provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. An image processing method, characterized in that: include: Get the image to be processed; Inputting the image to be processed into a first image processing model for image segmentation to obtain an initial segmentation result, a contour probability mask, and hidden layer feature parameters, wherein the contour probability mask is used to represent the probability that a pixel in the image to be processed belongs to a target contour; Using a second image processing model, intercepting a target image from the image to be processed according to the contour probability mask, and performing a refinement process on the target image according to the initial segmentation result and hidden layer feature parameters to obtain a target segmentation result; After intercepting a target image from the image to be processed according to the contour probability mask and performing a refinement process on the target image according to the initial segmentation result and the hidden layer feature parameters, the method further includes: Replacing the target image with the refined result; The second image processing model is used to intercept a target image from the image to be processed according to the contour probability mask, and the target image is refined according to the initial segmentation result and the hidden layer feature parameters to obtain a target segmentation result, including: reducing the image to be processed according to a second preset coefficient; Combining the initial segmentation result, the hidden layer feature parameters and the reduced image to be processed to obtain refined data; The refined data is input into the second image processing model, and the second image processing model is used to intercept the target image from the reduced image to be processed according to the contour probability mask, and the target image is refined according to the initial segmentation result and the hidden layer feature parameters to obtain the target segmentation result.

2. The method according to claim 1, characterized in that Inputting the image to be processed into the first image processing model for image segmentation to obtain an initial segmentation result, a contour probability mask and hidden layer feature parameters, includes: reducing the image to be processed according to a first preset coefficient; The reduced image to be processed is input into the first image processing model, and the Encoder module, ASPP module and Decoder module of the first image processing model are used to perform image segmentation to obtain an initial segmentation result, a contour probability mask and hidden layer feature parameters.

3. The method according to claim 1, characterized in that The first image processing model is trained and generated in the following manner: Get a sample image; The sample image is used to train an original image segmentation model; wherein the original image segmentation model includes a segmentation result network and a contour probability mask network, the segmentation result network is trained using a first loss function, and the contour probability mask network is trained using a second loss function; When it is determined through the first loss function and the second loss function that the original image segmentation model meets the requirements, the training of the original image segmentation model is stopped, and the original image segmentation model is determined to be the first image processing model.

4. The method according to claim 3, characterized in that The first loss function is expressed as follows: ； Wherein, F represents the segmentation result, F* represents the label of the image, ▽F represents the Sobel gradient operator of the segmentation result, and ▽F* represents the Sobel gradient operator of the image label.

5. The method according to claim 3, characterized in that The second loss function is expressed as follows: ； Wherein, E represents a contour probability mask, E* represents a label of E, and E*=|F - F*|.

6. The method according to claim 3, characterized in that The second image processing model is trained and generated in the following manner: According to the initial segmentation result, the contour probability mask and the hidden layer feature parameters, the original refined network module is trained by the first loss function to obtain a second image processing model.

7. An image processing device, characterized in that include: An image acquisition module, used for acquiring an image to be processed; an image segmentation module, configured to input the image to be processed into a first image processing model for image segmentation to obtain an initial segmentation result, a contour probability mask, and hidden layer feature parameters, wherein the contour probability mask is used to represent the probability that a pixel in the image to be processed belongs to a target contour; an image refinement module, configured to intercept a target image from the image to be processed according to the contour probability mask using a second image processing model, and refine the target image according to the initial segmentation result and hidden layer feature parameters to obtain a target segmentation result; After intercepting a target image from the image to be processed according to the contour probability mask and performing a refinement process on the target image according to the initial segmentation result and the hidden layer feature parameters, the method further includes: Replacing the target image with the refined result; The image refinement module includes: An image processing submodule, configured to reduce the image to be processed according to a second preset coefficient; A data merging submodule, configured to merge the initial segmentation result, the hidden layer feature parameters, and the reduced image to be processed to obtain refined data; The image refinement submodule is used to input the refined data into the second image processing model, adopt the second image processing model, intercept the target image from the reduced image to be processed according to the contour probability mask, and refine the target image according to the initial segmentation result and hidden layer feature parameters to obtain a target segmentation result.

8. An electronic device, characterized in that: include: one or more processors; and One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, cause the electronic device to perform the steps of any one of the methods of claims 1-6.

9. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the steps of any one of the methods of claims 1 to 6.

Images