Image Processing Method, Apparatus, Electronic Device, Medium and Program Product

Abstract

An embodiment of the present disclosure discloses an image processing method, including: obtaining a first intermediate image including a target object and a second intermediate image including the target object from an image to be processed; determining the degree of association between the first intermediate image and a reference target object image; obtaining an estimated mask of the image to be processed; and obtaining a mask of the image to be processed according to the degree of association, the estimated mask, and the second intermediate image, where the mask is used to segment the target object from the image to be processed.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technologies, and in particular, to an image processing method, apparatus, electronic device, medium, and program product. Background Art

[0002] With the rapid development of Internet and communication technologies, video has become one of the important means of information transmission. However, a video usually contains many objects, such as people, animals, other objects, etc., and not all objects are needed. Some videos may have some redundant objects, thus interfering with the information transmission of the entire video. To solve this problem, technical means are usually used to process the video, erase the unnecessary objects therein, and fill the part where the erased object is located with the surrounding information to obtain a clean video after erasing the target object, so as to reduce the time for post-processing. However, the currently used erasing methods include manual target erasing, erasing based on a segmentation algorithm, etc., and these methods are all very complex and time-consuming. Summary of the Invention

[0003] To solve the problems in the related technologies, embodiments of the present disclosure provide an image processing method, apparatus, electronic device, medium, and program product.

[0004] In a first aspect, an image processing method is provided in embodiments of the present disclosure, including:

[0005] Obtaining a first intermediate image including a target object and a second intermediate image including the target object from the image to be processed;

[0006] Determining the degree of association between the first intermediate image and a reference target object image;

[0007] Obtaining an estimated mask of the image to be processed;

[0008] Obtaining a mask of the image to be processed according to the degree of association, the estimated mask, and the second intermediate image, where the mask is used to segment the target object from the image to be processed.

[0009] In combination with the first aspect, in a first implementation manner of the first aspect, the obtaining a first intermediate image including a target object and a second intermediate image including the target object from the image to be processed includes:

[0010] Determining a target region in the image to be processed, where the target region includes the target object;

[0011] Taking the part of the image to be processed located in the target region as the first intermediate image;

[0012] Take the part of the image to be processed located in the target area and the part surrounding the target area as the second intermediate image.

[0013] Combined with the first implementation manner of the first aspect, in the second implementation manner of the first aspect of the present disclosure:

[0014] The image to be processed is an image in a video image sequence;

[0015] The target area is obtained by using a target tracking algorithm based on the previous frame image of the image to be processed in the video image sequence and the reference target object image.

[0016] Combined with the first implementation manner of the first aspect, in the third implementation manner of the first aspect of the present disclosure, wherein:

[0017] The image to be processed is an image in a video image sequence;

[0018] The target area is obtained by using a target tracking algorithm based on the previous frame image of the image to be processed in the video image sequence and the target object image, and correcting the target tracking result using the mask of the previous frame image.

[0019] Combined with the first aspect, in the fourth implementation manner of the first aspect of the present disclosure, determining the correlation degree between the first intermediate image and the reference target object image includes:

[0020] Extract the image features of the first intermediate image and the image features of the reference target object image;

[0021] Calculate the cross-correlation value of the image features of the first intermediate image and the image features of the reference target object image as the correlation degree.

[0022] Combined with the first aspect, in the fifth implementation manner of the first aspect of the present disclosure, obtaining the estimated mask of the image to be processed includes:

[0023] Input the image-mask pair set and the image to be processed into a mask estimation model to obtain the estimated mask, where the image-mask pair includes a previous image before the image to be processed in the video image sequence and the mask of the previous image, and the confidence level of the mask of the previous image is higher than a preset confidence level.

[0024] Combined with the fifth implementation manner of the first aspect, in the sixth implementation manner of the first aspect of the present disclosure, inputting the image-mask pair set and the image to be processed into a mask estimation model to obtain the estimated mask includes:

[0025] Perform feature encoding on each image-mask pair in the image-mask pair set;

[0026] Perform feature encoding on the image to be processed;

[0027] Perform feature matching on the feature encoding of each image-mask pair and the feature encoding of the image to be processed, and input the feature matching result and the feature encoding of the image to be processed into a first decoder to obtain the estimated mask.

[0028] Combined with the sixth implementation manner of the first aspect, in the seventh implementation manner of the first aspect of the present disclosure, the reference target object image is a pre-set target object image or a target object image obtained by segmenting the corresponding image using the mask in the image-mask pair.

[0029] Combined with the first aspect, in the eighth implementation manner of the first aspect of the present disclosure, the obtaining the mask of the image to be processed according to the correlation degree, the estimated mask, and the second intermediate image includes:

[0030] Input the feature encoding of the correlation degree, the estimated mask, and the second intermediate image into a second decoder to obtain the mask.

[0031] Combined with the first aspect, in the ninth implementation manner of the first aspect of the present disclosure, the method further includes:

[0032] Erase the target object in the image to be processed based on the mask;

[0033] Use an image inpainting algorithm to complete the image to be processed after erasing the target object.

[0034] Combined with the ninth implementation manner of the first aspect, in the tenth implementation manner of the first aspect of the present disclosure, the image to be processed is an image in a video image sequence, and the using an image inpainting algorithm to complete the image to be processed after erasing the target object includes:

[0035] Based on the completion result of a specified image before the image to be processed and the completion result of a specified image after the image to be processed in the video image sequence, use an image inpainting algorithm to process the image to be processed after erasing the target object to complete the image to be processed after erasing the target object.

[0036] In a second aspect, an embodiment of the present disclosure provides an image processing method, including:

[0037] Display a reference image;

[0038] Determine a reference target object image from the reference image according to a user's operation on the reference image, where the reference target object image includes a target object;

[0039] Display an output image, where the output image is an image obtained by removing the target object from the image to be processed.

[0040] In combination with the second aspect, in the first implementation manner of the second aspect of the present disclosure, the output image is obtained by erasing the target object from the image to be processed using a mask obtained by the method described in the first aspect of the present disclosure, and using an image inpainting algorithm to complete the image to be processed after erasing the target object.

[0041] In a third aspect, an image processing apparatus is provided in an embodiment of the present disclosure, including:

[0042] A first acquisition module configured to acquire a first intermediate image including a target object and a second intermediate image including the target object from the image to be processed;

[0043] A first determination module configured to determine the degree of association between the first intermediate image and a reference target object image;

[0044] A second acquisition module configured to acquire an estimated mask of the image to be processed;

[0045] A third acquisition module configured to acquire a mask of the image to be processed according to the degree of association, the estimated mask, and the second intermediate image, where the mask is used to segment the target object from the image to be processed.

[0046] In combination with the third aspect, in the first implementation manner of the third aspect of the present disclosure, acquiring the first intermediate image including the target object and the second intermediate image including the target object from the image to be processed includes:

[0047] Determine a target region in the image to be processed, where the target region includes the target object;

[0048] Take the part of the image to be processed located in the target region as the first intermediate image;

[0049] Take the part of the image to be processed located in the target region and the part surrounding the target region as the second intermediate image.

[0050] In combination with the first implementation manner of the third aspect, in the second implementation manner of the third aspect of the present disclosure:

[0051] The image to be processed is an image in a video image sequence;

[0052] The target region is obtained using a target tracking algorithm based on the previous frame image of the image to be processed in the video image sequence and the reference target object image.

[0053] Combined with the first implementation manner of the third aspect, in the third implementation manner of the third aspect of the present disclosure:

[0054] The image to be processed is an image in a video image sequence;

[0055] The target region is obtained by using a target tracking algorithm based on the previous frame image of the image to be processed in the video image sequence and the target object image, and correcting the target tracking result by using the mask of the previous frame image.

[0056] Combined with the third aspect, in the fourth implementation manner of the third aspect of the present disclosure, determining the degree of association between the first intermediate image and the reference target object image includes:

[0057] Extracting the image features of the first intermediate image and the image features of the reference target object image;

[0058] Calculating the cross-correlation value of the image features of the first intermediate image and the image features of the reference target object image as the degree of association.

[0059] Combined with the third aspect, in the fifth implementation manner of the third aspect of the present disclosure, obtaining the estimated mask of the image to be processed includes:

[0060] Inputting the image-mask pair set and the image to be processed into a mask estimation model to obtain the estimated mask, where the image-mask pair includes a previous image before the image to be processed in the video image sequence and the mask of the previous image, and the confidence level of the mask of the previous image is higher than a preset confidence level.

[0061] Combined with the fifth implementation manner of the third aspect, in the sixth implementation manner of the third aspect of the present disclosure, inputting the image-mask pair set and the image to be processed into a mask estimation model to obtain the estimated mask includes:

[0062] Performing feature encoding on each image-mask pair in the image-mask pair set;

[0063] Performing feature encoding on the image to be processed;

[0064] Performing feature matching on the feature encoding of each image-mask pair and the feature encoding of the image to be processed, and inputting the feature matching result and the feature encoding of the image to be processed into a first decoder to obtain the estimated mask.

[0065] Combined with the sixth implementation manner of the third aspect, in the seventh implementation manner of the third aspect of the present disclosure, the reference target object image is a preset target object image or a target object image obtained by segmenting the corresponding image by using the mask in the image-mask pair.

[0066] In combination with the third aspect, in the eighth implementation manner of the third aspect, obtaining the mask of the image to be processed according to the degree of association, the estimated mask, and the second intermediate image includes:

[0067] Inputting the feature encodings of the degree of association, the estimated mask, and the second intermediate image into a second decoder to obtain the mask.

[0068] In combination with the ninth implementation manner of the third aspect, in the tenth implementation manner of the third aspect, the apparatus further includes:

[0069] An erasing module configured to erase the target object in the image to be processed based on the mask;

[0070] A completion module configured to use an image inpainting algorithm to complete the image to be processed after erasing the target object.

[0071] In combination with the ninth implementation manner of the third aspect, in the tenth implementation manner of the third aspect, the image to be processed is an image in a video image sequence, and using the image inpainting algorithm to complete the image to be processed after erasing the target object includes:

[0072] Based on the completion results of the specified image before the image to be processed and the completion results of the specified image after the image to be processed in the video image sequence, using the image inpainting algorithm to process the image to be processed after erasing the target object to complete the image to be processed after erasing the target object.

[0073] Fourth aspect, an embodiment of the present disclosure provides an image processing apparatus, including:

[0074] A first display module configured to display a reference image;

[0075] A second determination module configured to determine a reference target object image including a target object from the reference image according to a user's operation on the reference image;

[0076] A second display module configured to display an output image, where the output image is an image obtained by removing the target object from the image to be processed.

[0077] In combination with the fourth aspect, in the first implementation manner of the fourth aspect, the output image is obtained by erasing the target object from the image to be processed using the mask obtained by the method according to the first aspect and using an image inpainting algorithm to complete the image to be processed after erasing the target object.

[0078] Fifth aspect, embodiments of the present disclosure provide an electronic device, including a memory and a processor. The memory is configured to store one or more computer instructions, and the one or more computer instructions are executed by the processor to implement the method according to any one of the first implementation manners of the first aspect to the second aspect.

[0079] Sixth aspect, embodiments of the present disclosure provide a computer-readable storage medium, on which computer instructions are stored. When the computer instructions are executed by a processor, the method according to the first implementation manner of the first aspect to the second aspect is implemented.

[0080] Seventh aspect, embodiments of the present disclosure provide a computer program product, including computer instructions. When the computer instructions are executed by a processor, the method steps according to the first implementation manner of the first aspect to the second aspect are implemented.

[0081] According to the technical solution provided by the embodiments of the present disclosure, based on the correlation between the first intermediate image and the reference target object image, the estimated mask, and the second intermediate image, the mask of the image to be processed is obtained, which takes less time than the existing manual target erasure solution, and the labor cost is significantly reduced. At the same time, since two reference quantities (i.e., the correlation and the estimated mask) are introduced in the process of generating the mask, the obtained mask can achieve a more accurate image segmentation result.

[0082] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In conjunction with the drawings, through the following detailed description of non-limiting embodiments, other features, objects, and advantages of the present disclosure will become more apparent. In the drawings:

[0084] Figure 1 A flowchart showing an image processing method according to an embodiment of the present disclosure is shown.

[0085] Figure 2A An overall flowchart showing an image processing method according to an embodiment of the present disclosure is shown.

[0086] Figure 2B A schematic diagram of a target area according to an embodiment of the present disclosure is shown.

[0087] Figure 2C A schematic diagram of the principle of a mask estimation model according to an embodiment of the present disclosure is shown.

[0088] Figure 2D A flowchart showing an image processing method according to an embodiment of the present disclosure is shown.

[0089] Figure 3AThe figure shows a schematic diagram of the effect of processing a video image sequence using the image processing method according to an embodiment of the present disclosure.

[0090] Figure 3B The figure shows a schematic diagram of the effect of one frame image obtained by processing a video image sequence using the image processing method according to an embodiment of the present disclosure.

[0091] Figure 4A The figure shows a structural block diagram of an image processing apparatus according to an embodiment of the present disclosure.

[0092] Figure 4B The figure shows a structural block diagram of an image processing apparatus according to an embodiment of the present disclosure.

[0093] Figure 5 The figure shows a structural block diagram of an electronic device according to an embodiment of the present disclosure.

[0094] Figure 6 The figure shows a schematic diagram of the structure of a computer system suitable for implementing the method according to an embodiment of the present disclosure. Detailed implementation manners

[0095] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for clarity, parts unrelated to the description of the exemplary embodiments are omitted in the drawings.

[0096] In the present disclosure, it should be understood that terms such as "including" or "having" are intended to indicate the presence of features, numbers, steps, actions, components, parts, or combinations thereof disclosed in this specification, and are not intended to exclude the possibility of the presence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0097] In addition, it should be noted that, without conflict, the embodiments in the present disclosure and the features in the embodiments can be combined with each other. Hereinafter, the present disclosure will be described in detail with reference to the drawings and in conjunction with the embodiments.

[0098] In the present disclosure, the acquisition of user information or user data is an operation that has been authorized and confirmed by the user or actively selected by the user.

[0099] As described above, when there are unwanted objects in the video, if there is an algorithm to automatically erase them to obtain a clean video, the subsequent video processing time can be greatly reduced. The embodiments of the present disclosure can be applied to fields such as interactive video target automatic erasure and advertisement automatic filtering products.

[0100] An embodiment of the present disclosure provides an image processing method, including: obtaining a first intermediate image including a target object and a second intermediate image including the target object from an image to be processed; determining the degree of association between the first intermediate image and a reference target object image; obtaining an estimated mask of the image to be processed; and obtaining a mask of the image to be processed according to the degree of association, the estimated mask, and the second intermediate image, where the mask is used to segment the target object from the image to be processed.

[0101] According to an embodiment of the present disclosure, obtaining the mask of the image to be processed based on the degree of association between the first intermediate image and the reference target object image, the estimated mask, and the second intermediate image takes less time than existing manual target erasure solutions, and the labor cost is significantly reduced. At the same time, since two reference quantities (i.e., the degree of association and the estimated mask) are introduced in the process of generating the mask, the obtained mask can achieve a more accurate image segmentation result.

[0102] Figure 1 The flowchart showing the image processing method according to an embodiment of the present disclosure is as follows. Figure 1 As shown, the image processing method includes the following steps S101 - S104:

[0103] In step S101, obtaining a first intermediate image including a target object and a second intermediate image including the target object from the image to be processed;

[0104] In step S102, determining the degree of association between the first intermediate image and the reference target object image;

[0105] In step S103, obtaining the estimated mask of the image to be processed;

[0106] In step S104, obtaining the mask of the image to be processed according to the degree of association, the estimated mask, and the second intermediate image, where the mask is used to segment the target object from the image to be processed.

[0107] Figure 2A The overall flowchart of the image processing method according to an embodiment of the present disclosure is shown.

[0108] Figure 2B The schematic diagram of the target area according to an embodiment of the present disclosure is shown.

[0109] Next, reference is made to Figure 2A and 2B to describe the image processing method according to an embodiment of the present disclosure. In the following, it is described that the image to be processed is an image in a video image sequence, but it can be understood that the embodiments of the present disclosure are not limited to processing video image sequences.

[0110] As Figure 2AAs shown, after obtaining the image to be processed, the image to be processed is cropped using the target region bbox to obtain a first intermediate image and a second intermediate image. The target region bbox includes the target object to be erased, such as the person wearing a white top in the figure. The part of the image to be processed located within the target region is used as the first intermediate image; the part of the image to be processed located within the target region and the part surrounding the target region are used as the second intermediate image.

[0111] According to an embodiment of the present disclosure, the target region is obtained using a target tracking algorithm based on the previous frame image of the image to be processed and the reference target object image in the video image sequence. For example, as Figure 2A shown by the tracking module of, based on the previous frame image of the image to be processed and the reference target object image in the video image sequence, a target box is obtained using a target tracking algorithm (such as the SiamFC++ algorithm), and the target box is the circumscribed rectangle of the target object, as Figure 2B shown. According to an embodiment of the present disclosure, other known target tracking algorithms may also be used to determine the target box.

[0112] According to an embodiment of the present disclosure, the target region is obtained by using a target tracking algorithm based on the previous frame image of the image to be processed and the target object image in the video image sequence and correcting the target tracking result using the mask of the previous frame image. As Figure 2A shown by the tracking module of, after obtaining the target box by using a target tracking algorithm based on the previous frame image of the image to be processed and the target object image in the video image sequence, the target box can be corrected using the mask of the previous frame image. Since the mask of the previous frame image contains the position information of the target object, the position of the target box can be corrected using the mask of the previous frame image. For example, the mean value of the position coordinates of the target box and the circumscribed rectangle of the target object determined according to the mask of the previous frame image can be taken as the position of the corrected target box.

[0113] After obtaining the target box, the range of the target box is expanded to include a part of the background image, thereby obtaining the target region. The part of the image to be processed located within the target region is used as the first intermediate image. The part of the image to be processed surrounding the target region is used as the second intermediate image. The second intermediate image may include more background image than the first intermediate image.

[0114] According to an embodiment of the present disclosure, since both the image of the target object and the background image contain information that can be used to calculate the correlation degree when calculating the correlation degree between the first intermediate image and the reference target object image, therefore, by making the first intermediate image contain the target object image and a part of the background image, a calculation result that can better reflect the correlation degree between the two can be obtained. By making the second intermediate image contain more background image than the first intermediate image, more information of the image to be processed can be provided when generating the mask, and a more accurate mask can be obtained. According to an embodiment of the present disclosure, the first intermediate image may also be the same image as the second intermediate image.

[0115] According to an embodiment of the present disclosure, the reference target object image is a pre-set target object image. For example, a frame image in a video image sequence can be selected, such as the first frame image containing the target object in the video image sequence, and a part containing the target object and optionally a part of the background image can be manually selected in the selected image as the reference target object image. Alternatively, an existing object detection and segmentation algorithm can also be used to segment a part containing the target object and optionally a part of the background image from the selected image as the reference target object image.

[0116] According to an embodiment of the present disclosure, determining the correlation degree between the first intermediate image and the reference target object image includes: extracting the image features of the first intermediate image and the image features of the reference target object image; calculating the cross-correlation value of the image features of the first intermediate image and the image features of the reference target object image as the correlation degree.

[0117] As Figure 2A shown by the tracking module of , the image features of the first intermediate image and the image features of the reference target object image are respectively extracted, and then the cross-correlation value of the image features of the first intermediate image and the image features of the reference target object image is calculated as the correlation degree and the correlation degree is output.

[0118] According to an embodiment of the present disclosure, when using the SiamFC++ algorithm to implement the tracking module, the target box bbox and the correlation degree can be obtained simultaneously.

[0119] According to an embodiment of the present disclosure, the image to be processed can be input into a trained mask estimation model to obtain an estimated mask of the image to be processed.

[0120] According to an embodiment of the present disclosure, existing various mask generation methods can be used to implement the mask estimation model to obtain an estimated mask of the image to be processed, such as instance segmentation algorithms, semantic segmentation algorithms, object detection algorithms combined with instance or semantic segmentation algorithms, object tracking algorithms combined with instance or semantic segmentation algorithms, etc.

[0121] Alternatively, an optical flow method may be used to perform a warp transformation on the mask of the previous frame image of the to-be-processed image in the video image sequence to obtain an estimated mask of the to-be-processed image.

[0122] Alternatively, obtaining the estimated mask of the to-be-processed image includes: inputting an image-mask pair set and the to-be-processed image into a mask estimation model to obtain the estimated mask, where the image-mask pair includes a previous image in the video image sequence before the to-be-processed image and the mask of the previous image, and the confidence level of the mask of the previous image is higher than a preset confidence level.

[0123] By obtaining the target mask of the to-be-processed image based on the previous image and its mask, since the confidence level of the mask of the previous image is higher than the preset confidence level, that is, the mask accuracy is higher, a more accurate estimated mask can be obtained.

[0124] According to an embodiment of the present disclosure, inputting the image-mask pair set and the to-be-processed image into the mask estimation model to obtain the estimated mask includes: performing feature encoding on each image-mask pair in the image-mask pair set; performing feature encoding on the to-be-processed image; performing feature matching on the feature encoding of each image-mask pair and the feature encoding of the to-be-processed image, and inputting the feature matching result and the feature encoding of the to-be-processed image into a first decoder to obtain the estimated mask.

[0125] For example, as Figure 2A shown, according to an embodiment of the present disclosure, an estimation module implements the mask estimation model. Image-mask pairs are stored in the memory of the estimation module. The to-be-processed image is adjusted to the size required by the mask estimation model and then input into a first encoder for feature encoding to obtain the feature encoding of the to-be-processed image. Feature matching is performed on the feature encoding of each image-mask pair and the feature encoding of the to-be-processed image, and the feature matching result and the feature encoding of the to-be-processed image are input into a first decoder to obtain the estimated mask.

[0126] Figure 2C FIG. shows a schematic diagram of the principle of the mask estimation model according to an embodiment of the present disclosure.

[0127] According to an embodiment of the present disclosure, the mask estimation model is implemented by a Space-Time Memory Networks.

[0128] Such as Figure 2CAs shown, for each image-mask pair in the set of image-mask pairs, the image and the mask are feature-encoded to obtain a feature encoding including Key features, Value features, and Query features. The image to be processed is feature-encoded to obtain a feature encoding including Key features, Value features, and Query features.

[0129] After concatenating the feature encodings of each image-mask pair, a matching operation is performed with the feature encoding of the image to be processed. The feature matching result and the feature encoding of the image to be processed are input into the first decoder to obtain an estimated mask of the image to be processed.

[0130] According to an embodiment of the present disclosure, the matching operation includes assigning weights to the Value features of the corresponding image-mask pairs according to the matching degree between the Query feature of the image to be processed and the Key features of each image-mask pair, and calculating the weighted sum of the Value features of all image-mask pairs as the feature matching result. In this feature matching result, the higher the matching degree between the image-mask pair and the image to be processed, the greater the proportion of its features. The matching result and the feature encoding of the image to be processed are output to the first decoder together, so that an estimated mask of the image to be processed can be generated with reference to the previously more accurately matched mask with the image to be processed.

[0131] Obtaining the mask of the image to be processed according to the relevance degree, the estimated mask, and the second intermediate image includes: inputting the feature encodings of the relevance degree, the estimated mask, and the second intermediate image into a second decoder to obtain the mask.

[0132] For example, as shown by the segmentation module in Figure 2A the second intermediate image is input into a second encoder to obtain a feature encoding of the second intermediate image, and then the feature encodings of the relevance degree, the estimated mask, and the second intermediate image are input into a second decoder to obtain the mask of the image to be processed.

[0133] According to an embodiment of the present disclosure, the second decoder outputs a mask matrix of the mask, where the value 0 represents background pixels and the value 1 represents pixels of the target object. The mask of the image to be processed can be obtained according to this mask matrix.

[0134] According to an embodiment of the present disclosure, the second decoder also outputs the confidence of the mask. The higher the confidence, the higher the accuracy of the mask. The masks with a confidence higher than the preset confidence and their corresponding images to be processed are stored in the memory of the estimation module as image-mask pairs.

[0135] According to an embodiment of the present disclosure, the first frame image including the target object and its mask in the above-described video image sequence may also be added to the memory of the estimation module.

[0136] According to an embodiment of the present disclosure, the reference target object image may be updated using the image to be processed corresponding to the mask with a confidence level higher than the preset confidence level. For example, each time a mask with a confidence level higher than the preset confidence level is obtained, the corresponding image to be processed may be segmented according to the mask to obtain an image including the target object and a part of the background image as the updated reference target object image. In this way, the reference target object image can be made closer to the target object image in the image to be processed, further improving the accuracy of the finally obtained mask.

[0137] According to an embodiment of the present disclosure, a more accurate mask can be obtained using the correlation between the first intermediate image and the reference target object image and the estimated mask generated based on the mask of the relatively accurate prior image.

[0138] According to an embodiment of the present disclosure, after determining the mask of the image to be processed, the target object in the image to be processed may be erased, and an image inpainting algorithm may be used to complete the erased part.

[0139] According to an embodiment of the present disclosure, the method further includes: erasing the target object in the image to be processed based on the mask; using an image inpainting algorithm to complete the image to be processed after erasing the target object.

[0140] According to an embodiment of the present disclosure, an image-level restoration solution based on deep learning, such as the two-stage image inpainting adversarial algorithm EdgeConnect, the encoder-decoder-based Vornet algorithm, etc., may be used to complete the image after erasing the target object.

[0141] Alternatively, variational methods, sequential filling algorithms with boundary priority, etc. may also be applied to complete the image after erasing the target object.

[0142] According to an embodiment of the present disclosure, the image to be processed is an image in a video image sequence, and using an image inpainting algorithm to complete the image to be processed after erasing the target object includes: based on the completion results of the specified image before the image to be processed and the completion results of the specified image after the image to be processed in the video image sequence, using an image inpainting algorithm to process the image to be processed after erasing the target object to complete the image to be processed after erasing the target object.

[0143] According to an embodiment of the present disclosure, the Deep Video Inpainting algorithm can be used to complete the processing of the image after erasing the target object. Specifically, assuming that the image to be processed is the k-th frame in the video image sequence, the completion results of the specified images in the video image sequence (for example, including the k-6-th frame, the k-3-th frame, the k-1-th frame, the k+3-th frame, and the k+6-th frame) are used as inputs. The features of each frame image are aligned through the optical flow method, and the motion information estimation of the k-1-th frame and the k-th frame is obtained by using the optical flow method to obtain the reference result of the k-th frame. The reference result is superimposed with the downsampled result of the k-th frame, and the superimposed result is input into the Convolutional Long Short-Term Memory Network (Conv-LSTM) to obtain the completion result of the k-th frame.

[0144] Figure 2D A flowchart showing an image processing method according to an embodiment of the present disclosure is as follows. Figure 2D As shown, the image processing method includes the following steps S201–S203:

[0145] In step S201, a reference image is displayed;

[0146] In step S202, according to the user's operation on the reference image, a reference target object image is determined from the reference image, and the reference target object image includes the target object;

[0147] In step S203, an output image is displayed, and the output image is an image obtained by removing the target object from the image to be processed.

[0148] According to an embodiment of the present disclosure, the output image is obtained by erasing the target object from the image to be processed using the mask obtained by the method described above, and completing the processing of the image to be processed after erasing the target object using an image inpainting algorithm. According to an embodiment of the present disclosure, the reference image can be an image in a video image sequence, for example, the first frame image in the video image sequence that includes the target object. The user can manually select a part that includes the target object and optionally includes a part of the background image as the reference target object image. Using the image processing method according to an embodiment of the present disclosure, the user can conveniently select the target object to be erased from the video image, and obtain the video image sequence after erasing the target object, automatically completing video editing.

[0149] The method according to an embodiment of the present disclosure can be used in scenarios such as automatic advertisement filtering products, automatic sensitive information filtering, automatic illegal information filtering, and various video editing fields. For example, the user can configure the attributes or features of the filtering object to automatically complete video editing.

[0150] For example, there may be unwanted products in an advertising video, and these products can be conveniently erased by the method of the present disclosure. Or, the video may contain sensitive or illegal content, items, etc., and the images of these sensitive or illegal content and items can be conveniently erased by the method of the present disclosure. Or, when performing video editing, the user can conveniently select a reference target object image containing the target object to be erased on the reference image with a selection box, or the user can configure the attributes or features of the target object, such as color, shape, etc., and determine the reference target object image from the reference image by means of image intelligent recognition, so as to automatically complete video editing.

[0151] Figure 3A The schematic diagram of the effect of processing a video image sequence by using the image processing method according to an embodiment of the present disclosure is shown.

[0152] Figure 3B The schematic diagram of the effect of one of the frames obtained by processing a video image sequence by using the image processing method according to an embodiment of the present disclosure is shown.

[0153] As Figure 3A and Figure 3B shown, the image processing method according to an embodiment of the present disclosure realizes intelligent target erasure, tracks the pixel positions of the target object to be erased in the video image, uses an algorithm for video image restoration to refill the erased target, making it as similar as possible to the surrounding background, realizes the seamless removal of the target object, and considers the inter-frame information both when generating the mask and performing image restoration, reducing the discomfort with the surrounding background when erasing the target object.

[0154] Figure 4A The structural block diagram of the image processing device according to an embodiment of the present disclosure is shown. Among them, the device can be implemented as part or all of an electronic device through software, hardware or a combination of both.

[0155] As Figure 4A shown, the image processing device 410 includes a first acquisition module 411, a first determination module 412, a second acquisition module 413, and a third acquisition module 414.

[0156] The first acquisition module 411 is configured to acquire a first intermediate image including the target object and a second intermediate image including the target object from the image to be processed;

[0157] The first determination module 412 is configured to determine the degree of association between the first intermediate image and the reference target object image;

[0158] The second acquisition module 413 is configured to acquire an estimated mask of the image to be processed;

[0159] The third acquisition module 414 is configured to acquire a mask of the image to be processed according to the degree of association, the estimated mask, and the second intermediate image, where the mask is used to segment the target object from the image to be processed.

[0160] According to an embodiment of the present disclosure, acquiring the first intermediate image including the target object and the second intermediate image including the target object from the image to be processed includes:

[0161] Determine a target region in the image to be processed, where the target region includes the target object;

[0162] Take the part of the image to be processed located in the target region as the first intermediate image;

[0163] Take the part of the image to be processed located in the target region and the part surrounding the target region as the second intermediate image.

[0164] According to an embodiment of the present disclosure, where:

[0165] The image to be processed is an image in a video image sequence;

[0166] The target region is obtained by using a target tracking algorithm based on the previous frame image of the image to be processed in the video image sequence and the reference target object image.

[0167] According to an embodiment of the present disclosure, where:

[0168] The image to be processed is an image in a video image sequence;

[0169] The target region is obtained by using a target tracking algorithm based on the previous frame image of the image to be processed in the video image sequence and the target object image, and correcting the target tracking result by using the mask of the previous frame image.

[0170] According to an embodiment of the present disclosure, where determining the degree of association between the first intermediate image and the reference target object image includes:

[0171] Extract the image features of the first intermediate image and the image features of the reference target object image;

[0172] Calculate the cross-correlation value of the image features of the first intermediate image and the image features of the reference target object image as the degree of association.

[0173] According to an embodiment of the present disclosure, where acquiring the estimated mask of the image to be processed includes:

[0174] Input a set of image-mask pairs and the image to be processed into a mask estimation model to obtain the estimated mask, where the image-mask pair includes a previous image before the image to be processed in a video image sequence and the mask of the previous image, and the confidence of the mask of the previous image is higher than a preset confidence level.

[0175] According to an embodiment of the present disclosure, wherein the inputting a set of image-mask pairs and the image to be processed into a mask estimation model to obtain the estimated mask includes:

[0176] Perform feature encoding on each image-mask pair in the set of image-mask pairs;

[0177] Perform feature encoding on the image to be processed;

[0178] Perform feature matching on the feature encoding of each image-mask pair and the feature encoding of the image to be processed, and input the feature matching result and the feature encoding of the image to be processed into a first decoder to obtain the estimated mask.

[0179] According to an embodiment of the present disclosure, wherein the reference target object image is a preset target object image or a target object image obtained by segmenting the corresponding image using the mask in the image-mask pair.

[0180] According to an embodiment of the present disclosure, wherein the obtaining the mask of the image to be processed according to the correlation degree, the estimated mask, and the second intermediate image includes:

[0181] Input the feature encoding of the correlation degree, the estimated mask, and the second intermediate image into a second decoder to obtain the mask.

[0182] According to an embodiment of the present disclosure, the apparatus 410 further includes:

[0183] An erasing module 415, configured to erase the target object in the image to be processed based on the mask;

[0184] A filling module 416, configured to fill the image to be processed after erasing the target object using an image inpainting algorithm.

[0185] According to an embodiment of the present disclosure, wherein the image to be processed is an image in a video image sequence, and the filling the image to be processed after erasing the target object using an image inpainting algorithm includes:

[0186] Based on the completion results of the specified image before the image to be processed and the completion results of the specified image after the image to be processed in the video image sequence, use an image inpainting algorithm to process the image to be processed after erasing the target object, so as to complete the image to be processed after erasing the target object.

[0187] Figure 4B The structural block diagram of an image processing device according to an embodiment of the present disclosure is shown. Among them, the device can be implemented as part or all of an electronic device through software, hardware, or a combination of both.

[0188] As Figure 4B shown, the image processing device 420 includes a first display module 421, a second determination module 422, and a second display module 423.

[0189] The first display module 421 is configured to display a reference image;

[0190] The second determination module 422 is configured to determine a reference target object image from the reference image according to the user's operation on the reference image, and the reference target object image includes the target object;

[0191] The second display module 423 is configured to display an output image, and the output image is an image obtained by removing the target object from the image to be processed.

[0192] According to an embodiment of the present disclosure, the output image is obtained by erasing the target object from the image to be processed using the mask obtained by the method described above, and using an image inpainting algorithm to complete the image to be processed after erasing the target object.

[0193] The present disclosure also discloses an electronic device, Figure 5 The structural block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0194] As Figure 5 shown, the electronic device 500 includes a memory 501 and a processor 502. Among them, the memory 501 is used to store one or more computer instructions, and the one or more computer instructions are executed by the processor 502 to implement the method according to an embodiment of the present disclosure.

[0195] An embodiment of the present disclosure provides an image processing method, including:

[0196] Obtain a first intermediate image including a target object and a second intermediate image including the target object from the image to be processed;

[0197] Determine the degree of association between the first intermediate image and the reference target object image;

[0198] Obtain an estimated mask of the image to be processed;

[0199] According to the degree of association, the estimated mask, and the second intermediate image, obtain a mask of the image to be processed, where the mask is used to segment the target object from the image to be processed.

[0200] According to an embodiment of the present disclosure, the obtaining the first intermediate image including the target object and the second intermediate image including the target object from the image to be processed includes:

[0201] Determine a target region in the image to be processed, where the target region includes the target object;

[0202] Take the part of the image to be processed located in the target region as the first intermediate image;

[0203] Take the part of the image to be processed located in the target region and the part surrounding the target region as the second intermediate image.

[0204] According to an embodiment of the present disclosure, where:

[0205] The image to be processed is an image in a video image sequence;

[0206] The target region is obtained by using a target tracking algorithm based on the previous frame image of the image to be processed in the video image sequence and the reference target object image.

[0207] According to an embodiment of the present disclosure, where:

[0208] The image to be processed is an image in a video image sequence;

[0209] The target region is obtained by using a target tracking algorithm based on the previous frame image of the image to be processed in the video image sequence and the target object image, and correcting the target tracking result by using the mask of the previous frame image.

[0210] According to an embodiment of the present disclosure, where the determining the degree of association between the first intermediate image and the reference target object image includes:

[0211] Extract the image features of the first intermediate image and the image features of the reference target object image;

[0212] Calculate the cross-correlation value of the image features of the first intermediate image and the image features of the reference target object image as the degree of association.

[0213] According to an embodiment of the present disclosure, where the obtaining the estimated mask of the image to be processed includes:

[0214] Input a set of image-mask pairs and the image to be processed into a mask estimation model to obtain the estimated mask, where the image-mask pair includes a previous image before the image to be processed in a video image sequence and a mask of the previous image, and the confidence of the mask of the previous image is higher than a preset confidence level.

[0215] According to an embodiment of the present disclosure, where the inputting a set of image-mask pairs and the image to be processed into a mask estimation model to obtain the estimated mask includes:

[0216] Perform feature encoding on each image-mask pair in the set of image-mask pairs;

[0217] Perform feature encoding on the image to be processed;

[0218] Perform feature matching on the feature encoding of each image-mask pair and the feature encoding of the image to be processed, and input the feature matching result and the feature encoding of the image to be processed into a first decoder to obtain the estimated mask.

[0219] According to an embodiment of the present disclosure, where the reference target object image is a preset target object image or a target object image obtained by segmenting the corresponding image using the mask in the image-mask pair.

[0220] According to an embodiment of the present disclosure, where the obtaining the mask of the image to be processed according to the degree of association, the estimated mask, and the second intermediate image includes:

[0221] Input the feature encoding of the degree of association, the estimated mask, and the second intermediate image into a second decoder to obtain the mask.

[0222] According to an embodiment of the present disclosure, the method further includes:

[0223] Erase the target object in the image to be processed based on the mask;

[0224] Use an image inpainting algorithm to complete the image to be processed after erasing the target object.

[0225] According to an embodiment of the present disclosure, where the image to be processed is an image in a video image sequence, and the using an image inpainting algorithm to complete the image to be processed after erasing the target object includes:

[0226] Based on the completion result of a specified image before the image to be processed and the completion result of a specified image after the image to be processed in the video image sequence, use an image inpainting algorithm to process the image to be processed after erasing the target object to complete the image to be processed after erasing the target object.

[0227] Figure 6 A schematic structural diagram of a computer system suitable for implementing the method according to an embodiment of the present disclosure is shown.

[0228] As Figure 6 shown, the computer system 600 includes a processing unit 601, which can perform various processes in the above embodiments according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage section 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The processing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0229] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as required. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as required, so that a computer program read from it can be installed into the storage section 608 as required. Among them, the processing unit 601 can be implemented as a processing unit such as a CPU, a GPU, a TPU, an FPGA, an NPU, etc.

[0230] In particular, according to an embodiment of the present disclosure, the method described above can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes computer instructions that, when executed by a processor, implement the method steps described above. In such an embodiment, the computer program product can be downloaded and installed from a network through the communication section 609, and / or installed from the removable medium 611.

[0231] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0232] The units or modules involved in the embodiments described in the present disclosure can be implemented in software or in programmable hardware. The described units or modules can also be provided in a processor, and the names of these units or modules do not, in some cases, constitute a limitation on the units or modules themselves.

[0233] On the other hand, the present disclosure also provides a computer-readable storage medium, which can be the computer-readable storage medium included in the electronic device or computer system in the above embodiments; or it can exist separately and be unassembled into the device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the methods described in the present disclosure.

[0234] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles applied. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, technical solutions formed by mutually replacing the above features with technical features (but not limited to) having similar functions disclosed in the present disclosure.

Claims

1. An image processing method, comprising: Obtaining a first intermediate image including a target object and a second intermediate image including the target object from an image to be processed, including: determining a target area in the image to be processed, the target area including the target object; taking the part of the image to be processed located in the target area as the first intermediate image; taking the part of the image to be processed located in the target area and the part surrounding the target area as the second intermediate image; Determining the degree of association between the first intermediate image and a reference target object image; Obtaining an estimated mask of the image to be processed, including: inputting a set of image-mask pairs and the image to be processed into a mask estimation model to obtain the estimated mask, where the image-mask pair includes a previous image before the image to be processed in a video image sequence and the mask of the previous image, and the confidence level of the mask of the previous image is higher than a preset confidence level; Obtaining a mask of the image to be processed according to the degree of association, the estimated mask, and the second intermediate image, the mask being used to segment the target object from the image to be processed.

2. The method according to claim 1, wherein: The image to be processed is an image in a video image sequence; The target area is obtained by using a target tracking algorithm based on the previous frame image of the image to be processed in the video image sequence and the reference target object image.

3. The method according to claim 1, wherein: The image to be processed is an image in a video image sequence; The target area is obtained by using a target tracking algorithm based on the previous frame image of the image to be processed in the video image sequence and the target object image, and correcting the target tracking result by using the mask of the previous frame image.

4. The method according to claim 1, wherein The determining the degree of association between the first intermediate image and the reference target object image includes: Extracting the image features of the first intermediate image and the image features of the reference target object image; Calculating the cross-correlation value of the image features of the first intermediate image and the image features of the reference target object image as the degree of association.

5. The method according to claim 1, wherein, The inputting the set of image-mask pairs and the image to be processed into the mask estimation model to obtain the estimated mask includes: Performing feature encoding on each image-mask pair in the set of image-mask pairs; Performing feature encoding on the image to be processed; Performing feature matching on the feature encoding of each image-mask pair and the feature encoding of the image to be processed, and inputting the feature matching result and the feature encoding of the image to be processed into a first decoder to obtain the estimated mask.

6. The method according to claim 5, wherein, The reference target object image is a pre-set target object image or a target object image obtained by segmenting the corresponding image by using the mask in the image-mask pair.

7. The method according to claim 1, wherein The obtaining the mask of the image to be processed according to the degree of association, the estimated mask, and the second intermediate image includes: Inputting the feature encoding of the degree of association, the estimated mask, and the second intermediate image into a second decoder to obtain the mask.

8. The method according to claim 1, further comprising: Erase the target object in the image to be processed based on the mask; Use an image inpainting algorithm to complete the image to be processed after erasing the target object.

9. The method according to claim 8, wherein The image to be processed is an image in a video image sequence. The using an image inpainting algorithm to complete the image to be processed after erasing the target object includes: Based on the completion result of a specified image before the image to be processed and the completion result of a specified image after the image to be processed in the video image sequence, use an image inpainting algorithm to process the image to be processed after erasing the target object, so as to complete the image to be processed after erasing the target object.

10. An image processing method, including: Display a reference image; According to the user's operation on the reference image, determine a reference target object image from the reference image, and the reference target object image includes a target object; Display an output image, where the output image is an image obtained by removing the target object from the image to be processed. Among them, the output image is obtained by erasing the target object from the image to be processed using the mask obtained by the method according to claim 1, and using an image inpainting algorithm to complete the image to be processed after erasing the target object.

11. An image processing device, including: A first acquisition module configured to acquire a first intermediate image including a target object and a second intermediate image including the target object from the image to be processed; A first determination module configured to determine the degree of association between the first intermediate image and the reference target object image; A second acquisition module configured to acquire an estimated mask of the image to be processed, including: inputting an image-mask pair set and the image to be processed into a mask estimation model to acquire the estimated mask, where the image-mask pair includes a previous image before the image to be processed in the video image sequence and the mask of the previous image, and the confidence level of the mask of the previous image is higher than a preset confidence level; A third acquisition module configured to acquire a mask of the image to be processed according to the degree of association, the estimated mask, and the second intermediate image, and the mask is used to segment the target object from the image to be processed.

12. The device according to claim 11, further including: An erasing module configured to erase the target object in the image to be processed based on the mask; A completion module configured to use an image inpainting algorithm to complete the image to be processed after erasing the target object.

13. An image processing device, including: A first display module configured to display a reference image; A second determination module configured to determine a reference target object image from the reference image according to the user's operation on the reference image, and the reference target object image includes a target object; A second display module, configured to display an output image, where the output image is an image obtained by removing the target object from the image to be processed. Among them, the output image is obtained by erasing the target object from the image to be processed using a mask obtained by the method according to claim 1, and using an image inpainting algorithm to complete the image to be processed after erasing the target object.

14. An electronic device, comprising a memory and a processor; wherein, The memory is used to store one or more computer instructions, where the one or more computer instructions are executed by the processor to implement the method steps described in any one of claims 1-10.

15. A readable storage medium, on which computer instructions are stored, and when the computer instructions are executed by a processor, the method steps described in any one of claims 1-10 are implemented.

16. A computer program product, including computer instructions, and when the computer instructions are executed by a processor, the method steps described in any one of claims 1-10 are implemented.

Images