Face cutting method and device, storage medium and electronic device

Abstract

The present disclosure relates to a method and device for image matting, a storage medium and an electronic device. The method comprises: in response to a matting instruction, displaying a first image, the first image being any image; determining a first mask region corresponding to a first object in the first image, the first object being any object in the first image; and performing image matting processing on the first mask region in the first image. The present disclosure can support any selected object for image matting processing, and is no longer limited to image matting of certain types of objects, thereby significantly expanding the use range of image matting processing and meeting the image matting needs of users for different objects.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of Internet, and particularly relates to a cutout method and device, a storage medium and an electronic device. BACKGROUND

[0002] With the development of image processing technology, cutout processing on video or image has been applied more and more. Specifically, the cutout can be based on explicit features in a picture, such as cutting out a portrait, a head, or the sky. However, the object of the cutout processing in the related art is relatively limited, and the cutout processing can usually be performed only on relatively limited objects such as a portrait, a head, or the sky, and it is difficult to support cutout processing on various objects. SUMMARY

[0003] To solve at least one of the above technical problems, the present disclosure provides a cutout method, device, storage medium and electronic device, and the technical solutions of the present disclosure are as follows.

[0004] According to a first aspect of an embodiment of the present disclosure, a cutout method is provided, comprising:

[0005] In response to a cutout instruction, a first image is displayed, the first image being any image;

[0006] A first mask region corresponding to a first object in the first image is determined, the first object being any object in the first image;

[0007] The first mask region in the first image is subjected to cutout processing.

[0008] In some possible embodiments, the cutout instruction comprises a foreground subject segmentation instruction or an interactive segmentation instruction, wherein the foreground subject segmentation instruction is used to segment a foreground subject region in the first image;

[0009] The interactive segmentation instruction is used to segment an object in the first image pointed to by the interactive segmentation instruction.

[0010] In some possible embodiments, when the cutout instruction comprises the foreground subject segmentation instruction, the determination of the first mask region corresponding to the first object comprises: in response to the foreground subject segmentation instruction, a foreground subject segmentation result of the first image is displayed; and in response to a selection instruction on the foreground subject segmentation result, the first mask region corresponding to the first object is determined.

[0011] In a case where the image matting instruction comprises the interactive segmentation instruction, the determining the first mask region corresponding to the first object comprises: in response to the interactive segmentation instruction, regarding an object in the first image occupying an area intersecting with a first action region of the interactive segmentation instruction as the first object, and determining the first mask region corresponding to the first object.

[0012] In some possible embodiments, the image matting instruction further comprises a free painting instruction, and the determining the first mask region corresponding to the first object further comprises:

[0013] obtaining a second action region corresponding to the free painting instruction;

[0014] updating the first mask region based on the second action region.

[0015] In some possible embodiments, the first image is a frame image in a video, and the video comprises at least one video segment, and after the first mask region in the first image is subjected to the image matting processing, the method further comprises:

[0016] sequentially performing tracking-based image matting processing on images in the video and located after a timestamp of the first image and located in a target video segment until a preset tracking stop condition is reached, the target video segment being a video segment in which the first image is located.

[0017] In some possible embodiments, the tracking stop condition is that a new image matting instruction is received or all frame images located after the timestamp of the first image in the target video segment are subjected to image matting.

[0018] In some possible embodiments, in a case where a new image matting instruction is received, the current tracking-based image matting processing is stopped, an image pointed to by the new image matting instruction is taken as a new first image, the new first image is subjected to image matting, and tracking-based image matting processing is performed based on the new first image.

[0019] In some possible embodiments, the sequentially performing tracking-based image matting processing on images in the video and located after the timestamp of the first image and located in the target video segment comprises:

[0020] tracking the first mask region to obtain a second mask region in a second image, the second image being a frame image in the video and located after the timestamp of the first image and adjacent to the first image;

[0021] subjecting the second mask region in the second image to image matting processing.

[0022] In some possible embodiments, the displaying the first image comprises: in a case where a zoom gesture is acquired for the first image, performing corresponding zoom display on the first image based on the zoom gesture.

[0023] In some possible embodiments, the acquiring the second action region corresponding to the free painting instruction comprises:

[0024] acquiring a trigger position of the free painting instruction in the first image;

[0025] determining a target region based on the trigger position;

[0026] performing zoom-in display on the target region;

[0027] in a case where a painting track in the target region is acquired, determining the second action region.

[0028] According to a second aspect of the embodiments of the present disclosure, a picture cutting device is provided, comprising:

[0029] an image display module configured to perform display of a first image in response to a picture cutting instruction, the first image being any image;

[0030] a mask determination module configured to perform determination of a first mask region corresponding to a first object in the first image, the first object being any object in the first image;

[0031] a picture cutting module configured to perform picture cutting processing on the first mask region in the first image.

[0032] In some possible embodiments, the picture cutting instruction comprises a foreground subject segmentation instruction or an interactive segmentation instruction, wherein the foreground subject segmentation instruction is used for segmentation of a foreground subject region in the first image.

[0033] the interactive segmentation instruction is used for segmentation of an object in the first image that is pointed to by the interactive segmentation instruction.

[0034] In some possible embodiments, in a case where the picture cutting instruction comprises the foreground subject segmentation instruction, the mask determination module is configured to perform display of a foreground subject segmentation result of the first image in response to the foreground subject segmentation instruction; and perform determination of the first mask region corresponding to the first object in response to a selection instruction on the foreground subject segmentation result.

[0035] In a case where the cutout instruction comprises the interaction segmentation instruction, the mask determination module is configured to perform, in response to the interaction segmentation instruction, determining a first object in the first image as the first object, the first object being an object occupying an area intersecting with a first action area of the interaction segmentation instruction.

[0036] In some possible embodiments, the cutout instruction further comprises a free painting instruction, and the mask determination module is configured to perform:

[0037] obtaining a second action area corresponding to the free painting instruction;

[0038] updating the first mask area based on the second action area.

[0039] In some possible embodiments, the first image is a frame image in a video, and the video comprises at least one video segment, and the cutout module is configured to perform:

[0040] performing, in sequence, tracking-based cutout processing on images in the video, the images being located after a timestamp of the first image and located in a target video segment, until a preset tracking stop condition is reached, the target video segment being a video segment in which the first image is located.

[0041] In some possible embodiments, the tracking stop condition is that a new cutout instruction is received or all frame images located after the timestamp of the first image in the target video segment are cut out.

[0042] In some possible embodiments, the cutout module is configured to perform, in a case where a new cutout instruction is received, stopping current tracking-based cutout processing, taking an image indicated by the new cutout instruction as a new first image, performing cutout processing on the new first image, and performing tracking-based cutout processing based on the new first image.

[0043] In some possible embodiments, the cutout module is configured to perform:

[0044] tracking the first mask area to obtain a second mask area in a second image, the second image being a frame image in the video, the second image being located after the timestamp of the first image and adjacent to the first image;

[0045] performing cutout processing on the second mask area in the second image.

[0046] In some possible embodiments, the image display module is configured to perform, in a case where a zoom gesture for the first image is acquired, corresponding zoom display of the first image based on the zoom gesture.

[0047] In some possible embodiments, the mask determination module is configured to perform:

[0048] acquiring a trigger position of the free painting instruction in the first image;

[0049] determining a target area based on the trigger position;

[0050] performing zoom display on the target area;

[0051] in a case where a painting track in the target area is acquired, determining the second action area.

[0052] According to a third aspect of embodiments of the present disclosure, an electronic device is provided, including: a processor; a memory for storing processor-executable instructions; and wherein the processor is configured to execute the instructions to implement the method of any one of the first aspect.

[0053] According to a fourth aspect of embodiments of the present disclosure, a computer-readable storage medium is provided, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method of any one of the first aspect.

[0054] According to a fifth aspect of embodiments of the present disclosure, a computer program product is provided, the computer program product including a computer program stored in a readable storage medium, at least one processor of a computer device reading and executing the computer program from the readable storage medium, so that the computer device performs the method of any one of the first aspect.

[0055] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects:

[0056] The method for cutting out an image provided in the embodiments of the present disclosure can display a first image in response to a cutting-out instruction; determine a first mask area corresponding to a first object in the first image, the first object being any object in the first image; and perform cutting-out processing on the first mask area in the first image. By supporting cutting-out processing on any selected object, the use range of the cutting-out processing is significantly widened, which is no longer limited to cutting-out of certain types of objects. To meet the cutting-out requirements of users on different objects, the cutting-out can be customized, one-key automatic identification of the subject of an image can be supported, and object tracking and tracking object key frame technologies can be used to perform real-time cutting-out on any object in a video. In the embodiments of the present disclosure, manual cutting-out and automatic cutting-out can be combined, any automatically identified cutting-out can be selected, and then fine manual cutting-out adjustment can be performed, so that the user can quickly cut out an image, the effect of the cutting-out is better, the memory consumption is lower, and the computing resource occupation is less.

[0057] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0058] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure, and do not constitute an undue limitation on the present disclosure.

[0059] Figure 1 is an implementation environment schematic diagram of a method for cutting out an image according to an exemplary embodiment;

[0060] Figure 2 is a flow schematic diagram of a method for cutting out an image according to an exemplary embodiment;

[0061] Figure 3 is a first mask area schematic diagram according to an exemplary embodiment;

[0062] Figure 4 is a first action area schematic diagram of an interactive segmentation instruction for triggering a first mask area according to an exemplary embodiment; Figure 3

[0063] Figure 5 is a flow schematic diagram of video cutting-out according to an exemplary embodiment;

[0064] Figure 6 is a cutting-out schematic diagram for a first image according to an exemplary embodiment;

[0065] Figure 7 is another flow schematic diagram of video cutting-out according to an exemplary embodiment;

[0066] ​Figure 8 is a mask tracking diagram for a first image according to an example embodiment;

[0067] Figure 9 is a first cutout interface diagram according to an example embodiment;

[0068] Figure 10 is a second cutout interface diagram according to an example embodiment;

[0069] Figure 11 is a third cutout interface diagram according to an example embodiment;

[0070] Figure 12 is a fourth cutout interface diagram according to an example embodiment;

[0071] Figure 13 is a cutout flowchart for a video according to an example embodiment;

[0072] Figure 14 is a block diagram of another cutout device according to an example embodiment;

[0073] Figure 15 is a block diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0074] In order to make the ordinary person skilled in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings.

[0075] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation described in the following example embodiments does not represent all implementations consistent with the present disclosure. Rather, they are only examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0076] The various data such as user information, images, etc. involved in the present disclosure are information authorized by the user or authorized by all parties.

[0077] In the related art, a user can be provided with a video or image matting service to obtain a video or image after matting. The matting technology can be used in various scenarios. For example, in the editing scenario of a video or image, the matting can be used as a link in the editing process of the video or image. For another example, in the secondary publishing scenario of a video or image, the video or image after matting can be further edited and then secondarily published. However, the matting service provided in the related art has great limitations. For example, only the region with a person image can be processed for matting, the matting object is limited, and custom modification is not supported. The matting region is fixed, and if the matting is not clean or accurate, it cannot be repaired. For some objects with fine granularity, such as some hairs, the user cannot freely choose whether to remove them, thereby affecting the matting effect.

[0078] Further, the related art can only perform matting on simple objects, such as person image, head, and sky, and cannot provide the ability to mat all objects. In the video matting field, only one object can be fixedly matted from the beginning to the end of a video, and different objects cannot be matted at different time points.

[0079] To provide the ability to mat all kinds of objects and expand the matting use scenario, the embodiment of the present disclosure provides a matting method, device, storage medium, and electronic equipment.

[0080] Figure 1 is a schematic diagram of an implementation environment of a matting method according to an example embodiment. Taking an electronic equipment provided as a terminal as an example, referring to Figure 1 , the implementation environment specifically includes a terminal 101 and a server 102.

[0081] The terminal 101 can be at least one of a smart phone, a smart watch, a desktop computer, a laptop computer, and a notebook computer. The terminal 101 can be installed and run with an application program providing an image matting or video matting service. A user can log in to the application program through the terminal 101 to obtain the image matting or video matting service provided by the application program. The terminal 101 can generally refer to one of a plurality of terminals, and the embodiment is only exemplified by the terminal 101. Those skilled in the art can know that the number of the above terminals can be more or less. For example, the above terminals can be only a few, or the above terminals can be dozens or hundreds, or more, and the number and type of the terminal are not limited in the embodiment of the present disclosure.

[0082] The server 102 can be at least one of a server, a plurality of servers, a cloud computing platform, and a virtualization center. The server 102 can be connected to the terminal 101 and other terminals through a wireless network or a wired network, and can receive a related request sent by the terminal 101 to provide image matting or video matting services in cooperation with the terminal 101. Alternatively, the number of servers can be more or less, and the embodiments of the present disclosure do not limit this. Of course, the server 102 can also include other functional servers to provide more comprehensive and diversified services. Figure 2 is a flowchart of a matting method according to an exemplary embodiment, as shown in Figure 2 The above method at least includes the following steps S10-S30.

[0083] In step S10, in response to a matting instruction, a first image is displayed, and the first image is any image.

[0084] The embodiments of the present disclosure do not limit the scenario of displaying the first image, and the first image can be displayed in a video clip, video editing, image editing, etc. The embodiments of the present disclosure do not limit the source of the first image, which can come from a local gallery of the terminal 101, an image captured by a camera of the terminal 101, or a network. The first image can come from a picture or a frame image in a video. The embodiments of the present disclosure do not limit the content in the first image, and the first image can be any image, which can include any object existing in nature or a virtually constructed object.

[0085] Taking a frame image in a video as an example, a matting function can be triggered in an application interface, and a control in a time axis corresponding to the video is slid to locate a frame of the video picture, and the video picture is taken as the first image.

[0086] In step S20, a first mask region corresponding to a first object in the first image is determined, and the first object is any object in the first image.

[0087] The first object in the embodiments of the present disclosure refers to an object in the first image that needs to be matted, and the number and type of the first object are not limited.

[0088] In related technologies, only portraits, avatars, sky, and other relatively limited objects in an image can be automatically recognized, and only these relatively limited objects can be supported for matting processing. Unlike related technologies, the embodiments of the present disclosure can support automatic recognition of any object, thereby determining the contour of any object and obtaining the first mask region based on the contour.

[0089] The functions of automatic portrait cutting, head cutting, background cutting, and sky cutting in the related art are based on a single algorithm model. For example, the portrait cutting function uses a trained portrait recognition model to cut the portrait based on the recognition result of the portrait recognition model. Unlike the related art, the algorithm model used in the embodiments of the present disclosure automatically recognizes multiple objects. The model is trained using various types of objects, and thus has the ability to automatically recognize various objects and extract the edges of the various objects.

[0090] The embodiments of the present disclosure do not limit the method used for identifying any object. For example, images in nature or virtually constructed images can be classified to obtain category labels corresponding to each object. The neural network is trained using the category labels corresponding to each object and images including the object, so that the neural network has the ability to recognize objects of various categories. The classification process can be implemented by itself or by referring to existing open source object classification results in the related art. The present disclosure does not limit this. The only thing that needs to be made clear is that the embodiments of the present disclosure differ from the related art, which can only cut images of a small number of objects such as portraits, head portraits, and sky. The embodiments of the present disclosure support cutting images of various types of objects, that is, they can support "cutting everything", thereby significantly expanding the scope of application of image cutting. For example, the embodiments of the present disclosure can not only cut faces, portraits, and sky, but also cut animals, plants, electronic devices, mechanical devices, household items, buildings, and the like. The present disclosure does not limit this. Because of this "cutting everything" capability, the first image in step S10 can be any image, and the first object in step S20 can be any object.

[0091] In step S30, the first mask region in the first image is subjected to image cutting processing.

[0092] The first mask region is a mask representation of the contour of the first image, that is, the first mask region can be considered as a region that covers only the first object. For example, Figure 3 a first mask region diagram is shown. Figure 3 The left side is a first image including a butterfly, which is a first object, Figure 3 The right side is a mask diagram corresponding to the first image. The white part in the mask diagram is the first mask region. As is obvious, the first mask region covers only the first object and can be determined based on the contour of the first object.

[0093] The embodiments of the present disclosure do not limit the specific method of image cutting processing of the first mask region in the first image. For example, the pixels corresponding to the first mask region in the mask diagram corresponding to the first image can be set to 1, and the other pixels can be set to 0. The mask diagram and the first image are fused to obtain the result of the image cutting processing.

[0094] The embodiments of the present disclosure can perform cutout processing on any object, and are not limited to cutout processing on certain types of objects, thereby significantly expanding the use range of cutout processing.

[0095] The embodiments of the present disclosure do not limit the triggering manner of the cutout instruction, for example, the cutout instruction includes a foreground subject segmentation instruction or an interactive segmentation instruction, where the foreground subject segmentation instruction is used to segment a foreground subject region in the first image, and the interactive segmentation instruction is used to segment an object pointed to by the interactive segmentation instruction in the first image. The foreground subject segmentation instruction and the interactive segmentation instruction can be triggered by triggering a corresponding control by a user, that is, automatic cutout can be implemented based on the two instructions, the application range and efficiency of cutout are improved by automatically cutting out everything, and user stickiness is significantly increased.

[0096] Of course, the embodiments of the present disclosure do not limit the use method of the two segmentation instructions, for example, only one of them can be used, or they can be used jointly, each segmentation instruction can be used at least once, and the present disclosure does not limit the number of uses. For example, a first mask region A1 is obtained by using the foreground subject segmentation, the corresponding first object is person A, a first mask region A2 is obtained by using the interactive segmentation instruction, and the corresponding first object is person B. Then, cutout processing of the first image can be performed based on A1 and A2 to obtain an image including only person A and person B. For another example, a first mask region A3 is obtained by using the foreground subject segmentation, and a first mask region A4 is obtained by using the foreground subject segmentation again. The corresponding first objects of the first mask regions A3 and A4 are person C and person D, respectively. Then, cutout processing of the first image can be performed based on A3 and A4 to obtain an image including only person C and person D.

[0097] In one specific embodiment, in the case where the cutout instruction includes the foreground subject segmentation instruction, determining the first mask region corresponding to the first object includes: in response to the foreground subject segmentation instruction, displaying a foreground subject segmentation result of the first image; and in response to a selection instruction on the foreground subject segmentation result, determining the first mask region corresponding to the first object. The foreground subject segmentation instruction can automatically segment a foreground subject region in the first image, and by displaying the foreground subject segmentation result, the user can clearly distinguish the foreground subject region and the background region. The user can select the foreground subject region or the background region for cutout, and by determining the first mask region based on the foreground subject segmentation instruction, the user can directly trigger foreground cutout and background cutout, thereby saving the user operation path, improving the cutout speed, and increasing the user stickiness.

[0098] In one specific embodiment, when the keying instruction includes the interactive segmentation instruction, determining the first mask region corresponding to the first object includes: in response to the interactive segmentation instruction, identifying the object whose area in the first image intersects with the first effective area of ​​the interactive segmentation instruction as the first object, and determining the first mask region corresponding to the first object. In this embodiment, the first effective area refers to the effective area of ​​the interactive segmentation instruction. If the user generates the interactive segmentation instruction by clicking on certain locations in the first image or by smearing on the first image, then those locations or the smeared trajectory can be considered the first effective area.

[0099] Please refer to Figure 4 It shows that the trigger was obtained. Figure 3 A schematic diagram of the first active area of ​​the interactive segmentation instruction in the first mask region. Figure 4 left side and Figure 4 The right side shows two different smearing paths, and it can be seen that both smearing paths are located in... Figure 3 Therefore, the object within the left-hand image, whose region in the first image intersects with the first effective region of the aforementioned interactive segmentation instruction, is... Figure 3 The butterflies in the picture, therefore, can be obtained Figure 3 The right side represents the first mask region of the butterfly. In this embodiment, the first object can be flexibly determined through interactive segmentation, thereby determining the first mask region. Interactive segmentation can significantly improve the flexibility of image matting.

[0100] Based on the above, a first mask region can be obtained through interactive segmentation and / or foreground subject segmentation. This embodiment of the present disclosure can further refine the first mask region, thereby significantly improving the precision of the keying, giving users greater autonomy in keying, and increasing user engagement. In one embodiment, the keying instruction further includes a free smearing instruction. Determining the first mask region corresponding to the first object further includes: obtaining a second active region corresponding to the free smearing instruction; and updating the first mask region based on the second active region. In this embodiment of the present disclosure, keying based on foreground subject segmentation instructions is automatic keying, while keying based on interactive instructions and keying based on free smearing instructions are manual custom keying. Combining automatic keying (also referred to as automatic recognition in this embodiment of the present disclosure) and manual custom keying (also referred to as user drawing in this embodiment of the present disclosure) can significantly improve keying accuracy.

[0101] The embodiments of the present disclosure do not limit the method of updating the first mask area based on the second action area. For example, the segmentation operation for the first object can be performed again based on the second action area and the existing first mask area to obtain a segmentation result, and the first mask area is updated based on the segmentation result. Taking the first object as a head portrait as an example, the head portrait can be roughly segmented through interactive segmentation and / or foreground subject segmentation, but there may still be deficiencies in details, for example, the hair part of the head portrait may not be segmented and processed, and the degree of detail is not enough. In this case, the hair part can be painted by triggering the free painting instruction, the second action area is determined according to the painting track, the segmentation model in the background identifies the head portrait again based on the second action area and the existing first mask area, and a more detailed head portrait recognition result is obtained, and the second action area is updated according to the head portrait recognition result.

[0102] In an embodiment, the first image is a frame image in a video, the video includes at least one video segment, and after the first mask area in the first image is subjected to the matting processing, the method further includes: sequentially performing tracking-based matting processing on images in the video whose timestamps are later than the timestamp of the first image and located in a target video segment until a preset tracking stop condition is reached, the target video segment being a video segment in which the first image is located. Specifically, the tracking stop condition is that a new matting instruction is received or all other frame images in the target video segment located after the timestamp of the first image have been subjected to matting. In the embodiments of the present disclosure, based on the matting operation on some frame images, automatic matting can be performed on other frame images after the some frame images, without the need for the user to perform image matting frame by frame, and through the rigorous video matting processing logic, the speed and intelligent degree of video matting are significantly improved.

[0103] Reference is made to Figure 5 which shows a flowchart of video matting. Figure 5 In the embodiment, the first image is a third frame image (frame3) in a video, a first mask image can be obtained in the first image based on user scribbling (interactive segmentation instruction and / or free painting instruction) or automatic recognition (foreground segmentation instruction), and a mask (Mask) can be obtained based on the first mask image. Based on the mask, the mask corresponding to each frame can be obtained in the subsequent frame images through tracking. Thus, the subsequent frame images are subjected to matting processing for the first object by fusing the mask corresponding thereto. Figure 5 It can be clearly seen from the embodiment that frame4 and frame5 are subjected to tracking-based matting processing after frame3. Of course, the embodiments of the present disclosure do not limit the fusion manner. Reference is made to Figure 6which shows a matting schematic diagram for the first image. Taking the first image as an example, the first image can be multiplied by the mask image corresponding to the first image, and the multiplication result can be rendered to obtain the matting result of the first image. The first image and the mask image or mask region obtained in the process of performing tracking-based matting processing can be stored. Of course, the specific storage method is not limited in the embodiments of the present disclosure.

[0104] Of course, in the case of receiving a new matting instruction, the current tracking-based matting processing can be stopped, the image pointed to by the new matting instruction is taken as a new first image, the new first image is subjected to matting processing, and tracking-based matting processing is performed based on the new first image. Of course, in the embodiments of the present disclosure, the specific operation of performing matting processing on the new first image and the operation of performing tracking-based matting processing based on the new first image can be referred to the foregoing, and will not be repeated here.

[0105] Please refer to Figure 7 which shows another flowchart of video matting. In the case of performing matting on frame3 to obtain a mask image, frame4, frame5 and frame6 can be processed based on the mask image by using the tracking-based matting method. The user performs new matting on frame7, frame7 can be taken as a new first image, and the mask image corresponding to frame7 is obtained, and frame8 and frame9 can be processed based on the mask image by using the tracking-based method. This processing method is based on the same inventive concept as “in the case of performing matting on frame3 to obtain a mask image, frame4, frame5 and frame6 can be processed based on the mask image by using the tracking-based matting method”, and will not be repeated here. The user performs new matting on frame10, frame10 can be taken as a new first image, and the mask image corresponding to frame10 is obtained, and subsequent frame images can be processed based on the mask image by using the tracking-based method until the video matting is completed. Moreover, different methods can be used for each time of matting, Figure 7 In the foregoing, the user scribble method is used for matting frame3 and frame10, and the foreground segmentation method is used for matting frame7.

[0106] In the video matting process, the first mask region can be determined based on the user inputted matting instruction, so as to obtain a mask graph for the first image. The mask graph or the first mask region belongs to non-reproducible data and can be saved as a draft material persistently. On the user timeline, if the mask graph or the first mask region is added at a certain time point, the subsequent frame images from the time point in the original video are sequentially subjected to the tracking-based matting processing, and the tracking-based matting processing is implemented based on the mask graph or the first mask region. Please refer to Figure 7 . Assuming that the user sequentially adds Mask1, Mask2, and Mask3 to the same segment, the range of the effect of each Mask on the original segment is also shown in Figure 7 .

[0107] The embodiments of the present disclosure do not limit the tracking-based matting processing method. For example, the above-mentioned tracking-based matting processing is sequentially performed on the images in the video whose timestamps are after the timestamp of the first image and are in the target video segment, including:

[0108] S101. Tracking the first mask region to obtain a second mask region in a second image, the second image being a frame image in the video whose timestamp is after the timestamp of the first image and is adjacent to the first image.

[0109] S102. Performing matting processing on the second mask region in the second image.

[0110] Please refer to Figure 8 , which shows a mask tracking diagram for the first image. Figure 8 The upper left image is the first image, Figure 8 The upper right image is a mask graph corresponding to the first image, and the white part in the mask graph is a first mask region of the first image, which represents the matting of the person riding a bicycle in the first image. Figure 8 The lower left image is a second image, Figure 8 The lower right image is a mask graph corresponding to the second image. The white part in the mask graph is a second mask region of the second image, which represents the tracking result of the first mask region. It is obvious that the second mask region can be used to mat the person riding a bicycle in the second image. Through tracking matting, the sequential matting of the same object in adjacent images can be realized.

[0111] Of course, in the process of carrying out the tracking-based matting processing, for a third image in the video, the timestamp of which is after the timestamp of the second image and adjacent to the second image, the matting processing manner of the third image is based on the same inventive concept as steps S101-S102, and details are not described herein. Specifically, the second mask region can be tracked to obtain a third mask region in the third image, and the third image is subjected to matting processing based on the third mask region. Through tracking-based matting processing, each frame image in the video that needs to be subjected to matting can be subjected to matting processing in turn, without the need for user intervention, and the matting effect is guaranteed, the video matting speed is significantly improved, and user stickiness is increased.

[0112] Please refer to Figure 9 , which shows a first schematic diagram of a matting interface. Different matting modes can be selected in the matting interface through automatic and manual selection. The default selection is the "manual" mode, that is, the "brush" in Figure 9 . The preview screen is divided into a preview state and an editing state, and the default is the editing state, which presents the picture of the first image and identifies the reserved area of matting. Of course, automatic foreground subject identification can also be achieved by selecting "automatic", and manual brushing is not allowed during the identification waiting process. The brush control is used for brushing in the first image, and the brush brushing area is the matting reserved area, that is, the semi-transparent area of the preview area in the editing state. In the editing state, the brush is selected, and the finger can be pressed to draw a line on the preview area. Figure 9 The "eraser" is also provided in Figure 9 , and the eraser brushing area is the matting removal area, that is, the unmarked area of the preview area in the editing state. In the editing state, the eraser is selected, and the marked area in the preview area can be removed. Figure 9 There is also a "preview" control. When the control is off, the non-preview state is presented, and the original picture of the first image can be displayed with the reserved area marked, wherein the semi-transparent block represents the reserved area. When the control is on, the preview state is presented, and only the area after the matting effect is displayed, and the semi-transparent block on the picture is hidden. In the case of entering custom matting, the preview is off by default, and in the case of exiting custom matting, the editing and preview can be switched back and forth.

[0113] In one embodiment, the display of the first image includes: in a case where a zoom gesture for the first image is acquired, performing corresponding zoom display on the first image based on the zoom gesture. This gesture-based zoom function can facilitate the user to more accurately control the display of the first image, help the user to accurately determine the matting position, so that the matting is more accurate, that is, it is beneficial to determine the accurate first mask region through brushing.

[0114] Please refer to Figure 10 , which shows another schematic diagram of a matting interface.Figure 10 The double-finger zoom view guide can be displayed to prompt the user to display the first image in a corresponding zoomed manner based on the zooming gesture described above. The guide can be displayed after the first use of the brush recognition, and can disappear by tapping anywhere on the screen.

[0115] In an embodiment, the second action area corresponding to the free painting instruction is obtained by: obtaining a trigger position of the free painting instruction in the first image; determining a target area based on the trigger position; displaying the target area in an enlarged manner; and determining the second action area in a case where a painting track in the target area is obtained. The determination of the second action area can achieve a magnifying glass effect. Specifically, a target area around the trigger position can be displayed, and the target area can be displayed in an enlarged manner to help the user better control the drawing direction during free painting.

[0116] After automatic recognition, the semi-transparent color block is used to cover the matting area in the preview area, and the matting preprocessing is started. Of course, the sliding track can be used to add or modify the matting area in multiple frames at the time axis position of a video segment. Matting is effective only for the current frame and subsequent frames in the video segment. When a frame is positioned and automatically or manually operated, the current frame and subsequent frames are processed based on the matting result of the current frame. If the current frame and subsequent frames are processed at a certain time before the current time, the current matting processing will perform an overlay operation. The preprocessed frames are not affected. The processed frames are played frame by frame, and the unprocessed frames are played frame by frame. If it is necessary to play the unprocessed frame image, a related prompt can be given, such as "matting processing, there may be slight lag".

[0117] Please refer to Figure 11 Another schematic diagram of the matting interface is shown. The first mask area of single-time addition / erasure can be automatically processed by finger painting in the preview area. The magnifying glass effect can be triggered when the finger is pressed and painted at a certain position in the preview area. The magnifying glass can also automatically adjust the position, such as when the finger draws a line to the upper left corner of the preview area and affects the magnifying glass, the magnifying glass can be displayed in the upper right corner. The magnifying glass can display the image in several times magnification, and the target area around the trigger position can be displayed in an enlarged manner.

[0118] In one embodiment, in the preview state, the preview picture cannot be zoomed in, and in the custom keyframe preview state, it cannot be zoomed out by double-finger zooming, nor can it trigger the free painting instruction. Only in the editing state can automatic keying or manual keying be performed. After free painting, the trajectory spreading calculation can be triggered when the finger is released, and after the calculation is completed, the complete picture is displayed, which is covered with a semi-transparent block to represent the keying area. If no new operation is performed within a predetermined time, the subsequent frame image can be automatically processed based on tracking keying.

[0119] Please refer to Figure 12 , which shows another schematic diagram of the keying interface. Figure 12 Two different video clips in the same video file are displayed in the video display module 10, and the keying operation for the first video clip has no effect on the second video clip. This design can achieve fine-grained video keying, and the video keying of a certain video clip will not affect other video clips.

[0120] Please refer to Figure 13 , which shows a keying flowchart for a video. After entering the keying function, a certain frame image (first image) can be positioned by sliding the timeline, the video is paused, and the first mask area for the first image can be determined by triggering foreground segmentation, interactive segmentation, and / or free painting. Based on the first mask area, a mask image (template mask) for the first image can be obtained. After editing is completed, the "confirm" instruction can be sent to trigger tracking-based keying operation for subsequent frames until keying is complete, or a new keying instruction is obtained from the user. Specifically, the user can pause the video playback and jump to the frame image to be keyed, and trigger the above-mentioned new keying instruction for the frame image to be keyed. Specifically, the keying method of the embodiments of the present disclosure can meet the keying needs of different objects for users, can customize keying, can support one-key automatic recognition of the main body of the image, and can use object tracking and tracking object keyframe technology to perform real-time keying on any object in the video. In the embodiments of the present disclosure, manual keying and automatic keying can be combined, any automatically recognized keying can be selected, and fine manual keying adjustment can be performed, so that the user can quickly key, the keying effect is better, the memory consumption is lower, and the computing resource occupation is less.

[0121] Please refer to Figure 14 , which shows a keying device, the device comprising:

[0122] The image display module 10 is configured to display a first image in response to a keying instruction, the first image being any image;

[0123] The mask determining module 20 is configured to determine a first mask region corresponding to a first object in the first image, the first object being any object in the first image.

[0124] The image matting module 30 is configured to perform image matting processing on the first mask region in the first image.

[0125] In some possible embodiments, the image matting instruction comprises foreground subject segmentation instruction or interactive segmentation instruction,

[0126] The foreground subject segmentation instruction is used to segment a foreground subject region in the first image.

[0127] The interactive segmentation instruction is used to segment an object in the first image pointed by the interactive segmentation instruction.

[0128] In some possible embodiments, when the image matting instruction comprises the foreground subject segmentation instruction, the mask determining module is configured to display a foreground subject segmentation result of the first image in response to the foreground subject segmentation instruction, and determine the first mask region corresponding to the first object in response to a selection instruction on the foreground subject segmentation result.

[0129] When the image matting instruction comprises the interactive segmentation instruction, the mask determining module is configured to determine the first mask region corresponding to the first object by taking an object in the first image occupying a region intersecting a first action region of the interactive segmentation instruction as the first object in response to the interactive segmentation instruction.

[0130] In some possible embodiments, the image matting instruction further comprises a free painting instruction, and the mask determining module is configured to:

[0131] Obtain a second action region corresponding to the free painting instruction.

[0132] Update the first mask region based on the second action region.

[0133] In some possible embodiments, the first image is a frame image in a video, the video comprises at least one video segment, and the image matting module is configured to:

[0134] Perform tracking-based image matting processing on images in the video whose timestamps are later than a timestamp of the first image and located in a target video segment until a preset tracking stop condition is reached, the target video segment being a video segment where the first image is located.

[0135] In some possible embodiments, the tracking stop condition is that a new matting instruction is received or all other frame images after the timestamp of the first image in the target video segment are completely matted.

[0136] In some possible embodiments, the matting module is configured to perform the following operations: in a case where a new matting instruction is received, stopping the current tracking-based matting processing, taking the image pointed to by the new matting instruction as a new first image, performing matting processing on the new first image, and performing tracking-based matting processing based on the new first image.

[0137] In some possible embodiments, the matting module is configured to perform the following operations:

[0138] tracking the first mask region to obtain a second mask region in a second image, the second image being a frame image in the video and adjacent to the first image and having a timestamp after the timestamp of the first image;

[0139] performing matting processing on the second mask region in the second image.

[0140] In some possible embodiments, the image display module is configured to perform the following operation: in a case where a zoom gesture for the first image is acquired, performing corresponding zoom display on the first image based on the zoom gesture.

[0141] In some possible embodiments, the mask determination module is configured to perform the following operations:

[0142] acquiring a trigger position of the free painting instruction in the first image;

[0143] determining a target region based on the trigger position;

[0144] performing zoomed-in display on the target region;

[0145] In a case where a painting track in the target region is acquired, determining the second action region. As to the device in the above embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and will not be described here in detail.

[0146] Figure 15 FIG. 6 is a block diagram of an electronic device 600 for image matting according to an example embodiment.

[0147] The electronic device can be a server, and can also be a terminal device. The internal structure diagram of the electronic device can be as shown in FIG. 6. Figure 15As shown in the figure. The electronic device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the electronic device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement a matting method.

[0148] Those skilled in the art can understand that, Figure 15 The structure shown in the figure is only a block diagram of part of the structure related to the present disclosure scheme, and does not constitute a limitation on the electronic device to which the present disclosure scheme is applied. The specific electronic device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0149] In an example embodiment, an electronic device is also provided, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a matting method as in the embodiments of the present disclosure.

[0150] In an example embodiment, a computer readable storage medium is also provided, when the instructions in the computer readable storage medium are executed by the processor of the electronic device, the electronic device can execute the matting method in the embodiments of the present disclosure. Specifically, the above-mentioned matting method includes:

[0151] In response to the matting instruction, display a first image, the first image being any image;

[0152] Determine a first mask region corresponding to a first object in the first image, the first object being any object in the first image;

[0153] Perform matting processing on the first mask region in the first image.

[0154] In some possible embodiments, the matting instruction includes a foreground subject segmentation instruction or an interactive segmentation instruction,

[0155] The foreground subject segmentation instruction is used to segment a foreground subject region in the first image;

[0156] The interactive segmentation instruction is used to segment an object in the first image pointed to by the interactive segmentation instruction.

[0157] In some possible embodiments, in a case where the image matting instruction comprises the foreground subject segmentation instruction, the determining the first mask region corresponding to the first object comprises: in response to the foreground subject segmentation instruction, displaying a foreground subject segmentation result of the first image; and in response to a selection instruction on the foreground subject segmentation result, determining the first mask region corresponding to the first object.

[0158] In a case where the image matting instruction comprises the interaction segmentation instruction, the determining the first mask region corresponding to the first object comprises: in response to the interaction segmentation instruction, regarding an object occupying an area intersecting with a first action region of the interaction segmentation instruction in the first image as the first object, and determining the first mask region corresponding to the first object.

[0159] In some possible embodiments, the image matting instruction further comprises a free painting instruction, and the determining the first mask region corresponding to the first object further comprises:

[0160] obtaining a second action region corresponding to the free painting instruction;

[0161] updating the first mask region based on the second action region.

[0162] In some possible embodiments, the first image is a frame image in a video, and the video comprises at least one video segment, and after the first mask region in the first image is subjected to the image matting processing, the method further comprises:

[0163] performing, in sequence, the tracking-based image matting processing on images in the video that have a timestamp later than a timestamp of the first image and that are located in a target video segment, until a preset tracking stop condition is reached, the target video segment being a video segment in which the first image is located.

[0164] In some possible embodiments, the tracking stop condition is that a new image matting instruction is received or that all frame images in the target video segment that have a timestamp later than the timestamp of the first image have been subjected to the image matting processing.

[0165] In some possible embodiments, in a case where the new image matting instruction is received, the current tracking-based image matting processing is stopped, an image pointed to by the new image matting instruction is taken as a new first image, the new first image is subjected to the image matting processing, and the tracking-based image matting processing is performed based on the new first image.

[0166] In some possible embodiments, the performing, in sequence, the tracking-based image matting processing on images in the video that have a timestamp later than a timestamp of the first image and that are located in a target video segment comprises:

[0167] Tracking the first mask region in the first image, a second mask region in a second image is obtained, the second image is a frame image adjacent to the first image and having a timestamp later than that of the first image;

[0168] Performing cutout processing on the second mask region in the second image.

[0169] In some possible embodiments, the displaying the first image comprises: in a case where a zoom gesture for the first image is acquired, performing corresponding zoom display on the first image based on the zoom gesture.

[0170] In some possible embodiments, the acquiring the second action region corresponding to the free painting instruction comprises:

[0171] Acquiring a trigger position of the free painting instruction in the first image;

[0172] Determining a target region based on the trigger position;

[0173] Performing zoom display on the target region;

[0174] In a case where a painting track in the target region is acquired, the second action region is determined.

[0175] In an example embodiment, a computer program product is also provided, the computer program product comprising a computer program stored in a readable storage medium, at least one processor of a computer device reading and executing the computer program from the readable storage medium, so that the computer device performs the cutout method of the embodiments of the present disclosure.

[0176] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0177] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure that are deemed to fall within the general principles of the disclosure and include commonly known or customary practice in the art. The specification and examples are to be considered exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

[0178] It should be understood that the present disclosure is not limited to the precise structures herein described and illustrated in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A keying method, characterized in that, include: In response to a keying instruction, a first image is displayed, the keying instruction including a free smearing instruction; A first mask region corresponding to a first object in the first image is determined. The first object can be any object in the first image. The first mask region includes the region determined by the user's click or smear trajectory. Determining the first mask region corresponding to the first object includes: Obtain the trigger position of the free smearing command in the first image; determine a target area centered on the trigger position based on the trigger position; magnify and display the target area; if the smearing trajectory in the target area is obtained, determine a second action area; update the first mask area based on the second action area; The first mask region in the first image is subjected to keying processing, where the first image is a frame image in a video; The first mask region is tracked to obtain the second mask region in the second image. The second mask region in the second image is then chroma keyed. The second image is a frame image in the video whose timestamp is after the timestamp of the first image and is adjacent to the first image. The second mask region is tracked to obtain the third mask region in the third image. The third image is then processed by keying based on the third mask region. The third image is a frame image in the video whose timestamp is after the timestamp of the second image and is adjacent to the second image.

2. The keying method according to claim 1, characterized in that, The keying instructions include foreground subject segmentation instructions or interactive segmentation instructions. The foreground subject segmentation instruction is used to segment the foreground subject region in the first image; The interactive segmentation instruction is used to segment the object pointed to by the interactive segmentation instruction in the first image.

3. The keying method according to claim 2, characterized in that, The keying instruction includes the In the case of a foreground subject segmentation instruction, determining the first mask region corresponding to the first object includes: in response to the foreground subject segmentation instruction, displaying the foreground subject segmentation result of the first image; and in response to a selection instruction for the foreground subject segmentation result, determining the first mask region corresponding to the first object. When the keying instruction includes the interactive segmentation instruction, determining the first mask region corresponding to the first object includes: in response to the interactive segmentation instruction, taking the object whose area in the first image intersects with the first action area of ​​the interactive segmentation instruction as the first object, and determining the first mask region corresponding to the first object.

4. The keying method according to claim 1, characterized in that, After performing image matting on the first mask region in the first image, the method further includes: Images in the video whose timestamps are after the timestamp of the first image and are located in the target video segment are sequentially subjected to tracking-based keying processing until a preset tracking stop condition is reached. The target video segment is the video segment where the first image is located.

5. The keying method according to claim 4, characterized in that, The tracking stops when a new keying instruction is received or when keying is completed for all other frames in the target video segment after the timestamp of the first image.

6. The keying method according to claim 5, characterized in that, The method further includes: upon receiving a new keying instruction, stopping the current tracking-based keying process, taking the image pointed to by the new keying instruction as a new first image, performing keying processing on the new first image, and performing tracking-based keying processing based on the new first image.

7. The keying method according to claim 1, characterized in that, The display of the first image includes: upon receiving a zoom gesture for the first image, displaying the first image in a corresponding zoomed or scaled manner based on the zoom gesture.

8. A keying device, characterized in that, include: The image display module is configured to display a first image in response to a keying instruction, the keying instruction including a free smearing instruction; The mask determination module is configured to determine a first mask region corresponding to a first object in the first image, wherein the first object is any object in the first image, and the first mask region includes a region determined by a user click or smear trajectory. Determining the first mask region corresponding to the first object includes: obtaining the trigger position of the free smearing command in the first image; determining a target region centered on the trigger position based on the trigger position; magnifying and displaying the target region; determining a second active region when a smearing trajectory is obtained in the target region; and updating the first mask region based on the second active region. The keying module is configured to perform keying processing on a first mask region in a first image, where the first image is a frame image in a video; track the first mask region to obtain a second mask region in a second image, perform keying processing on the second mask region in the second image, where the second image is a frame image in the video whose timestamp is after the timestamp of the first image and adjacent to the first image; track the second mask region to obtain a third mask region in a third image, perform keying processing on the third image based on the third mask region, where the third image is a frame image in the video whose timestamp is after the timestamp of the second image and adjacent to the second image.

9. The keying device according to claim 8, characterized in that, The keying instructions include foreground subject segmentation instructions or interactive segmentation instructions. The foreground subject segmentation instruction is used to segment the foreground subject region in the first image; The interactive segmentation instruction is used to segment the object pointed to by the interactive segmentation instruction in the first image.

10. The keying device according to claim 9, characterized in that, When the keying instruction includes the foreground subject segmentation instruction, the mask determination module is configured to perform a foreground subject segmentation result of the first image in response to the foreground subject segmentation instruction; In addition, in response to the selection instruction for the foreground subject segmentation result, the first mask region corresponding to the first object is determined; When the keying instruction includes the interactive segmentation instruction, the mask determination module is configured to perform an action in response to the interactive segmentation instruction, taking the object whose area in the first image intersects with the first action area of ​​the interactive segmentation instruction as the first object, and determining the first mask area corresponding to the first object.

11. The keying device according to claim 8, characterized in that, The keying module is configured to execute: For the video whose timestamp is after the timestamp of the first image and is located in the target video... The images of the segments are sequentially subjected to tracking-based keying processing until a preset tracking stop condition is reached, and the target video segment is the video segment containing the first image.

12. The keying device according to claim 11, characterized in that, The tracking stops when a new keying instruction is received or when keying is completed for all other frames in the target video segment after the timestamp of the first image.

13. The keying device according to claim 12, characterized in that, The keying module is configured to, upon receiving a new keying instruction, stop the current tracking-based keying process, take the image pointed to by the new keying instruction as a new first image, perform keying processing on the new first image, and perform tracking-based keying processing based on the new first image.

14. The keying device according to claim 8, characterized in that, The image display module is configured to perform corresponding scaling display on the first image based on the scaling gesture when a scaling gesture for the first image is obtained.

15. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the keying method as described in any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the keying method as described in any one of claims 1 to 7.

17. A computer program product, characterized in that, The computer program product includes a computer program stored in a readable storage medium, wherein at least one processor of a computer device reads from the readable storage medium and executes the computer program, causing the computer device to perform the keying method as described in any one of claims 1 to 7.

Images