Method, device, storage medium and electronic device for displaying matte effect

Abstract

The present disclosure relates to a method, device, storage medium and electronic device for displaying matte effects. The method includes displaying the matte effect of a first image when a matte effect display instruction for a target video is obtained, wherein the first image is the first image to be matted in the target video in response to the target matte instruction; and displaying the matte effect of a third image when a second image does not exist; wherein the second image is an image in the target video whose timestamp is after the first image and is matted in response to the target matte instruction, and the third image is an image in the target video whose timestamp is after the first image and is not adjacent to the first image. The embodiment of the present disclosure can display the matte effect at intervals of some image frames, and this display can quickly display the entire video matte compared to displaying the matte effect frame by frame.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a matte effect display method, device, storage medium and electronic device. Background Art

[0002] With the development of image processing technology, the matting process of videos has been applied more and more. In the related technology, a matting instruction can be triggered for a certain frame in the video, and then the corresponding matting is performed on other frames in the video after the frame. After the matting of each frame in the video is completed, the matting effect of the video is displayed. When the matting process is performed, the user can be reminded to wait by means of a screen prompt. However, the display method of the video matting effect requires the user to wait for a long time, which reduces the user experience and affects the user stickiness. Summary of the invention

[0003] In order to solve at least one of the above technical problems, the present disclosure provides a method, device, storage medium and electronic device for displaying matte effects. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a method for displaying a matte effect is provided, comprising:

[0005] In the case of obtaining a matte effect display instruction for a target video, displaying a matte effect of a first image, where the first image is the first matte image in the target video in response to the target matte instruction;

[0006] In the absence of the second image, displaying the matting effect of the third image;

[0007] The second image is an image in the target video whose timestamp is after the first image and which is matted in response to the target matting instruction.

[0008] The third image is an image in the target video whose time stamp is located after the first image and is not adjacent to the first image.

[0009] In some possible embodiments, after displaying the matting effect of the third image, the method further includes:

[0010] The matting effect of the fourth image is displayed, where the fourth image is an image whose time stamp in the target video is located after the third image and is not adjacent to the third image.

[0011] In some possible embodiments, the method further includes:

[0012] In the case that there is at least one second image, the matting effects of the second images are displayed in sequence according to the order of the second images in the target video.

[0013] In some possible embodiments, after sequentially displaying the matting effect of the second image, the method further includes:

[0014] In the case that there are still images in the target video that have not been matted according to the target matting instruction, displaying the matting effect of the fifth image;

[0015] The fifth image is an image in the target video whose timestamp is after the target second image and is not adjacent to the target second image;

[0016] The target second image is the second image with the largest timestamp.

[0017] In some possible embodiments, after displaying the matting effect of the fifth image, the method further includes:

[0018] The matting effect of the sixth image is displayed, where the sixth image is an image whose time stamp in the target video is located after the fifth image and is not adjacent to the fifth image.

[0019] In some possible embodiments, the method further includes:

[0020] When displaying the matte effect of the matte-ed image, if there is an image in the target video that has not been matte-ed according to the target matte-ing instruction, perform video matte-ing processing in parallel;

[0021] The video matting process includes:

[0022] Determining at least one image to be matted in the target video according to an image in the target video that has been matted in response to the target matting instruction;

[0023] The at least one image to be matted is matted according to the target matting instruction to obtain a matting effect of the image to be matted.

[0024] In some possible embodiments, performing matting on the at least one image to be matted according to the target matting instruction to obtain a matting effect of the image to be matted includes:

[0025] For each image to be matted, determine a reference image, the reference image being an image in the target video that has a timestamp closest to the image to be matted and is located before the timestamp of the image to be matted and has been matted in response to the target matting instruction;

[0026] The matting effect corresponding to the reference image is tracked to obtain the matting effect of the image to be matted.

[0027] In some possible embodiments, the method further includes:

[0028] When no new matting instruction is received within a preset time and there are still images in the target video that have not been matted according to the target matting instruction, the video matting process is performed.

[0029] In some possible embodiments, the method further includes:

[0030] After performing matting on any image in the target video according to the target matting instruction, the matting effect of any image is stored.

[0031] In some possible embodiments, the method further includes:

[0032] In the case that there are still images in the target video that have not been matted according to the target matting instruction, prompt information is displayed, and the prompt information is used to prompt that the target video has not yet completed all the matting and that the matting effect display of the target video is stuck.

[0033] According to a second aspect of an embodiment of the present disclosure, a matte effect display device is provided, comprising:

[0034] A first display module is configured to display a matte effect of a first image when a matte effect display instruction for a target video is obtained, wherein the first image is a first matte image in the target video in response to the target matte instruction;

[0035] A second display module is configured to perform a matte effect of displaying a third image in the absence of the second image;

[0036] The second image is an image in the target video whose timestamp is after the first image and which is matted in response to the target matting instruction.

[0037] The third image is an image in the target video whose time stamp is located after the first image and is not adjacent to the first image.

[0038] In some possible embodiments, the second display module is configured to execute:

[0039] The matting effect of the fourth image is displayed, where the fourth image is an image whose time stamp in the target video is located after the third image and is not adjacent to the third image.

[0040] In some possible embodiments, the second display module is configured to execute:

[0041] In the case that there is at least one second image, the matting effects of the second images are displayed in sequence according to the order of the second images in the target video.

[0042] In some possible embodiments, the second display module is configured to execute:

[0043] In the case that there are still images in the target video that have not been matted according to the target matting instruction, displaying the matting effect of the fifth image;

[0044] The fifth image is an image in the target video whose timestamp is after the target second image and is not adjacent to the target second image;

[0045] The target second image is the second image with the largest timestamp.

[0046] In some possible embodiments, the second display module is configured to execute:

[0047] The matting effect of the sixth image is displayed, where the sixth image is an image whose time stamp in the target video is located after the fifth image and is not adjacent to the fifth image.

[0048] In some possible embodiments, the device further includes a matting module configured to execute:

[0049] When displaying the matte effect of the matte-ed image, if there is an image in the target video that has not been matte-ed according to the target matte-ing instruction, perform video matte-ing processing in parallel;

[0050] The video matting process includes:

[0051] Determining at least one image to be matted in the target video according to an image in the target video that has been matted in response to the target matting instruction;

[0052] The at least one image to be matted is matted according to the target matting instruction to obtain a matting effect of the image to be matted.

[0053] In some possible embodiments, the matting module is configured to determine a reference image for each image to be matted, the reference image being an image in the target video that has a timestamp closest to the image to be matted and is located before the timestamp of the image to be matted and has been matted in response to the target matting instruction;

[0054] The matting effect corresponding to the reference image is tracked to obtain the matting effect of the image to be matted.

[0055] In some possible embodiments, the matting module is configured to perform

[0056] When no new matting instruction is received within a preset time and there are still images in the target video that have not been matted according to the target matting instruction, the video matting process is performed.

[0057] In some possible embodiments, the matting module is configured to store the matting effect of any image in the target video after matting the target matting instruction is performed on the any image in the target video.

[0058] In some possible embodiments, the second display module is configured to execute:

[0059] In the case that there are still images in the target video that have not been matted according to the target matting instruction, prompt information is displayed, and the prompt information is used to prompt that the target video has not yet completed all the matting and that the matting effect display of the target video is stuck.

[0060] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement a method as described in any one of the first aspects above.

[0061] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any one of the methods in the first aspect of the embodiment of the present disclosure.

[0062] According to the fifth aspect of an embodiment of the present disclosure, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a readable storage medium, at least one processor of a computer device reading and executing the computer program from the readable storage medium, so that the computer device performs any one of the methods in the first aspect of the embodiment of the present disclosure.

[0063] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0064] The matte effect display method provided by the embodiment of the present disclosure can display the matte effect of the first image when the matte effect display instruction for the target video is obtained, wherein the first image is the first image to be matted in the target video in response to the target matte instruction; and when the second image does not exist, the matte effect of the third image is displayed; wherein the second image is an image in the target video whose timestamp is after the first image and is matted in response to the target matte instruction, and the third image is an image in the target video whose timestamp is after the first image and is not adjacent to the first image. The embodiment of the present disclosure can display the matte effect at intervals of some image frames. Compared with displaying the matte effect frame by frame, this display can quickly display the whole picture of the video matte, and the number of images to be matted is small, so the display speed is faster, the jamming is lighter, and there is no need to wait for matte for a long time. The matte effect can be displayed immediately in response to the matte effect display instruction, which significantly reduces the user waiting time, improves the matte effect display speed, and improves user stickiness.

[0065] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0067] Figure 1 is a schematic diagram of an implementation environment of a matte effect display method according to an exemplary embodiment;

[0068] Figure 2 is a schematic flow chart of a method for displaying a matte effect according to an exemplary embodiment;

[0069] Figure 3 is a tracking schematic diagram shown according to an exemplary embodiment;

[0070] Figure 4 is a schematic diagram of a matting interface according to an exemplary embodiment;

[0071] Figure 5 is a block diagram of a matte effect display device according to an exemplary embodiment;

[0072] Figure 6 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0073] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0074] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar first objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0075] All user information, images and other data involved in this disclosure are authorized by the user or fully authorized by all parties.

[0076] Matting usually uses artificial intelligence algorithms to perform object matting, which consumes relatively more computing resources and storage resources. Especially for video matting, the consumption of such resources may cause the display of the video matting effect to be stuck or to wait for a long time. Especially in the case of low-end equipment, this disadvantage will be more prominent. In order to provide the ability to play the matting effect of the video in real time, reduce the user's waiting time, and improve the user experience, the embodiments of the present disclosure provide a matting effect display method, device, storage medium and electronic device.

[0077] Figure 1 FIG. 1 is a schematic diagram of an implementation environment of a method for displaying a matte effect according to an exemplary embodiment. Figure 1 , the implementation environment specifically includes: a terminal 101 and a server 102.

[0078] The terminal 101 may be at least one of a smart phone, a smart watch, a desktop computer, a laptop computer and a portable computer. An application that provides a video matting service and a video matting effect playback service may be installed and run on the terminal 101. A user may log in to the application through the terminal 101 to obtain the video matting service and the video matting effect playback service provided by the application. The terminal 101 may generally refer to one of a plurality of terminals. This embodiment only takes the terminal 101 as an example. Those skilled in the art may know that the number of the above-mentioned terminals may be more or less. For example, the above-mentioned terminals may be only a few, or the above-mentioned terminals may be dozens or hundreds, or a greater number. The embodiment of the present disclosure does not limit the number of terminals and device types.

[0079] The server 102 may be at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server 102 may be connected to the terminal 101 and other terminals via a wireless network or a wired network. The server 102 may receive relevant requests sent by the terminal 101, and cooperate with the terminal 101 to provide video matting services and video matting effect playback services. Optionally, the number of the above servers may be more or less, and the embodiments of the present disclosure are not limited thereto. Of course, the server 102 may also include other functional servers to provide more comprehensive and diversified services. Figure 2 is a flow chart of a method for displaying a matte effect according to an exemplary embodiment. Figure 2 As shown, the above method at least includes the following steps S10-S30.

[0080] In step S10, when a matte effect display instruction for a target video is obtained, a matte effect of a first image is displayed, where the first image is the first matte image in the target video in response to the target matte instruction.

[0081] The disclosed embodiment does not limit the scene for displaying the matte effect of the target video, and the matte effect of the target video can be displayed in scenes such as video clipping and video editing. The disclosed embodiment does not limit the source of the target video, for example, it can come from the local gallery of the terminal 101, from the video captured by the camera of the terminal 101, or from the network. In the disclosed embodiment, the target video can be a complete video or a video clip in a video.

[0082] In the disclosed embodiment, a certain frame of the target video can be matted, and the images after the frame in the target video will also be matted based on the same principle, and the image corresponding to the certain frame is the first image mentioned above. For example, in the target video "Kitten Drinking Water", the kitten in the image of the kitten lowering its head to the small bowl is matted, and the subsequent images of the kitten drinking water from the small bowl and the kitten raising its head after drinking water will also be matted.

[0083] After the first image is matted, when a matting effect display instruction for the target video is obtained, the matting result can be displayed, that is, the matting effect of the first image can be displayed.

[0084] In step S20, in the absence of a second image, the matting effect of a third image is displayed; wherein the second image is an image in the target video whose timestamp is after the first image and which is matted in response to the target matting instruction, and the third image is an image in the target video whose timestamp is after the first image and which is not adjacent to the first image.

[0085] In the disclosed embodiment, the second image refers to an image in the target video whose timestamp is after the first image and which is matted in response to the target matting instruction. For example, the target video includes 10 frames of images in total, and the target matting instruction mattes the beauty in the second frame of the image. That is to say, among the above 10 frames of images, the image corresponding to frame 2 is the first image in step S10, and the beauty in the images of frames 3-10 should also be matted in response to the target matting instruction. If there is an image of a beauty matted in frame 3-10, the image of the beauty matted is the second image. If there is no image of a beauty matted in frame 3-10, the second image does not exist.

[0086] In fact, if the matte effect for the target matte instruction is displayed for the first time, the second image must not exist, because the matte processing for the target matte instruction has not been performed on frame 3-10 before. In this case, the matte effect of the third image can be displayed. Still using the above example, the third image is an image in the target video whose timestamp is after the first image and is not adjacent to the first image, that is, the third image can be any image in frame 4-10. After displaying the matte effect of the first image, the matte effect of the third image is displayed. The first image and the third image are not adjacent. Therefore, this matte display is a frame skipping display.

[0087] Of course, after displaying the matte effect of the third image, the matte effect of the fourth image may continue to be displayed, and the fourth image is an image whose timestamp in the target video is after the third image and is not adjacent to the third image. Assuming that the third image is frame4, the fourth image may be any image in frame5-10. Similarly, for 10 frames of images, only frame2, frame4, frame8 and frame10 may be displayed in step S20.

[0088] In other words, the matte effect can be displayed at intervals of some image frames. Compared with displaying the matte effect frame by frame, this type of display can quickly display the entire video matte, and the number of images to be matte is smaller, so the display speed is faster, the lag is less, and there is no need to wait for a long time for matte. The matte effect can be displayed instantly in response to the matte effect display instruction, which significantly reduces the user's waiting time, increases the display speed of the matte effect, and improves user stickiness.

[0089] In step S30, when there is at least one of the second images, the matting effects of the second images are displayed in sequence according to the order of the second images in the target video.

[0090] If after acquiring the target matting instruction, the matting effect display instruction is triggered, then step S20 has been executed at least once, that is, at least some images have been matted in response to the above target matting instruction. These images can be referred to as second images in the embodiment of the present disclosure, that is, the above second image is an image in the above target video whose timestamp is after the above first image and is matted in response to the above target matting instruction. Taking frame4, frame6 and frame8 displayed in step S20 as an example, frame4, frame6 and frame8 all belong to the second image. In this case, if the matting effect display instruction is triggered again, steps S10 and S30 can be executed to directly display the matting effects of frame2, frame4, frame6 and frame8 without matting frame4, frame6 and frame8 again, which significantly improves the display speed of the matting effect.

[0091] In one embodiment, after the matte effect of the second image is displayed in sequence, the method further includes: if there are still images in the target video that have not been matted according to the target matte instruction, displaying the matte effect of the fifth image; wherein the fifth image is an image in the target video whose timestamp is after the target second image and is not adjacent to the target second image; the target second image is the second image with the largest timestamp. Taking the above text as an example, after the matte effects of frame2, frame4, frame6 and frame8 are displayed in sequence, there are still images that have not been matted according to the target matte instruction, such as frame9 and frame10. In this case, the same frame skipping idea as step S20 can be used to improve the display speed, for example, the matte effect of frame10 can be displayed.

[0092] Of course, if there are still images after the fifth image that have not been matted according to the target matting instruction, the frame skipping display can continue. Specifically, the matting effect of the sixth image can be displayed. The sixth image is an image whose time stamp in the target video is after the fifth image and is not adjacent to the fifth image. Similarly, the idea of ​​the frame skipping display has been described in the previous text and will not be repeated here.

[0093] That is to say, when there are frames that have been matted, the matte effect is directly displayed; if it is necessary to display frames that have not been matted, in order to increase the display speed of the matte effect, the embodiment of the present disclosure performs frame skipping display, thereby making the display of the matte effect of the target video smoother, improving the smoothness of the picture, reducing the screen freeze, and increasing the display speed of the matte effect. The user no longer needs to wait, thereby significantly improving the user experience.

[0094] In the disclosed embodiments, concurrent resources or idle resources can also be fully utilized for matting processing. In one embodiment, when the matting effect of an image that has been matted is displayed, if there are still images in the target video that have not been matted according to the above target matting instruction, the video matting processing can be performed in parallel. By performing the video matting processing in parallel when the matting effect is displayed, if there are still images in the target video that have not been matted according to the above target matting instruction, better computing resources and time gaps can be used to accelerate the matting operation, complete the matting as soon as possible, and reduce the playback jam of the matting effect.

[0095] Specifically, the video matting processing includes: determining at least one image to be matted in the target video based on the image that has been matted in the target video in response to the target matting instruction; matting the at least one image to be matted according to the target matting instruction to obtain the matting effect of the image to be matted.

[0096] The disclosed embodiments do not limit the method for determining the image to be matted. For example, the target video has a total of 10 frames. When playing, it starts from the 1st frame. Assuming that the 1st frame has been matted and has an effect, it is played and the user sees the matting effect. The next 2-4 frames may not have enough computing resources to perform matting calculations in time, so the matting is directly skipped to the 5th frame and the effect of the 5th frame is displayed. Therefore, the effect seen is the skipped frame matting effect of 1, 5, and 9.

[0097] The images that have been matted in the video matting process are the 1st frame, the 5th frame and the 9th frame, and the other frames can be used as at least one of the above-mentioned images to be matted. Alternatively, the image to be matted can be selected by skipping frames from the above-mentioned other frames. For example, the above-mentioned other frames are the 2nd frame, the 3rd frame, the 4th frame, the 6th frame, the 7th frame, the 8th frame and the 10th frame. Then the 3rd frame, the 6th frame and the 8th frame can be selected as the images to be matted to perform the above-mentioned video matting process, and the 2nd frame, the 4th frame, the 7th frame and the 10th frame are processed as the images to be matted when the video matting process is performed again. The advantage of this is that the distribution of the matted images in the target video is as uniform as possible. If the matting effect needs to be displayed again in this process, the display of the matting effect can better reflect the overall matting effect of the video. For example, if after selecting the 3rd, 6th and 8th frames as the images to be matted and performing the above-mentioned video matting processing, a matting effect display instruction is received again, then in step S30, the 3rd, 5th, 6th, 8th and 9th frames can be displayed in sequence. These frames are distributed as evenly as possible in the target video, and can represent the matting effect for the video more holistically.

[0098] In a specific embodiment, the above-mentioned matting of at least one of the images to be matted is performed according to the above-mentioned target matting instruction to obtain the matting effect of the above-mentioned image to be matted, including: for each image to be matted, determining a reference image, the above-mentioned reference image is an image in the above-mentioned target video that is closest to the timestamp of the image to be matted and is located before the timestamp of the above-mentioned image to be matted, and has been matted in response to the above-mentioned target matting instruction; tracking the matting effect corresponding to the above-mentioned reference image to obtain the matting effect of the above-mentioned image to be matted. Taking the example that the images of the 1st frame, the 5th frame and the 9th frame have been matted, and the image to be matted is the image of the 3rd frame, the reference image is the image of the 1st frame, and the matting effect of the 3rd frame can be obtained by tracking the matting effect of the 1st frame. The disclosed embodiment improves the matting speed by tracking the matting effect.

[0099] Please refer to Figure 3 , which shows a tracking schematic diagram. Figure 3 The upper left image is the reference image. Figure 3 The upper right picture is the mask image corresponding to the reference image. The mask image represents the matting of the person riding a bicycle in the reference image, that is, the mask image reflects the matting effect. Figure 3 The lower left picture is the image to be keyed. Figure 3 The lower right picture is the mask image corresponding to the image to be matted. The mask image represents the Figure 3 The tracking result of the mask image in the upper right image. Obviously, based on the tracking result of the mask image in the upper right image, the image of a person riding a bicycle can be keyed out. By tracking keying, the keying speed can be significantly improved.

[0100] In another embodiment, when no new matting instruction is received after a preset time, and there are still images in the target video that have not been matted according to the target matting instruction, the above-mentioned video matting process is performed. In the case that no new matting instruction is received within the preset time, it can be considered that the computing resources are relatively idle, and background preprocessing can be performed in the idle state, that is, the above-mentioned video matting process is performed, and all images that have not been matted in the video are matted, so that the next time the matting effect of the video needs to be displayed, it can be displayed quickly and directly, reducing the user's waiting time. In other words, when the background is idle, all frames that have not been matted can be matted by background preprocessing, so that the matting effect of each frame in the target video can be fully displayed during the next playback without frame skipping, making the picture smoother.

[0101] Normally, the matting operation is likely to work in conjunction with other image processing methods. For example, after the user triggers the target matting command, he or she may further perform other operations such as adjusting the picture brightness and beautifying the image. These operations do not significantly consume computing resources. It can be considered that the computing resources are relatively idle in this case. By utilizing this idle situation, video matting processing can be performed and the matting results can be cached, thereby making the subsequent playback of the matting effect smoother.

[0102] Of course, any matting effect obtained by matting an image in a target video in the foregoing text can be stored in time for future use. That is to say, after matting any image in the above target video according to the above target matting instruction, the matting effect of any of the above images is stored. The disclosed embodiments do not limit the storage method. For example, the storage can be based on a cache pool, or based on a page replacement algorithm, for example, using an LRU algorithm, where LRU is the abbreviation of Least Recently Used. By storing the matting effect, the matting effect can be directly displayed when it is needed in the future, without the need to re-matte, thereby improving the display speed and efficiency of the matting effect.

[0103] In the disclosed embodiment, when there are images in the target video that have not been matted according to the target matting instruction, a prompt message is displayed, and the prompt message is used to prompt that the target video has not completed all matting and there is a freeze in the matting effect display of the target video. For example, when the matting effect is displayed, the part that has completed matting is played frame by frame, and the part that has not completed matting is played by skipping frames, and a prompt message can be displayed: "There may be slight freezes during the matting process." The user is reminded by displaying prompt messages to optimize the user experience.

[0104] The objects of matting are not limited in the embodiments of the present disclosure. In the related art, only relatively limited objects such as portraits, head portraits, and the sky in the image can be automatically identified, and only these relatively limited objects can be matted. Different from the related art, the embodiments of the present disclosure can support the automatic identification of any object, so as to determine the outline of any object and perform matting based on the outline. The functions of automatically matting portraits, heads, backgrounds, and skies in the related art are based on a single algorithm model. For example, the matting portrait function uses a trained portrait recognition model to matte the portrait based on the recognition result of the portrait recognition model. Different from the related art, the embodiments of the present disclosure use an algorithm model for automatically identifying multiple objects. Various types of objects are used when the model is trained, so it has the ability to automatically identify various objects and extract the edges of the above-mentioned various objects. The embodiments of the present disclosure can arbitrarily select objects for matting, and are no longer limited to matting certain types of objects, which significantly broadens the scope of use of matting.

[0105] The disclosed embodiments do not limit the triggering method of the target matting instruction. For example, the target matting instruction includes a foreground subject segmentation instruction or an interactive segmentation instruction, wherein the foreground subject segmentation instruction is used to segment the foreground subject area in the first image; the interactive segmentation instruction is used to segment the object pointed to by the interactive segmentation instruction in the first image. The foreground subject segmentation instruction and the interactive segmentation instruction can be triggered by the user triggering the corresponding control. That is to say, automatic matting can be achieved based on these two instructions. By automatically matting everything, the application scope and efficiency of matting are improved, and user stickiness is significantly increased.

[0106] The disclosed embodiments can also provide refined matting capabilities, giving users greater matting autonomy and improving user stickiness. In one embodiment, the foreground subject segmentation instruction or the interactive segmentation instruction generates a first mask area, the first mask area represents the target action area of ​​the matting, and the target matting instruction also includes a free smear instruction, which can obtain the action area corresponding to the free smear instruction; based on the action area corresponding to the free smear instruction, the first mask area is updated. In the disclosed embodiments, the foreground subject segmentation instruction is automatic matting, while the matting based on the interactive segmentation instruction and the matting based on the free smear instruction are manual custom matting. The combination of automatic matting and manual custom matting can significantly improve the accuracy of matting.

[0107] For example, if you want to extract a head portrait from the first image, the head portrait can be roughly segmented through interactive segmentation and / or foreground subject segmentation, but there may still be deficiencies in details, for example, the hair part of the head portrait may not be segmented and processed, and the level of detail is not enough. In this case, you can trigger the free smearing command to smear the hair part, determine the action area according to the smearing trajectory, trigger the background segmentation model to recognize the head portrait again based on the above action area and the existing first mask area, and obtain a more detailed head portrait recognition result. According to the head portrait recognition result, the above first mask area is updated, and the first image is subjected to the matting process based on the first mask area, so that the matting effect of the first image can be displayed accurately.

[0108] Please refer to Figure 4 , which shows a schematic diagram of the matte interface. In the matte interface, you can select different matte modes through automatic and manual, and these matte modes can trigger the above target matte instructions. The default selection is "manual", that is, Figure 4 The brush control is used to paint in the first image. The brush painted area is the keying reserved area, that is, the semi-transparent area in the preview area in the editing state. In the editing state, select the brush and press your finger in the preview area above to draw a line. Figure 4 There is also an "eraser" in the tool. The area smeared by the eraser is the area to be removed from the keying, that is, the unmarked area in the preview area in the editing state. In the editing state, select the eraser to remove the marked area in the preview area. Figure 4 There is also a "preview" control. When this control is turned off, the non-preview state is displayed. The non-preview state can display the original image of the first image and mark the reserved area, where the semi-transparent color block represents the reserved area. When this control is turned on, the preview state is displayed. The preview state only displays the area after the matte effect, and the semi-transparent color block on the screen is hidden. When entering the custom matte, the default is the preview off state. When exiting the custom matte, you can switch back and forth between editing and preview. Figure 4 Clicking the triangle button in the image will generate a keying effect display instruction, which instructs the user to play the keying effect of the target video.

[0109] Please refer to Figure 5 , which shows a matte effect display device, the device comprising:

[0110] The first display module 10 is configured to execute, in the case of obtaining a matte effect display instruction for a target video, displaying a matte effect of a first image, wherein the first image is the first matte image in the target video in response to the target matte instruction;

[0111] The second display module 20 is configured to perform a matte effect of displaying a third image in the absence of the second image;

[0112] The second image is an image in the target video whose timestamp is after the first image and which is matted in response to the target matting instruction.

[0113] The third image is an image in the target video whose timestamp is located after the first image and is not adjacent to the first image.

[0114] In some possible embodiments, the second display module is configured to execute:

[0115] The matting effect of the fourth image is displayed, where the fourth image is an image whose time stamp in the target video is located after the third image and is not adjacent to the third image.

[0116] In some possible embodiments, the second display module is configured to execute:

[0117] In the case where there is at least one of the second images, the matting effects of the second images are displayed in sequence according to the order of the second images in the target video.

[0118] In some possible embodiments, the second display module is configured to execute:

[0119] In the case that there are still images in the target video that have not been matted according to the target matting instruction, displaying the matting effect of the fifth image;

[0120] The fifth image is an image in the target video whose timestamp is after the second target image and is not adjacent to the second target image;

[0121] The target second image is the second image with the largest timestamp.

[0122] In some possible embodiments, the second display module is configured to execute:

[0123] The matting effect of the sixth image is displayed, where the sixth image is an image whose timestamp in the target video is after the fifth image and is not adjacent to the fifth image.

[0124] In some possible embodiments, the above apparatus further includes a matte module configured to execute:

[0125] When displaying the matte effect of the matte-ed image, if there is an image in the target video that has not been matted according to the target matte-ed instruction, the video matte-ed processing is performed in parallel;

[0126] The above-mentioned video matting processing includes:

[0127] Determining at least one image to be matted in the target video according to the image in the target video that has been matted in response to the target matting instruction;

[0128] The at least one image to be matted is matted according to the target matting instruction to obtain a matting effect of the image to be matted.

[0129] In some possible embodiments, the matting module is configured to determine a reference image for each image to be matted, where the reference image is an image in the target video that has a timestamp closest to the image to be matted and is located before the timestamp of the image to be matted, and has been matted in response to the target matting instruction;

[0130] The matting effect corresponding to the reference image is tracked to obtain the matting effect of the image to be matted.

[0131] In some possible embodiments, the above-mentioned matting module is configured to perform

[0132] When no new matting instruction is received within a preset time and there are still images in the target video that have not been matted according to the target matting instruction, the video matting process is performed.

[0133] In some possible embodiments, the matting module is configured to store the matting effect of any image in the target video after matting the target matting instruction is performed on any image in the target video.

[0134] In some possible embodiments, the second display module is configured to execute:

[0135] In the case that there are still images in the target video that have not been matted according to the target matting instruction, prompt information is displayed, and the prompt information is used to prompt that the target video has not yet completed all the matting and that the matting effect display of the target video is stuck.

[0136] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0137] Figure 6 is a block diagram of an electronic device 600 for matting according to an exemplary embodiment.

[0138] The electronic device may be a server or a terminal device, and its internal structure diagram may be as follows: Figure 6As shown. The electronic device includes a processor, a memory and a network interface connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a matte effect display method is implemented.

[0139] Those skilled in the art will understand that Figure 6 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present disclosure, and does not constitute a limitation on the electronic device to which the scheme of the present disclosure is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0140] In an exemplary embodiment, an electronic device is also provided, including: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the matte effect display method as in the embodiment of the present disclosure.

[0141] In an exemplary embodiment, a computer-readable storage medium is also provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the matte effect display method in the embodiment of the present disclosure. Specifically, the matte effect display method includes:

[0142] In the case of obtaining a matte effect display instruction for a target video, displaying a matte effect of a first image, where the first image is the first matte image in the target video in response to the target matte instruction;

[0143] In the absence of the second image, displaying the matting effect of the third image;

[0144] The second image is an image in the target video whose timestamp is after the first image and which is matted in response to the target matting instruction.

[0145] The third image is an image in the target video whose timestamp is located after the first image and is not adjacent to the first image.

[0146] In some possible embodiments, after displaying the matting effect of the third image, the method further includes:

[0147] The matting effect of the fourth image is displayed, where the fourth image is an image whose time stamp in the target video is located after the third image and is not adjacent to the third image.

[0148] In some possible embodiments, the above method further includes:

[0149] In the case where there is at least one of the second images, the matting effects of the second images are displayed in sequence according to the order of the second images in the target video.

[0150] In some possible embodiments, after sequentially displaying the matting effect of the second image, the method further includes:

[0151] In the case that there are still images in the target video that have not been matted according to the target matting instruction, displaying the matting effect of the fifth image;

[0152] The fifth image is an image in the target video whose timestamp is after the second target image and is not adjacent to the second target image;

[0153] The target second image is the second image with the largest timestamp.

[0154] In some possible embodiments, after displaying the matting effect of the fifth image, the method further includes:

[0155] The matting effect of the sixth image is displayed, where the sixth image is an image whose timestamp in the target video is after the fifth image and is not adjacent to the fifth image.

[0156] In some possible embodiments, the above method further includes:

[0157] When displaying the matte effect of the matte-ed image, if there is an image in the target video that has not been matted according to the target matte-ed instruction, the video matte-ed processing is performed in parallel;

[0158] The above-mentioned video matting processing includes:

[0159] Determining at least one image to be matted in the target video according to the image in the target video that has been matted in response to the target matting instruction;

[0160] The at least one image to be matted is matted according to the target matting instruction to obtain a matting effect of the image to be matted.

[0161] In some possible embodiments, performing matting on the at least one image to be matted according to the target matting instruction to obtain a matting effect of the image to be matted includes:

[0162] For each image to be matted, determine a reference image, the reference image being an image in the target video that has a timestamp closest to the image to be matted and is located before the timestamp of the image to be matted, and has been matted in response to the target matting instruction;

[0163] The matting effect corresponding to the reference image is tracked to obtain the matting effect of the image to be matted.

[0164] In some possible embodiments, the above method further includes:

[0165] When no new matting instruction is received within a preset time and there are still images in the target video that have not been matted according to the target matting instruction, the video matting process is performed.

[0166] In some possible embodiments, the above method further includes:

[0167] After performing matting on any image in the target video according to the target matting instruction, the matting effect of any image is stored.

[0168] In some possible embodiments, the above method further includes:

[0169] In the case that there are still images in the target video that have not been matted according to the target matting instruction, prompt information is displayed, and the prompt information is used to prompt that the target video has not yet completed all the matting and that the matting effect display of the target video is stuck.

[0170] In an exemplary embodiment, a computer program product is also provided, which includes a computer program, the computer program is stored in a readable storage medium, and at least one processor of a computer device reads and executes the computer program from the readable storage medium, so that the computer device executes the matte effect display method of the embodiment of the present disclosure.

[0171] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0172] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0173] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for displaying a matte effect, characterized in that: include: In the case of obtaining a matte effect display instruction for a target video, displaying a matte effect of a first image, where the first image is the first matte image in the target video in response to the target matte instruction; In the absence of the second image, displaying the matting effect of the third image; The second image is an image in the target video whose timestamp is after the first image and is matted in response to the target matting instruction. The third image is an image in the target video whose time stamp is located after the first image and is not adjacent to the first image.

2. The method for displaying matte effects according to claim 1, characterized in that: After displaying the matting effect of the third image, the method further includes: The matting effect of the fourth image is displayed, where the fourth image is an image whose time stamp in the target video is located after the third image and is not adjacent to the third image.

3. The method for displaying matte effects according to claim 1, characterized in that: The method further comprises: In the case that there is at least one second image, the matting effects of the second images are displayed in sequence according to the order of the second images in the target video.

4. The method for displaying matte effects according to claim 3, characterized in that: After sequentially displaying the matting effect of the second image, the method further includes: In the case that there are still images in the target video that have not been matted according to the target matting instruction, displaying the matting effect of the fifth image; The fifth image is an image in the target video whose timestamp is after the target second image and is not adjacent to the target second image; The target second image is the second image with the largest timestamp.

5. The method for displaying matte effects according to claim 4, characterized in that: After displaying the matting effect of the fifth image, the method further includes: The matting effect of the sixth image is displayed, where the sixth image is an image whose time stamp in the target video is located after the fifth image and is not adjacent to the fifth image.

6. The method for displaying matte effect according to any one of claims 1 to 5, characterized in that: The method further comprises: When displaying the matte effect of the matte-ed image, if there is an image in the target video that has not been matte-ed according to the target matte-ing instruction, perform video matte-ing processing in parallel; The video matting process includes: Determining at least one image to be matted in the target video according to an image in the target video that has been matted in response to the target matting instruction; The at least one image to be matted is matted according to the target matting instruction to obtain a matting effect of the image to be matted.

7. The method for displaying matte effects according to claim 6, characterized in that: The step of performing matting on the at least one image to be matted according to the target matting instruction to obtain a matting effect of the image to be matted includes: For each image to be matted, determine a reference image, the reference image being an image in the target video that has a timestamp closest to the image to be matted and is located before the timestamp of the image to be matted, and has been matted in response to the target matting instruction; The matting effect corresponding to the reference image is tracked to obtain the matting effect of the image to be matted.

8. The method for displaying matte effects according to claim 6, characterized in that: The method further comprises: When no new matting instruction is received within a preset time and there are still images in the target video that have not been matted according to the target matting instruction, the video matting process is performed.

9. The method for displaying matte effects according to claim 8, characterized in that: The method further comprises: After performing matting on any image in the target video according to the target matting instruction, the matting effect of any image is stored.

10. The method for displaying matte effects according to claim 1, characterized in that: The method further comprises: In the case that there are still images in the target video that have not been matted according to the target matting instruction, prompt information is displayed, and the prompt information is used to prompt that the target video has not yet completed all the matting and that the matting effect display of the target video is stuck.

11. A matte effect display device, characterized in that: include: A first display module is configured to execute, in the case of obtaining a matte effect display instruction for a target video, displaying a matte effect of a first image, wherein the first image is a first matte image in the target video in response to the target matte instruction; A second display module is configured to perform a matte effect of displaying a third image in the absence of the second image; The second image is an image in the target video whose timestamp is after the first image and is matted in response to the target matting instruction. The third image is an image in the target video whose time stamp is located after the first image and is not adjacent to the first image.

12. The matte effect display device according to claim 11, characterized in that: The second display module is configured to execute: The matting effect of the fourth image is displayed, where the fourth image is an image whose time stamp in the target video is located after the third image and is not adjacent to the third image.

13. The matte effect display device according to claim 11, characterized in that: The second display module is configured to execute: In the case that there is at least one second image, the matting effects of the second images are displayed in sequence according to the order of the second images in the target video.

14. The matte effect display device according to claim 13, characterized in that: The second display module is configured to execute: In the case that there are still images in the target video that have not been matted according to the target matting instruction, displaying the matting effect of the fifth image; The fifth image is an image in the target video whose timestamp is after the target second image and is not adjacent to the target second image; The target second image is the second image with the largest timestamp.

15. The matte effect display device according to claim 14, characterized in that: The second display module is configured to execute: The matting effect of the sixth image is displayed, where the sixth image is an image whose time stamp in the target video is located after the fifth image and is not adjacent to the fifth image.

16. The matte effect display device according to any one of claims 11 to 15, characterized in that: The device also includes a matting module configured to execute: When displaying the matte effect of the matte-ed image, if there is an image in the target video that has not been matte-ed according to the target matte-ing instruction, perform video matte-ing processing in parallel; The video matting process includes: Determining at least one image to be matted in the target video according to an image in the target video that has been matted in response to the target matting instruction; The at least one image to be matted is matted according to the target matting instruction to obtain a matting effect of the image to be matted.

17. The matte effect display device according to claim 16, characterized in that: The matting module is configured to determine a reference image for each image to be matted, wherein the reference image is an image in the target video that has a timestamp closest to the image to be matted and is located before the timestamp of the image to be matted and has been matted in response to the target matting instruction; The matting effect corresponding to the reference image is tracked to obtain the matting effect of the image to be matted.

18. The matte effect display device according to claim 16, characterized in that: The matte module is configured to perform When no new matting instruction is received within a preset time and there are still images in the target video that have not been matted according to the target matting instruction, the video matting process is performed.

19. The matte effect display device according to claim 18, characterized in that: The matting module is configured to store the matting effect of any image in the target video after matting the target matting instruction is performed on the image.

20. The matte effect display device according to claim 11, characterized in that: The second display module is configured to execute: In the case that there are still images in the target video that have not been matted according to the target matting instruction, prompt information is displayed, and the prompt information is used to prompt that the target video has not yet completed all the matting and that the matting effect display of the target video is stuck.

21. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instruction to implement the matte effect display method as described in any one of claims 1 to 10.

22. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the matte effect display method as described in any one of claims 1 to 10.

23. A computer program product, characterized in that The computer program product includes a computer program, which is stored in a readable storage medium. At least one processor of a computer device reads and executes the computer program from the readable storage medium, so that the computer device executes the matte effect display method as described in any one of claims 1 to 10.

Images