Wallpaper display method and electronic equipment

Abstract

The embodiment of the invention provides a wallpaper display method and electronic equipment, in the method, the electronic equipment can display an AOD interface, a screen locking interface and an unlocking interface in sequence, wallpaper is displayed on the AOD interface, the screen locking interface and a desktop, the wallpaper can be generated on the basis of a video, and video frames adopted by the AOD interface, the screen locking interface and the wallpaper on the desktop are arranged in sequence. When the user switches from the AOD interface to the lock screen interface and then switches to the desktop, a coherent and dynamic wallpaper playing picture is formed; moreover, the wallpaper is well fused with interfaces such as AOD, a lock screen and a desktop, and a main object in the wallpaper cannot be shielded by non-interactive elements such as time in the interfaces. Besides, the electronic equipment can update the wallpaper based on user-defined or editing operation, so that the user can conveniently set the personalized wallpaper, and the user operation is simple.

Description

Wallpaper display method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on November 2, 2023, with application number 202311458292.9 and application name “Wallpaper Display Method and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of electronic technology, and in particular to a wallpaper display method and an electronic device. Background Art

[0003] When using electronic devices like mobile phones, wallpaper is the interface users see most often and can highlight their personality. Currently, wallpapers are primarily used in user interfaces such as the always-on display (AOD), lock screen, and desktop. Users often prefer to set wallpapers across the AOD, lock screen, and desktop for aesthetics and personalization.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a wallpaper display method and electronic device, which can generate coherent and dynamic wallpapers from AOD to lock screen to desktop based on user customized operations, meet the user's demand for customized wallpapers, and provide users with a rich and colorful wallpaper creation space.

[0006] In a first aspect, an embodiment of the present application provides a wallpaper display method, which may include: an electronic device in a locked or unlocked state detects a first operation, and in response to the first operation, the electronic device displays a first interface, where the first interface includes an AOD wallpaper. When the electronic device displays the first interface, in response to a second operation, the electronic device displays a second interface, where the second interface includes the lock screen wallpaper. In response to a third operation for unlocking, the electronic device displays a third interface, where the third interface includes the desktop wallpaper.

[0007] Among them, the AOD wallpaper, lock screen wallpaper and desktop wallpaper can correspond to the first video frame, the second video frame and the third video frame in the first video respectively, the second video frame is after the first video frame, and the third video frame is after the second video frame.

[0008] In the first aspect, the first interface, the second interface, and the third interface can be an AOD interface, a lock screen interface, and a desktop interface, respectively.

[0009] In the first aspect, the first operation may be a screen-off operation, such as pressing the power button to turn off the screen of an electronic device in a locked or unlocked state. In both the locked and unlocked states, the screen of the electronic device is illuminated. In the locked state, the lock screen interface is displayed on the screen, and in the unlocked state, the desktop interface or the user interface of the application opened by the user is displayed on the screen. The electronic device may also display the AOD interface if the user does not operate the phone for a long time in the locked or unlocked state, without being limited to being triggered by a screen-off operation.

[0010] In the first aspect, the second operation may be a user operation to light up the screen, such as a touch on the screen, a power button press, etc., but is not limited thereto. Lighting up the screen in the AOD interface may also be timed, and the user may set a time to light up the screen, and when the time arrives, the electronic device lights up the screen. The embodiments of the present application do not limit the operation of lighting up the screen in the AOD interface.

[0011] By implementing the method provided in the first aspect, from AOD to lock screen and then to desktop, AOD wallpaper, lock screen wallpaper and desktop wallpaper are linked together and are no longer separated from each other.

[0012] In conjunction with the first aspect, in some embodiments, the first video frame, the second video frame, and the third video frame are each one frame, the frame interval between the first video frame and the second video frame is greater than the first frame interval, and the frame interval between the second video frame and the third video frame is greater than the second frame interval. For example, the first video frame, the second video frame, and the third video frame can be extracted from the first video at a certain frame interval (e.g., more than 1 second). In this way, the images of these multiple video frames can be different, presenting a dynamic change effect when they are played continuously.

[0013] In conjunction with the first aspect, in some embodiments, the first video frame, the second video frame, and the third video frame are all multiple frames, the starting frame of the second video frame is the ending frame of the first video frame, and the starting frame of the third video frame is the ending frame of the second video frame. In other words, the video frames used for the AOD wallpaper, lock screen wallpaper, and desktop wallpaper need to be connected end to end according to the timing of interface switching, achieving an overall coherent dynamic effect from AOD to lock screen to desktop.

[0014] In conjunction with the first aspect, in some embodiments, the wallpaper can be well integrated with interfaces such as the AOD, lock screen, and desktop, and the main object in the wallpaper (such as a person) will not be obscured by non-interactive elements such as the time in the AOD, lock screen, and desktop interfaces. Interactive elements in interfaces such as the AOD, lock screen, or desktop, such as the fingerprint unlock button in the AOD interface, the password keyboard in the lock screen interface, and application icons on the desktop, can be set above the main object in the wallpaper to facilitate user operation of these interactive elements and ensure the implementation of interface interactive functions.

[0015] In combination with the first aspect, in some embodiments, the AOD wallpaper, lock screen wallpaper and desktop wallpaper can directly use the original video frame in the first video, or use the image frame after the original video frame is processed by cropping, filtering, etc.

[0016] In conjunction with the first aspect, in some embodiments, the first video can be selected from a gallery of videos. For example, as shown in FIG3B , a user can select video clip 14 as the first video for generating a dynamic wallpaper. The first video can be selected from a video in the gallery or a Live Photo in the gallery. Live Photos are also a type of video. This can meet the user's need for custom wallpapers.

[0017] In conjunction with the first aspect, in some embodiments, the first video is selected from videos provided by a settings application.

[0018] In some embodiments, not limited to jumping from AOD to lock screen and then to desktop, the method provided in the first aspect can also be applied to other adjacent interface jump scenarios, such as AOD to desktop and then to certain application interfaces, which will especially occur when the user has not set a lock screen, and for example, from desktop to application main interface and then to application secondary interface. Of course, these application interfaces can support wallpaper display, rather than application interfaces such as game display interface and video playback interface that are not suitable for wallpaper display. For example, the chat interface of "WeChat" can set chat wallpaper. That is: the electronic device can display multiple user interfaces of different levels in turn, and the electronic device also displays wallpapers at multiple user interfaces. The wallpaper is generated based on the first video, and the video frames used in the wallpaper at multiple user interfaces at different levels are arranged in sequence; wherein, the main object in the wallpaper is displayed above the non-interactive elements in the user interface and below the interactive elements in the user interface. In this way, when the user jumps to multiple levels of interfaces, a coherent and dynamic wallpaper display effect is formed as a whole.

[0019] In conjunction with the first aspect, in some embodiments, the wallpaper display method may further include: the electronic device displaying a fourth interface, the fourth interface including a video frame sequence of the second video. Selecting a first video from the video frame sequence, wherein the first video is a partial video clip of the second video or the entire second video. The fourth interface may be, for example, the "Video Wallpaper Preview" shown in FIG. 10E , the second video may be video 16, and the video frame sequence of the second video may be video frame sequence 18.

[0020] In some embodiments, there may be a selection box on the video frame sequence, such as the selection box 19 in Figure 10E, and the video segment within the selection box is the first video. The user operation of selecting the video segment is specifically an operation of sliding the selection box along the video frame sequence.

[0021] In some embodiments, the fourth interface also includes a first preview, which is used to display the display status of the AOD wallpaper, lock screen wallpaper, and desktop wallpaper on the first interface, second interface, and third interface, respectively. The first preview can be preview 17 in Figure 10E. The first preview can be a video, and when played, the first preview presents a dynamic display process of switching from the AOD wallpaper to the lock screen wallpaper and then to the desktop wallpaper.

[0022] In combination with the first aspect, in some embodiments, the wallpaper display method may further include: the electronic device displays a fifth interface, and the main object and background of the wallpaper in the fifth interface are allowed to be selected separately by the user. The fifth interface may be, for example, the "Wallpaper Editing" interface shown in Figure 11A. In some embodiments, the wallpaper display method may further include: in response to the user operation of selecting the main object to edit the main object, the electronic device updates the image of the main object, and refreshes the AOD wallpaper, lock screen wallpaper, and desktop wallpaper based on the updated main object image. In some embodiments, the wallpaper display method may further include: in response to the user operation of selecting the background of the wallpaper to edit the background, the electronic device updates the image of the background, and refreshes the AOD wallpaper, lock screen wallpaper, and desktop wallpaper based on the updated background image.

[0023] In some embodiments, before the electronic device displays the fifth interface, the wallpaper display method may further include: the electronic device performs a cutout operation on the main object in the first video to separate the main object from the background.

[0024] In combination with the first aspect, in some embodiments, the electronic device performs a cutout on the first video for the main object, specifically including: compressing information of the target video frame in the video to obtain a first image feature; the video is a video clip; based on the first image feature, the target video frame is segmented to obtain features of the contour mask image of the object obtained in the segmentation process and the first contour mask image of the object in the target video frame; the object is the main object; based on the first image feature and the features of the obtained contour mask image, feature reconstruction is performed, the reconstructed features and the first hidden state information of the video are fused and the first hidden state information is updated to obtain a fusion result, wherein the first hidden state information represents: the fusion features of the transparency mask image of the object edge in the video frame that is cutout before the target video frame; based on the fusion result, a target transparency mask image of the object edge in the target video frame is obtained; according to the target transparency mask image and the first contour mask image, the target video frame is regionally cutout to obtain a cutout result.

[0025] In combination with the first aspect, in some embodiments, based on the fusion result, a target transparency mask image of the edge of the object in the target video frame is obtained, including: based on the fusion result, obtaining a second contour mask image of the object in the target video frame; fusing the second contour mask image and the fusion result to obtain a target fusion feature; based on the target fusion feature, obtaining a target transparency mask image of the edge of the object in the target video frame.

[0026] In conjunction with the first aspect, in some embodiments, segmenting a target video frame based on first image features to obtain features of a contour mask image of the object obtained during the segmentation process and a first contour mask image of the object in the target video frame includes: performing cascade feature reconstruction based on the first image features to obtain features of contour mask images of successively larger scales of the object, and obtaining the first contour mask image of the object in the target video frame based on the features obtained from the last processing; the first hidden state information includes: a plurality of first sub-hidden state information, each first sub-hidden state information representing a fusion feature of a transparency mask image at a certain scale;

[0027] Feature reconstruction is performed based on the first image feature and the feature of the obtained contour mask image, and the reconstructed feature and the first hidden state information of the video are fused and the first hidden state information is updated to obtain a fusion result, including: performing a preset number of transparency information fusions in the following manner, and determining the feature obtained by the last processing as the fusion result: feature reconstruction is performed based on the first target feature and the second target feature in the feature of the obtained contour mask image to obtain a second image feature with an increased scale, wherein the first target feature is the first image feature when the information fusion is performed for the first time, and the first target feature is the feature obtained by the previous information fusion when the information fusion is performed for other times, and the scale of the first target feature is the same as the scale of the second target feature; the second image feature and the first sub-state information in the first hidden state information are fused and the first sub-state information is updated to obtain a third image feature, wherein the scale of the transparency mask image corresponding to the fusion feature represented by the first sub-state information is the same as the scale of the second image feature.

[0028] In combination with the first aspect, in some embodiments, feature reconstruction is performed based on the first target feature and the second target feature in the features of the obtained contour mask image to obtain a second image feature with an enlarged scale, including: screening the representative features of the object edge from the second target feature in the features of the obtained contour mask image; and feature reconstruction is performed based on the first target feature and the representative feature to obtain a second image feature with an enlarged scale.

[0029] In combination with the first aspect, in some embodiments, cascade feature reconstruction is performed based on the first image feature to obtain features of the contour mask image of the object with successively increasing scales, including: performing contour information fusion a preset number of times in the following manner to obtain features of the contour mask image of the object with successively increasing scales: performing feature reconstruction based on the third target feature to obtain a fourth image feature with increasing scales, wherein the third target feature is the first image feature when the information fusion is performed for the first time, and the third target feature is the feature obtained by the previous feature reconstruction when the information fusion is performed for other times; fusing the fourth image feature with the second sub-state information in the second hidden state information and updating the second sub-state information to obtain features of the contour mask image of the object, wherein the second hidden state information represents: the fused features of the contour mask image of the object in the video frame segmented before the target video frame, the second hidden state information includes: multiple second sub-hidden state information, each second sub-hidden state information represents the fused features of the contour mask image of a scale, and the scale of the contour mask image corresponding to the fused features represented by the second sub-state information is the same as the scale of the fourth image feature.

[0030] In conjunction with the first aspect, in some embodiments, the first image feature includes multiple first sub-image features. Compressing information on a target video frame in a video to obtain the first image feature includes: performing cascade information compression on the target video frame in the video to obtain first sub-image features of successively decreasing scales; and when performing feature reconstruction for the first time, the third target feature is the first sub-image feature with the smallest scale. Reconstructing features based on the third target feature to obtain a fourth image feature of an increased scale includes: performing feature reconstruction based on the third target feature and the first sub-image feature with the same scale as the third target feature during additional feature reconstructions to obtain a fourth image feature of an increased scale.

[0031] In combination with the first aspect, in some embodiments, fusing the second image feature with the first sub-state information in the first hidden state information and updating the first sub-state information to obtain the third image feature includes: segmenting the second image feature to obtain the second sub-image feature and the third sub-image feature; fusing the second sub-image feature with the first sub-state information in the first hidden state information and updating the first sub-state information to obtain the fourth sub-image feature; and concatenating the fourth sub-image feature and the third sub-image feature to obtain the third image feature.

[0032] The fourth image feature and the second sub-state information in the second hidden state information are fused and the second sub-state information is updated to obtain the feature of the contour mask image of the object, including: dividing the fourth image feature to obtain the fifth sub-image feature and the sixth sub-image feature; fusing the fifth sub-image feature and the second sub-state information in the second hidden state information and updating the second sub-state information to obtain the seventh sub-image feature; and splicing the seventh sub-image feature and the sixth sub-image feature to obtain the feature of the contour mask image of the object.

[0033] In conjunction with the first aspect, in some embodiments, information compression is performed on a target video frame in a video to obtain a first feature, including: inputting the target video frame in the video into an information compression network in a pre-trained video matting model to obtain a first image feature output by the information compression network, wherein the video matting model further includes: a first image generation network, a second image generation network, a result output network, a plurality of groups of the same number of contour feature generation networks and transparency feature generation networks, each group of contour feature generation networks corresponding to a scale of a contour mask image, including a first reconstruction subnetwork and a first fusion subnetwork, and each group of transparency feature generation networks corresponding to a scale of a transparency mask image, including a second reconstruction subnetwork and a second fusion subnetwork;

[0034] Performing feature reconstruction based on the first target feature and the second target feature among the features of the obtained contour mask image to obtain a scale-enlarged second image feature, including: inputting the first target feature and the second target feature among the features of the obtained contour mask image into a target second reconstruction subnetwork in a target transparency feature generation network, obtaining the scale-enlarged second image feature output by the target second reconstruction subnetwork, wherein the scale of the transparency mask image corresponding to the target transparency feature generation network is the same as the scale of the second image feature;

[0035] fusing the second image feature with the first sub-state information in the first hidden state information and updating the first sub-state information to obtain a third image feature, including: inputting the second image feature into a target second fusion sub-network in the target transparency feature generation network, so that the target second fusion sub-network fuses the second image feature with the first sub-state information provided by itself and updates the first sub-state information, thereby obtaining a third image feature output by the target second fusion sub-network;

[0036] Based on the fusion result, a target transparency mask image of the edge of the object in the target video frame is obtained, including: inputting the fusion result into a second image generation network to obtain a target transparency mask image of the object in the target video frame output by the second image generation network;

[0037] Performing feature reconstruction based on the third target feature to obtain a scale-enlarged fourth image feature, including: inputting the third target feature into a target first reconstruction subnetwork in a target contour feature generation network, obtaining a scale-enlarged fourth image feature output by the target first reconstruction subnetwork, wherein the scale of the contour mask image corresponding to the target contour feature generation network is the same as the scale of the fourth image feature;

[0038] fusing the fourth image feature with the second sub-state information in the second hidden state information and updating the second sub-state information to obtain features of a contour mask image of the object, including: inputting the fourth image feature into a target first fusion sub-network in a target contour feature generation network, so that the target first fusion sub-network fuses the fourth image feature with the second sub-state information provided by itself and updates the second sub-state information, thereby obtaining features of the contour mask image of the object output by the target first fusion sub-network;

[0039] Obtaining a first contour mask image of the object in the target video frame based on the features obtained by the last processing, including: inputting the features obtained by the last processing into a first image generation network, and obtaining a first contour mask image of the object in the target video frame output by the first image generation network;

[0040] According to the target transparency mask image and the first contour mask image, the target video frame is subjected to regional cutout to obtain a cutout result, including: inputting the target transparency mask image and the first contour mask image into a result output network, so that the result output network performs regional cutout on the target video frame based on the obtained image, and obtains a cutout result output by the result output network.

[0041] In conjunction with the first aspect, in some embodiments, the transparency feature generation network further includes a feature screening subnetwork;

[0042] Before inputting the first target feature and the second target feature among the features of the obtained contour mask image into the target second reconstruction subnetwork in the target transparency feature generation network, the method further includes: inputting the second target feature among the features of the obtained contour mask image into the target feature screening subnetwork in the target transparency feature generation network, and obtaining a target screening feature that is representative of the edge contour of the object among the second target features output by the target feature screening subnetwork;

[0043] Inputting the first target feature and the second target feature in the features of the obtained contour mask image into the target second reconstruction subnetwork in the target transparency feature generation network, including: inputting the first target feature and the target screening feature into the target second reconstruction subnetwork in the target transparency feature generation network.

[0044] In combination with the first aspect, in some embodiments, the first fusion subnetwork is: a gated recurrent unit GRU or a long short-term memory LSTM unit; and / or, the second fusion subnetwork is: a gated recurrent unit GRU or a long short-term memory LSTM unit; and / or, the first reconstruction subnetwork is implemented based on the QARepVGG network structure, or, the first reconstruction subnetwork in the specific contour feature generation network is implemented based on the QARepVGG network structure, wherein the specific contour feature generation network is: a contour feature generation network whose scale of the corresponding contour mask image is smaller than the first preset scale; and / or, the second reconstruction subnetwork is implemented based on the QARepVGG network structure, or, the second reconstruction subnetwork in the specific transparency feature generation network is implemented based on the QARepVGG network structure, wherein the specific transparency feature generation network is: a transparency feature generation network whose scale of the corresponding transparency mask image is smaller than the second preset scale.

[0045] In conjunction with the first aspect, in some embodiments, the video cutout model is trained in the following manner:

[0046] Inputting a first sample video frame in the sample video into an initial model of the video matting model for processing, obtaining a first sample contour mask image of the object in the first sample video frame output by a first image generation network in the initial model;

[0047] Obtaining a first difference between the annotated mask image corresponding to the first sample video frame and the annotated mask image corresponding to the second sample video frame, wherein the second sample video frame is a video frame in the sample video that precedes the first sample video frame and is separated by a preset number of frames;

[0048] Obtaining a second difference between the first sample contour mask image and the second sample contour mask image, wherein the second sample contour mask image is: a mask image output by the first image generation network when the initial model processes the second sample video frame;

[0049] Obtaining a third difference between a first sample transparency mask image and a second sample transparency mask image, wherein the first sample transparency mask image is a mask image output by the second image generation network when the initial model processes the first sample video frame, and the second sample transparency mask image is a mask image output by the second image generation network when the initial model processes the second sample video frame;

[0050] Calculate training loss based on first difference, second difference and third;

[0051] Based on the training loss, the model parameters of the initial model are adjusted to obtain the video cutout model.

[0052] In combination with the first aspect, in some embodiments, the first sample contour mask image includes: a first mask sub-image that identifies the area where the object is located in the first sample video frame and a second mask sub-image that identifies the area outside the object in the first sample video frame; the second sample contour mask image includes: a third mask sub-image that identifies the area where the object is located in the second sample video frame and a fourth mask sub-image that identifies the area outside the object in the second sample video frame; obtaining a second difference between the first sample contour mask image and the second sample contour mask image includes: obtaining a difference between the first mask sub-image and the third mask sub-image, and obtaining a difference between the second mask sub-image and the fourth mask sub-image, to obtain a second difference including the obtained differences.

[0053] In combination with the first aspect, in some embodiments, information compression is performed on a target video frame in a video to obtain a first feature, including: performing a convolution transformation on the target video frame in the video to obtain a fifth image feature; performing a linear transformation on the fifth image feature based on the convolution kernel to obtain a sixth image feature; performing batch normalization processing on the sixth image feature to obtain a seventh image feature; performing a nonlinear transformation on the seventh image feature to obtain an eighth image feature; and performing a linear transformation on the eighth image feature based on the convolution kernel to obtain the first feature of the target video frame.

[0054] In combination with the first aspect, in some embodiments, the convolution kernel is: a 1x1 convolution kernel; and / or, a nonlinear transformation is performed on the seventh image feature to obtain an eighth image feature, including: performing a nonlinear transformation on the seventh image feature based on a RELU activation function to obtain the eighth image feature.

[0055] In a second aspect, the present application provides an electronic device comprising a processor and a memory; wherein the memory is coupled to the processor, and the memory is used to store computer program code, and the computer program code comprises computer instructions. When the processor executes the computer instructions, the electronic device executes the method described in the first aspect and any possible implementation of the first aspect.

[0056] In a third aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, which are used to call computer instructions to enable the electronic device to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0057] In a fourth aspect, the present application provides a computer-readable storage medium comprising a computer executable program. When the above executable program runs on an electronic device, the above electronic device executes the method described in the first aspect and any possible implementation method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] FIG1 shows a setting interface for off-screen display provided in an embodiment of the present application;

[0059] FIG2 shows the overall process of the wallpaper display method provided in an embodiment of the present application;

[0060] FIG3A exemplarily shows a wallpaper provided in an embodiment of the present application;

[0061] FIG3B shows a diagram of generating a wallpaper provided by an embodiment of the present application based on a video clip;

[0062] FIG4A shows a method of sequentially arranging video frames used by the AOD wallpaper, the lock screen wallpaper, and the desktop wallpaper;

[0063] FIG4B shows another method of sequentially arranging video frames used by the AOD wallpaper, the lock screen wallpaper, and the desktop wallpaper;

[0064] FIG5 shows a method for integrating wallpaper and user interface provided by an embodiment of the present application;

[0065] FIG6 shows a method flow for editing a wallpaper background (such as changing the background) provided in an embodiment of the present application;

[0066] FIG7 shows the display of the new wallpaper after the background is changed on the AOD, lock screen or desktop interface;

[0067] FIG8 shows a method flow for editing a foreground (such as adding decorations) provided by an embodiment of the present application;

[0068] FIG9 shows the display of a new wallpaper with decorations added to a main object on an AOD, lock screen, or desktop interface;

[0069] 10A-10H exemplarily illustrate a human-computer interaction for a user to select a video clip to generate a wallpaper;

[0070] 11A-11D exemplarily illustrate human-computer interaction for a user to edit a wallpaper subject;

[0071] 12A-12E exemplarily illustrate human-computer interaction for a user to edit a wallpaper background;

[0072] 13A-13B exemplarily illustrate a human-computer interaction for a user to change the position of a subject;

[0073] 14A-14B exemplarily illustrate a human-computer interaction for a user to change the scaling ratio of a wallpaper subject and background;

[0074] FIG15 shows a first video cutout method provided by an embodiment of the present application;

[0075] FIG16 shows an image change provided by an embodiment of the present application;

[0076] FIG17 shows a second video cutout method provided by an embodiment of the present application;

[0077] FIG18 shows a first feature processing method provided in an embodiment of the present application;

[0078] FIG19 shows a second feature processing method provided in an embodiment of the present application;

[0079] FIG20 shows a first cascade feature reconstruction method provided by an embodiment of the present application;

[0080] FIG21 shows a second cascade feature reconstruction method provided in an embodiment of the present application;

[0081] FIG22 shows a third video cutout method provided in an embodiment of the present application;

[0082] FIG23 shows the structure of the first video cutout model provided in an embodiment of the present application;

[0083] FIG24 shows the structure of an information compression network provided by an embodiment of the present application;

[0084] FIG25 shows the structure of a second video cutout model provided in an embodiment of the present application;

[0085] FIG26 shows a fourth video cutout method provided by an embodiment of the present application;

[0086] FIG27 shows the structure of the third video cutout model provided in an embodiment of the present application;

[0087] FIG28 shows the structure of the fourth video cutout model provided in an embodiment of the present application;

[0088] FIG29 shows a first model training method provided in an embodiment of the present application;

[0089] FIG30 shows a second model training method provided in an embodiment of the present application;

[0090] FIG31 shows a mask provided in an embodiment of the present application;

[0091] FIG32 shows the structure of a second image generation network provided by an embodiment of the present application;

[0092] FIG33 shows a comparison of cutout results provided by an embodiment of the present application;

[0093] FIG34 shows an electronic device provided by an embodiment of the present application;

[0094] FIG35 shows a software system architecture applied to an electronic device provided by an embodiment of the present application;

[0095] Figure 36 shows a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0096] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0097] As described in the background technology section, mobile phones and other electronic devices can provide wallpaper functions, supporting users to set wallpapers in AOD, lock screen, desktop, etc. One wallpaper function can support users to set wallpapers independently in multiple places such as AOD, lock screen, desktop, etc., but the AOD wallpaper, lock screen wallpaper, desktop wallpaper and other wallpapers are relatively separated, and no coherent display effect of the wallpaper is provided. Another wallpaper function, also known as "super wallpaper" or "screen-off one-shot", can provide a coherent and dynamic wallpaper display effect from AOD to lock screen to desktop, but the wallpaper transition effect is relatively stiff and the user experience is limited.

[0098] In this document, the AOD interface is the interface displayed by an electronic device when the screen is off. It can be used to display information such as time, date, text messages and call reminders, saving user operations and being very intuitive and convenient. In an embodiment of the present application, the AOD interface may also include an AOD wallpaper. To provide the AOD interface, the screen of the electronic device needs to have pixel-level luminescence capabilities and support lighting only some pixels of the screen, such as an organic light-emitting diode (OLED) screen. The lock screen interface is the interface displayed by the electronic device when the screen is on but not unlocked. It may include the system status bar, time, date, weather, lock screen logo, unlock prompt, and icons for shortcut functions that do not require user authorization (such as flashlight, camera, etc.), etc. The desktop is an interface displayed by the electronic device when the screen is on and unlocked. It can also be called the main interface. It may include desktop icons of various applications, the system status bar, etc. Not limited to the desktop, the interface displayed by the electronic device when the screen is on and unlocked can also be other interfaces, such as the user interface of an application, the user interface displayed in the last unlocked state (before the lock screen). The interface displayed when the screen is on and unlocked is collectively referred to as the unlock interface.

[0099] As shown in Figure 1, an electronic device may provide an AOD function switch in the user interface of a settings application, such as providing an "off screen display" switch in the "off screen display" interface of the settings application. The embodiments of the present application do not restrict the interface presentation of the AOD function switch, or whether it may also appear in other application interfaces. The user can choose whether to use the AOD function by operating the switch. When the AOD function switch is turned on, the electronic device can display the AOD interface when the screen is off, otherwise the AOD interface is not displayed. As shown in Figure 1, when the AOD function switch is turned on, the electronic device can also display two off screen modes: "full screen" and "partial" in the "off screen display" interface. In the "full screen" off screen mode, the entire screen area displays the AOD interface with a dimmed effect to save screen power consumption; in the "partial" off screen mode, only a portion of the screen is illuminated to display the AOD interface. In some embodiments, in the "full screen" off screen mode, the AOD wallpaper in the AOD interface can be changed by setting the lock screen wallpaper, that is, the image content of the AOD wallpaper is the same as the image content of the lock screen wallpaper.

[0100] The embodiment of the present application provides a wallpaper display method, which can achieve a dynamic wallpaper display effect with a coherent AOD to lock screen and then to the desktop and other multi-level interfaces, and there is no need to design an additional transition method or transition method from AOD to lock screen, lock screen to desktop for the wallpaper. The transition effect between them is inherent in the video and is more vivid and natural. Moreover, the wallpaper is well integrated with interfaces such as AOD, lock screen, and desktop, and the main object in the wallpaper will not be obscured by non-interactive elements such as time in the interface. In addition, the wallpaper can be generated based on user customization or editing operations, which is convenient for users to set personalized wallpapers, and the user operation is simple, and one editing operation can take effect on the entire dynamic wallpaper.

[0101] FIG2 shows the overall process of the wallpaper display method provided by an embodiment of the present application. As shown in FIG2 , the method may include the following steps:

[0102] S11: The electronic device in a locked or unlocked state detects a first operation. In response to the first operation, the electronic device displays an AOD interface. The AOD interface includes an AOD wallpaper. In addition to the AOD wallpaper, the AOD interface may also display brief information such as date, time, and weather.

[0103] The first operation may be a screen-off operation, such as pressing the power button to turn off the screen of an electronic device in a locked or unlocked state. In both the locked and unlocked states, the screen of the electronic device is in a lit state. In the locked state, the lock screen interface is displayed on the screen, and in the unlocked state, the desktop interface or the user interface of the application opened by the user is displayed on the screen. Not limited to being triggered by a screen-off operation, if the user does not operate the mobile phone for a long time in the locked or unlocked state, the electronic device may also display the AOD interface.

[0104] S12, when the electronic device displays the AOD interface, the electronic device detects a second operation, such as a user operation of lighting up the screen.

[0105] Specifically, the user operation for lighting up the screen in the AOD interface may be, but is not limited to, a touch on the screen, pressing the power button, or the like. Lighting up the screen in the AOD interface may also be timed, with the user setting a time for lighting up the screen, and the electronic device lighting up the screen when the time arrives. The embodiments of the present application do not limit the operation for lighting up the screen in the AOD interface.

[0106] S13. In response to the second operation, the electronic device displays a lock screen interface, where the lock screen interface includes a lock screen wallpaper.

[0107] S14, the electronic device detects a third operation for unlocking, and the third operation unlocks the screen successfully.

[0108] S15. In response to the third operation, the electronic device displays a desktop interface, where the desktop interface includes a first desktop wallpaper.

[0109] Among them, AOD wallpaper, lock screen wallpaper and desktop wallpaper can respectively use the first video frame, second video frame and third video frame in a video, and the second video frame is after the first video frame, and the third video frame is after the second video frame. As the user switches from the AOD interface to the lock screen interface and then to the desktop, a coherent and dynamic wallpaper playback screen is formed.

[0110] In an embodiment of the present application, the AOD interface, the lock screen interface, and the desktop interface may be referred to as the first interface, the second interface, and the third interface, respectively; the video (or video clip) used to generate the dynamic wallpaper may be referred to as the first video.

[0111] AOD wallpapers, lock screen wallpapers and desktop wallpapers can directly use the original video frames in the first video, or use image frames that have been processed by cropping, filtering, etc. on the original video frames.

[0112] Figure 3A exemplifies the wallpaper provided in an embodiment of the present application. As shown in Figure 3A, (a), (b), and (c) respectively show the display of the AOD wallpaper 11, the lock screen wallpaper 12, and the desktop wallpaper 13. The AOD wallpaper 11, the lock screen wallpaper 12, and the desktop wallpaper 13 are linked in sequence to form a dynamic wallpaper. As shown in Figure 3A, from AOD to lock screen and then to desktop, the AOD wallpaper 11, the lock screen wallpaper 12, and the desktop wallpaper 13 are linked together, and they are no longer separated from each other.

[0113] The video clip used to generate the wallpaper can be selected by the user. For example, as shown in Figure 3B, the user can select video clip 14 to generate the wallpaper, and video clip 14 comes from video 15. Video clip 14 can also be selected from a live photo.

[0114] The human-computer interaction method for users to select video clips to generate wallpapers will be described in detail in the following embodiments of Figures 10A to 10H, which will not be expanded here.

[0115] Not limited to jumping from AOD to lock screen and then to desktop, the wallpaper generated by the embodiment of the present application can also be applied to other adjacent interface jump scenarios, such as AOD to desktop and then to certain application interfaces, which will especially occur when the user has not set a lock screen, and for example, from desktop to application main interface and then to application secondary interface. Of course, these application interfaces can support wallpaper display, rather than game display interface, video playback interface and other application interfaces that are not suitable for wallpaper display. For example, the chat interface of "WeChat" can set chat wallpaper. That is, the electronic device can display multiple user interfaces of different levels in turn, and display wallpapers at these multiple user interfaces; the wallpaper is generated based on the video clip selected by the user, and the video frames used in the wallpapers at these multiple user interfaces are arranged in sequence, forming a coherent and dynamic wallpaper display effect as a whole when the user jumps to multiple levels of interfaces.

[0116] The wallpaper display solution provided in the embodiment of the present application will be further explained below from the aspects of wallpaper linkage, main object highlighting, background editing, and main object editing.

[0117] Wallpaper linkage

[0118] Since the video frames used in the AOD interface, lock screen interface, and desktop wallpapers are arranged sequentially, the embodiment of the present application does not require additional design of a transition method or transition method from AOD to lock screen, and from lock screen to desktop for the wallpaper. The transition effect between them is inherent in the video clips, which is more vivid and natural.

[0119] As an example, the video frames used in the AOD interface, lock screen interface, and desktop wallpaper in this application can be arranged in the following order:

[0120] The first implementation method

[0121] The AOD wallpaper 11, the lock screen wallpaper 12, and the desktop wallpaper 13 may all be static wallpapers, that is, the video frames used by the AOD wallpaper, the lock screen wallpaper, and the desktop wallpaper are all one frame.

[0122] As shown in Figure 4A, during the duration of the AOD interface (t1 to t2), the video frame used by AOD wallpaper 11 is always frame "1". For example, AOD wallpaper 11 specifically only uses the facial image in frame "1". During the duration of the lock screen interface (t2 to t3), the video frame used by lock screen wallpaper 12 is always frame "2". During the duration of the desktop interface (t3 to t4), the video frame used by desktop wallpaper 13 is always frame "3". Frames "1", "2", and "3" are several video frames arranged in sequence in the video clip, where frame "1" precedes frame "2", and frame "2" precedes frame "3".

[0123] The frame interval between the first video frame and the second video frame can be greater than the first frame interval, and the frame interval between the second video frame and the third video frame can be greater than the second frame interval. For example, the first video frame, the second video frame, and the third video frame can be extracted from the first video at a certain frame interval (e.g., more than 1 second). This allows the images of these multiple video frames to be different, presenting a dynamic effect when played continuously.

[0124] Using the first implementation method, although the wallpaper seen by the user when he stays at a certain place such as AOD, lock screen or desktop is static, as the user switches from AOD to lock screen and then to desktop, the AOD wallpaper 11, lock screen wallpaper 12, and desktop wallpaper 13 are displayed in sequence, which will produce a coherent dynamic effect as a whole.

[0125] Moreover, in the embodiment of the present application, when the user switches interfaces in reverse, that is, from the desktop to the lock screen and then to the AOD interface, a coherent dynamic effect will be produced as a whole.

[0126] The second implementation method

[0127] The AOD wallpaper 11, the lock screen wallpaper 12, and the desktop wallpaper 13 can each be a dynamic wallpaper, that is, the video frames used by the AOD wallpaper, the lock screen wallpaper, and the desktop wallpaper are all multiple frames.

[0128] When the user stops at a certain place such as AOD, lock screen or desktop, the electronic device plays the dynamic wallpaper at that place, and the wallpaper the user sees is dynamic. In order to achieve a coherent dynamic effect from AOD to lock screen to desktop as a whole, the video frames used by AOD wallpaper 11, lock screen wallpaper 12, and desktop wallpaper 13 need to be connected end to end according to the timing of interface switching. That is: when switching from AOD to the lock screen interface, the starting frame used by the lock screen wallpaper 12 can be the next frame of the ending frame used by the AOD wallpaper 11; when switching from the lock screen interface to the description interface, the starting frame used by the desktop wallpaper 13 can be the next frame of the ending frame used by the lock screen wallpaper 12. In other words, in two user interfaces of adjacent levels, the starting video frame used by the wallpaper at the user interface of the next level can be the next frame of the ending video frame used by the wallpaper at the user interface of the previous level.

[0129] For example, assume that AOD wallpaper 11, lock screen wallpaper 12, and desktop wallpaper 13 are all generated from frames "1", "2", and "3". As shown in Figure 4B, the stages and timings of a wallpaper display cycle from AOD to lock screen to desktop can be as follows:

[0130] t1 to t2: Play the AOD wallpaper 11. As shown in FIG4B, if the time from t1 to t2 is very long, and the user stays on the AOD interface for a long time and does not press the power button to light up the screen, the AOD wallpaper 11 may be played in a loop.

[0131] t2: Detecting that the user switches from the AOD interface to the lock screen interface, the AOD wallpaper 11 ends and the lock screen wallpaper 12 begins. The starting frame (such as frame "3") used by the lock screen wallpaper 12 can be the next frame of the ending frame (such as frame "2") used by the AOD wallpaper 11.

[0132] t2 to t3: Play the lock screen wallpaper 12. As shown in FIG4B, if the time from t2 to t3 is very short and the user quickly unlocks and successfully enters the desktop, the lock screen wallpaper 12 will not be played in a loop; otherwise, the lock screen wallpaper 12 will be played in a loop.

[0133] t3: Detecting that the user switches from the lock screen interface to the desktop, the lock screen wallpaper 12 ends and the desktop wallpaper 13 begins. The starting frame (such as frame "3") used by the desktop wallpaper 13 is the next frame after the ending frame (such as frame "2") used by the lock screen wallpaper 12.

[0134] t3 to t4: Play desktop wallpaper 13. As shown in FIG4B , if the time from t1 to t2 is very long and the user stays on the desktop for a long time and does not press the power button to turn off the screen, the AOD wallpaper 11 may be played in a loop.

[0135] t4: Detecting that the user switches from the desktop to the AOD interface, the desktop wallpaper 13 ends playing, and the AOD wallpaper 11 starts playing, and the next round of wallpaper display begins. The starting frame (such as frame "1") used by the next round of AOD wallpaper 11 is the next frame of the ending frame (such as frame "3") used by the previous round of desktop wallpaper 13. Of course, the user may also switch to the AOD interface from other application interfaces (such as the video playback interface). At this time, the starting frame used by the next round of AOD wallpaper 11 can default to frame "1", that is, the first frame in the video clip.

[0136] Moreover, in the embodiment of the present application, when the user switches interfaces in reverse, that is, from the desktop to the lock screen and then to the AOD interface, a coherent dynamic effect will be produced as a whole. Specifically, the beginning and the end can be connected according to the timing of the interface switching. That is: when switching from the desktop to the lock screen, the starting frame used by the lock screen wallpaper can be the next frame of the ending frame used by the desktop wallpaper; when switching from the lock screen to the AOD, the starting frame used by the AOD wallpaper can be the next frame of the ending frame used by the lock screen wallpaper.

[0137] As shown in Figure 4B, the frame rates of the AOD wallpaper 11, the lock screen wallpaper 12, and the desktop wallpaper 13 can be the same, such as changing the video frame used twice every 1 second. But not limited to this, their frame rates can also be different. For example, the frame rate of the AOD dynamic wallpaper can be larger, such as changing the video frame used 5 times every 1 second, to reduce the risk of damage to the fixed area of ​​the screen due to long-term lighting. Originally, the AOD wallpaper 11 is dynamic, which can reduce this risk compared to the static AOD wallpaper 11. In Figures 4A and 4B, "T" represents the screen refresh rate. Figures 4A and 4B are only examples. The wallpaper frame rate can actually be greater than T, such as changing the video frame used for the wallpaper every 10 T; moreover, the duration of AOD can be much greater than several T; based on human reaction delay and operation delay, the duration of the lock screen wallpaper and desktop wallpaper can also be much greater than several T.

[0138] Main object highlighting

[0139] In the embodiment of the present application, the wallpaper can be well integrated with interfaces such as AOD, lock screen, and desktop, and the main objects in the wallpaper (such as characters) will not be blocked by non-interactive elements such as time in interfaces such as AOD, lock screen, and desktop.

[0140] Figure 5 shows a method for fusing wallpaper and user interface provided by an embodiment of the present application. As shown in Figure 5, the electronic device can first complete the cutout of the main object in the video clip used to generate the wallpaper, separate the main object and its background, and obtain the foreground layer and the background layer. The foreground layer can be a layer of the main object (such as a person), and the background layer is a layer of image content other than the main object. Then, the electronic device can insert layers of non-interactive elements such as time between the foreground layer and the background layer, synthesize these layers, and render interfaces such as AOD, lock screen or desktop. In this way, the image of the main object will be displayed on non-interactive elements such as time, and will not be blocked by non-interactive elements such as time, which can better show the style of the main object in the wallpaper.

[0141] Layers for interactive elements in interfaces such as the AOD, lock screen, or desktop, such as the fingerprint unlock button in the AOD interface, the password keyboard in the lock screen interface, and application icons on the desktop, can be set above the foreground layer to facilitate user operation of these interactive elements and ensure the implementation of interface interactive functions. In other words, the layers of interfaces such as the AOD, lock screen, or desktop can be decomposed into layers for non-interactive elements and layers for interactive elements.

[0142] Regarding how to cut out the main object in a video, please refer to the subsequent embodiments, which will not be expanded here.

[0143] In addition to ensuring that the main object is not blocked by non-interactive elements such as time, the cutout electronic device can further provide background editing and foreground editing functions.

[0144] Background

[0145] Figure 6 shows a method flow for editing wallpaper background (such as changing the background) provided by an embodiment of the present application. As shown in Figure 6, after obtaining the foreground layer and background layer of the wallpaper by cutting out, the electronic device can replace the background layer, and then insert a layer of non-interactive elements such as time between the foreground layer and the new background layer, synthesize these layers, and render an AOD, lock screen or desktop interface for changing the wallpaper background. Thereafter, as shown in Figure 7, the wallpaper displayed on the AOD, lock screen or desktop interface completes the background change.

[0146] Furthermore, in order to simplify the editing operation and improve the efficiency of background editing, after detecting that the user edits the background of the wallpaper at a certain user interface (such as the lock screen interface), the electronic device can synchronously apply the edit to the wallpaper at other user interfaces, that is, the same editing operation is also performed on the wallpaper background at other user interfaces, so that one editing operation takes effect on the entire wallpaper. Here, other user interfaces refer to user interfaces other than the user interface (such as the lock screen interface) that receives the editing operation in the multi-level user interface (such as the AOD interface, the lock screen interface, and the desktop) that can display wallpapers. In other words, the editing operation implemented by the user can be an operation in which the user selects the wallpaper background at a certain user interface in the multi-level user interface for background editing; it is not limited to updating the wallpaper background at the certain user interface, and the electronic device can also update the background image at other user interfaces. That is to say, in response to the user's operation of editing the wallpaper background at a certain user interface, the electronic device can update the wallpaper background image at the multi-level user interface.

[0147] Not limited to changing the background, background editing can also include changing the background scaling, blurring the background, defocusing the background, changing the background color, brightness, and so on. The electronic device can perform corresponding image processing on the background layer separately, such as image scaling, image blurring, image defocusing, adjusting image brightness, changing image color, etc., to update the background layer, thereby obtaining a new wallpaper, and applying the new wallpaper on the AOD interface, lock screen interface, and desktop. Here, the specific meaning of application may refer to synthesizing the background layer, foreground layer of the new wallpaper and the non-interactive element layers and interactive element layers of the AOD interface, lock screen interface, and desktop, and rendering the screen displayed by the new wallpaper on these interfaces.

[0148] The following embodiments of Figures 12A to 12E will exemplarily introduce the human-computer interaction method for users to edit wallpaper backgrounds, which will not be expanded here.

[0149] Main object

[0150] Figure 8 shows a method flow for editing the foreground (such as adding decorations) provided by an embodiment of the present application. As shown in Figure 8, after obtaining the foreground layer and background layer of the wallpaper by cutout, the electronic device can add an image of a decoration (such as a hat) to the foreground layer to obtain a new foreground layer, and then insert layers of non-interactive elements such as time between the new foreground layer and the background layer, synthesize these layers, and render out interfaces such as AOD, lock screen or desktop that decorate the main object of the wallpaper. Thereafter, as shown in Figure 9, the wallpaper displayed at interfaces such as AOD, lock screen or desktop realizes the decoration of the main object. The specific implementation of adding decorations to the main object can be layer synthesis, such as synthesizing the decoration layer and the foreground layer to obtain a new foreground layer.

[0151] Furthermore, in order to simplify the editing operation and improve the efficiency of editing the main object, after detecting that the user edits the main object of the wallpaper at a certain user interface (such as the lock screen interface), the electronic device can synchronously apply the edit to the wallpaper at other user interfaces, that is, the same editing operation is also performed on the wallpaper foreground objects at other user interfaces, so that the main object editing operation is effective for the entire wallpaper. Here, other user interfaces refer to user interfaces other than the user interface (such as the lock screen interface) that receives the editing operation in the multi-level user interface (such as the AOD interface, the lock screen interface, and the desktop) that can display the wallpaper. In other words, the editing operation implemented by the user can be an operation in which the user selects the wallpaper main object at a certain user interface in the multi-level user interface to edit the main object; it is not limited to updating the wallpaper main object at the certain user interface, and the electronic device can also update the main object image at other user interfaces. That is to say, in response to the user's operation of editing the wallpaper main object at a certain user interface, the electronic device can update the main object image of the wallpaper at the multi-level user interface.

[0152] Editing the foreground is not limited to adding decorative objects. It can also include changing the scale, position, color, and brightness of the main object. The electronic device can perform corresponding image processing on the foreground layer separately, such as image scaling, changing the position of the main object image in the foreground layer, adjusting the image brightness, changing the image color, etc., to update the foreground layer containing the main object image, thereby obtaining a new wallpaper and applying the new wallpaper to the AOD interface, lock screen interface, and desktop.

[0153] The following embodiments of Figures 11A to 11D will exemplarily introduce the human-computer interaction method for users to edit the main object, which will not be expanded here.

[0154] Electronic devices can also edit the wallpaper as a whole, such as adding filters and changing the scaling of the main object and its background at the same time. Editing as a whole means that the editing effect takes effect on the wallpaper as a whole, not limited to the foreground or background. For example, when synthesizing layers, add a filter layer on top of the wallpaper background layer, the non-interactive element layer of the lock screen interface, the wallpaper foreground layer, and the interactive element layer of the lock screen interface, so that the filter effect takes effect on the entire lock screen wallpaper. For another example, scale the main object and its background according to the selected scaling ratio to obtain a new foreground layer and background layer, and then synthesize them with the layer of the lock screen interface so that the main object and its background in the lock screen wallpaper are scaled synchronously.

[0155] Based on the wallpaper display solution introduced above, next, a series of human-computer interaction methods provided by the embodiments of the present application are introduced.

[0156] 10A-10H exemplarily illustrate a human-computer interaction method for a user to select a video clip to generate a wallpaper.

[0157] First, the electronic device can display the "Desktop and Wallpaper" settings interface shown in Figure 10A and provide settings options such as "Wallpaper" in this interface. After detecting that the user selects the "Wallpaper" setting option 101, the electronic device can display the "Wallpaper" settings interface shown in Figure 10B and provide wallpaper setting options such as "Video Wallpaper" 102 in this interface. In this article, "Video Wallpaper" is just a name that refers to the wallpaper setting function that uses videos to generate wallpapers, and its naming is not restricted. After detecting that the user selects the "Video Wallpaper" option 102, the electronic device can display the "Video Wallpaper" settings interface shown in Figure 10C and display some recommended videos in this interface. Recommended videos can come from a gallery, from the Internet, or from some locally pre-stored videos. To enrich user choices, the electronic device can also provide a gallery entry "Select from Gallery" 103 in the "Video Wallpaper" settings interface shown in Figure 10C. In terms of technical implementation, this gallery entry can be a control (such as a button), a link, or other types of interface interactive elements. After detecting that the user clicks on the gallery entrance, the electronic device can display the interface shown in Figure 10D and display the videos in the gallery in the interface. To further facilitate the user's selection, in the interface shown in Figure 10D, the videos in the gallery can be categorized, such as "People" and "Architecture".

[0158] The user can select a video 16 for generating a wallpaper from a plurality of videos. Once the user's operation of selecting the video 16 is detected, the electronic device can display a "video wallpaper preview" interface as shown in FIG. 10E.

[0159] As shown in Figure 10E, a preview 17 may be displayed in the "Video Wallpaper Preview" interface, which can be used to display the display status of AOD wallpaper, lock screen wallpaper, and desktop wallpaper at the AOD, lock screen, desktop and other interfaces respectively. The preview can be a video, as shown in Figures 10F to 10H. When the video is played, the dynamic display process of the wallpaper from AOD to lock screen and then to the desktop can be presented. The video can be composed of multiple frames of images, wherein the front image frame can present the display status of the wallpaper at the AOD interface, the middle image frame can present the display status of the wallpaper at the lock screen interface, and the rear image frame can present the display status of the wallpaper at the desktop.

[0160] Furthermore, the preview can also present the reverse dynamic display process of the wallpaper from the desktop to the lock screen and then to the AOD.

[0161] As shown in Figure 10E, the "Video Wallpaper Preview" interface may also include: a video frame sequence 18 of the selected video 16, and a selection box 19. The video frames in the video frame sequence may be presented as video frame thumbnails. The video segment within the selection box 19 is the selected first video, which may be part or all of video 16. In this embodiment of the present application, the "Video Wallpaper Preview" interface may be referred to as the fourth interface, and video 16 may be referred to as the second video. The user may slide the selection box 19 along the video frame sequence 18 to change the selected video segment. The length of the selection box 19 may be fixed, such as a fixed selection of a video segment of 240 frames or 4 seconds. The length of the selected video segment may also be variable. The selected video segment is at least longer than the sum of the frame interval 1 and the frame interval 2, so that three video frames spaced farther apart can be selected from the video segment for use as the AOD wallpaper, lock screen wallpaper, and desktop wallpaper, respectively. This allows the images of these multiple video frames to be different, presenting a dynamic changing effect when played consecutively. Frame Interval 1 is the frame interval between the video frames used by the AOD wallpaper and the lock screen wallpaper, respectively. Frame Interval 2 is the frame interval between the video frames used by the lock screen wallpaper and the desktop wallpaper, respectively. The interface interaction element for selecting a video segment from the video frame sequence 18 is not limited to the selection box 19. Alternatively, the user may select a video segment by directly selecting multiple consecutive frames from the video frame sequence 18, rather than designing a dedicated control for selecting a video segment.

[0162] The above content illustrates that the first video can be selected by the user from the gallery. However, the settings app can also provide a video for wallpaper generation, and the user can select the first video from the videos provided by the settings app. The first video can also be provided by the system without user intervention.

[0163] As shown in Figure 10E, the "Video Wallpaper Preview" interface may also include: a cancel control 20, a completion control 21, a "depth of field effect" control, an "edit" control, and so on. Among them, the cancel control 20 can be used by the user to cancel the video clip in the selection box 19 from generating wallpaper; the completion control 21 can be used by the user to confirm the video clip in the selection box 19 from generating wallpaper; the "depth of field effect" control can be used by the user to adjust the depth of field of the wallpaper. The "edit" control can be the entrance to wallpaper editing. Through this entrance, the user can perform a series of wallpaper editing operations, such as adding decorations to the main object, changing the background, etc. The following Figures 11A-11D and Figures 12A-12E will introduce wallpaper editing, which will not be expanded here.

[0164] After the electronic device detects that the user has changed the video clip used to generate the wallpaper, such as by sliding the selection box 19 to change the selected video clip, the electronic device can generate a new wallpaper based on the new video clip and update the preview 17. The updated preview 17 can be used to display the display status of the new wallpaper in interfaces such as the AOD, lock screen, and desktop. Specifically, when the updated preview 17 is played, the dynamic display process of the new wallpaper from the AOD to the lock screen and then to the desktop can be displayed. In this way, it is convenient for users to promptly understand the dynamic display effect of the new wallpaper and facilitate users to create their favorite wallpapers.

[0165] The video clip used to generate the wallpaper is not limited to user selection. The video clip used to generate the wallpaper can also be a default one, such as the first three video frames of the selected video constitute the video clip by default, or the video clip used to generate the wallpaper can be composed of several video frames before and after the highlight video frame (including the highlight frame itself). As for how to identify the highlight video frame from the video, this embodiment of the application does not limit it.

[0166] After the wallpaper is updated by changing the video clip, when the AOD, lock screen, and desktop are displayed again, the electronic device will apply the new wallpaper on these interfaces, that is, the updated wallpaper will also be displayed on the AOD, lock screen, and desktop, so as to ultimately achieve the purpose of changing the wallpaper.

[0167] 11A-11D exemplarily illustrate a human-computer interaction method for a user to edit a main object of a wallpaper.

[0168] First, the electronic device can display the "Wallpaper Editing" interface shown in Figure 11A, and display a preview 23 in the interface. In an embodiment of the present application, the "Wallpaper Editing" interface can be referred to as the fifth interface. Like the aforementioned preview 17, preview 23 can be used to present the display status of AOD wallpapers, lock screen wallpapers, and desktop wallpapers at interfaces such as AOD, lock screen, and desktop, respectively. Preview 23 can be a video, which can present the dynamic display process of the wallpaper from AOD to lock screen and then to the desktop when the video is played. In addition, in preview 23, the main object of the wallpaper and its background can be displayed separately, such as by indicating this separation through mark 24. In this way, the user can select the main object or background for editing respectively. This separate display is achieved based on the cutout of the main object on the video clip that generates the wallpaper.

[0169] Specifically, the mark 24 can be a highlighted dividing line designed along the boundary of the main object, or a semi-transparent layer with some dynamic effect covering the main object, etc. The embodiment of the present application does not limit the interface expression form of the mark 24.

[0170] The user can click the "Edit" control in the interface shown in Figure 10E to enter the "Wallpaper Edit" interface shown in Figure 11A. Not limited to this, the entrance to the "Wallpaper Edit" interface can also be set in other locations, and this embodiment of the application does not limit this.

[0171] As shown in Figures 11A and 11B, when the main object of the wallpaper and its background are displayed separately, the electronic device can detect that the user has selected the main object and, in response, provide an editing option called "Decoration." The electronic device can further detect that the user has clicked on the "Decoration" editing option and, in response, display one or more decorative objects, such as "Element 1," "Element 2," "Element 3," and "Element 4," as shown in Figure 11C.

[0172] Subsequently, the electronic device may detect the user's operation of adding a decoration (such as "element 1") to the main object. In response, the electronic device may generate a new wallpaper and update the preview 23 as shown in Figure 11D. The main object in the new wallpaper is added with the decoration selected by the user. The updated preview 23 can be used to show the display status of the new wallpaper on the AOD, lock screen, and desktop. Furthermore, the user can also adjust the position of the decoration, such as moving the "hat" to the top of the character's head.

[0173] Furthermore, in order to simplify the editing operation and improve the efficiency of editing the main object, after detecting the user's operation of adding decorations to the main object of a wallpaper (such as AOD wallpaper, lock screen wallpaper or desktop wallpaper), the electronic device can synchronously apply the editing to other wallpapers, that is, add decorations to the main objects of other wallpapers in the same way, so that the main object editing operation can take effect on the entire dynamic wallpaper.

[0174] After adding decorations to the main object of the wallpaper, when the AOD, lock screen, and desktop are displayed again, the electronic device will apply the new wallpaper with the main object decorated on these interfaces, that is, the new wallpaper will also be displayed on the AOD, lock screen, and desktop, so as to ultimately achieve the purpose of adding decoration to the main object of the wallpaper.

[0175] It is not limited to adding decorations to the main object. Selecting the main object for editing can also include: replacing the main object, adding filters to the main object, and other editing operations.

[0176] 12A-12E exemplarily illustrate a human-computer interaction method for a user to edit a wallpaper background.

[0177] The electronic device can display the "wallpaper editing" interface shown in Figure 12A. Its implementation can refer to the relevant description of the interface shown in Figure 11A, which will not be repeated here.

[0178] As shown in Figures 12A and 12B, when the main object of the wallpaper and its background are displayed separately, the electronic device can detect the user's operation of selecting the wallpaper background, and in response, provide an editing option of "Switch Background". The electronic device can further detect the user's operation of clicking the "Switch Background" editing option, and in response, as shown in Figure 12C, provide background editing options such as "Switch Background Color" and "Switch Background Image", and provide background color options such as "Color 1", "Color 2", "Color 3", and "Color 4", as well as a background image switching option of "Select from Gallery".

[0179] Subsequently, the electronic device can detect that the user has selected the option "Select from Gallery" and display the gallery call interface shown in Figure 12C, which displays photos in the gallery. As shown in Figure 12D, the electronic device can further detect that the user has selected photo 26 as a new wallpaper background. In response, a new wallpaper can be generated based on the new background and preview 23 can be updated as shown in Figure 12E. The background of the new wallpaper has been changed, and the updated preview 23 can be used to show the display status of the new wallpaper in interfaces such as AOD, lock screen, and desktop.

[0180] After changing the wallpaper background, when the AOD, lock screen, and desktop are displayed again, the electronic device will apply the new wallpaper with the changed background on these interfaces, that is, the new wallpaper will also be displayed on the AOD, lock screen, and desktop, so as to ultimately achieve the purpose of changing the wallpaper background.

[0181] In addition, the electronic device can detect the user's operation of adjusting the position of the subject relative to the background, such as dragging the subject to the left as shown in Figure 13A. In response to this, a new foreground layer can be generated based on the subject after the updated position, and then a new wallpaper can be generated. As shown in Figure 13B, the position of the subject in the new wallpaper is the adjusted position. After changing the position of the subject in the wallpaper, when the AOD, lock screen, and desktop are displayed again, the electronic device will apply the new wallpaper with the changed position of the subject at these interfaces, that is, it will also display the new wallpaper at the AOD, lock screen, and desktop, so as to ultimately achieve the purpose of changing the position of the subject in the wallpaper.

[0182] The electronic device can also detect the user's operation of adjusting the scale ratio of the main object or background of the wallpaper, such as dragging the slider to change the scale ratio of the main object when the main object is selected as shown in Figure 14A, and dragging the slider to change the scale ratio of the main object when the background is selected as shown in Figure 14B. In response, the electronic device can generate a new foreground layer or background layer based on the scale ratio selected by the user, and then generate a new wallpaper. Not limited to Figures 14A and 14B, the electronic device can also detect the user's operation of adjusting the scale ratio of the main object and background at the same time, such as zooming in on the main object while reducing the background to create a movie visual experience. In response, the foreground layer and background layer can be updated at the same time, and a new wallpaper can be generated based on the new foreground layer and the new background layer. After changing the scale ratio of the main object and background in the wallpaper, when the AOD, lock screen, and desktop are displayed again, the electronic device will apply the new wallpaper with the changed scale ratio of the main object and background on these interfaces, that is, it will also display the new wallpaper on the AOD, lock screen, and desktop, so as to ultimately achieve the purpose of changing the scale ratio of the main object and background in the wallpaper.

[0183] The embodiment of the present application also provides a video cutout method to cut out the main object in the video clip used to generate wallpaper, separate the main object and its background, and then insert a layer of non-interactive elements such as time between the foreground layer and the background layer to avoid the main object image being blocked by non-interactive elements such as time, which can better highlight the style of the main object in the wallpaper.

[0184] The video cutout method provided in the embodiment of the present application can also improve the accuracy of video cutout.

[0185] First, the video cutout process is explained.

[0186] A video contains multiple frames. The frame to be cropped is called the target frame. This means any frame in the video that needs to be cropped. Video cropping involves extracting the object region within a video frame with transparency. These objects can be people, animals, vehicles, lane lines, and more.

[0187] In the video cutout process, first, the first target video frame is determined, which can be the first video frame or other video frames in the video, and the determined target video frame is cutout to obtain the object area in the target video frame; then, the next video frame of the target video frame is determined as a new target video frame, and the new target video frame is cutout; in this way, each time the object area in the target video frame is obtained, the next video frame is determined as the new target video frame, until the segmentation result of the last video frame of the video is obtained, and the object cutout of the entire video is completed.

[0188] Next, the application scenarios of the video cutout solution provided in the embodiments of the present application are illustrated with examples.

[0189] 1. Real-time video scene

[0190] In this scenario, the video to be played is cut out to obtain the object area in each video frame in the video. In this way, when playing the video, only the area content of the object area in each video frame in the video can be played.

[0191] 2. Video editing scenes

[0192] In this scenario, after segmenting the video to obtain the object regions within each frame, you can then perform editing operations on the video frames, such as background replacement, object erasure, background blurring, and color preservation, based on the object region's location and image content, to create a new video. Furthermore, after editing the video frames to create a new video, you can also implement other applications based on the new video, such as video creation and terminal lock screen applications.

[0193] In the scenario where the new video obtained by video editing is applied to the terminal lock screen, after the video is segmented to obtain the area where the object is located in each video frame in the video, and the video is edited according to the information of the area where the object is located in the video frame to obtain a new video, a dynamic lock screen wallpaper for the terminal can be generated according to the picture content of each video frame in the new video, so that the dynamic lock screen wallpaper can be displayed when the terminal is in the lock screen state.

[0194] 3. Video surveillance scenarios

[0195] In this scenario, after the monitoring device captures a video of a specific area, it can detect objects in the specific area by cropping the video.

[0196] Next, the video segmentation solution provided in the embodiment of the present application is described in detail through specific examples.

[0197] In one embodiment of the present application, referring to FIG15 , a flow chart of a first video cutout method is provided. In this embodiment, the method includes the following steps S301 - S305 .

[0198] Step S301: compressing information of a target video frame in a video to obtain a first image feature.

[0199] The target video frame may be any video frame among the video frames included in the video.

[0200] Information compression on the target video frame can be understood as extracting features from the target video frame to obtain first image features with a scale smaller than the target video frame. Feature extraction on the target video frame can extract edge information from the image content, which can reflect the area where the object is located in the video frame.

[0201] Furthermore, when extracting features from a target video frame, multiple cascaded feature extractions can be performed. As the number of feature extractions increases, the scale of the resulting features decreases. From a scale perspective, a larger scale for the first image feature contains more detailed edge information. Excessive detailed edge information can, in some cases, hinder the determination of the object's region within the video frame. Conversely, a smaller scale for the first feature contains more macroscopic edge information, making it more effective at determining the object's region within the video frame.

[0202] Furthermore, the dimension of the above-mentioned first image feature can be the same as the dimension of the target video frame, that is, the target video frame is a two-dimensional image, so its dimension is 2, then the first image feature can also be two-dimensional data. In this case, the first image feature can also be considered as a feature map.

[0203] In one embodiment of the present application, when information compression is performed on a target video frame, it may be implemented based on an encoding method, for example, based on an encoding network.

[0204] In another embodiment of the present application, the target video frame can be compressed by performing a convolution transformation on the target video frame. During the convolution transformation of the target video frame, the target video frame can be subjected to multiple convolution transformations, thereby continuously reducing the scale of the features obtained by the convolution transformation.

[0205] In addition, the target video frame may be compressed by combining convolution transformation, linear transformation, batch normalization, nonlinear transformation and other processing. For details, please refer to steps S301A-S301E in the embodiment shown in the subsequent Figure 22, which will not be described in detail here.

[0206] Step S302: Segment the target video frame based on the first image feature, and obtain features of the object's contour mask image obtained during the segmentation process and a first contour mask image of the object in the target video frame.

[0207] Among them, the above-mentioned first contour mask image is obtained by segmenting the target video frame. Therefore, there is a correspondence between the pixel points in the first contour mask image and the pixel points in the target video frame. The first contour mask image can be understood as a binary image indicating the position of the area where the object is located in the target video frame. The area where the object is located indicated by the first contour mask image is determined based on the approximate outline of the object. Therefore, the area where the object is located can be considered as the approximate area where the object is located.

[0208] For example, the first contour mask image may be a mask image containing pixel points with two pixel values, 0 and 1. In the target video frame, the pixel points corresponding to the pixel points with a pixel value of 1 in the first contour mask image may be the pixel points within the area where the object is located, and the pixel points corresponding to the pixel points with a pixel value of 0 in the first contour mask image may be the pixel points in an area other than the area where the object is located.

[0209] Specifically, the target video frame may be segmented by any one of the following three implementation methods.

[0210] In the first implementation, based on the first image feature, the target video frame can be segmented by the segmentation method mentioned in the embodiment shown in the subsequent FIG. 18 , which will not be described in detail here.

[0211] In the second implementation method, based on the first image feature, the target video frame can be segmented using a pre-trained network model, such as the contour feature generation network and the first image generation network in the video segmentation model in the embodiment shown in the subsequent Figure 23, which will not be described in detail here.

[0212] In a third implementation manner, based on the first image feature, the target video frame may be segmented using an image segmentation algorithm, a video segmentation algorithm, or the like.

[0213] Step S303: reconstructing features based on the first image features and the features of the obtained contour mask image, fusing the reconstructed features with the first hidden state information of the video, and updating the first hidden state information to obtain a fusion result.

[0214] The first hidden state information represents: a fusion feature of a transparency mask image of an object edge in a video frame that is cut out before the target video frame.

[0215] The edge of an object in a video frame may reflect the image content details of the object in the video frame. The image content details may include the specific location of the region where the object is located and the transparency of the object image content presented by the pixels in the region where the object is located.

[0216] The transparency mask image of the object edge in the video frame can be understood as a mask image indicating the image content details of the object in the video frame, and the pixel value of the pixel point in the transparency mask image can represent the transparency of the object image content presented by the pixel point at the same position in the video frame.

[0217] For example, the pixel value range of the pixel in the transparency mask image can be 0-1. If the pixel value of the pixel in the transparency mask image is 0, it means that the pixel at the same position in the video frame belongs to an area outside the area where the object is located, and the image content presented by the pixel at the same position does not contain the image content of the object; if the pixel value of the pixel in the transparency mask image is 0.6, it means that the pixel at the same position in the video frame belongs to the area where the object is located, and the transparency of the image content of the object presented by the pixel at the same position is 0.6; if the pixel value of the pixel in the transparency mask image is 1, it means that the pixel at the same position in the video frame belongs to the area where the object is located, and the image content presented by the pixel at the same position is entirely the image content of the object.

[0218] The video frames mentioned in this step that are cut out before the target video frame include at least two video frames. Of course, they can also be all video frames that are cut out before the target video frame.

[0219] When the target video frame is the first video frame in the video, there is no previous video frame to be cut out. In this case, the first hidden state information may be preset data, for example, preset all-zero data.

[0220] Specifically, the first hidden state information may be expressed in a tensor form or in a matrix form.

[0221] As can be seen from the description of step S301, the first image feature is a feature that is scaled down relative to the target video frame, and the first image feature can reflect the region within the target video frame where the object resides. To successfully extract the region within the target video frame, feature mapping is required for these small-scale first image features. The ultimate goal is to map them to the target video frame, thereby obtaining the region within the target video frame where the object resides. To this end, upsampling is required for these first image features.

[0222] Specifically, feature reconstruction is performed based on the first image feature to reconstruct a scaled feature, and then the reconstructed feature is fused with the first hidden state information to obtain a fusion result. Since the first hidden state information represents the fusion feature of the transparency mask image of the object edge in the video frame that was cut out before the target video frame, that is, the first hidden state information can represent the information of the object edge in the video frame before the target video frame, and the object edge information can include the specific location information of the region where the object is located, after the reconstructed feature and the first hidden state information are fused, the resulting fusion result can not only reflect the region where the object is located in the target video frame, but can also adjust the region where the object is located in the target video frame in combination with the region where the object is located in the previous video frame, thereby ensuring the smoothness of the region where the object is located between adjacent video frames, or in other words, the temporal correlation.

[0223] Because the first hidden state information is used in subsequent video frame cutouts, it must be updated based on information about the object in the target video frame. Specifically, the hidden state information can be updated based on the first image features or based on the fusion results. For example, the reconstructed features can be fused with the first hidden state information to create the new first hidden state information.

[0224] Specifically, when performing feature reconstruction based on the first image feature, the first image feature can be transformed using an upsampling algorithm to obtain the reconstructed first image feature; the above-mentioned first image feature can be deconvolution transformed to obtain the reconstructed first image feature; the first image feature can be reconstructed based on a decoding network to obtain the reconstructed first image feature. For example, the above-mentioned decoding network can be the decoding part in the U-Net network architecture, or it can be the decoder part in the U2-Net network architecture.

[0225] The reconstructed first image feature and the first hidden state information may be fused by any one of the following two implementation methods.

[0226] In a first implementation manner, a fusion algorithm, a network, etc. may be used to fuse the reconstructed first image features and the first hidden state information to obtain a fusion result.

[0227] For example, a long short-term memory (LSTM) network, a gated recurrent unit (GRU), etc. are used to fuse the reconstructed first image features and the first hidden state information to obtain a fusion result.

[0228] In the second implementation, the reconstructed first image feature and the first hidden state information may be directly subjected to operations such as superposition, concatenation, or dot multiplication to obtain a processing result as a fusion result.

[0229] For other implementations of the above step S303, please refer to the subsequent embodiments and will not be described in detail here.

[0230] Step S304: Based on the fusion result, a target transparency mask image of the object edge in the target video frame is obtained.

[0231] The target transparency mask image may be understood as a mask image indicating image content details of an object in a target video frame. The pixel values ​​of pixels in the target transparency mask image may range from 0 to 1, and the scale thereof may be the same as that of the target video frame.

[0232] In one implementation of the present application, the fusion result may include the confidence level that each pixel in the target video frame belongs to the object. In this case, after obtaining the fusion result, a target transparency mask image may be obtained based on the confidence level corresponding to each pixel included in the fusion result.

[0233] For example, the confidence level corresponding to each pixel in the fusion result can be used as the pixel value of each pixel at the same position in the mask image to obtain the target transparency mask image.

[0234] For another example, a first threshold and a second threshold may be pre-set, where the first threshold is greater than the second threshold. If the confidence corresponding to a pixel included in the fusion result is greater than or equal to the first threshold, it indicates that the confidence corresponding to the pixel is close to 1, and the confidence that the pixel belongs to the object is high. In this case, the pixel value of the pixel at the same position in the mask image can be determined to be 1. If the confidence corresponding to a pixel included in the fusion result is less than or equal to the second threshold, it indicates that the confidence corresponding to the pixel is close to 0, and the confidence that the pixel belongs to the object is low. In this case, the pixel value of the pixel at the same position in the mask image can be determined to be 0. If the confidence corresponding to a pixel included in the fusion result is between the second threshold and the first threshold, the confidence corresponding to the pixel can be mapped to the 0-1 interval based on the mapping relationship between the confidence interval from the second threshold to the first threshold and the 0-1 interval. The resulting mapped value is the pixel value of the pixel at the same position in the mask image.

[0235] In another implementation of the present application, the target transparency mask image may be obtained through steps S304A-S304C in the embodiment shown in FIG17 .

[0236] Step S305: performing region cutout on the target video frame according to the target transparency mask image and the first contour mask image to obtain a cutout result.

[0237] Specifically, the first contour mask image indicates the approximate area where the object is located in the target video frame, and the target transparency mask image indicates the image content details of the object in the target video frame. In this way, based on the target transparency mask image and the first contour mask image, the target video frame can be regionally cut out in combination with the approximate area where the object is located in the target video frame and the image content details to obtain the cutout result.

[0238] In one implementation of the present application, the target transparency mask image and the first contour mask image can be dot-multiplied according to the position of each pixel point to obtain a mask image after dot multiplication. The obtained mask image and the target video frame are then dot-multiplied again according to the position of each pixel point to obtain a dot multiplication result as the cutout result, thereby realizing regional cutout of the target video frame.

[0239] In addition, see Figure 16, which shows a schematic diagram of the process from target video frames to target transparency mask images and first contour mask images, and then to the resulting cutout image, when multiple video frames are used as target video frames. In Figure 16, the first row of images represents the target video frames; the second row of images represents the first contour mask images corresponding to the target video frames; the third row of images represents the target transparency mask images corresponding to the target video frames; and the fourth row of images represents the resulting cutout image corresponding to the target video frames.

[0240] As can be seen from the above, in the solution provided by this embodiment, when the target video frame is segmented based on the first image feature, the feature of the contour mask image of the object during the segmentation process is obtained. This feature is representative of the contour of the object in the contour mask image. The first hidden state information represents the fusion feature of the transparency mask image of the edge of the object in the video frame that was cut out before the target video frame. In this way, feature reconstruction is performed based on the first image feature and the feature of the obtained contour mask image, and the reconstructed feature and the first hidden state information are fused. The resulting fusion result not only incorporates the contour information of the object in the contour mask image, but also incorporates the information of the edge of the object in the video frame that was cut out before the target video frame. Since there is often temporal correlation between video frames in a video, when obtaining the target transparency mask image based on the fusion result, the information of the object in the video frame with temporal correlation is considered on the basis of the target video frame, thereby improving the accuracy of the obtained target transparency mask image. On this basis, based on the target transparency mask image and the first contour mask image, the target video frame can be accurately cut out. It can be seen that the application of the video cutout solution provided by the embodiment of the present application can improve the accuracy of video cutout.

[0241] In addition, when obtaining the target transparency mask image, the fusion features of the transparency mask images of the object edges in the video frames that were previously cut out of the target video frame are taken into account. That is, the image information of the object edges in these video frames is taken into account, rather than just the image information of the target video frame itself. This can improve the inter-frame smoothness of the changes in the object edge region in the target transparency mask image corresponding to each video frame within the video, thereby improving the inter-frame smoothness of the changes in the object edge region in the cutout results corresponding to each video frame. Furthermore, in the case of a video of a moving object, because this solution takes into account the information of the video frames that have already been cut out when cutting out the target video frame, it can achieve a higher smoothness between the cutout results of the target video frame and the cutout results of the video frames that have already been cut out.

[0242] Other implementations of obtaining the target transparency mask image in the above step S304 are described below.

[0243] In one embodiment of the present application, referring to FIG17 , a flow chart of a second video cutout method is provided. In this embodiment, the above step S304 can be implemented through the following steps S304A-S304C.

[0244] Step S304A: Based on the fusion result, a second contour mask image of the object in the target video frame is obtained.

[0245] The second contour mask image may be a binary image, and its scale is the same as that of the target video frame.

[0246] In one implementation of the present application, it can be seen from the description of the above step S304 that the above fusion result may include the confidence that each pixel point in the target video frame belongs to the object. In this case, after obtaining the above fusion result, the fusion result can be binarized based on a preset third threshold to obtain a second contour mask image.

[0247] When performing the above-mentioned binarization processing, the values ​​in the fusion result that are greater than the third threshold value may be set to 0, and the values ​​that are not greater than the third threshold value may be set to 1. Of course, the values ​​in the fusion result that are less than the third threshold value may also be set to 0, and the values ​​that are not less than the third threshold value may be set to 1. This embodiment of the present application is not limited to this.

[0248] Step S304B: Fuse the second contour mask image and the fusion result to obtain the target fusion feature.

[0249] Specifically, the second contour mask image and the fusion result may be fused by any one of the following two implementation methods.

[0250] In the first implementation, the second contour mask image and the fusion result can be fused using a fusion algorithm, a network, etc. to obtain a fused target fusion feature.

[0251] In the second implementation method, the second contour mask image and the fusion result can be directly processed by superposition, splicing or dot multiplication to obtain the processing result as the target fusion feature.

[0252] Step S304C: Based on the target fusion features, a target transparency mask image of the object edge in the target video frame is obtained.

[0253] The target fusion feature is obtained by fusing the second contour mask image and the fusion result. The second contour mask image indicates the approximate area where the object is located in the target video frame. The fusion result may include the confidence that each pixel point in the target video frame belongs to the object. In this way, the target fusion feature obtained by fusing the two can include the confidence that each pixel point in the approximate area where the object of the target video frame belongs to the object. Based on the target fusion feature, the target transparency mask image can be obtained with the confidence corresponding to each pixel point included in the feature.

[0254] Specifically, the implementation method of obtaining the target transparency mask image based on the target fusion feature can refer to the implementation method of obtaining the target transparency mask image based on the fusion result in the above step S304, which will not be repeated here.

[0255] As can be seen from the above, in the solution provided by this embodiment, a second contour mask image of the object is obtained based on the fusion result, and the second contour mask image can indicate the approximate area where the object is located determined by the approximate outline of the object in the target video frame. In this way, the second contour mask image is fused with the fusion result to obtain the target fusion feature. The target fusion feature only needs to focus on the detailed information of the image in the approximate area where the object is located. Based on the target fusion feature, the target transparency mask image of the edge of the object can be accurately obtained while only focusing on the image content in the approximate area where the object is located. Therefore, video cutout can be performed based on the target transparency mask image, which can improve the accuracy of video cutout.

[0256] The first implementation method of segmenting the target video frame mentioned in step S302 is described below.

[0257] In one embodiment of the present application, referring to FIG18 , a flow chart of a first feature processing method is provided. The feature processing process shown in FIG18 includes the segmentation process of step S302 and the reconstruction and fusion process of step S303 . The number of feature reconstructions included in the segmentation process is the same as the number of transparency information fusions included in the reconstruction and fusion process.

[0258] In Figure 18, the segmentation process includes two feature reconstructions, and the reconstruction fusion process includes two transparency information fusions. In addition, the number of feature reconstructions included in the segmentation process and the number of transparency information fusions included in the reconstruction fusion process can also be other numbers, such as 3, 4, 5, etc., which are not limited in this embodiment.

[0259] The segmentation process of step S302 and the reconstruction and fusion process of step S303 are described below in conjunction with FIG. 18 .

[0260] First, for the segmentation process of the above-mentioned step S302, when the target video frame is segmented based on the first image feature, cascade feature reconstruction can be performed based on the first image feature to obtain the features of the contour mask image with successively increasing scales of the object, and based on the features obtained by the last processing, the first contour mask image of the object in the target video frame is obtained.

[0261] Cascade feature reconstruction can be understood as multiple feature reconstructions, where the result of each feature reconstruction is the feature of a contour mask image of a certain scale. Moreover, the object of the first feature reconstruction is the first image feature mentioned above, and the objects of other feature reconstructions are the features obtained from the previous feature reconstruction.

[0262] In Figure 18, the cascade feature reconstruction process includes two feature reconstructions. In the first feature reconstruction process, the object of feature reconstruction is the first image feature. After feature reconstruction of the first image feature, contour feature 1 of the scaled contour mask image can be obtained. At this point, the first feature reconstruction process ends.

[0263] In the second feature reconstruction process, the object of feature reconstruction is the above-mentioned contour feature 1. After feature reconstruction of contour feature 1, contour feature 2 of the contour mask image with the scale increased again can be obtained. At this time, the second feature reconstruction process ends, and contour feature 2 is the feature obtained by the last processing. In this way, based on contour feature 2, the first contour mask image of the object in the target video frame can be obtained.

[0264] The following describes how to implement each feature reconstruction and how to obtain the first contour mask image based on the features obtained in the last processing.

[0265] In each feature reconstruction process, feature reconstruction can be achieved through either of the following two implementation methods.

[0266] In the first implementation, feature reconstruction can be achieved by feature reconstruction, feature fusion and updating as mentioned in subsequent embodiments, which will not be described in detail here.

[0267] In the second implementation method, an upsampling algorithm, a deconvolution transform, or a feature decoding network can be used to process the object of each feature reconstruction.

[0268] After obtaining the feature obtained by the last feature reconstruction process, the feature may include the confidence that each pixel point in the target video frame belongs to the object. In this case, after obtaining the feature, the feature can be binarized based on the preset fourth threshold to obtain the first contour mask image.

[0269] Secondly, regarding the reconstruction and fusion process in step S303, in this embodiment, the first hidden state information includes: multiple first sub-hidden state information, each of which represents the fusion characteristics of a transparency mask image of a different scale. The multiple first sub-hidden state information may represent the fusion characteristics of transparency mask images of successively larger scales. For example, the first hidden state information may include three first sub-hidden state information, each of which may represent the fusion characteristics of transparency mask images of scales of 24*24, 28*28, and 32*32, respectively.

[0270] In this case, when reconstructing, fusing, and updating features based on the first image features and the features of the obtained contour mask image, a preset number of transparency information fusions may be performed in the following manner, and the features obtained from the last processing are determined as the fusion results:

[0271] Feature reconstruction is performed based on the first target feature and the second target feature in the features of the obtained contour mask image to obtain a scale-enlarged second image feature, the second image feature and the first sub-state information in the first hidden state information are fused and the first sub-state information is updated to obtain a third image feature.

[0272] The above-mentioned preset number is a pre-set number and is the same as the number of reconstructions of the cascade feature reconstruction.

[0273] When information fusion is performed for the first time, the first target feature is the first image feature. When information fusion is performed for other times, the first target feature is the feature obtained from the last information fusion.

[0274] The scale of the first target feature is the same as the scale of the second target feature.

[0275] The scale of the transparency mask image corresponding to the fusion feature represented by the first sub-state information is the same as the scale of the second image feature.

[0276] In FIG18 , the preset number is 2, meaning two transparency information fusion processes are performed. In the first transparency information fusion process, the first target feature is the first image feature, and the second target feature is the aforementioned outline feature 1. The first image feature and outline feature 1 have the same scale. Feature reconstruction is performed based on the first target feature and the second target feature. That is, feature reconstruction is performed based on the first image feature and outline feature 1 to obtain a scaled-up second image feature 1. This second image feature 1 is then fused with the first sub-state information 1 corresponding to the second image feature 1, and the first sub-state information 1 is updated to obtain a third image feature 1. At this point, the first transparency information fusion process ends.

[0277] After obtaining the third image feature 1 in the first transparency information fusion process, a second transparency information fusion is performed. In this second transparency information fusion process, the first target feature is the third image feature 1, and the second target feature is the outline feature 2. The third image feature 1 and the outline feature 2 have the same scale. Feature reconstruction is performed based on the first and second target features. That is, feature reconstruction is performed based on the third image feature 1 and the outline feature 2 to obtain a second image feature 2 with a further increased scale. This second image feature 2 is then fused with the first sub-state information 2 corresponding to the second image feature 2, and the first sub-state information 2 is updated to obtain the third image feature 2. At this point, the second transparency information fusion process ends, and the third image feature 2 is the final fusion result.

[0278] For the implementation method of each feature reconstruction based on the first target feature and the second target feature, please refer to the implementation method of feature reconstruction based on the first image feature and the feature of the contour mask image in step S303 in the embodiment shown in Figure 15 above; for the implementation method of each time the second image feature and the first sub-state information are fused and the first sub-state information is updated, please refer to the implementation method of step S303 in the embodiment shown in Figure 15 above to fuse the reconstructed feature and the first hidden state information and update the first hidden state information, which will not be repeated here.

[0279] As can be seen from the above, in the solution provided by this embodiment, the target video frame is segmented based on the first image feature in a cascade feature reconstruction manner, and the cascade feature reconstruction includes multiple feature reconstruction processes, which can improve the accuracy of the first contour mask image; after obtaining the first image feature, multiple transparency information fusions are performed, and each transparency information fusion process includes three processing processes: feature reconstruction, feature and hidden state information fusion, and updating of hidden state information. This can improve the accuracy of the final fusion result, thereby realizing regional cutout of the target video frame based on a relatively accurate first contour mask image and fusion result, which can improve the accuracy of regional cutout, and thus improve the accuracy of video cutout.

[0280] The amount of data of the two features required for feature reconstruction in each transparency information fusion process is usually large, so the computational complexity of feature reconstruction is also large, and the computational complexity of performing a preset number of transparency information fusions is also large.

[0281] In view of this, in one embodiment of the present application, when performing feature reconstruction based on the first target feature and the second target feature in the features of the obtained contour mask image, the representative features of the object edge can be screened from the second target feature in the features of the obtained contour mask image, and feature reconstruction can be performed based on the first target feature and the representative feature to obtain a second image feature with an enlarged scale.

[0282] The representative feature is a feature included in the second target feature that is representative of edge details of an object in the target video frame.

[0283] In the process of feature reconstruction based on the first image feature, the attributes of the object represented by the features of the reconstructed contour mask image can be determined. In this way, in each transparency information fusion process, after determining the second target feature with the same scale as the first target feature in the features of each contour mask image, the features representative of the object edge in the second target feature can be determined according to the attributes of the object represented by the second target feature, thereby extracting the determined features from the second target feature, and the extracted features are the representative features.

[0284] Specifically, after determining that the second target feature has representative features for the object edge, the representative features can be extracted from the second target feature using a feature screening algorithm, a feature screening network, or an attention mechanism.

[0285] After the representative features are screened out, the implementation method of feature reconstruction based on the first target feature and the representative features can refer to the implementation method of feature reconstruction based on the first target feature and the second target feature in the aforementioned embodiment, which will not be repeated here.

[0286] See Figure 19, which has been adjusted based on Figure 18. The specific adjustments are as follows: During the first transparency information fusion process, when reconstructing features based on the first image feature and outline feature 1, first, representative feature 1 is selected from outline feature 1. Feature reconstruction is then performed based on the first image feature and representative feature 1, resulting in a scaled-up second image feature 1. During the second transparency information fusion process, when reconstructing features based on the third image feature 1 and outline feature 2, first, representative feature 2 is selected from outline feature 2. Feature reconstruction is then performed based on the third image feature 1 and representative feature 2, resulting in a further scaled-up second image feature 2.

[0287] From the above, it can be seen that in the solution provided by this embodiment, in the process of feature reconstruction based on the first target feature and the second target feature, representative features with smaller data volume are screened out from the second target feature, so that feature reconstruction is performed based on the first target feature and the representative feature. This can reduce the computational amount of feature reconstruction, thereby reducing the computational amount of performing a preset number of transparency information fusions, improving the efficiency of obtaining fusion results, and thereby improving the efficiency of video cutout.

[0288] The following describes the method of achieving feature reconstruction by feature reconstruction, feature fusion and updating mentioned in the embodiment shown in Figure 18.

[0289] In one embodiment of the present application, after obtaining the first image feature, contour information fusion may be performed a preset number of times in the following manner to obtain features of contour mask images of objects with increasing scales, wherein one contour information fusion process is equivalent to one feature reconstruction process:

[0290] Feature reconstruction is performed based on the third target feature to obtain a scale-enlarged fourth image feature, the fourth image feature is fused with the second sub-state information in the second hidden state information, and the second sub-state information is updated to obtain the feature of the object's contour mask image.

[0291] Among them, when the information fusion is performed for the first time, the third target feature is the first image feature, and when the information fusion is performed for other times, the third target feature is the feature obtained by the previous feature reconstruction.

[0292] The second hidden state information representation is: the fusion features of the contour mask image of the object in the video frame segmented before the target video frame.

[0293] The second hidden state information includes: multiple second sub-hidden state information, each second sub-hidden state information representing the fusion features of a contour mask image of a different scale. The multiple second sub-hidden state information may represent the fusion features of contour mask images of successively larger scales. For example, the second hidden state information may include three second sub-hidden state information, each of which may represent the fusion features of contour mask images of scales of 24*24, 28*28, and 32*32, respectively.

[0294] The scale of the contour mask image corresponding to the fused feature represented by the second sub-state information is the same as the scale of the fourth image feature. For example, if the second hidden state information includes three second sub-hidden state information, and these three second sub-hidden state information represent fused features of contour mask images with scales of 24*24, 28*28, and 32*32, respectively, then the preset number is three, and three fusions are required. The scales of the fourth image features obtained during the three fusions are 24*24, 28*28, and 32*32, respectively.

[0295] The following describes the above contour information fusion process in detail, assuming that the preset number is 3, in conjunction with FIG20 .

[0296] Referring to FIG. 20 , a flowchart illustrating a cascade feature reconstruction method is provided. In FIG. 20 , after obtaining the first image feature, a first contour information fusion is performed. During this first contour information fusion process, the third target feature is the first image feature, and feature reconstruction is performed based on the third target feature, that is, feature reconstruction is performed based on the first image feature to obtain a scaled fourth image feature 1. The fourth image feature 1 and the second sub-state information 1 corresponding to the fourth image feature 1 are fused, and the second sub-state information 1 is updated to obtain contour feature 3 of the object's contour mask image. At this point, the first contour information fusion process ends.

[0297] After obtaining contour feature 3, a second contour information fusion is performed. In this second contour information fusion process, the third target feature is contour feature 3. Feature reconstruction is performed based on the third target feature, that is, feature reconstruction is performed based on contour feature 3 to obtain a fourth image feature 2 with a further increase in scale. The fourth image feature 2 is then fused with the second sub-state information 2 corresponding to the fourth image feature 2, and the second sub-state information 2 is updated to obtain contour feature 4 of the object's contour mask image. At this point, the second contour information fusion process ends.

[0298] After obtaining contour feature 4, a third contour information fusion is performed. In the third contour information fusion process, the third target feature is contour feature 4. Feature reconstruction is performed based on the third target feature, that is, feature reconstruction is performed based on contour feature 4 to obtain a fourth image feature 3 with a further increase in scale. The fourth image feature 3 and the second sub-state information 3 corresponding to the fourth image feature 3 are fused and the second sub-state information 3 is updated to obtain contour feature 5 of the object's contour mask image. At this point, the second contour information fusion process ends, and the obtained contour feature 5 is the feature of the final contour mask image of the object.

[0299] The implementation method of feature reconstruction based on the third target feature in each contour information fusion process can be referred to the implementation method of feature reconstruction based on the first image feature in step S303 in the embodiment shown in Figure 15 above; the implementation method of fusing and updating the second sub-state information based on the fourth image feature and the second sub-state information can be referred to the implementation method of fusing and updating the first hidden state information based on the reconstructed first image feature and the first hidden state information in step S303 in the embodiment shown in Figure 15 above, which will not be repeated here.

[0300] As can be seen from the above, in the solution provided by this embodiment, the second hidden state information represents the fusion features of the contour mask image of the object in the video frame segmented before the target video frame, and each second sub-hidden state information represents the fusion features of the contour mask image of a scale. Therefore, in each contour information fusion process, the fourth image feature obtained by feature reconstruction is fused with the second sub-state information, which is to fuse the information in the fusion features of the contour mask image of a scale of the object in the video frame segmented before the target video frame with the fourth image feature. In this way, the features of the contour mask image of the object finally obtained not only include the information of the object in the target video frame, but also include the information of the object in the video frame segmented before the target video frame, so that the first contour mask image is obtained based on the finally obtained features, which can improve the accuracy of the first contour mask image, and thus improve the accuracy of the video cutout.

[0301] When performing feature reconstruction during the contour information fusion process, the processing may be based on the third target feature and may also be based on the third target feature and other information.

[0302] In one embodiment of the present application, the first image feature includes a plurality of first sub-image features. When compressing information of a target video frame in a video, the target video frame may be subjected to cascade information compression to obtain first sub-image features of successively smaller sizes.

[0303] Cascade information compression can be understood as multiple information compressions, where the result of each information compression is a first sub-image feature. Moreover, the object of the first information compression is the target video frame, and the objects of other information compressions are the first sub-image features obtained from the previous information compression.

[0304] The implementation method of each information compression can refer to the implementation method of information compression of the target video frame in step S301 shown in Figure 15.

[0305] For example, multiple convolution transformations may be performed on the object to be compressed each time information is compressed.

[0306] For another example, each time information is compressed, the information compression can be achieved by executing the processing flow shown in steps S301A-S301E in the embodiment shown in Figure 22 once or multiple times.

[0307] After obtaining the first sub-image features with successively decreasing scales, contour information fusion may be performed a preset number of times based on the first sub-image features.

[0308] When performing feature reconstruction in the first contour information fusion process, the first sub-image feature with the smallest scale among the first sub-image features may be used as the target feature, and feature reconstruction is performed based on the target feature.

[0309] When performing feature reconstruction in other contour information fusion processes, the feature obtained in the previous contour information fusion can be used as the third target feature, and feature reconstruction is performed based on the third target feature and the first sub-image feature with the same scale as the third target feature to obtain a fourth image feature with an increased scale.

[0310] When reconstructing features based on the third target feature and the first sub-image feature with the same scale as the third target feature, the third target feature and the first sub-image feature with the same scale as the third target feature can be fused into one feature through fusion methods such as superposition and dot product, and then feature reconstruction is performed based on the fused feature.

[0311] The following takes the preset number of 3 as an example and describes the contour information fusion process in detail in conjunction with FIG21 .

[0312] Referring to FIG21 , a flowchart of another cascade feature reconstruction method is provided. In FIG21 , after obtaining the first sub-image features of successively decreasing scales, the first contour information fusion is performed. During the first contour information fusion process, the third target feature is the first sub-image feature 1 with the smallest scale among the first sub-image features. Feature reconstruction is performed based on the third target feature, that is, feature reconstruction is performed based on the first sub-image feature 1 to obtain a fourth image feature 4 with an increased scale. The fourth image feature 4 is fused with the second sub-state information 4 corresponding to the fourth image feature 4, and the second sub-state information 4 is updated to obtain the contour feature 6 of the contour mask image of the object. At this point, the first contour information fusion process ends.

[0313] After obtaining the contour feature 6 in the first contour information fusion process, a second contour information fusion is performed. In the second contour information fusion process, the third target feature is the contour feature 6, and the first sub-image feature having the same scale as the third target feature is the first sub-image feature 2. Feature reconstruction is performed based on the third target feature and the first sub-image feature having the same scale as the third target feature. That is, feature reconstruction is performed based on the contour feature 6 and the first sub-image feature 2 to obtain a fourth image feature 5 with a further increased scale. The fourth image feature 5 is fused with the second sub-state information 5 corresponding to the fourth image feature 5, and the second sub-state information 5 is updated to obtain the contour feature 7 of the contour mask image of the object. At this point, the second contour information fusion process ends.

[0314] After obtaining the contour feature 7 in the second contour information fusion process, the third contour information fusion is performed. In the third contour information fusion process, the third target feature is the above-mentioned contour feature 7, and the first sub-image feature having the same scale as the third target feature is the first sub-image feature 3. Feature reconstruction is performed based on the third target feature and the first sub-image feature having the same scale as the third target feature. That is, feature reconstruction is performed based on the contour feature 7 and the first sub-image feature 3 to obtain a fourth image feature 6 with a further increased scale. The fourth image feature 6 is fused with the second sub-state information 6 corresponding to the fourth image feature 6, and the second sub-state information 6 is updated to obtain the contour feature 8 of the contour mask image of the object. At this point, the third contour information fusion process ends, and the contour feature 8 obtained in this process is the feature of the contour mask image of the object finally obtained.

[0315] As can be seen from the above, in the solution provided by this embodiment, the target video frame is cascaded information compressed to obtain the first sub-image features of successively decreasing scales. In the subsequent contour information fusion processes except the first one, feature reconstruction can be performed based on the third target feature and the first sub-image feature with the same scale as the third target feature. This can improve the accuracy of feature reconstruction, thereby improving the accuracy of the features finally obtained after contour information fusion, and further improving the accuracy of video cutout.

[0316] According to the above content, the number of transparency information fusion times and the number of contour information fusion times are both preset numbers. The larger the preset number, the more accurate the fusion result obtained by transparency information fusion and the more accurate the features obtained by contour information fusion. However, the amount of calculation is also greater.

[0317] In view of this, in one embodiment of the present application, the preset number is 4, 5 or 6. This can not only improve the accuracy of video cutout, but also avoid excessive calculation, thereby ensuring that video cutout is achieved with high efficiency, while also saving computing resources of the terminal. Therefore, the solution provided in the embodiment of the present application can be applied to the terminal and is friendly to the application of the solution on the terminal, thereby realizing a lightweight application of the video cutout solution in the terminal.

[0318] Since the data volume of the second image feature and the fourth image feature itself is relatively large, the computational complexity of fusing the second image feature with the first sub-state information in the first hidden state information and updating the first sub-state information is usually large, and the computational complexity of fusing the fourth image feature with the second sub-state information in the second hidden state information and updating the second sub-state information is usually also large.

[0319] In view of the above situation, in order to reduce the amount of calculation for fusing the second image feature and the first sub-state information and updating the first sub-state information, in one embodiment of the present application, when fusing the second image feature and the first sub-state information and updating the first sub-state information, the second image feature is divided to obtain the second sub-image feature and the third sub-image feature; the second sub-image feature and the first sub-state information in the first hidden state information are fused and the first sub-state information is updated to obtain the fourth sub-image feature; and the fourth sub-image feature and the third sub-image feature are spliced ​​to obtain the third image feature.

[0320] The features can be represented in the form of matrices or tensors. Taking a tensor as an example, segmenting the second image feature can be understood as segmenting the feature tensor representing the second image feature into two sub-tensors in any dimensional direction of the feature tensor.

[0321] For example, for a feature tensor with a scale of H*C*W, the feature tensor can be split in the W dimension direction to obtain two sub-feature tensors with scales of H*C*W1 and H*C*W2, where W1+W2=W.

[0322] Specifically, when segmenting the second image feature, the second image feature can be segmented in equal proportions to obtain two sub-features of equal scale, or the second image feature can be segmented in arbitrary proportions to obtain two sub-features of different scales. Furthermore, after segmenting the second image feature to obtain two sub-features, one of the sub-features can be determined as the second sub-image feature, and the other sub-feature can be determined as the third sub-image feature.

[0323] After the second sub-image feature and the third sub-image feature are obtained by segmentation, the second sub-image feature and the first sub-state information in the first hidden state information can be fused and the first sub-state information can be updated to obtain the fourth sub-image feature. The specific implementation method can be referred to the implementation method of step S303 in the embodiment shown in Figure 15 above to fuse the reconstructed feature and the first hidden state information and update the first hidden state information. It will not be repeated here.

[0324] The splicing feature can be regarded as the reverse processing of feature segmentation. After the fourth sub-image feature is obtained by fusion, the fourth sub-image feature and the third sub-image feature can be spliced. When splicing these two features, the fourth sub-image feature and the third sub-image feature can be spliced ​​into one feature in the dimensional direction based on the segmentation, that is, in the dimensional direction based on the segmentation, the third sub-image feature is spliced ​​behind the fourth sub-image feature, or the fourth sub-image feature is spliced ​​behind the third sub-image feature. The feature obtained by splicing is the third image feature.

[0325] For example, if the scale of the third sub-image feature is H*C*W3 and the scale of the fourth sub-image feature is H*C*W4, then when splicing the fourth sub-image feature and the third sub-image feature, the fourth sub-image feature and the third sub-image feature can be spliced ​​into a feature with a scale of H*C*W5 in the W dimension direction, where W3+W4=W5.

[0326] As can be seen from the above, in the solution provided by this embodiment, the second image feature is segmented to obtain the second sub-image feature and the third sub-image feature. The data volume of the second sub-image feature and the third sub-image feature are both smaller than the data volume of the second image feature. In this way, the second sub-image feature and the first sub-state information are fused, which can reduce the computational complexity of the fusion and improve the fusion efficiency, thereby improving the efficiency of obtaining the third image feature, and further improving the efficiency of video cutout. At the same time, it also saves the computing resources of the terminal, thereby realizing a lightweight application of the video cutout solution in the terminal.

[0327] In order to reduce the amount of calculation for fusing the fourth image feature and the second sub-state information and updating the second sub-state information, in one embodiment of the present application, when fusing the fourth image feature and the second sub-state information and updating the second sub-state information, the fourth image feature is divided to obtain the fifth sub-image feature and the sixth sub-image feature; the fifth sub-image feature and the second sub-state information in the second hidden state information are fused and the second sub-state information is updated to obtain the seventh sub-image feature; the seventh sub-image feature and the sixth sub-image feature are spliced ​​to obtain the features of the contour mask image of the object.

[0328] For the implementation method of segmenting the fourth image feature, please refer to the aforementioned implementation method of segmenting the second image; for the implementation method of fusing the fifth sub-image feature with the second sub-state information and updating the second sub-state information, please refer to the aforementioned implementation method of fusing the second sub-image feature with the first sub-state information and updating the first sub-state information; for the implementation method of splicing the seventh sub-image feature and the sixth sub-image feature, please refer to the aforementioned implementation method of splicing the fourth sub-image feature and the third sub-image feature, which will not be repeated here.

[0329] As can be seen from the above, in the solution provided by this embodiment, the fourth image feature is segmented to obtain the fifth sub-image feature and the sixth sub-image feature. The data volume of the fifth sub-image feature and the sixth sub-image feature are both smaller than the data volume of the fourth image feature. In this way, the fifth sub-image feature and the first sub-state information are fused, which can reduce the amount of fusion calculation and improve the fusion efficiency, thereby improving the efficiency of obtaining the features of the contour mask image of the object, and further improving the efficiency of video cutout. At the same time, it also saves the computing resources of the terminal, thereby realizing a lightweight application of the video cutout solution in the terminal.

[0330] Furthermore, to address the computationally intensive nature of both fusion processes, in one embodiment of the present application, when fusing the second image feature with the first sub-state information and updating the first sub-state information, the second image feature can be segmented to obtain a second sub-image feature and a third sub-image feature; the second sub-image feature can be fused with the first sub-state information in the first hidden state information and updating the first sub-state information to obtain a fourth sub-image feature; and the fourth sub-image feature and the third sub-image feature can be concatenated to obtain a third image feature. Furthermore, when fusing the fourth image feature with the second sub-state information and updating the second sub-state information, the fourth image feature can be segmented to obtain a fifth sub-image feature and a sixth sub-image feature; the fifth sub-image feature can be fused with the second sub-state information in the second hidden state information and updating the second sub-state information to obtain a seventh sub-image feature; and the seventh sub-image feature and the sixth sub-image feature can be concatenated to obtain the features of the object's contour mask image. This can reduce the computational complexity of both fusion processes, thereby further improving fusion efficiency, and thus further improving video cutout efficiency, thereby achieving a lightweight video cutout solution for terminal applications.

[0331] The following describes an implementation method for compressing information on the target video frame in the above step S301 by combining convolution transformation, linear transformation, batch normalization processing, nonlinear transformation and other processing.

[0332] In one embodiment of the present application, referring to FIG. 22 , a flow chart of a third video cutout method is provided. In this embodiment, the above-mentioned step S301 can be implemented through the following steps S301A-S301E.

[0333] Step S301A: performing a convolution transformation on the target video frame in the video to obtain a fifth image feature.

[0334] Specifically, a pre-set convolution kernel can be used to perform convolution calculation on the target video frame to obtain the fifth image feature, or a trained convolutional neural network can be used to perform convolution transformation on the target video frame to obtain the fifth image feature output by the model.

[0335] Step S301B: performing a linear transformation on the fifth image feature based on the convolution kernel to obtain a sixth image feature.

[0336] Among them, the above convolution kernel is a pre-set convolution kernel.

[0337] Specifically, based on the convolution kernel, the fifth image feature can be linearly transformed by performing a convolution transformation on the fifth image feature. Since the network processor (NPU) in the terminal has strong computing power for convolution transformation, performing linear transformation using convolution transformation can shorten the time consumption of linear transformation, thereby shortening the time consumption of video cutout and improving the efficiency of video cutout.

[0338] In one embodiment of the present application, the above-mentioned convolution kernel is: a 1x1 convolution kernel. Since the 1x1 convolution kernel itself has a small amount of data, the fifth image feature is linearly transformed based on the 1x1 convolution kernel. On the premise that the fifth image feature can be linearly transformed, the computational amount of the linear transformation can be reduced, the computational efficiency of the linear transformation can be improved, and thus the efficiency of video cutout can be improved. In addition, the video cutout solution provided by this embodiment is applied to the terminal. In the terminal, the fifth image feature is linearly transformed based on the 1x1 convolution kernel. There is no need to occupy more computing resources of the terminal, thereby facilitating the terminal to achieve linear transformation and promoting lightweight video cutout on the terminal side.

[0339] Step S301C: performing batch normalization processing on the sixth image feature to obtain the seventh image feature.

[0340] Specifically, a batch normalization algorithm, model, etc. may be used to perform batch normalization processing on the sixth image feature to obtain the seventh image feature.

[0341] For example, the BatchNorm2d algorithm may be used to perform batch normalization processing on the sixth image feature.

[0342] Step S301D: performing a nonlinear transformation on the seventh image feature to obtain an eighth image feature.

[0343] Specifically, a nonlinear transformation function, algorithm, activation function, etc. may be used to perform a nonlinear transformation on the seventh image feature to obtain the eighth image feature.

[0344] For example, the GELU activation function or the RELU activation function can be used to perform a nonlinear transformation on the seventh image feature. When the RELU activation function is used to perform a nonlinear transformation on the seventh image feature, since the RELU activation function has a better quantization effect on data processing, the use of the RELU activation function to perform a nonlinear transformation on the seventh image feature can improve the transformation effect of the nonlinear transformation, thereby improving the accuracy of the eighth image feature.

[0345] Step S301E: performing a linear transformation on the eighth image feature based on the convolution kernel to obtain a first feature of the target video frame.

[0346] The implementation method of the linear transformation in this step is the same as the implementation method of the linear transformation in the above step S301B, and will not be repeated here.

[0347] In addition, when obtaining the first feature, the process shown in steps S301A-S301E may be performed once, or multiple times, for example, four, five, or any other number of times.

[0348] When the processing flow shown in steps S301A-S301E is executed multiple times, the input of the first processing flow is the target video frame in the video, and the input of other processing flows is the features output by the previous processing flow. The feature output by the last processing flow is the above-mentioned first feature. In this case, as the above-mentioned processing flow is executed multiple times, the scale of the features output by each processing flow continues to decrease.

[0349] As can be seen from the above, in the solution provided by this embodiment, when compressing the information of the target video frame, the target video frame is subjected to multiple processing such as convolution transformation, linear transformation, batch normalization processing, nonlinear transformation, etc., so that more accurate information compression of the target video can be achieved, thereby improving the accuracy of the first image feature, and then performing video segmentation based on the first image feature, which can improve the accuracy of video cutout.

[0350] In addition, in the solution provided in the embodiment of the present application, the sixth image feature is first batch-normalized and then the seventh image feature obtained by processing is nonlinearly transformed. This can prevent the loss of feature quantization accuracy during information compression, thereby improving the quantization accuracy of information compression, and further improving the accuracy of the first image feature and the accuracy of video cutout.

[0351] The solution provided in the embodiment of the present application is applied to the terminal, and convolution transformation, linear transformation, batch normalization processing, nonlinear transformation and other processing are relatively friendly to the terminal's computing power. Therefore, performing convolution transformation, linear transformation, batch normalization processing, nonlinear transformation and other processing in the terminal can facilitate the terminal to compress information, thereby promoting lightweight video cutout on the terminal side.

[0352] The video cutout solution provided in the embodiment of the present application can also be implemented based on a neural network model. The video cutout solution is described below in conjunction with the neural network model.

[0353] In one embodiment of the present application, the above steps can be implemented using a pre-trained video cutout model.

[0354] Referring to Figure 23, a structural schematic diagram of the first video cutout model is provided. It can be seen from Figure 23 that the video cutout model includes an information compression network, a first image generation network, a second image generation network, a result output network, three groups of contour feature generation networks, and three groups of transparency feature generation networks. Each group of contour feature generation networks corresponds to the scale of a contour mask image, including a first reconstruction subnetwork and a first fusion subnetwork. Each group of transparency feature generation networks corresponds to the scale of a transparency mask image, including a second reconstruction subnetwork and a second fusion subnetwork.

[0355] Figure 23 shows a video cutout model in which the number of contour feature generation networks included is three. In addition, the number of contour feature generation networks included in the video cutout model can also be four, five or other numbers. The number of transparency feature generation networks is the same as that of contour feature generation networks, and this embodiment does not limit this.

[0356] The following describes the connection relationship between the various networks in the video cutout model shown in FIG23 .

[0357] The three groups of contour feature generation networks in the video cutout model are contour feature generation network 1, contour feature generation network 2 and contour feature generation network 3, whose scales of the corresponding contour mask images increase in sequence. The first reconstruction subnetwork contained in each group of contour feature generation networks is connected to the first fusion subnetwork; the three groups of transparency feature generation networks are transparency feature generation network 1, transparency feature generation network 2 and transparency feature generation network 3, whose scales of the corresponding transparency mask images increase in sequence. The second reconstruction subnetwork contained in each group of transparency feature generation networks is connected to the second fusion subnetwork.

[0358] The video cutout model consists of two branches: the segmentation branch and the cutout branch. The segmentation branch primarily includes three contour feature generation networks and a first image generation network, while the cutout branch primarily includes three transparency feature generation networks and a second image generation network. The first layer of the video cutout model is an information compression network, which is connected to both branches and is also interconnected.

[0359] The following describes the connection relationship between the networks contained in the segmentation branch and the cutout branch, as well as the connection relationship between the two branches.

[0360] First, the connection relationship of the networks included in the segmentation branch itself is explained. The information compression network is connected to the first reconstruction subnetwork 1 included in the contour feature generation network 1, the first fusion subnetwork 1 included in the contour feature generation network 1 is connected to the first reconstruction subnetwork 2 included in the contour feature generation network 2, the first fusion subnetwork 2 included in the contour feature generation network 2 is connected to the first reconstruction subnetwork 3 included in the contour feature generation network 3, and the first fusion subnetwork 3 included in the contour feature generation network 3 is connected to the first image generation network.

[0361] Next, the connection relationship between the networks included in the cutout branch itself is explained. The information compression network is connected to the second reconstruction subnetwork 1 included in the transparency feature generation network 1, the second fusion subnetwork 1 included in the transparency feature generation network 1 is connected to the second reconstruction subnetwork 2 included in the transparency feature generation network 2, the second fusion subnetwork 2 included in the transparency feature generation network 2 is connected to the second reconstruction subnetwork 3 included in the transparency feature generation network 3, and the second fusion subnetwork 3 included in the transparency feature generation network 3 is connected to the second image generation network.

[0362] Finally, the connection between the segmentation branch and the cutout branch is explained. The first fusion subnetwork 1 included in the contour feature generation network 1 is connected to the second reconstruction subnetwork 1 included in the transparency feature generation network 1; the first fusion subnetwork 2 included in the contour feature generation network 2 is connected to the second reconstruction subnetwork 2 included in the transparency feature generation network 2; and the first fusion subnetwork 3 included in the contour feature generation network 3 is connected to the second reconstruction subnetwork 3 included in the transparency feature generation network 3. Furthermore, the first image generation network and the second image generation network are each connected to the result output network.

[0363] The following describes each network and sub-network in the video cutout model.

[0364] For the information compression network, when compressing information on a target video frame, the target video frame is input into the information compression network, and the information compression network compresses information on the target video frame, thereby obtaining a first feature output by the information compression network.

[0365] The implementation method of the information compression network for compressing the target video frame can be found in the above content and will not be repeated here.

[0366] Refer to Figure 24, which is a structural diagram of an information compression network. In the information compression network shown in Figure 24, the network layers are, from top to bottom: convolutional layer, linear layer 1, batch normalization layer, nonlinear layer and linear layer 2.

[0367] Among them, the convolution layer is used to perform convolution transformation on the target video frame to obtain the fifth image feature.

[0368] The linear layer 1 is used to perform a linear transformation on the fifth image feature based on the convolution kernel to obtain the sixth image feature.

[0369] The batch normalization layer is used to perform batch normalization on the sixth image feature to obtain the seventh image feature.

[0370] The nonlinear layer is used to perform a nonlinear transformation on the seventh image feature to obtain the eighth image feature.

[0371] The linear layer 2 is used to perform a linear transformation on the eighth image feature based on the convolution kernel to obtain the first feature.

[0372] The implementation method of data processing by the convolution layer, linear layer 1, batch normalization layer, nonlinear layer and linear layer 2 can be found in the above content and will not be repeated here.

[0373] When reconstructing features based on the third target feature, the first target reconstruction subnetwork in the target contour feature generation network inputs the third target feature into the first target reconstruction subnetwork, which then performs feature reconstruction based on the third target feature, thereby obtaining a scaled fourth image feature output by the first target reconstruction subnetwork. The scale of the contour mask image corresponding to the target contour feature generation network is the same as the scale of the fourth image feature.

[0374] The implementation method of the target first reconstruction sub-network performing feature reconstruction based on the third target feature can be found in the above embodiment and will not be repeated here.

[0375] In one embodiment of the present application, the first reconstruction sub-network is implemented based on the QARepVGG network structure.

[0376] Since the quantization calculation accuracy of the QARepVGG network is relatively high, implementing the above-mentioned first reconstruction subnetwork based on the QARepVGG network structure can improve the quantization calculation capability of the first reconstruction subnetwork, thereby improving the accuracy of feature reconstruction performed by the first reconstruction subnetwork, and further improving the accuracy of video cutout.

[0377] In another embodiment of the present application, the first reconstruction subnetwork in the specific contour feature generation network is implemented based on the QARepVGG network structure.

[0378] The specific contour feature generation network is a contour feature generation network whose corresponding contour mask image has a scale smaller than a first preset scale.

[0379] The first preset scale may be a pre-set scale.

[0380] When constructing the above-mentioned video cutout model, the scale of the contour mask image corresponding to each set of contour feature generation networks can be determined. In this way, the contour feature generation network whose corresponding contour mask image scale is smaller than the first preset scale can be determined as a specific contour feature generation network. Therefore, when constructing the specific contour feature generation network, the specific contour feature generation network is constructed based on the QARepVGG network structure. Other contour feature generation networks can be constructed based on other network structures.

[0381] Since the computational cost of the U-shaped residual block in the specific contour feature generation network constructed based on the QARepVGG network structure increases as the scale of the contour mask image corresponding to the network increases, when constructing each contour feature generation network, only the specific contour feature generation network whose corresponding contour mask image has a scale smaller than the first preset scale can be used, and the first reconstruction subnetwork in the specific contour feature generation network can be implemented based on the QARepVGG network structure. This can reduce the computational cost of each contour feature generation network, improve the efficiency of obtaining the features of the contour mask image of the object, thereby improving the efficiency of video cutout, and can also enable lightweight deployment of the above-mentioned video cutout model in the terminal.

[0382] For the target first fusion sub-network in the target contour feature generation network, when the fourth image feature and the second sub-state information are fused and the second sub-state information is updated, the fourth image feature can be input into the target first fusion sub-network, and the second sub-hidden state information provided by the target first fusion sub-network is the second sub-state information. In this way, the target first fusion sub-network fuses the fourth image feature and the second sub-hidden state information provided by itself and updates the second sub-hidden state information provided by itself, thereby obtaining the features of the contour mask image of the object output by the target first fusion sub-network.

[0383] The implementation manner in which the target first fusion sub-network fuses the fourth image feature and the second sub-hidden state information and updates the second sub-hidden state information can be referred to the aforementioned embodiment and will not be repeated here.

[0384] In one embodiment of the present application, the first fusion sub-network is a gated recurrent unit (GRU) or a long short-term memory (LSTM) unit.

[0385] Both GRU and LSTM units have information memory functions. If either of these two units is used as the first fusion sub-network, the unit itself can store hidden state information of the fusion features of the contour mask image representing the object in the segmented video frame, so that the fourth image feature and the second sub-hidden state information provided by itself can be accurately fused, thereby improving the accuracy of the features of the contour mask image of the object output by the sub-network, thereby improving the accuracy of the video cutout.

[0386] For the above-mentioned first image generation network, when obtaining the first contour mask image based on the features of the contour mask image of the object finally obtained, the features of the contour mask image of the object finally obtained are input into the first image generation network, and the first image generation network generates an image based on the features, thereby obtaining the image output by the network as the above-mentioned first contour mask image.

[0387] The implementation method of the first image generation network generating an image based on the feature can be found in the aforementioned embodiment and will not be repeated here.

[0388] For the target second reconstruction subnetwork in the target transparency feature generation network, when performing feature reconstruction based on the first target feature and the second target feature, the first target feature and the second target feature are input into the target second reconstruction subnetwork, and the target second reconstruction subnetwork performs feature reconstruction based on the first target feature and the second target feature, thereby obtaining a scale-enlarged second image feature output by the target second reconstruction subnetwork, wherein the scale of the transparency mask image corresponding to the target transparency feature generation network is the same as the scale of the second image feature.

[0389] The implementation method of the target second reconstruction sub-network performing feature reconstruction based on the first target feature and the second target feature can be found in the above content and will not be repeated here.

[0390] In one embodiment of the present application, the second reconstruction sub-network is implemented based on the QARepVGG network structure.

[0391] Since the quantization calculation accuracy of the QARepVGG network is relatively high, implementing the above-mentioned second reconstruction subnetwork based on the QARepVGG network structure can improve the quantization calculation capability of the second reconstruction subnetwork, thereby improving the accuracy of the feature reconstruction performed by the second reconstruction subnetwork based on the first target feature and the second target feature, thereby improving the accuracy of the video cutout.

[0392] In another embodiment of the present application, the second reconstruction subnetwork in the specific transparency feature generation network is implemented based on the QARepVGG network structure.

[0393] The specific transparency feature generation network is a transparency feature generation network whose corresponding transparency mask image has a scale smaller than a second preset scale.

[0394] The second preset scale may be a pre-set scale, and may be the same as or different from the first preset scale.

[0395] When constructing the aforementioned video matting model, the scale of the transparency mask image corresponding to each set of transparency feature generation networks can be determined. This allows the transparency feature generation networks whose corresponding transparency mask image scale is smaller than a second preset scale to be identified as specific transparency feature generation networks. Consequently, when constructing the specific transparency feature generation networks, the specific transparency feature generation networks can be constructed based on the QARepVGG network structure. Other transparency feature generation networks can be constructed based on other network structures.

[0396] Since the computational complexity of the U-shaped residual block in the specific transparency feature generation network constructed based on the QARepVGG network structure increases as the scale of the transparency mask image corresponding to the network increases, when constructing each transparency feature generation network, only the specific transparency feature generation network whose scale of the corresponding transparency mask image is smaller than the second preset scale can be used to implement the second reconstruction subnetwork in the specific transparency feature generation network based on the QARepVGG network structure. This can reduce the computational complexity of each transparency feature generation network, improve the efficiency of obtaining the fusion result, thereby improving the efficiency of video cutout, and also enable lightweight deployment of the above-mentioned video cutout model in the terminal.

[0397] For the target second fusion subnetwork in the target transparency feature generation network, when the second image feature and the first sub-state information are fused and the first sub-state information is updated, the second image feature can be input into the target second fusion subnetwork, and the first sub-hidden state information provided by the target second fusion subnetwork is the first sub-state information. In this way, the target second fusion subnetwork fuses the second image feature and the first sub-hidden state information provided by itself and updates the first sub-hidden state information provided by itself, thereby obtaining the third image feature output by the target second fusion subnetwork.

[0398] The implementation manner in which the target second fusion sub-network fuses the second image feature and the first sub-hidden state information and updates the first sub-hidden state information can be referred to the aforementioned embodiment and will not be repeated here.

[0399] In one embodiment of the present application, the second fusion sub-network is a gated recurrent unit (GRU) or a long short-term memory (LSTM) unit.

[0400] Both GRU and LSTM units have information memory functions. If either of these two units is used as the second fusion sub-network, the unit itself can store the hidden state information of the fusion features of the transparency mask image representing the object in the segmented video frame, so that the second image features and the first sub-hidden state information provided by itself can be accurately fused, thereby improving the accuracy of the third image features output by the sub-network, thereby improving the accuracy of video cutout.

[0401] For the above-mentioned second image generation network, when the first contour mask image is obtained based on the final fusion result, the fusion result is input into the second image generation network, and the second image generation network generates an image based on the fusion result, thereby obtaining the image output by the network as the above-mentioned target transparency mask image.

[0402] The implementation method of the second image generation network generating an image based on the fusion result can be found in the aforementioned embodiment and will not be repeated here.

[0403] For the result output network, when performing region cutout, the target transparency mask image and the first contour mask image are input into the result output network, and the result output network performs region cutout on the target video frame according to the target transparency mask image and the first contour mask image to obtain the cutout result output by the result output network.

[0404] The implementation method of the result output network for performing region cutout on the target video frame can be found in the aforementioned embodiment and will not be described in detail here.

[0405] As can be seen from the above, in the solution provided by this embodiment, the various networks and sub-networks contained in the video cutout model are used to perform video cutout. Since the video cutout model is a pre-trained video cutout model, the use of the video cutout model can improve the accuracy of video cutout. Moreover, the video cutout model does not require any interaction with other devices. Therefore, the video cutout model can be deployed in an offline device, which can improve the convenience of video segmentation.

[0406] Since the data volume of the first target feature and the second target feature itself is usually large, the computational complexity of the target second reconstruction sub-network for feature reconstruction is large, which results in low efficiency of feature reconstruction by the second reconstruction sub-network, and thus low efficiency of video cutout using the video cutout model.

[0407] In view of this, in one embodiment of the present application, referring to Figure 25, a structural diagram of a second video cutout model is provided. Compared with Figure 23, in the video cutout model shown in Figure 25, each transparency feature generation network also includes a feature screening subnetwork, the first fusion subnetwork 1 is connected to the second reconstruction subnetwork 1 through the feature screening subnetwork 1, the first fusion subnetwork 2 is connected to the second reconstruction subnetwork 2 through the feature screening subnetwork 2, and the first fusion subnetwork 3 is connected to the second reconstruction subnetwork 3 through the feature screening subnetwork 3.

[0408] For the target feature screening subnetwork in the target transparency feature generation network, before the feature output by the target first fusion subnetwork is input into the target second reconstruction subnetwork, the feature can be input into the target feature screening subnetwork, and the target feature screening subnetwork filters out target screening features that are representative of the edge contour of the object from the feature, and inputs the target screening features into the target second reconstruction subnetwork in the target transparency feature generation network, and the target second reconstruction subnetwork performs feature reconstruction based on the target screening features.

[0409] Specifically, the implementation method of the target feature screening subnetwork screening features, and the implementation method of the target second reconstruction subnetwork performing feature reconstruction based on the target screening features and the first target features can be referred to the above embodiments and will not be repeated here.

[0410] From the above, it can be seen that in the solution provided by this embodiment, by adding a feature screening subnetwork to the transparency feature generation network, the computational complexity of the second reconstruction subnetwork in the transparency feature generation network for feature reconstruction can be reduced, and the efficiency of the second reconstruction subnetwork for feature reconstruction can be improved, thereby improving the efficiency of the video cutout model for video cutout.

[0411] In one embodiment of the present application, referring to FIG26 , a flow chart of a fourth video cutout method is provided. In FIG26 , the video cutout model processes the video frame 1 and the video frame 2 contained in the video in sequence. When the video cutout model processes the video frame 1, the video frame 1 is processed by each network and sub-network in the model respectively to obtain the cutout result 1 corresponding to the video frame 1, wherein each first fusion sub-network and second fusion sub-network in the model outputs information to other network layers on the one hand, and updates the hidden state information contained in itself on the other hand, which is used for the model to fuse with the input features when processing the next frame of video frame 2. When the video cutout model processes the video frame 2, the video frame 2 is processed by each network and sub-network in the model respectively to obtain the cutout result 2 corresponding to the video frame 2.

[0412] In one embodiment of the present application, when the video cutout model includes a large number of contour feature generation networks and transparency feature generation networks, the computational complexity of the video cutout model in processing video frames is also large. In view of this, the first fusion sub-network in the last group or groups of contour feature generation networks included in the video cutout model can be removed, and the second fusion sub-network in the last group or groups of transparency feature generation networks included in the video cutout model can also be removed, thereby reducing the computational complexity of the video cutout model in processing video frames, and also enabling lightweight deployment of the video cutout model in the terminal.

[0413] Taking the first fusion sub-network in the network for generating the last set of contour features as an example, see Figure 27, which provides a structural diagram of the third video cutout model. Compared with the video cutout model shown in Figure 23, the last set of contour feature generation network 3 in the video cutout model shown in Figure 27 only includes the first reconstruction sub-network 3, and the output result of the first reconstruction sub-network 3 is the feature of the contour mask image of the object finally obtained.

[0414] In one embodiment of the present application, referring to FIG28 , a structural diagram of a fourth video cutout model is provided. The video cutout model shown in FIG26 includes a multi-layer cascaded information compression network, the output result of each layer of the information compression network is a first sub-feature, and one layer of the information compression network is connected to the first reconstruction sub-network in a group of contour feature generation networks, and the last layer of the information compression network is connected to the first group of contour feature generation networks. The scale of the first sub-feature output by the connected information compression network is the same as the scale of the third target feature to be processed by the contour feature generation network. The first sub-feature output by the last layer of the information compression network serves as the third target feature to be processed by the first group of contour feature generation networks. The first reconstruction sub-network in other groups of contour feature generation networks performs feature reconstruction based on the third target feature and the first sub-feature output by the information compression network connected to the network, thereby improving the accuracy of feature reconstruction and improving the accuracy of video cutout.

[0415] The training process of the above video cutout model is described below.

[0416] In one embodiment of the present application, referring to FIG29 , a flow chart of a first model training method is provided. In this embodiment, the method includes the following steps S1701-S1706.

[0417] Step S1701: input the first sample video frame in the sample video into the initial model of the video cutout model for processing, and obtain a first sample contour mask image of the object in the first sample video frame output by the first image generation network in the initial model.

[0418] The sample video may be any video obtained through the Internet, a video library, or other channels. In addition, after obtaining a video through the Internet, a video library, or other channels, multiple videos may be spliced ​​into one video to obtain the spliced ​​video as the sample video.

[0419] The first sample contour mask image has the same scale as the first sample video frame, and the pixel values ​​of the pixels in the first sample contour mask image represent the confidence level predicted by the model that the pixels at the same position in the first sample video frame belong to the area where the object is located.

[0420] The initial model is used to process the video frames input to the model according to its own configured untrained model parameters. During the process of the model processing the video frames, the image output by the first image generation network in the model can be obtained.

[0421] Specifically, after the first sample video frame is input into the initial model, the initial model can process the first sample video frame according to the model parameters configured by itself, and obtain the image output by the first image generation network in the model as the first sample contour mask image of the object in the first sample video frame.

[0422] Step S1702: Obtain a first difference between the annotated mask image corresponding to the first sample video frame and the annotated mask image corresponding to the second sample video frame.

[0423] The second sample video frame is a video frame in the sample video that precedes the first sample video frame and is separated by a preset number of frames.

[0424] The preset frame number is a pre-set frame number, for example, 3 frames, 4 frames or other numerical frames.

[0425] The first sample video frame may be any video frame that is a preset number of video frames or a video frame after a preset number of video frames in the sample video.

[0426] The first difference may be calculated by the terminal itself, or may be calculated by another device, and the terminal device then obtains the calculated first difference from the other device.

[0427] The following describes how the terminal or other device calculates the first difference.

[0428] The terminal or other device can obtain the annotated mask image corresponding to each sample video frame in the sample video. The annotated mask image corresponding to the sample video frame can be understood as the actual mask image of the object in the sample video frame. In this way, after determining the second sample video frame based on the frame number of the first sample video frame and the preset frame number, the annotated mask image corresponding to the first sample video frame and the annotated mask image corresponding to the second sample video frame can be obtained from the annotated mask images corresponding to each obtained sample video frame, thereby calculating the first difference between the two obtained annotated mask images.

[0429] When calculating the first difference between two annotated mask images, in one implementation method, the pixel values ​​of the pixels at the same position in the two images can be subtracted, and the number of results that are not "0" in the calculation results of each pixel point can be counted as the above-mentioned first difference, or the ratio of the number of results that are not "0" to the total number of pixels in the annotated mask image can be counted as the above-mentioned first difference; in another implementation method, the similarity of the two images can be calculated, and the calculated similarity can be subtracted from 1 to obtain the calculation result as the above-mentioned first difference.

[0430] Step S1703: Obtain a second difference between the first sample contour mask image and the second sample contour mask image, wherein the second sample contour mask image is: a mask image output by the first image generation network when the initial model processes the second sample video frame.

[0431] Specifically, similar to the aforementioned video matting process, during model training, each sample video frame in the sample video can be input into the model frame by frame to obtain a sample contour mask image of the object in each sample video frame output by the first image generation network in the model. After obtaining the first sample contour mask image, a second sample video frame that is a preset number of frames away from the first sample video frame can be determined from the video frames that precede the first sample video frame and have been processed by the model. The second sample contour mask image output by the first image generation network when the model processes the second sample video frame can then be obtained, and a second difference between the first sample contour mask image and the second sample contour mask image can be calculated.

[0432] The implementation method of calculating the above-mentioned second difference is the same as the implementation method of calculating the first difference in the aforementioned step S1702, and will not be repeated here.

[0433] Step S1704: Obtain a third difference between the first sample transparency mask image and the second sample transparency mask image.

[0434] The first sample transparency mask image is: a mask image output by the second image generation network when the initial model processes the first sample video frame.

[0435] The second sample transparency mask image is: the mask image output by the second image generation network when the initial model processes the second sample video frame.

[0436] Specifically, after the first sample video frame is input into the initial model, the initial model can process the first sample video frame according to the model parameters configured by itself, and obtain the image output by the second image generation network in the model as the above-mentioned first sample transparency mask image; after the second sample video frame is input into the initial model, the initial model can process the second sample video frame according to the model parameters configured by itself, and obtain the image output by the second image generation network in the model as the above-mentioned second sample transparency mask image. In this way, after obtaining the first sample transparency mask image and the second sample transparency mask image, the third difference between the first sample transparency mask image and the second sample transparency mask image can be calculated.

[0437] The implementation method of calculating the third difference is the same as the implementation method of calculating the first difference in the aforementioned step S1702, and will not be repeated here.

[0438] Step S1705: Calculate the training loss based on the first difference, the second difference, and the third difference.

[0439] The training loss is calculated based on the first difference, the second difference, and the third difference. The training loss can be calculated using a loss function, an algorithm, etc.

[0440] Step S1706: Based on the training loss, adjust the model parameters of the initial model to obtain a video cutout model.

[0441] Specifically, based on the training loss, the model parameters of the initial model can be adjusted by any of the following three implementation methods.

[0442] In the first implementation method, for each model parameter in the initial model, the correspondence between the training loss and the adjustment range of the model parameter can be set in advance. In this way, after calculating the training loss, the actual adjustment range of the model parameter can be calculated according to the correspondence, and the model parameter can be adjusted according to the actual adjustment range.

[0443] In the second implementation method, the initial model usually needs to be trained using a large amount of sample data. During the training process, the training loss needs to be continuously calculated, and the model parameters of the initial model need to be continuously adjusted based on the training loss. In view of this, after calculating the training loss, the training loss change difference can be determined based on the training loss and the previously calculated training loss, and then the model parameters of the initial model can be adjusted based on the change difference.

[0444] In the third implementation method, based on the training loss, the model parameter adjustment algorithm, function, etc. can be used to adjust the model parameters of the initial model.

[0445] As can be seen from the above, in the solution provided by this embodiment, since there is often a time domain correlation between the first sample video frame and the second sample video frame separated by a preset number of frames, a first difference between the annotated mask image corresponding to the first sample video frame and the annotated mask image corresponding to the second sample video frame, a second difference between the first sample contour mask image and the second sample contour mask image, and a third difference between the first sample transparency mask image and the second sample transparency mask image are obtained. The training loss is calculated based on the first difference, the second difference and the third difference. When the model parameters of the initial model are adjusted based on the training loss, the initial model can learn the time domain correlation between different video frames of the video, thereby improving the accuracy of the trained model, and then using the model to perform video cutout, which can improve the accuracy of video cutout.

[0446] When obtaining the second difference, in addition to the method mentioned in step S1703, it can also be obtained by the method mentioned in step S1703A in the embodiment shown in FIG. 30 .

[0447] In one embodiment of the present application, referring to FIG30 , a flow chart of a second model training method is provided.

[0448] In this embodiment, the first sample contour mask image includes: a first mask sub-image identifying the area where the object is located in the first sample video frame and a second mask sub-image identifying the area outside the object in the first sample video frame.

[0449] The pixel values ​​of the pixels in the first mask sub-image represent the confidence that the pixel at the same position in the first sample video frame predicted by the model belongs to the area where the object is located, and the pixel values ​​of the pixels in the second mask sub-image represent the confidence that the pixel at the same position in the first sample video frame predicted by the model belongs to the area outside the object.

[0450] As shown in FIG31 , FIG31 is a mask image provided in an embodiment of the present application, and the mask image is a first sample mask image.

[0451] The mask image shown in FIG31 includes two sub-images: a first mask sub-image identifying the area where the object is located in the first sample video frame and a second mask sub-image identifying the area outside the object in the first sample video frame.

[0452] The second sample contour mask image includes: a third mask sub-image identifying the area where the object is located in the second sample video frame and a fourth mask sub-image identifying the area outside the object in the second sample video frame.

[0453] The pixel values ​​of the pixels in the third mask sub-image represent the confidence that the pixel at the same position in the second sample video frame predicted by the model belongs to the area where the object is located, and the pixel values ​​of the pixels in the fourth mask sub-image represent the confidence that the pixel at the same position in the second sample video frame predicted by the model belongs to the area outside the object.

[0454] In this case, the above step S1703 can be implemented by the following step S1703A.

[0455] Step S1703A: Obtain a difference between the first mask sub-image and the third mask sub-image, and obtain a difference between the second mask sub-image and the fourth mask sub-image, to obtain a second difference including the obtained differences.

[0456] The implementation manner of obtaining the difference between the first mask sub-image and the third mask sub-image and the difference between the second mask sub-image and the fourth mask sub-image is the same as the implementation manner of obtaining the first difference or the second difference, which will not be repeated here.

[0457] After obtaining the two differences, namely the difference between the first mask sub-image and the third mask sub-image and the difference between the second mask sub-image and the fourth mask sub-image, the two differences can be accumulated to obtain a second difference containing the two differences. Alternatively, the average value of the two differences can be used as the second difference. Alternatively, the larger difference between the two differences can be determined as the second difference, and so on.

[0458] From the above, it can be seen that in the solution provided by this embodiment, since the area in the video frame is composed of two types of areas, namely the area where the object is located and the area outside the object, the greater the difference in the area where the object is located in different video frames, the greater the difference in the area outside the object in different video frames. It can be seen that the difference in the area outside the object can also reflect the difference in the area where the object is located. Therefore, the second difference is obtained based on the two differences, namely the difference between the first mask sub-image and the third mask sub-image and the difference between the second mask sub-image and the fourth mask sub-image. The second difference is comprehensively calculated from two different angles. This can improve the accuracy of the second difference, thereby improving the accuracy of model training, and improving the accuracy of video cutout using the model.

[0459] When the second fusion sub-network in the above-mentioned video cutout model fuses the features and the hidden state information provided by itself, it can ensure that the features of the transparency mask image corresponding to the segmented video frame of the video are considered in the process of cutout of the target video frame, that is, the temporal continuity between the video frames is guaranteed. However, in this case, if the second image generation network in the model has no hard restriction that the output image is a binary image, then there may be semi-transparent areas in the image finally output by the second image generation network, and the pixel value range of the pixel points in the target transparency mask image output by the second image generation network is 0-1. Semi-transparent areas may appear in the target transparency mask image output by itself. In this way, it is difficult to determine the reason for the semi-transparent area in the target transparency mask image, and it is also difficult to train the various networks and sub-networks in the cutout branch of the model.

[0460] In view of this, in one embodiment of the present application, referring to FIG32 , a structural diagram of a second image generation network is provided. In FIG32 , the second image generation network includes a hard segmentation subnetwork, a result fusion subnetwork, and an image generation subnetwork.

[0461] Among them, the input of the hard segmentation sub-network is the fusion result of the output of the second fusion sub-network in the last set of transparency feature generation network.

[0462] The hard segmentation subnetwork is used to obtain a second contour mask image of the object in the target video frame based on the fusion results.

[0463] The result fusion sub-network is used to fuse the second contour mask image and the fusion result to obtain the target fusion feature.

[0464] The image generation subnetwork is used to obtain the target transparency mask image of the object edge in the target video frame based on the target fusion features.

[0465] During model training, not only can the initial model be trained based on the first, second, and third differences described above, but mask images corresponding to different sample video frames output by the hard segmentation subnetwork can also be obtained, and a fourth difference between the obtained mask images can be calculated, thereby training the initial model based on the fourth difference. Because the hard segmentation subnetwork outputs a contour mask image that is a binary image, this eliminates the possibility of semi-transparent areas in the mask image output by the second image generation network itself during training, thereby enabling accurate and rapid training of the initial model.

[0466] As shown in FIG33 , the left side of FIG33 shows the final cutout result obtained when the fourth difference is not used for model training, and the right side of FIG33 shows the final cutout result obtained when the fourth difference is used for model training.

[0467] Next, the electronic device provided by the embodiments of the present application is introduced.

[0468] Electronic equipment can be equipped or portable terminal devices with other operating systems, such as mobile phones, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, in-vehicle devices, smart home devices and / or smart city devices, etc.

[0469] 34 exemplarily shows an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may support AOD, and its screen has pixel-level luminescence capability, supporting lighting up only some pixels of the screen, such as an organic light emitting diode (OLED) screen.

[0470] The electronic device 100 may include a processor 110, a display screen 120, a camera 130, an internal memory 140, a SIM (Subscriber Identification Module) card interface 150, a USB (Universal Serial Bus) interface 160, a charging management module 170, a power management module 171, a battery 172, a sensor module 180, a mobile communication module 190, a wireless communication module 200, an antenna 1, and an antenna 2. The sensor module 180 may include a pressure sensor 180A, a fingerprint sensor 180B, a touch sensor 180C, an ambient light sensor 180D, and the like.

[0471] The structures illustrated in the embodiments of this application do not constitute specific limitations on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0472] The processor 110 may include one or more processing units. For example, the processor 110 may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent components or integrated into one or more processors. In some embodiments, the electronic device 100 may also include one or more processors 110. The controller may generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. In other embodiments, the processor 110 may also include a memory for storing instructions and data. For example, the memory in the processor 110 may be a cache memory. This memory may store instructions or data that have just been used or are being recycled by the processor 110. If the processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the electronic device 100 in processing data or executing instructions.

[0473] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General-Purpose Input / Output (GPIO) interface, a SIM card interface, and / or a USB interface. Among them, the USB interface 160 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 160 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and peripheral devices. The USB interface 160 can also be used to connect headphones to play audio through the headphones.

[0474] The interface connection relationship between the modules illustrated in the embodiment of the present application is for illustrative purposes only and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0475] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 190, the wireless communication module 200, the modem processor, the baseband processor, and the like.

[0476] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0477] Electronic device 100 implements display functionality through a GPU, display screen 120, and an application processor. The GPU is a microprocessor for image processing that connects display screen 120 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0478] The display screen 120 is used to display images, videos, etc. The display screen 120 includes a display panel, which can be an organic light emitting diode (OLED), an active matrix organic light emitting diode (AMOLED), or other display panel with pixel-level light emitting capability.

[0479] The electronic device 100 can implement a shooting function through an ISP, a camera 130, a video codec, a GPU, a display screen 120, and an application processor, wherein the camera 130 includes a front camera and a rear camera.

[0480] The ISP processes data fed back by the camera 130. For example, when shooting, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can perform algorithmic optimization on image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within the camera 130.

[0481] The camera 130 is used to take photos or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard red, green, blue (RGB), YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 130, where N is a positive integer greater than 1.

[0482] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0483] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0484] The NPU is a neural network (NN) computing processor that rapidly processes input information and continuously self-learns by drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain. The NPU enables intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0485] The internal memory 140 can be used to store one or more computer programs, which include instructions. The processor 110 can execute the above instructions stored in the internal memory 140, thereby enabling the electronic device 100 to perform the video cutout method provided in some embodiments of the present application, as well as various applications and data processing. The internal memory 140 may include a program storage area and a data storage area. The program storage area may store an operating system; the program storage area may also store one or more applications (such as a gallery, contacts, etc.). The data storage area may store data created during the use of the electronic device 100 (such as photos, contacts, etc.). In addition, the internal memory 140 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage components, a flash memory component, a universal flash memory (UFS), etc. In some embodiments, the processor 110 can execute the video cutout method provided in the embodiments of the present application, as well as other applications and data processing by executing the instructions stored in the internal memory 140 and / or the instructions stored in the memory provided in the processor 110.

[0486] The internal memory 140 can be used to store program codes or program instructions for implementing the wallpaper display method and video cutout method provided in the embodiments of the present application. The processor 110 can call the program codes or program instructions of the wallpaper display method and video cutout method stored in the internal memory 140 to execute the wallpaper display method and video cutout method of the embodiments of the present application.

[0487] The sensor module 180 may include a pressure sensor 180A, a fingerprint sensor 180B, a touch sensor 180C, an ambient light sensor 180D, and the like.

[0488] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be located on display screen 120. There are many types of pressure sensors 180A, including resistive, inductive, and capacitive pressure sensors. A capacitive pressure sensor may comprise at least two parallel plates made of conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes, and electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation is applied to display screen 120, electronic device 100 detects the touch operation based on pressure sensor 180A. Electronic device 100 can also calculate the touch location based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch location but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, a command to view short messages is executed; when a touch operation with an intensity greater than or equal to the first pressure threshold is applied to a short message application icon, a command to create a new short message is executed.

[0489] The fingerprint sensor 180B is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement functions such as unlocking, accessing application locks, taking photos, and answering calls.

[0490] Touch sensor 180C, also known as a touch-sensitive device, can be provided on display screen 120. Touch sensor 180C and display screen 120 form a touch screen, also known as a touchscreen. Touch sensor 180C is used to detect touch operations applied thereto or in the vicinity thereof. Touch sensor 180C can transmit the detected touch operations to an application processor to determine the type of touch event. Visual output related to the touch operations can be provided via display screen 120. In other embodiments, touch sensor 180C can also be provided on the surface of electronic device 100 and at a different location from display screen 120.

[0491] Ambient light sensor 180D is used to sense ambient light brightness. Electronic device 100 can adaptively adjust the brightness of display screen 120 based on the perceived ambient light brightness. Ambient light sensor 180D can also be used to automatically adjust white balance during photography. Ambient light sensor 180D can also transmit information about the device's surroundings to the GPU.

[0492] The ambient light sensor 180D is also used to obtain the brightness, light ratio, color temperature, etc. of the acquisition environment in which the camera 130 captures images.

[0493] 35 shows a software system architecture used by the electronic device 100. The software system architecture may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture.

[0494] A layered architecture divides the terminal's software system into several layers, each with distinct roles and responsibilities. Layers communicate with each other via software interfaces. In some embodiments, the software system can be divided into five layers: applications, application framework, system libraries, hardware abstraction layer (HAL), and kernel.

[0495] The application layer can include a series of application packages, which run applications by calling the application programming interface (API) provided by the application framework layer. As shown in Figure 35, the application package can include applications such as browser, gallery, music, and video. It can be understood that the port of each of the above applications can be used to receive data.

[0496] The application framework layer provides an API and programming framework for applications in the application layer. The application framework layer includes some predefined functions. As shown in Figure 35, the application framework layer can include a window manager, content provider, view system, resource manager, notification manager, and DHCP (Dynamic Host Configuration Protocol) module.

[0497] The system library can include multiple functional modules, such as surface manager, 3D graphics processing library, 2D graphics engine and file library.

[0498] The hardware abstraction layer can contain multiple library modules, such as a display library module and a motor library module. The terminal system can load the corresponding library module for the device hardware, thereby enabling the application framework layer to access the device hardware.

[0499] The kernel layer is the layer between hardware and software. The kernel layer is used to drive the hardware, making it work. The kernel layer includes at least a display driver, an audio driver, a sensor driver, and a motor driver, etc., which are not limited in the embodiments of the present application. It is understandable that the display driver, audio driver, sensor driver, and motor driver, etc. can all be considered as a driver node. Each of the above-mentioned driver nodes includes an interface that can be used to receive data.

[0500] The term "user interface (UI)" in the specification, claims and drawings of this application refers to the media interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is a source code written in a specific computer language such as Java and Extensible Markup Language (XML). The interface source code is parsed and rendered on the terminal device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, pictures and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as <head> and <body>. A node corresponds to a control or attribute in the interface, and the node is presented as user-visible content after parsing and rendering. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is a source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code, such as HTML through <body>. 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.

[0501] A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operations that uses graphics. It can be an icon, window, control, or other interface element displayed on the display of an electronic device. Controls can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, and other visual interface elements.

[0502] Each step in the above method embodiments provided in this application can be completed by hardware integrated logic circuits in a processor or by software instructions. The method steps disclosed in the embodiments of this application can be directly implemented as hardware processor execution, or can be implemented by a combination of hardware and software modules in the processor.

[0503] The present application also provides an electronic device, which may include a memory and a processor, wherein the memory may be used to store a computer program, and the processor may be used to call the computer program in the memory so that the electronic device executes the method in any one of the above embodiments.

[0504] The present application also provides a chip system, as shown in Figure 36, the chip system may include a processor 2201 for implementing the functions involved in the method executed by the electronic device in any of the above embodiments. The chip system may also include a memory, the memory is used to store program instructions and data, and the memory is located inside or outside the processor. The chip system may also include an input and output interface, which can serve as a bridge connecting the processor and peripheral input and output devices. Specifically, when the peripheral input device collects input data, such as the touch screen collects the user's touch operation, the audio input circuit collects audio data, the camera collects image data, etc., the input and output interface can transmit the input data to the processor for processing; when the processor completes the processing of the input data and generates a processing result, the input and output interface can transmit the processing result to the peripheral output device for output, such as transmitting the processing result to the display screen for display, transmitting the processing result to the audio output circuit for voice output, etc.

[0505] The chip system can be composed of chips, or can include chips and other discrete devices.

[0506] Optionally, there may be one or more processors in the chip system. The processor may be implemented in hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0507] Optionally, the memory in the chip system may be one or more. The memory may be integrated with the processor or may be provided separately from the processor, which is not limited in the embodiments of the present application. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or provided on different chips. The embodiments of the present application do not specifically limit the type of memory or the configuration of the memory and the processor.

[0508] Exemplarily, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0509] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which enables a computer to execute the method executed by the electronic device in any of the above embodiments when the computer program is executed.

[0510] The present application also provides a computer-readable storage medium storing a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the electronic device in any of the above embodiments.

[0511] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. Available media can be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state drive Solid State Disk), etc.

[0512] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0513] In short, the above are only embodiments of the technical solution of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made based on the disclosure of the present invention should be included in the scope of protection of the present invention.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A wallpaper display method, characterized in that: include: An electronic device in a locked state or unlocked state detects a first operation, and in response to the first operation, the electronic device displays a first interface, wherein the first interface includes an AOD wallpaper; When the electronic device displays the first interface, in response to a second operation, the electronic device displays a second interface, where the second interface includes a lock screen wallpaper; In response to a third operation for unlocking, the electronic device displays a third interface, wherein the third interface includes a desktop wallpaper; Among them, the AOD wallpaper, the lock screen wallpaper and the desktop wallpaper respectively correspond to the first video frame, the second video frame and the third video frame in the first video, the second video frame is after the first video frame, and the third video frame is after the second video frame.

2. The method according to claim 1, characterized in that The first video frame, the second video frame, and the third video frame are all one frame, the frame interval between the first video frame and the second video frame is greater than the first frame interval, and the frame interval between the second video frame and the third video frame is greater than the second frame interval.

3. The method according to claim 1, characterized in that The first video frame, the second video frame, and the third video frame are all multiple frames, the start frame in the second video frame is the end frame in the first video frame, and the start frame in the third video frame is the end frame in the second video frame.

4. The method according to any one of claims 1 to 3, characterized in that The first video is selected from a gallery of videos.

5. The method according to any one of claims 1 to 4, characterized in that The first video is selected from videos provided by a settings application.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: The electronic device displays a fourth interface, wherein the fourth interface includes a video frame sequence of the second video; The first video is selected from the video frame sequence, wherein the first video is a partial video clip in the second video or the entire second video.

7. The method according to claim 6, characterized in that There is a selection box on the video frame sequence, and a video segment in the selection box is the first video; the user operation of selecting the video segment is specifically an operation of sliding the selection box along the video frame sequence.

8. The method according to claim 6 or 7, characterized in that The fourth interface also includes a first preview, which is used to display the display status of the AOD wallpaper, the lock screen wallpaper and the desktop wallpaper at the first interface, the second interface and the third interface respectively.

9. The method according to claim 8, characterized in that The first preview is a video, and when played, the first preview presents a dynamic display process of switching from the AOD wallpaper to the lock screen wallpaper and then to the desktop wallpaper.

10. The method according to any one of claims 1 to 9, characterized in that The method further includes: the electronic device displays a fifth interface, and the main object and the background of the wallpaper in the fifth interface are allowed to be selected separately by the user.

11. The method according to claim 10, characterized in that Before the electronic device displays the fifth interface, the method further includes: the electronic device cuts out the main object in the first video to separate the main object from the background.

12. The method according to claim 11, characterized in that The method further comprises: In response to a user operation of selecting the subject object for subject object editing, the electronic device updates the image of the subject object, and refreshes the AOD wallpaper, the lock screen wallpaper, and the desktop wallpaper based on the updated subject object image.

13. The method according to claim 11 or 12, characterized in that Also includes: In response to a user operation of selecting a background of the wallpaper for background editing, the electronic device updates an image of the background, and refreshes the AOD wallpaper, the lock screen wallpaper, and the desktop wallpaper based on the updated background image.

14. An electronic device, characterized in that: The method comprises one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program codes, wherein the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the method as described in any one of claims 1 to 13 is executed.

15. A chip system, the chip system is applied to electronic equipment, the chip system comprises one or more processors, characterized in that: The processor is configured to call computer instructions so as to execute the method according to any one of claims 1 to 13.

16. A computer-readable storage medium, comprising a computer-executable program, characterized in that: When the computer executable program is run on an electronic device, the method according to any one of claims 1 to 13 is executed.

Images