Video processing method, electronic equipment and storage medium

Abstract

The invention discloses a video processing method, electronic equipment and a storage medium, and belongs to the technical field of video editing. The method comprises the following steps: displaying a first video picture of a video; in response to setting of a to-be-tracked first target on the first video picture by a user and confirmation of execution of sticker tracking operation, determining a first target frame position corresponding to the first target; tracking a first video clip of the video according to the first reference video frame, the first target frame position and the first sticker material, so that the first sticker material is added to the position of a first target in the first video clip; and displaying the first video clip added with the first sticker material. Thus, the special effect of the sticker can be added to the same target of the video according to the tracking target set by the user on any video picture, the flexibility of sticker adding is improved, and the sticker adding effect is enriched.

Description

Technical Field

[0001] The present application relates to the technical field of video editing, and in particular to a video processing method, an electronic device and a storage medium. Background Art

[0002] With the rise of streaming platforms and short video platforms, people's demand for video editing has gradually increased. At present, people's demand for video editing is no longer satisfied with simple operations such as synthesis, segmentation, and adding filters. The demand for more professional and detailed operations is increasing.

[0003] For example, during video editing, users can add stickers in the form of pictures or text on the video screen. Currently, in response to the user's operation of adding stickers at a certain position on the video screen, the video editing application will add stickers at the same position on all video screens of the video, generating a video with stickers added at the same screen position. This sticker adding method has low flexibility and the sticker adding effect is relatively simple. Summary of the invention

[0004] The present application provides a video processing method, an electronic device and a storage medium, which can improve the flexibility of adding stickers and enrich the effects of adding stickers. The technical solution is as follows:

[0005] In a first aspect, a video processing method is provided, the method comprising: displaying a first video screen of a video, and in the first video screen, a user can set a first target to be tracked on the first video screen. In response to the user setting the first target to be tracked on the first video screen and confirming the operation of performing sticker tracking, first determining the target frame position corresponding to the first target, and then performing tracking processing on the first video segment of the video according to the target frame position, the video frame corresponding to the first video screen, and the first sticker material, so that the first sticker material is added to the position of the first target in the first video segment, and then displaying the first video segment with the first sticker material added.

[0006] Through the video processing method provided by the embodiment of the present application, the user can set a tracking target on any video frame of the video. After the user sets the tracking target, the target frame position corresponding to the tracking target determined by the user is determined, and then the other video frames of the video are tracked and processed according to the video frame, the target frame position and the sticker material, so that the sticker material is added to the position of the tracking target on the other video frames, and the video frame with the sticker material added is displayed. In this way, according to the tracking target set by the user, sticker special effects can be added to the same target in the video, which improves the flexibility of adding stickers and enriches the effect of adding stickers.

[0007] Among them, the first video picture is the video picture of the target to be tracked, and can be the video picture corresponding to any video frame in the video. For the sake of convenience, the video frame corresponding to the first video picture is referred to as the first reference video frame in the embodiment of the present application. The target frame position corresponding to the first target can be referred to as the first target frame position, and the first target frame position is used to indicate the position of the first target on the first reference video frame, such as used to indicate the boundary of the image area where the first target is located. The target frame is usually a rectangular frame. The target frame position can be the coordinates of the target frame, such as the coordinates of at least two vertices of the target frame, such as the coordinates of the upper left and lower right vertices of the target frame, or the coordinates of the upper right and lower left vertices, or the coordinates of four vertices, etc.

[0008] The first video segment includes all or part of the video frames in the video except the first reference video frame. For example, the first video segment may include a video segment after the first reference video frame and / or a video segment before the first reference video frame.

[0009] The displaying of the first video clip with the first sticker material added thereto includes: previewing and displaying the first video clip with the first sticker material added thereto. In this way, a real-time preview of the sticker adding effect can be achieved, so that the user can view the sticker adding effect in a timely manner.

[0010] In the embodiment of the present application, the electronic device can display the first video screen during the video playback, or can display the first video screen according to the user's preview operation. For example, the user can drag the preview progress bar of the video to adjust the video preview progress to the preview time point corresponding to the first reference video frame to preview and display the first video screen.

[0011] When the electronic device displays the first video screen, the user can set the target to be tracked on the first video screen. For example, the first target in the first video screen is set as the target to be tracked on the first video screen. The target to be tracked refers to the target to be tracked and to which sticker materials are added.

[0012] In one embodiment of the present application, after selecting the first sticker material, the user can set the first target as the first target to be tracked by dragging the first sticker material to the position of the first target in the first video frame, and confirm to perform sticker tracking on the first target. Sticker tracking refers to tracking the first target in other video frames and adding the first sticker material to the tracked first target.

[0013] In this way, it is convenient for users to flexibly set the target to be tracked in the first video screen, which increases the flexibility of adding stickers and improves the user experience.

[0014] In one embodiment of the present application, in response to a user triggering an operation on a sticker option, the electronic device displays a first interface, the first interface including a plurality of sticker materials. In response to a user selecting a first sticker material from a plurality of sticker materials, the first sticker material is displayed at a preset position on a first video screen. In response to a user dragging the first sticker material from a preset position to a position where a first target is located on the first video screen and triggering an intelligent tracking option, the first target frame position is determined.

[0015] In this way, it is convenient for users to select sticker materials and flexibly set the target to be tracked in any video screen, which improves the flexibility of adding stickers, can meet the user's personalized sticker adding needs, and improves the user experience.

[0016] In the embodiment of the present application, the video may be a separate video or a video generated by stitching together multiple image materials.

[0017] In one embodiment of the present application, before displaying the first video frame of a video, in response to a first operation of the user, videos and pictures in a gallery are loaded and displayed; in response to the user selecting an image material in the gallery and confirming an operation to perform video editing, a video editing interface of the video is displayed, the video editing interface including a video preview window, and the first video frame is displayed in the video preview window.

[0018] In this way, during the process of video stitching of multiple image materials, sticker effects can be added to the stitched video, making it convenient for users to add stickers to the video during the video editing process, thereby expanding the application scenarios of adding stickers.

[0019] The video preview window is used to preview and display the video screen of the video obtained according to the image material, and the image material includes a video, or includes a video and a picture, or includes multiple pictures. For example, the video is a video generated by splicing multiple image materials.

[0020] In response to the user setting the first target to be tracked on the first video screen and confirming the operation of performing sticker tracking, the electronic device can determine the first target frame position according to the material position of the first sticker material. For example, the first target covered by the first sticker material can be determined according to the material position of the first sticker material, and then the first target frame position can be determined according to the first target.

[0021] It should be noted that the first target can be one or more. That is, the user can set one first target on the first video screen, or multiple first targets. In the case of multiple first targets, the first target frame position corresponding to each first target can be determined, thereby obtaining multiple first target frame positions, so that other video frames can be tracked and processed later according to the first reference video frame, multiple first target frame positions and the first sticker material.

[0022] Wherein, in the case where multiple first targets are set on the first video screen, the first sticker material may include one or more. Each first target corresponds to a first sticker material to be added, and the first sticker materials corresponding to the multiple first targets may be the same or different. For example, other video frames may be tracked based on the first reference video frame, multiple first target frame positions, and the first sticker material corresponding to each of the multiple first target frame positions.

[0023] In one embodiment of the present application, a first video segment includes at least one video segment; tracking processing is performed on the first video segment of the video according to a first reference video frame, a first target frame position and a first sticker material, including: sequentially acquiring a target video frame to be processed from the target video segment, the target video segment being any one of the at least one video segment; performing target tracking on the acquired first video frame according to the first reference video frame and the first target frame position, the first video frame being any target video frame in the target video segment; in the case of performing target tracking on the first video frame to obtain a target frame position of the first video frame, determining sticker material information of the first video frame according to the target frame position of the first video frame, the sticker material information at least including a material position of the first sticker material; adding the first sticker material to the first video frame according to the sticker material information of the first video frame to generate a second video frame, the first sticker material being added to the position of the first target in the second video frame.

[0024] In this way, according to the first reference video frame and the first target frame position, the target tracking and sticker adding operations can be performed on the video frames in the first video clip in sequence, thereby improving the video tracking processing efficiency and realizing the real-time preview of the tracking processing results. For example, after the second video frame is generated, the second video frame can be previewed.

[0025] As an example, the video frames to be processed can be obtained from the first video segment in sequence, and each time a video frame is obtained, the video frame is tracked and processed. After the tracking process of the video frame is completed, the next video frame is obtained and the tracking process is performed on the next video frame.

[0026] That is, after the tracking process of one video frame is completed, the tracking process of the next video frame is performed. In this way, the complexity of the video processing algorithm can be reduced.

[0027] As another example, the target video frames to be processed can be obtained from the target video clip in sequence, and the obtained target video frames can be stored in a tracking queue, which is used to store the video frames to be tracked. Afterwards, on the one hand, the video frames to be tracked are obtained from the tracking queue in sequence, and target tracking is performed on the obtained video frames according to the first reference video frame and the first target frame position, and the obtained video frames include the first video frame. On the other hand, the target frame position of the latest tracked video frame is obtained from the tracking results corresponding to the tracking queue in sequence, and the sticker material information of the video frame is determined according to the obtained target frame position of the video frame, and the video frame includes the first video frame.

[0028] In this way, the target can be tracked on the video frame to be tracked while the sticker processing can be performed on the video frame that has been tracked. The two can be executed simultaneously, which further improves the video processing efficiency and facilitates the real-time preview of the tracking processing effect.

[0029] In one embodiment of the present application, sequentially obtaining target video frames to be processed from a target video segment may include: when the target video segment is a video segment located after a first reference video frame, sequentially obtaining the target video frames to be processed from the target video segment in a forward playback order of the video; when the target video segment is a video segment located before the first reference video frame, sequentially obtaining the target video frames to be processed from the target video segment in a reverse playback order of the video.

[0030] That is, the video clip after the first reference video frame can be tracked forward, and the video clip before the first reference video frame can be tracked backward. By taking the first reference video frame as a reference, the front and back video clips are tracked backward and forward respectively, so as to ensure the tracking accuracy and the continuity of the tracking target change.

[0031] The target video frame is a video frame or a key frame in the target video clip. That is, the video frames to be processed can be obtained from the target video clip in sequence, and the obtained video frames can be tracked in sequence according to the first reference video frame, the first target frame position and the first sticker material. The key frames to be processed can also be obtained from the target video clip in sequence, and the obtained key frames can be tracked in sequence according to the first reference video frame, the first target frame position and the first sticker material.

[0032] In the implementation method of sequentially obtaining key frames to be processed from the target video clip, and sequentially tracking and processing the obtained key frames according to the first reference video frame, the first target frame position and the first sticker material, for the sticker material information of the intermediate video frame between the two key frames, the sticker material information of the intermediate video frame can also be determined, and then the first sticker material is added to the intermediate video frame according to the sticker material information of the intermediate video frame to generate the intermediate video frame with the first sticker material added. The determination of the sticker material information of the intermediate video frame can be obtained by the following two implementation methods:

[0033] The first implementation method: the target video frame is a key frame, and the first video frame is the first key frame. When the target frame position of the first key frame is obtained by tracking the first key frame, linear interpolation can be performed based on the target frame position of the first key frame and the target frame position of the third video frame to obtain the target frame position of the intermediate video frame, and the intermediate video frame is a video frame between the third video frame and the first key frame. Wherein, when the first key frame is the first key frame tracked in the first video clip, the third video frame is the first reference video frame, and when the first key frame is not the first key frame, the third video frame is the most recently tracked key frame before the first key frame.

[0034] The second implementation method: when the sticker material information of the first key frame is obtained, linear interpolation is performed according to the sticker material information of the first key frame and the sticker material information of the third video frame to obtain the sticker material information of the middle video frame.

[0035] In an embodiment of the present application, the sticker material information of the key frame is determined by tracking the target frame position of the key frame in the target video clip, and for the intermediate video frames between the key frames, the sticker material information of the intermediate video frames is generated by performing linear interpolation according to the target frame positions of the previous and next key frames, or by performing linear interpolation according to the sticker material information of the previous and next key frames. In this process, only the key frames in the target video clip need to be tracked, which reduces the processing amount of target tracking, saves the target tracking time, and thus improves the video processing efficiency.

[0036] In one embodiment of the present application, displaying a first video clip with a first sticker material added thereto includes: upon generating an intermediate video frame with the first sticker material added thereto and a first key frame with the first sticker material added thereto, previewing and displaying the intermediate video frame with the first sticker material added thereto and the first key frame with the first sticker material added thereto.

[0037] That is, in the process of tracking the acquired key frames in sequence, for the intermediate video frame between two adjacent key frames, the sticker material information of the intermediate video frame can also be determined according to the target frame position or sticker material information of the two adjacent video frames, and then the first sticker material is added to the intermediate video frame according to the sticker material information of the intermediate video frame. After obtaining the intermediate video frame and key frame with the first sticker material added, the intermediate video frame and key frame with the first sticker material added are previewed and displayed in sequence.

[0038] In one embodiment of the present application, each time a second video frame with the first sticker material added is generated, the second video frame can be previewed and displayed. In this way, a real-time preview of the sticker adding effect can be achieved, so that the user can view the sticker adding effect in time.

[0039] In one embodiment of the present application, when an abnormal mark is obtained by target tracking of the first video frame, the historical sticker material information is determined as the sticker material information of the first video frame, and the abnormal mark is used to indicate that the first target is lost or missing in the first video frame. Wherein, when there is a target video frame that has been tracked before the first video frame, the historical sticker material information is the sticker material information of the most recently tracked target video frame before the first video frame, and when there is no target video frame that has been tracked before the first video frame, the historical sticker material information is the sticker material information of the first reference video frame.

[0040] In this way, when the target is lost or missing, by using the sticker material information of the previous video frame as the sticker material information of the current video frame, it is convenient to subsequently use the sticker material information of the previous video frame to add the first sticker material to the first video frame, thereby improving the smoothness and accuracy of adding stickers on continuous video frames and avoiding the situation of missing sticker special effects on continuous video frames.

[0041] In addition, in the embodiment of the present application, it is also possible to support setting tracking targets on multiple video screens of a video, such as setting different tracking targets on multiple video screens. As an example, after setting a tracking target on a video frame, it is also possible to support setting a new tracking target on other video frames.

[0042] In one embodiment of the present application, in response to a user setting a first target to be tracked on a first video screen and confirming the operation of performing target tracking, after determining the first target frame position, a second video screen of the video can also be displayed, the second video screen being a video screen corresponding to any video frame in the video except the first reference video frame. In response to a user setting a second target to be tracked on a second video screen and confirming the operation of performing target tracking, the second target frame position is determined, and the second target frame position is used to indicate the position of the second target on the second reference video frame corresponding to the second video screen. According to the second reference video frame, the second target frame position, and the second sticker material, target tracking is performed on the second video clip in the video, so that the second sticker material is added to the position of the second target on the second video clip, and the second video clip is all or part of the video frames in the video frame except the second reference video frame. The second video clip with the second sticker material added is displayed.

[0043] The first video segment includes a video segment located on a first side of a first reference video frame, and the second video segment includes a video segment located on a second side of a second reference video frame. The first side refers to a side that is not adjacent to the second reference video frame, and the second side refers to a side that is not adjacent to the first reference video frame.

[0044] That is, for the video clip located on the first side of the first reference video frame, tracking processing can be performed with the first target as the tracking target, and for the video clip located on the second side of the first reference video frame, tracking processing can be performed with the second target as the tracking target.

[0045] In addition, the first video segment also includes an intermediate video segment between the first reference video frame and the second reference video frame, or the second video segment also includes an intermediate video segment.

[0046] That is, for the intermediate video segment between the first reference video frame and the second reference video frame, the tracking process may be performed with the first target as the tracking target, or the tracking process may be performed with the second target as the tracking target.

[0047] As an example, the tracking target of the middle video segment can be specified by the user from the first target and the second target. For example, in response to the user setting the second target to be tracked on the second video screen and confirming the operation of performing target tracking, after determining the position of the second target frame, a prompt information can be displayed, and the prompt information is used to prompt the user to specify the tracking target for the middle video segment from the first target and the second target. If the tracking target specified by the user is the first target, the first video segment also includes the middle video segment; if the tracking target specified by the user is the second target, the second video segment also includes the middle video segment.

[0048] In this way, the flexibility of adding stickers can be further improved to ensure the user's personalized sticker adding needs.

[0049] In addition, the embodiment of the present application also supports the cancellation of the operation of executing sticker tracking. For example, in response to the user canceling the operation of executing sticker tracking, the sticker adding effect of the first video clip is canceled. Alternatively, the sticker adding effect of the first video clip is canceled, and the first sticker material is added to the preset position of each video frame in the first video clip, and the first video clip with the first sticker material added at the preset position is displayed. Among them, the preset position can be the center position of the video frame, etc., which is not limited by the embodiment of the present application.

[0050] For example, in response to the user canceling the target tracking operation, a first sticker material is added to a preset position of each video frame in the first video segment; and the first video segment with the first sticker material added to the preset position is displayed.

[0051] In this way, it is convenient for users to modify the executed sticker adding operation, further improving the flexibility of sticker adding and ensuring the user's personalized sticker adding needs.

[0052] In one embodiment of the present application, before determining the first target frame position, an editing preview pipeline, a special effects rendering pipeline and a target tracking pipeline may also be created, the editing preview pipeline including a decoding node and a display node connected in sequence, the special effects rendering pipeline is located after the decoding node and before the display node, the special effects rendering node includes at least a sticker rendering node, the target tracking pipeline is located after the decoding node and before the sticker rendering node, the target tracking pipeline includes at least a target tracking node; then, by running the editing preview pipeline, the special effects rendering pipeline and the target tracking pipeline, the first video clip of the video is tracked according to the first reference video frame, the first target frame position and the first sticker material.

[0053] For example, the first video clip can be decoded through a decoding node, and target video frames to be processed can be obtained in sequence from the decoding results; through a target tracking node, target tracking is performed on the obtained first video frame according to the first reference video frame and the first target frame position to obtain the target frame position of the first video frame, where the first video frame is any target video frame in the first video clip; through a sticker rendering node, the first sticker material is rendered on the first video frame according to the sticker material information of the first video frame to generate a second video frame, where the first sticker material is added to the position of the first target in the second video frame, and the sticker material information of the first video frame is determined according to the target frame position of the first video frame, and the sticker material information of the first video frame at least includes the material position; through a display node, the second video frame is displayed.

[0054] Among them, the pipeline is a linear communication model, which can be regarded as an assembly line. In simple terms, it is a comprehensive solution for a system that provides a one-stop service for obtaining results. By running the editing preview pipeline, target tracking pipeline and special effect rendering pipeline to realize special effect tracking processing of the video, and preview display of the special effect tracking processing results, when the user performs intelligent special effect tracking on the video, it can achieve real-time preview of the video file after rendering the sticker special effect on the tracking target.

[0055] In a second aspect, a video processing device is provided, wherein the video processing device has the function of implementing the video processing method in the first aspect. The video processing device comprises at least one module, wherein the at least one module is used to implement the video processing method provided in the first aspect.

[0056] In a third aspect, a video processing device is provided, wherein the structure of the video processing device includes a processor and a memory, wherein the memory is used to store a program that supports the video processing device to execute the video processing method provided in the first aspect, and to store data involved in implementing the video processing method described in the first aspect. The processor is configured to execute the program stored in the memory. The video processing device may also include a communication bus, wherein the communication bus is used to establish a connection between the processor and the memory.

[0057] In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes the video processing method described in the first aspect.

[0058] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the video processing method described in the first aspect.

[0059] The technical effects obtained by the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0061] Figure 2 is a schematic diagram of a software structure of an electronic device provided in an embodiment of the present application;

[0062] Figure 3 is a schematic diagram of a software structure of another electronic device provided in an embodiment of the present application;

[0063] Figure 4This is an example diagram of the effect of adding sticker materials to a video provided by an embodiment of the present application;

[0064] Figure 5 is a set of example diagrams of graphical user interfaces provided in an embodiment of the present application;

[0065] Figure 6 is a set of example diagrams of graphical user interfaces provided in an embodiment of the present application;

[0066] Figure 7 is a set of example diagrams of graphical user interfaces provided in an embodiment of the present application;

[0067] Figure 8 is a set of example diagrams of graphical user interfaces provided in an embodiment of the present application;

[0068] Fig. 9 It is a schematic diagram of a group of canceling intelligent tracking processing effects provided by an embodiment of the present application;

[0069] Fig.10 is a set of example diagrams of graphical user interfaces provided in an embodiment of the present application;

[0070] Fig.11 is a set of example diagrams of graphical user interfaces provided in an embodiment of the present application;

[0071] Fig.12 It is a flowchart of a video processing method provided by an embodiment of the present application;

[0072] Fig.13 It is a schematic diagram of a forward tracking process of a video provided by an embodiment of the present application;

[0073] Fig.14 It is a schematic diagram of a reverse tracking process of a video provided by an embodiment of the present application;

[0074] Fig.15 It is a schematic diagram of performing forward tracking processing and reverse tracking processing on a video provided by an embodiment of the present application;

[0075] Fig.16 is a schematic diagram of interpolating sticker positions of intermediate video frames between adjacent key frames provided by an embodiment of the present application;

[0076] Fig.17 It is a schematic diagram of a video processing effect of setting a tracking target on multiple video frames of a video provided by an embodiment of the present application;

[0077] Fig.18 It is a schematic flow chart of forward tracking processing of a video in a synchronous tracking interface mode provided by an embodiment of the present application;

[0078] Fig.19 is a schematic flow chart of forward tracking processing of a video in an asynchronous tracking interface mode provided by an embodiment of the present application;

[0079] Fig. 20 is a schematic flow chart of reverse tracking processing of a video clip in a synchronous tracking interface mode provided by an embodiment of the present application;

[0080] Fig.21 It is a schematic flowchart of reverse tracking processing of a video clip in an asynchronous tracking interface mode provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0082] It should be understood that the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of this application, the words "first" and "second" are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not limit them to be different. In the description of this embodiment, unless otherwise specified, the meaning of "multiple" is two or more.

[0083] The video processing method provided in the embodiment of the present application is applied to electronic devices. The electronic device may also be referred to as a terminal, a user equipment (UE), a mobile station (MS), a mobile terminal (MT), etc. The electronic device may be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with a wireless transceiver function, a virtual reality (VR) electronic device, an augmented reality (AR) electronic device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The embodiment of the present application does not limit the specific technology and specific device form adopted by the electronic device.

[0084] Figure 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 1 The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0085] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0086] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0087] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0088] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0089] In some embodiments, the processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0090] The electronic device 100 can realize the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0091] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is an integer greater than 1.

[0092] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor. ISP is used to process the data fed back by camera 193. ISP can also perform algorithm optimization on the noise, brightness and skin color of the image. ISP can also optimize the parameters such as exposure and color temperature of the shooting scene. In some embodiments, ISP can be set in camera 193. Camera 193 is used to capture static images or videos. In some embodiments, electronic device 100 may include 1 or N cameras 193, where N is an integer greater than 1.

[0093] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0094] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0095] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0096] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0097] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created by the electronic device 100 during use (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0098] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170 and the application processor.

[0099] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0100] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor 180K can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0101] Understandably, Figure 1 The components included in the hardware structure shown do not constitute a specific limitation on the electronic device 100. The electronic device 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.

[0102] Next, the software system of the electronic device 100 will be described.

[0103] The electronic device provided in the embodiments of the present application may be a user equipment (UE), for example, a mobile terminal (such as a mobile phone), a tablet computer, a desktop computer, a laptop computer, a handheld computer, a netbook, a personal digital assistant (PDA), and the like.

[0104] In addition, an operating system is running on the above components, such as the iOS operating system developed by Apple, the Android open source operating system developed by Google, and the Windows operating system developed by Microsoft. Application programs can be installed and run on the operating system.

[0105] The operating system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0106] Figure 2 1 is a schematic diagram of a software structure of an electronic device 100 provided in an embodiment of the present application. The software structure includes several layers, each layer has a clear role and division of labor, and the layers communicate with each other through software interfaces. In some embodiments, the software structure is divided from top to bottom into an application layer, a media middle platform framework layer, a smart middle platform framework layer, and an application framework layer.

[0107] The application layer can include a series of application packages, such as camera, gallery, calendar, map, navigation, etc. When these application packages are running, they can access the various service modules provided by the media platform framework layer through the application programming interface (API) and execute corresponding services.

[0108] In some embodiments, the camera is used to take photos, videos, slow-motion images, panoramic images, etc. in response to user operations. After these images are taken by the camera, or after the user triggers the electronic device to take a screenshot, or after the user triggers the electronic device to record the screen, or after the electronic device downloads images from other devices, the electronic device can save these images in a gallery, so that the user can perform video editing operations on the images in the gallery, such as smart cropping operations.

[0109] The embodiment of the present application divides the gallery into: a business layer, an application function layer and a basic function layer from top to bottom.

[0110] Among them, the business layer provides a variety of services such as one-click filming, free creation, and intelligent special effects tracking. These services are presented in the form of controls in the user interface (UI) of the gallery. Users can trigger the gallery to perform corresponding video processing actions by operating a certain control. Intelligent special effects tracking is used to track and process a video composed of one or more image materials after the user selects one or more image materials and confirms the execution of the intelligent special effects tracking service, so as to add sticker materials such as pictures or texts to specific targets in the video. Among them, intelligent special effects tracking can provide a tracking target setting function, and the tracking target setting function is used to provide users with the function of autonomously setting tracking targets. For example, users can set tracking targets on any video screen of the video based on the tracking target setting function. After the user sets the tracking target, the tracking target setting function can determine the target frame position of the tracking target set by the user, so as to track and process each video frame according to the target frame position of the tracking target set by the user.

[0111] The application function layer may include an engineering data model, which may store data related to the intelligent special effect tracking service, such as caching sticker material information such as the position and size of sticker materials corresponding to each video frame in the processed video, so that sticker material information of the video frame can be directly obtained from the engineering data model when it is needed later. The sticker material information corresponding to each video frame can be determined based on the target frame position of each video frame.

[0112] The basic function layer includes a clipping framework. Each business in the business layer can call the clipping framework to provide clipping services for pictures and videos. Exemplarily, the clipping framework includes an editing preview module and a special effect rendering module. The editing preview module is used to preview and display the video. For example, the editing preview model includes functional modules such as decoding, texture conversion, and display transmission. Decoding is a process of using a specific method to restore digital data to the content it represents or convert electrical pulse signals, light signals, radio waves, etc. into the information, data, etc. it represents. Decoding is the process in which the receiver restores the received symbols or codes to information, corresponding to the encoding process. For example, decoding is used to decode the video encoding data of the video to obtain the original video data. The texture conversion module is used to convert the image frame into an image texture, such as converting the video frame into an image texture. Display transmission is used to display the data to be displayed so as to display it on the display screen. The special effect rendering module is used to perform special effect rendering on images or videos. For example, the special effect rendering module includes special effect rendering modules such as transition rendering, picture-in-picture synthesis, and sticker rendering. Transition rendering is used to add transition special effects. Picture-in-picture compositing is used to display a small area of ​​another image. Sticker rendering is used to add sticker effects.

[0113] The media middle platform framework layer and the smart middle platform framework layer are software layers set between the application layer and the application framework layer. The media middle platform framework layer may include a resolution reduction module, a texture to image frame module and a target tracking algorithm interface. The resolution reduction module is used to reduce the resolution of the image to reduce the resolution of the image and obtain an image with a smaller resolution. The texture to image frame module is used to convert the image texture into image frame data, such as converting the image texture of the video frame into video frame data. The target tracking algorithm interface is used to call the target tracking algorithm of the target tracking module in the smart middle platform framework layer. The smart middle platform framework layer includes a target tracking module for tracking the target of the video frame to determine the target frame position of the tracking target on the video frame. For example, the target tracking module is integrated with a target tracking algorithm, and the target tracking algorithm is used to track the target of other video frames according to a specific video frame and the target frame position of the specific tracking target on the video frame to determine the target frame position of other video frames, that is, the target frame position of the tracking target on other video frames.

[0114] The application framework layer, referred to as the framework layer for short, can be used to support the operation of the media middle platform framework layer and various modules in it.

[0115] Understandably, Figure 2 The layers in the software structure shown and the components contained in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown in the figure, and each layer may include more or fewer components. For example, the media platform framework layer may also include a decapsulation module, an encapsulation module, etc. For another example, the software structure may also include a hardware abstraction layer, a system library, a kernel layer, etc., and the present application does not limit this.

[0116] The hardware abstraction layer is a package of the Linux kernel driver, providing an interface to the upper layer. It hides the hardware interface details of a specific platform and provides a virtual hardware platform for the operating system, making it hardware-independent and portable on multiple platforms. The kernel layer uses file descriptors to access files. File descriptors are non-negative integers. When opening an existing file or creating a new file, the kernel returns a file descriptor. Reading and writing files also requires the use of file descriptors to specify the files to be read and written.

[0117] It is understandable that in order to implement the video processing method in the embodiment of the present application, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments.

[0118] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.

[0119] The following is combined with the above Figure 2 The software architecture shown briefly summarizes the video processing method provided in the embodiment of the present application.

[0120] In an embodiment of the present application, the free creation service is located in the business layer of the application layer, which is a sub-service of video editing. The image material to be processed is selected in the business layer, and the video composed of the image material is sent to the application function layer, and the application function layer sends the video to the basic function layer. In the business layer, a tracking target can be set in any video screen of the video, and then the target frame position of the set tracking target and the video frame corresponding to the video screen are sent to the target tracking algorithm of the smart middle platform framework layer. The basic function layer can send the video frames in the video to the media middle platform framework layer in sequence. For example, in the basic function layer, the video can be decoded, and the decoded video frames can be converted into image textures and then sent to the media middle platform framework layer in sequence. The media middle platform framework layer can reduce the resolution of the image texture sent by the basic function layer, convert the image texture to the video frame, and then call the target tracking algorithm interface, and call the target tracking algorithm of the smart middle platform framework layer through the target tracking algorithm interface to track the processed video frame, and return the target frame position of the obtained video frame to the application function layer. At the application function layer, the engineering data model can determine the sticker material information of the video frame according to the target box position of the video frame, and send the sticker material information to the basic engineering layer. The basic function layer adds sticker material to the video frame according to the sticker material information of the video frame, and displays the video frame with the added sticker material, thereby realizing the addition of sticker special effects on the tracking target of the video.

[0121] In another embodiment, an editing preview pipeline and a special effect rendering pipeline can be constructed in the basic function layer of the application layer based on the capabilities of the basic function layer, and a target tracking pipeline can be constructed in the media platform framework layer based on the capabilities of the media platform framework layer, so as to realize special effect tracking processing of the video and preview display of the special effect tracking processing results by running the editing preview pipeline, target tracking pipeline and special effect rendering pipeline. For example, an editing preview pipeline is constructed according to the capabilities of the editing preview function in the basic function layer, and a special effect rendering pipeline is constructed according to the capabilities of the special effect rendering module in the basic function layer.

[0122] It should be noted that the pipeline is a linear communication model and can be regarded as an assembly line. In simple terms, it is a comprehensive solution for a system that provides a one-stop service for obtaining results. By running the editing preview pipeline, target tracking pipeline, and special effects rendering pipeline, special effects tracking processing of the video and preview display of the special effects tracking processing results can be realized. When the user performs intelligent special effects tracking on the video, it can achieve real-time preview of the video file after rendering sticker special effects on the tracking target.

[0123] Please refer to Figure 3 , Figure 3 1 is a schematic diagram of the software structure of another electronic device 100 provided in an embodiment of the present application. Figure 3 As shown, the software structure is divided from top to bottom into application layer, media middle platform framework layer, smart middle platform framework layer, smart engine framework layer, and application framework layer.

[0124] The application layer includes the gallery, which is divided into the business layer, application function layer and basic function layer from top to bottom. Figure 2 The related description of the embodiments of the present application will not be repeated here. The basic function layer includes a video editing pipeline built based on the video editing capabilities of the basic function layer, and a special effect rendering pipeline built based on the special effect rendering capabilities. The video editing pipeline includes a decoding node, a texture conversion node, and a display node connected in sequence. The special effect rendering pipeline includes at least a sticker rendering node. The sticker rendering node is located after the texture conversion node and before the display node.

[0125] The media platform framework layer includes a target tracking pipeline built based on the capabilities of the media platform framework layer. The target tracking pipeline includes a resolution reduction node, a texture conversion image frame node, and a target tracking algorithm node connected in sequence. The resolution reduction node is located after the texture conversion node and before the sticker rendering node. The connection relationship between each node can be as follows: Figure 3 shown.

[0126] The business layer can send the edited video files to the basic function layer according to the user's video editing operation. In the business layer, users can also set the tracking target in any video screen of the video through the intelligent special effects tracking module. The intelligent special effects tracking module determines the target frame position of the tracking target set by the user, and sends the video frame where the tracking target is located and the target frame position of the tracking target to the target tracking algorithm node of the media middle platform framework layer, and then sends it to the target tracking module of the smart middle platform framework layer through the target tracking algorithm node.

[0127] The decoding node is used to decode the video or video clip sent by the business layer in sequence to obtain a video frame. The texture conversion node is used to convert the video frame decoded by the decoding node into an image texture, and send the image texture to the resolution reduction node of the media middle platform framework. For example, the texture conversion node is a surface texture node, and the surface texture node can render the video frame decoded by the decoding node onto the surface and bind it to the corresponding image texture. The resolution reduction node is used to reduce the resolution of the image texture to obtain an image texture with a lower resolution. The texture to image frame node is used to convert the image texture after the resolution reduction process into a video frame, and send the video frame to the target tracking algorithm node. The target tracking algorithm node is used to call the target tracking module of the smart middle platform framework layer according to the video frame. After the target tracking module is called, the target tracking algorithm is used to track the target of the video frame according to the video frame where the tracking target is set by the user and the target frame position of the tracking target, and the target frame position of the video frame is obtained, and the target frame position of the video frame is returned to the target tracking algorithm node. The target tracking algorithm node sends the target frame position of the video frame to the intelligent special effects tracking module of the application layer through the intelligent special effects tracking module, and the intelligent special effects tracking module determines the sticker material information of the video frame according to the target frame position of the video frame, and writes the sticker material information of the video frame into the engineering data model, which is cached by the engineering data model so that it can be directly read from the engineering data model when needed later. Among them, the sticker material information includes information such as the material position and material size of the sticker material to be added. The intelligent special effects tracking module can send the sticker material information of the video frame to the sticker rendering node, or the sticker rendering node can read the sticker material information of the video frame from the engineering data model. After the sticker rendering node obtains the sticker material information of the video frame, the sticker material can be added to the corresponding video frame according to the sticker material information of the video frame to generate a video frame with the sticker material added, and the video frame with the sticker material added is sent to the display node for display, so that the video frame with the sticker material added is displayed on the display screen. By way of example, the display node can be a texture view node.

[0128] It is understandable that the special effects rendering pipeline may also include other special effects rendering nodes, such as transition rendering nodes, picture-in-picture synthesis nodes, etc. Other special effects rendering nodes may be located after the texture conversion node and before the sticker rendering node. The texture conversion node may first send the converted texture image to other special effects rendering nodes, which perform special effects rendering on the texture image and then send the texture image after special effects rendering to the resolution reduction node for processing. Among them, the transition rendering node is used to add transition special effects, and the picture-in-picture synthesis node is used to synthesize and display another image frame in an image frame, or to synthesize and display an image frame in another image frame.

[0129] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.

[0130] The execution subject of the video processing method provided in the embodiment of the present application can be the above-mentioned electronic device, or it can be a functional module and / or functional entity in the electronic device that can implement the video processing method, and the present application scheme can be implemented by hardware and / or software, which can be determined according to actual use requirements, and the embodiment of the present application is not limited. The following takes an electronic device as an example, and combines the accompanying drawings to exemplarily illustrate the video processing method provided in the embodiment of the present application.

[0131] The technical problems to be solved by this application and the inventive concept of this application are introduced below.

[0132] During video editing, users usually have the need to add stickers to the video screen. Currently, in response to the user's operation of adding stickers at a certain position on the video screen, the video editing application usually adds stickers at the same position of all video screens of the video, and generates a video with stickers added at the same screen position. This sticker adding method has low flexibility and a relatively simple sticker adding effect, and cannot meet the user's needs for personalized sticker adding.

[0133] For example, Figure 4 This is an example diagram of the effect of adding sticker material to a video provided in an embodiment of the present application. Figure 4 The video clip to be processed shown in (a) in FIG. 1 includes 5 video frames, and the video pictures corresponding to each video frame include a father, a mother and a child. Figure 4 Figure (b) in FIG. 1 shows the effect of adding stickers after the video clip is processed using the video processing method of the related technology. Figure 4As shown in Figure (b), if the user adds a sticker at the father's face position (upper left corner of the screen) on the first video frame, in response to the user's operation of adding a sticker, the current video processing method will add sticker material at the same screen position (upper left corner of the screen) of all video frames of the video, so that the five processed video frames all have sticker special effects added to the upper left corner of the screen. However, the actual demand of the user may be to block the father's face in all video frames in the video clip, but according to the current video processing method, when the position of the father's face in the video screen changes, the sticker material will not be able to block it, which cannot meet the user's needs.

[0134] In response to the above problems, an embodiment of the present application provides a video processing method. In order to improve the flexibility of adding stickers and enrich the effects of adding stickers, the present application proposes an intelligent special effects tracking solution. In this solution, the user can set a tracking target on a certain video screen of the video. After the user sets the tracking target, the first target frame position is determined (the target frame position of the tracking target on the video frame corresponding to the video screen), and the other video frames of the video are tracked based on the video frame, the first target frame position and the sticker material, so that the sticker material is added to the position of the tracking target on the other video frames, and the video frame with the added sticker material is displayed.

[0135] For example, Figure 5 Figure (c) shows the effect of adding stickers after the video clip is processed using the video processing method provided in the embodiment of the present application. Figure 5 As shown in Figure (b), if the user adds a sticker to the father's face position on the first video frame, in response to the user's operation of adding stickers, the solution provided in the embodiment of the present application can determine the tracking target (the target of adding stickers) according to the user's operation of adding stickers on any video frame, and then track the same target in other video frames, so that sticker materials are added to the same target in other video frames, so that sticker special effects are added to the same target (father's face) in the 5 processed video frames. In this way, it is possible to add sticker special effects to the same target in a video, meet the user's demand for using sticker special effects to cover the same target in a video, improve the flexibility of adding stickers, enrich the effect of adding stickers, and improve the user experience.

[0136] The other video frames may be all or part of the video frames except the video frame (reference video frame) in which the tracking target is set. For example, the other video frames may include a video segment after the reference video frame and / or a video segment before the reference video frame.

[0137] The user can set a target as a tracking target by dragging the sticker material to be added to a target in the reference video frame. Of course, the tracking target can also be set in other ways.

[0138] In addition, after receiving the operation of adding sticker special effects, the current electronic devices or image processing software provided by third parties usually need to finish all the processing of the video before exporting and displaying the processed video, resulting in low flexibility of processing and display, and users cannot see the processing results in time, which leads to poor user experience. For example, after the video is fully processed, the user can see the exported video with the sticker special effects added, and then judge whether the processing result meets the user's needs. If it does not meet the user's needs, the position of adding the sticker special effects needs to be readjusted, which may result in the need to process the video multiple times, wasting processing resources and low processing efficiency.

[0139] In an embodiment of the present application, while tracking other video frames of the video so that sticker materials are added to other video frames at the position of the same tracking target, the video frames with sticker materials added obtained by the tracking process can also be displayed. For example, the video frames with sticker materials added obtained by the tracking process are previewed to achieve real-time preview of the effects of adding sticker special effects, thereby improving the flexibility of the display of adding sticker special effects, ensuring that users can see the effects of adding sticker special effects in a timely manner, and improving user experience.

[0140] In the embodiment of the present application, the video to which the sticker material is to be added can be any video material, or a video generated by stitching together multiple image materials. The image material can be a picture or a video. For example, the video to which the sticker material is to be added can be any video imported by the user, or a video generated by stitching together multiple image materials imported by the user. The embodiment of the present application does not limit this.

[0141] the following Figures 5 to 8 It is a set of example diagrams of a graphical user interface (GUI) provided in an embodiment of the present application.

[0142] In one application scenario, for example, a user uses an electronic device to pre-shoot multiple image materials, and the electronic device stores the multiple image materials in a gallery. Figure 5 As shown in Figure (a) of FIG. 1 , an icon of a gallery is displayed on the desktop of the electronic device. When the user wants to edit a video based on the image material in the gallery, the user can click the icon of the gallery on the desktop of the electronic device. In response to the user's click operation on the icon, the electronic device displays the following Figure 5 The gallery interface shown in Figure (b) in FIG. 1 provides a "free creation" option, such as Figure 5As shown in Figure (c) of FIG. 1 , the user can click on the “free creation” option. In response to the user clicking on the “free creation” option, the electronic device displays the following Figure 5 The gallery interface shown in Figure (d) in FIG. 1 may include multiple image materials taken recently, such as multiple image materials taken on October 1, 2020. Figure 5 As shown in FIG. 1(e), the user can click on any one of the multiple image materials. In response to the user's click operation on the any one of the image materials, the electronic device displays the following Figure 5 The gallery interface shown in Figure (f) includes all the materials in the gallery. Figure 5 Select a video material or multiple image materials in the gallery interface shown in Figure (f) to edit the video. Figure 5 As shown in Figure (f) in , assume that the user selects 3 image materials. Figure 5 The gallery interface shown in Figure (f) also includes confirmation options, such as Figure 5 As shown in Figure (f), the confirmation option is checked.

[0143] like Figure 6 As shown in Figure (a) of FIG. 1 , after selecting an image material, the user can click a confirmation option, such as clicking a check mark. In response to the user's click operation on the check mark option, the electronic device displays the following Figure 6 The gallery interface shown in Figure (b) is a video editing interface for videos spliced ​​together from the selected 3 image materials. For example, a 20-second video material and 2 image materials can be spliced ​​together to generate a 24-second video. The video editing interface includes a video preview window, a video track, a preview progress bar, and a "sticker" option. When the user wants to set a tracking target for a sticker effect on a certain video frame of the video, the user can quickly preview the video by dragging the preview progress bar to adjust the video preview progress. For example, drag the preview progress bar forward to adjust the video preview progress forward, and drag the preview progress bar backward to adjust the video preview progress backward, so as to adjust the video preview progress to any video frame where the tracking target is desired. Figure 6 As shown in Figure (c) in FIG. 1 , when the preview progress is adjusted to "00:02" and the video preview window displays the video frame corresponding to "00:02", the "sticker" option is clicked. Of course, the user can also click the play button and click the "sticker" option when the video preview window plays to any video frame that the tracking target is desired to be set. In response to the user's click operation on the "sticker" option, the electronic device displays the following Figure 6The gallery interface shown in FIG. (d) includes multiple sticker materials. The user can click on the sticker material he wants to add, such as the sticker material "strawberry head". In response to the user clicking on the sticker material "strawberry head", the electronic device displays the following Figure 6 As shown in the gallery interface of Figure (e), the sticker material "Strawberry Head" can be displayed at a preset position of the current video frame, such as the center of the current video frame, and the user can set the tracking target by dragging the sticker material. The gallery interface also includes a "special effect tracking" option. Figure 6 As shown in Figure (f) in FIG. 1 , if the user wants to set the father's face in the current video frame as the tracking target, the user can drag the sticker material to the father's face position and then click the "special effect tracking" option. In response to the user clicking the "special effect tracking" option, the electronic device can determine the target frame position according to the position of the sticker material "strawberry head" to determine the target corresponding to the position of the sticker material "strawberry head" as the tracking target, and then track other video frames according to the current video frame, the target frame position and the sticker material "strawberry head" so that the sticker material "strawberry head" is added to the position of the same tracking target (father's face) in other video frames.

[0144] During the process of tracking other video frames, the electronic device can also preview and display other video frames with sticker materials added. Figure 7 , you can preview other video frames of the sticker material "Strawberry Head" displayed on the tracking target (father's face) in the video preview window. Figure 7 As shown, the current gallery interface also includes a sticker track associated with the video track, and the sticker track is used to indicate the sticker material added corresponding to the video frame on the video track. In addition, the current gallery interface also includes a "Cancel Tracking" option, and the "Cancel Tracking" option is used to cancel the intelligent tracking processing effect. The user can click the "Cancel Tracking" option, and in response to the operation of clicking "Cancel Tracking", the electronic device can cancel the intelligent tracking processing effect. It should be understood that the "Cancel Tracking" option can also be set in other locations of the current gallery interface, or triggered by other options. For example, please refer to Figure 8 In the (a) figure, during the process of previewing the intelligent tracking processing effect in the video preview window, the user can click the "sticker" option. In response to the user clicking the "sticker" option, the electronic device displays Figure 8 The gallery interface shown in Figure (b) includes multiple sticker materials and a "Cancel Tracking" option. The user can click the "Cancel Tracking" option to cancel the current smart tracking processing effect.

[0145] Please refer to Fig. 9 , Fig. 91 is a schematic diagram of a group of cancellation intelligent tracking processing effects provided by an embodiment of the present application. Fig. 9 As shown in Figure (a) in FIG. 1 , after the user clicks the “Cancel Tracking” option, the electronic device can cancel the intelligent tracking processing effect and add stickers to the same position of other video frames according to the position where the user added the sticker (the position where the “Strawberry Head” sticker was added on the first video frame) in accordance with the conventional sticker adding method in the related art, thereby obtaining a video with the “Strawberry Head” sticker added on the same position. Fig. 9 As shown in Figure (b) in FIG. 1 , after the user clicks the “Cancel Tracking” option, the electronic device can also cancel the intelligent tracking processing effect and add stickers at the center of the video screen of all video frames according to the default sticker adding position (the center of the video screen), thereby obtaining a video with the “Strawberry Head” sticker added at the center of the video screen. Fig. 9 As shown in Figure (c), after the user clicks the "Cancel Tracking" option, the electronic device can cancel the intelligent tracking processing effect and no longer perform other sticker adding processing, thereby restoring the original video without adding stickers.

[0146] In addition, after the electronic device completes the tracking processing of all other video frames, it can automatically export the video generated after the tracking processing and store the video in the gallery, and the user can view the video from the gallery. Figure 6 In Figure (f) in the figure, the user can also click on the "Export" option. In response to the user's click on the "Export" option, the electronic device exports the generated video after completing the tracking and processing of all the videos, and stores the video in the gallery. In this way, the user can view the video from the gallery.

[0147] It should be noted that Figure 6 It is merely taken as an example that a target is set as a tracking target by dragging a sticker material to the position of the target. It should be understood that the tracking target may also be set in other ways, and the embodiments of the present application are not limited to this.

[0148] In addition, after the sticker material selected by the user is displayed at the preset position of the current video frame, the user can also perform a scaling operation (zoom-out operation or zoom-in operation) on the sticker material, such as zooming out the sticker material by dragging the border of the sticker material inward, or zooming in the sticker material by dragging the border of the sticker material outward. After scaling the sticker material, the scaled sticker material can be dragged to the position of the tracking target. Alternatively, the sticker material can be dragged to the position of the tracking target first, and then the sticker material can be scaled.

[0149] In one example, Figure 6The gallery interface displayed in Figure (b) or (c) also includes other functional options, which may include but are not limited to options such as editing, filters, music, text, picture-in-picture, and special effects. After the electronic device tracks and processes the video to generate a video with stickers added, the user can perform operations such as filtering, adding music, adding text, picture-in-picture processing, and adding special effects on the video with stickers added based on these functional options. Alternatively, the user can also perform operations such as filtering, adding music, adding text, picture-in-picture processing, and adding special effects on the video with stickers added based on these functional options, and then add stickers to the processed video, which is not limited in the embodiments of the present application.

[0150] Fig.10 and Fig.11 1 is a set of example diagrams of GUI provided in an embodiment of the present application. In another application scenario, for example, a user uses an electronic device to pre-shoot multiple image materials, and the electronic device stores the multiple image materials in a gallery. In this way, the user can select an image material in the gallery to enter the video editing interface of the video corresponding to the selected image material.

[0151] For example, Fig.10 As shown in Figure (a) of FIG. 1 , after entering the gallery interface, the gallery interface includes some recently taken image materials, and the user can click on multiple materials in these image materials, such as clicking on the first image material. In response to the user's click operation on any of the materials, the electronic device displays the following Fig.10 The gallery interface shown in (b) of FIG. 1 includes all image materials in the gallery, including pictures and videos. When the user wants to edit a video in the image materials, the user can select the video to be edited from the multiple image materials. In response to the user's selection of the video, the gallery interface can display an "Edit" option. The user can click on the "Edit" option, and in response to the user clicking on the "Edit" option, the electronic device displays the following Fig.10 The gallery interface shown in (c) is the video editing interface of the video, and the video editing interface includes a "sticker" option. After that, the user can edit the video in the video editing interface, for example, click the "sticker" option to enter the sticker adding interface to trigger the electronic device to execute the video processing method provided by the embodiment of the present application on the video.

[0152] For example, Fig.11 As shown in Figure (a) of FIG. 1 , after entering the gallery interface, the gallery interface includes some recently taken image materials, and the user can click on multiple materials in these image materials, such as clicking on the first image material. In response to the user's click operation on any of the materials, the electronic device displays the following Fig.11The gallery interface shown in (b) of FIG. 1 includes all the image materials in the gallery. The user can also select multiple image materials from these image materials, such as Fig.11 As shown in Figure (b) of FIG. 1 , assuming that the user selects the first three image materials, in response to the user's selection of the three image materials, the gallery interface may display a "Create" option. The user may click on the "Create" option, and in response to the user's clicking on the "Create" option, the electronic device may display the following Fig.11 The gallery interface shown in (c) of FIG. 1 includes a "free creation" option, and the user can choose to click the "free creation" option. In response to the user clicking the "free creation" option, the electronic device can stitch the three image materials together and display a video editing interface of the video stitched together from the three image materials, such as displaying Figure 6 The video editing interface shown in (b) of FIG. Afterwards, the user can edit the video in the video editing interface, such as clicking the "sticker" option to enter the sticker adding interface to trigger the electronic device to execute the video processing method provided in the embodiment of the present application on the video.

[0153] It should be understood that the embodiments of the present application may also use other operating methods to enter the video editing interface, so as to add sticker special effects to the video through the video editing interface, thereby triggering the electronic device to execute the video processing method provided by the embodiments of the present application on the video. The embodiments of the present application do not limit the operating method for entering the video editing interface of the video.

[0154] Next, the video processing algorithm involved in the embodiments of the present application is described in detail.

[0155] In order to improve the efficiency of video tracking processing and realize real-time preview of tracking processing results, in the process of tracking processing other video frames of the video according to the reference video frame with the tracking target set, the target frame position of the tracking target and the sticker material, the video frame to be processed can be obtained from the other video frames in turn, and the target frame position can be tracked on the obtained video frame according to the reference video frame and the target frame position. When the target frame position of the video frame is obtained by target tracking, the sticker material information of the video frame is determined according to the target frame position of the video frame, and the sticker material is added to the video frame according to the sticker material information of the video frame to generate a second video frame with the sticker material added at the position of the tracking target. After the second video frame is generated, the second video frame can also be previewed. The sticker material information includes the material position of the sticker material and can also include the material size of the sticker material.

[0156] Please refer to Fig.12 , Fig.12 is a flow chart of a video processing method provided by an embodiment of the present application, such as Fig.12As shown, the video processing method includes the following steps A1 to A8:

[0157] A1: Display a first video picture of the video, where the first video picture may be a video picture corresponding to any video frame of the video.

[0158] The first video screen is a video screen of a target to be tracked. The electronic device may display the first video screen during the video playback process, or may display the first video screen according to a preview operation of the user.

[0159] For example, please refer to Figure 6 In Figure (b), the user can adjust the video preview progress to preview and display the first video screen by dragging the preview progress bar of the video.

[0160] A2: In response to the user setting the first target to be tracked on the first video screen and confirming the operation of performing sticker tracking, the first target frame position is determined, and the first target frame position is used to indicate the position of the first target on the first reference video frame corresponding to the first video screen.

[0161] In an embodiment of the present application, the user can set the first target in the first video screen as the target to be tracked on the first video screen, and the target to be tracked refers to the target to be tracked and to which the sticker material is added. For example, after selecting the first sticker material, the user can set the first target as the first target to be tracked by dragging the first sticker material to the position of the first target in the first video screen, and confirm to perform sticker tracking on the first target. Sticker tracking refers to tracking the first target in other video frames and adding the first sticker material to the tracked first target.

[0162] After the user sets the first target to be tracked on the first video screen and confirms the operation of performing sticker tracking, the electronic device can determine the first target frame position, which refers to the target frame position of the first target, and is used to indicate the boundary of the image area where the first target is located. The target frame is usually a rectangular frame. The target frame position can be the coordinates of the target frame, such as the coordinates of at least two vertices of the target frame, such as the coordinates of the upper left and lower right vertices of the target frame, or the coordinates of the upper right and lower vertices, or the coordinates of four vertices, etc.

[0163] For example, after the user drags the first sticker material to the location of the first target in the first video screen and confirms to perform the sticker tracking operation, the first target frame position can be determined according to the material position of the first sticker material. For example, the first target covered by the first sticker material can be determined according to the material position of the first sticker material, and then the first target frame position can be determined according to the first target.

[0164] It should be noted that the first target can be one or more. That is, the user can set one first target on the first video screen, or multiple first targets. In the case of multiple first targets, the first target frame position corresponding to each first target can be determined, thereby obtaining multiple first target frame positions, so that other video frames can be tracked and processed later according to the first reference video frame, multiple first target frame positions and the first sticker material.

[0165] Wherein, in the case where multiple first targets are set on the first video screen, the first sticker material may include one or more. Each first target corresponds to a first sticker material to be added, and the first sticker materials corresponding to the multiple first targets may be the same or different. For example, other video frames may be tracked based on the first reference video frame, multiple first target frame positions, and the first sticker material corresponding to each of the multiple first target frame positions.

[0166] In an embodiment of the present application, after determining the position of the first target frame, the first video segment of the video can be tracked based on the first reference video frame, the first target frame position and the first sticker material. The first video segment is a video segment in the video that needs to be tracked. The first video segment includes at least one video segment. For example, the first video segment includes a video segment after the first reference video frame, and / or a video segment before the first reference video frame. The first video segment includes at least one video segment, and the at least one video segment can be tracked separately.

[0167] In addition, when the first sticker material is not added to the first reference video frame, the first video segment may also include the first reference video frame. For example, the first video segment may be all video frames of the video.

[0168] Next, the tracking process of the target video segment in the first video segment is described in detail.

[0169] A3: Obtain the video frames to be processed from the target video segment of the video in sequence.

[0170] The target video segment is any video segment in the video to be tracked, and the target video segment may include one or more video frames. For example, the target video segment may be a video segment after the first reference video frame, or a video segment before the first reference video frame.

[0171] After determining the first target frame position, the video frames to be processed can be obtained from the target video segment of the video in sequence, so as to track the video frames in the target video segment in sequence according to the first reference video frame, the first target frame position and the first sticker material.

[0172] In one embodiment, sequentially acquiring the video frames to be processed from the target video segments of the video may also include the following implementation methods:

[0173] The first implementation method: when the target video segment is a video segment located after the first reference video frame, the target video frames to be processed are obtained from the target video segment in sequence according to the forward playback order of the video to perform forward tracking processing on the target video segment.

[0174] For example, when the first reference video frame is the first video frame of the video, the video frames to be processed are obtained from the video frames after the first reference video frame in sequence according to the forward playback order of the video.

[0175] Please refer to Fig.13 , Fig.13 FIG. 1 is a schematic diagram of a forward tracking process for a video provided by an embodiment of the present application. Fig.13 As shown, when a tracking target is specified on the first video frame of a video, the video can be forward tracked, that is, the video frames to be processed are obtained from the video frames after the first video frame in the forward playback order of the video, and the obtained video frames are tracked, that is, forward tracking.

[0176] The second implementation method: when the target video segment is a video segment before the first reference video frame, the target video frames to be processed are obtained from the target video segment in sequence according to the reverse video playback order to perform reverse tracking processing on the target video segment.

[0177] For example, when the first reference video frame is the last video frame of the video, the video frames to be processed are obtained from the video frames before the first reference video frame in sequence according to the reverse playback order of the video.

[0178] Please refer to Fig.14 , Fig.14 Schematic diagram of a reverse tracking process of a video provided by an embodiment of the present application. Fig.14 As shown, when a tracking target is specified on the last video frame of a video, the video can be subjected to reverse tracking processing, that is, the video frames to be processed are obtained from the video frames before the first video frame in sequence according to the reverse playback order of the video, and tracking processing is performed on the obtained video frames, that is, reverse tracking processing.

[0179] In one embodiment, the first video segment includes a video segment after the first reference video frame and a video segment before the first reference video frame. The video segment after the first reference video frame can be forward tracked, and the video segment before the first reference video frame can be backward tracked.

[0180] For example, when the first reference video frame is neither the first video frame nor the last video frame, the target video frame to be processed is obtained from the video frames after the first reference video frame in a forward playback order of the video (increasing direction of video time), and / or, the target video frame to be processed is obtained from the video frames before the first reference video frame in a reverse playback order of the video (decreasing direction of video time).

[0181] Please refer to Fig.15 , Fig.15 Schematic diagram of a forward tracking process and a reverse tracking process of a video provided by an embodiment of the present application. Fig.15 As shown, when a tracking target is specified on a video frame at a middle position, the video can be tracked forward and backward. For example, first obtain the video frame to be processed from the video frames after the video frame in the forward playback order of the video, and track the obtained video frame. Then, obtain the video frame to be processed from the video frames before the video frame in the reverse playback order of the video, and track the obtained video frame. Of course, it is also possible to perform backward tracking first and then forward tracking, which is not limited in the embodiments of the present application. Alternatively, it is also possible to perform only forward tracking or only backward tracking.

[0182] By taking the first reference video frame as a reference and performing backward tracking processing and forward tracking processing on the front and rear video frames respectively, the tracking accuracy and the continuity of the change of the tracking target can be guaranteed.

[0183] A4: According to the first reference video frame and the first target frame position, target tracking is performed on the acquired first video frame, where the first video frame is any video frame in the target video segment.

[0184] In an embodiment of the present application, target tracking can be performed on the acquired first video frame based on the first reference video frame and the first target frame position to determine the target frame position of the first video frame, that is, the target frame position corresponding to the first target in the first video frame.

[0185] As an example, target tracking can be performed on the acquired first video frame according to the first reference video frame and the first target frame position. For example, the image area corresponding to the first target frame in the first reference video frame can be determined according to the first reference video frame and the first target frame position, and then the first target in the first video frame can be tracked according to the image area to determine the target frame position corresponding to the first target in the first video frame.

[0186] That is, the first reference video frame and the first target frame position can be used to characterize the target features of the first target, so according to the first reference video frame and the first target frame position, the target to be tracked can be identified as the first target, and then the first target in the first video frame is tracked to determine the target frame position corresponding to the first target in the first video frame. In this way, the accuracy of target tracking can be improved.

[0187] In some embodiments, a target tracking algorithm may be used to track a target in the acquired first video frame according to the first reference video frame and the first target frame position. The target tracking algorithm is used to determine the target frame position of a specific target in the video frame to be tracked according to the reference video frame and the target frame position of the specific target in the reference video frame.

[0188] In some embodiments, the target tracking algorithm may be a machine learning model, which may be trained through a training sample set. The training sample set may include multiple image samples and annotation information corresponding to each image sample, and the annotation information includes the target frame position.

[0189] Furthermore, before step A4, the target type of the first target can also be determined, and based on the target type, a target tracking algorithm that matches the target type is determined from a plurality of target tracking algorithms, and then, based on the first reference video frame and the first target frame position, the determined target tracking algorithm is used to perform target tracking on the acquired video frame.

[0190] In the embodiment of the present application, multiple target tracking algorithms corresponding to different target types (human faces, other objects, etc.) can be pre-stored, and multiple target tracking algorithms can be used to track targets of different target types. When actually tracking the target, different target tracking algorithms can be used to track the target for different targets. In this way, the accuracy of target tracking can be improved.

[0191] A5: When the target frame position of the first video frame is obtained by target tracking on the first video frame, the sticker material information of the first video frame is determined according to the target frame position of the first video frame.

[0192] The sticker material information includes at least the material position of the first sticker material to be added. The first sticker material to be added may be a preset sticker material or a sticker material selected by a user. The sticker material information may also include information such as the material size (material size) of the first sticker material, which is not limited in the embodiment of the present application. The material position is used to indicate the adding position of the first sticker material, and the material size is used to indicate the target size of the first sticker material.

[0193] Two tracking results may be obtained when performing target tracking on the first video frame. The first tracking result is that the target frame position of the first video frame is tracked (the target frame position of the first target in the first video frame), for example, when the first target exists in the first video frame, the target frame position of the first video frame can be tracked. The second tracking result is that the target frame position of the first video frame is not tracked, for example, when the first target is missing or lost, the target frame position of the first video frame cannot be tracked. In a possible implementation, the second tracking result can be indicated by an exception mark. For example, performing target tracking on the first video frame can obtain an exception mark indicating that the target frame position of the first video frame is not tracked, for example, the exception mark can indicate that the target is missing or missing.

[0194] When the first tracking result is obtained, the target frame position of the first video frame obtained by tracking can be converted into sticker material information of the first video frame, so that the first sticker material can be added to the first video frame according to the sticker material information of the first video frame.

[0195] As an example, the material position of the first sticker material can be determined according to the target frame position of the first video frame using a preset conversion algorithm, or the material position and material size (material size) of the first sticker material can be determined, so that when a sticker is added to the first video frame according to the obtained sticker material information, the first sticker material can cover the image area corresponding to the target frame position.

[0196] A6: When target tracking is performed on the first video frame and the target frame position of the first video frame is not tracked, the historical sticker material information (ie, the sticker material information of the previous video frame) is determined as the sticker material information of the first video frame.

[0197] Among them, when there is a tracked video frame before the first video frame, the historical sticker material information is the sticker material information of the most recently tracked video frame before the first video frame; when there is no tracked video frame before the first video frame, the historical sticker material information is the sticker material information of the first reference video frame.

[0198] In this way, when the second tracking result mentioned above is obtained, that is, when the target is lost or missing, by using the sticker material information of the previous video frame as the sticker material information of the current video frame, it is convenient to subsequently use the sticker material information of the previous video frame to add the first sticker material to the first video frame, thereby improving the smoothness and accuracy of adding stickers on continuous video frames and avoiding the situation where sticker special effects are missed on continuous video frames.

[0199] A7: Adding the first sticker material to the first video frame according to the sticker material information of the first video frame to generate a second video frame having the first sticker material added to the first object.

[0200] In the case where the sticker material information includes a material position, the first sticker material may be added to the position indicated by the material position in the first video frame. The material size of the added first sticker material may be an initial size, and the initial size may be a preset size (default size) or the size of the first sticker material added to the first reference video frame.

[0201] In the case where the sticker material information includes a material position and a material size, if the initial size of the first sticker material is not the material size, the first sticker material may be scaled according to the material size, and then the scaled first sticker material may be added to the position indicated by the material position in the first video frame. If the initial size of the first sticker material is the material size, the first sticker material of the original size may be directly added to the position indicated by the material position in the first video frame.

[0202] A8: Preview and display the second video frame.

[0203] For example, the second video frame can be displayed on the video preview interface, so that the user can check the sticker adding effect corresponding to the first video frame in time.

[0204] That is, in the embodiment of the present application, the video frame with the first sticker material added after the tracking process is completed can be previewed and displayed, so that the user can check the sticker adding effect in time. For example, whenever a second video frame with the first sticker material added is generated, the second video frame is previewed and displayed.

[0205] For example, please refer to Figure 6 As shown in Figure (f) in FIG. 1 , after the user clicks the smart tracking option, the electronic device can perform smart tracking processing on other video frames in sequence according to the video processing method provided in the embodiment of the present application, and Figure 6 The video preview window shown in Figure (f) shows the video frame with the first sticker material added after the tracking process is completed.

[0206] It should be noted that Fig. 9The embodiment is only described by taking the example of obtaining the video frames to be processed from the target video segment of the video in sequence after step A2, and performing tracking processing on the obtained video frames in sequence according to the first reference video frame, the first target frame position and the first sticker material. It should be understood that in some embodiments, after step A2, the key frames to be processed can also be obtained in sequence from the target video segment of the video, and the obtained key frames can be tracked in sequence according to the first reference video frame, the first target frame position and the first sticker material.

[0207] That is, in an embodiment of the present application, video frames in the target video clip can be acquired in sequence to perform target tracking on the acquired video frames, or key frames in the target video clip can be acquired in sequence to perform tracking processing on the acquired key frames, so that the key frames after tracking processing have the first sticker material added at the position of the first target.

[0208] In the process of sequentially tracking the acquired key frames, for the intermediate video frame between two adjacent key frames, the sticker material information of the intermediate video frame can also be determined according to the target frame positions or sticker material information of the two adjacent video frames, and then the first sticker material is added to the intermediate video frame according to the sticker material information of the intermediate video frame. After obtaining the intermediate video frame and the first key frame with the first sticker material added, the intermediate video frame and the first key frame with the first sticker material added are previewed and displayed in sequence.

[0209] By tracking the key frames in the video to determine the sticker position information of the key frames, and then determining the sticker material information of the middle video frame based on the target frame position or sticker material information of the two adjacent video frames, the continuity of the sticker position between the two adjacent video frames can be ensured, thereby ensuring the continuity of the target movement. In addition, the target tracking processing volume can be reduced, the target tracking efficiency can be improved, and the efficiency of determining sticker material information and sticker rendering can be improved.

[0210] Among them, determining the sticker material information of the middle video frame according to the target frame positions or sticker material information of two adjacent video frames may include the following two implementation methods:

[0211] The first implementation method is to perform linear interpolation based on the sticker material information of two adjacent key frames to obtain the sticker material information of the middle video frame.

[0212] That is, linear interpolation can be performed based on the sticker material information of two adjacent key frames, and the sticker material information obtained by the linear interpolation is used as the sticker material information of the middle video frame.

[0213] For example, after obtaining the sticker material information of a certain key frame (the first key frame), for the intermediate video frame between the first key frame and the third video frame, linear interpolation can be performed according to the sticker material information of the first key frame and the sticker material information of the third video frame to obtain the target frame position of the intermediate video frame. Then, according to the target frame positions of the intermediate video frame and the first key frame, the sticker material information of the intermediate video frame and the first key frame are determined respectively, and then according to the sticker material information of the intermediate video frame and the first key frame, the first sticker material is added to these video frames respectively, so that these intermediate video frames have the first sticker material added to the first target. After all these video frames (intermediate video frames and the first key frame) are processed, these video frames are previewed and displayed in sequence. Among them, when the first key frame is the first key frame tracked in the target video segment, the third video frame is the first reference video frame, and when the first key frame is not the first key frame, the third video frame is the key frame that has been tracked most recently before the first key frame.

[0214] Please refer to Fig.16 , Fig.16 Schematic diagram of interpolating sticker positions of intermediate video frames between adjacent key frames provided by an embodiment of the present application. Fig.16 As shown in the figure, after determining the sticker positions of two adjacent key frames, for the intermediate video frame between the two key frames, linear interpolation can be performed based on the sticker positions of the two key frames before and after to calculate the sticker position of the current video frame, and then the sticker is rendered on the current video frame based on the sticker position, thereby obtaining a video frame with stickers added. In this way, the continuous movement of the sticker position between two adjacent key frames can be achieved.

[0215] The second implementation method is: linear interpolation is performed according to the target frame positions of two adjacent key frames to obtain the target frame position of the middle video frame, and then the sticker material information of the middle video frame is determined according to the target frame position of the middle video frame.

[0216] That is, linear interpolation can be performed according to the target frame positions of two adjacent key frames, and the target frame position obtained by linear interpolation is used as the target frame position of the middle video frame.

[0217] For example, after tracking a certain key frame (first key frame) to obtain the target frame position of the first key frame, for the intermediate video frame between the first key frame and the third video frame, linear interpolation can be performed based on the target frame position of the first key frame and the target frame position of the third video frame to obtain the target frame position of the intermediate video frame. Then, based on the target frame positions of the intermediate video frame and the first key frame, the sticker material information of the intermediate video frame and the first key frame is determined respectively, and then based on the sticker material information of the intermediate video frame and the first key frame, the first sticker material is added to these video frames respectively, so that these intermediate video frames have the first sticker material added to the first target. After all these video frames (intermediate video frames and first key frames) are processed, these video frames are previewed and displayed in sequence.

[0218] In addition, in the embodiment of the present application, in the process of tracking the target video clip, the synchronous tracking interface mode can be used for tracking, and the asynchronous tracking interface mode can also be used for tracking, which is not limited in the embodiment of the present application. Next, the synchronous tracking interface mode and the asynchronous tracking interface mode are respectively described.

[0219] 1) Synchronous tracking interface mode: The video frames to be processed are obtained from the target video clip in sequence, and each time a video frame is obtained, the video frame is tracked and processed. After the tracking process of the video frame is completed, the next video frame is obtained and the tracking process is performed on the next video frame.

[0220] That is, after the tracking process of one video frame is completed, the tracking process of the next video frame is performed. In this way, the complexity of the video processing algorithm can be reduced.

[0221] For example, when a video frame is acquired, target tracking is performed on the video frame according to the first reference video frame and the first target frame position, the target frame position of the video frame is obtained according to the tracking, the sticker material information of the video frame is determined, and the first sticker material is added to the video frame according to the sticker material information of the video frame to generate a second video frame. After the second video frame is generated, the next video frame is acquired, and target tracking and subsequent processing are performed on the next video frame.

[0222] Furthermore, after the tracking process is completed for the video frame, a second video frame generated after the tracking process is completed is displayed, and then the next video frame is acquired and the tracking process is performed on the next video frame.

[0223] 2) Asynchronous tracking interface mode: The video frames to be processed are obtained from the target video clip in sequence, and the obtained video frames are input into the tracking queue, which is used to store the video frames to be tracked. On the one hand, the video frames to be tracked are obtained from the tracking queue in sequence, and the obtained video frames are subjected to target tracking according to the first reference video frame and the first target frame position. On the other hand, the target frame position of the tracked video frame is obtained from the tracking results corresponding to the tracking queue in sequence, and the sticker material information of the video frame is determined according to the obtained target frame position of the video frame, and the first sticker material is added to the video frame according to the sticker material information of the video frame to generate a second video frame.

[0224] For the sake of convenience, the process of determining the sticker material information of a video frame according to the target frame position of the video frame, and adding the first sticker material to the video frame according to the sticker material information of the video frame to generate a second video frame is called the sticker processing process. In the asynchronous tracking interface mode, the target can be tracked on the video frame to be tracked while the sticker processing can be performed on the video frame that has been tracked, and the two can be performed synchronously. In this way, the video processing efficiency can be improved.

[0225] In some embodiments, in the asynchronous trace interface mode, a buffer smoothing mode is also supported to ensure the smoothness of the trace processing output through the buffer smoothing mode. The buffer smoothing mode may include the following processing situations:

[0226] The first processing situation: When the sticker processing speed is greater than the target tracking speed, the callback frequency is adjusted according to the target tracking speed so that the input frequency of the sticker processing is adapted to the slower target tracking speed, thereby ensuring the continuous smoothness of the tracking processing output.

[0227] The callback frequency refers to the frequency of obtaining the tracking result of the latest video frame from the tracking result corresponding to the tracking queue. The sticker processing speed can be the real-time sticker speed or the average speed of sticker processing. The target tracking speed is the real-time target tracking speed or the average speed of target tracking.

[0228] The second case: when the sticker processing speed is lower than the target tracking speed, the callback frequency is adjusted according to the sticker processing speed so that the input frequency of the sticker processing is adapted to the slower sticker processing speed, thereby ensuring the continuous smoothness of the tracking processing output.

[0229] In addition, in the embodiment of the present application, it is also possible to support setting tracking targets on multiple video screens of a video, such as setting different tracking targets on multiple video screens. As an example, after setting a tracking target on a video frame, it is also possible to support setting a new tracking target on other video frames.

[0230] After a new tracking target is set on another video frame, the video segment located at a specified side of the video frame can be tracked according to the newly set tracking target, where the specified side refers to a side that is not adjacent to the video frame with the set tracking target. The intermediate video segment between the video frame and the video frame with the set tracking target can be tracked according to the newly set tracking target, according to the set tracking target, or according to the tracking target selected by the user from the newly set tracking target and the set tracking target.

[0231] For example, after the user sets the first target to be tracked on the first video screen and confirms the operation of executing sticker tracking, the user can also set the second target to be tracked on the displayed second video screen and confirm the operation of executing sticker tracking. The second video screen is a video screen corresponding to any video frame in the video except the first reference video frame. The operation of the user setting the second target to be tracked on the displayed second video screen and confirming the operation of executing sticker tracking is the same as the operation of setting the first target to be tracked on the first video screen and confirming the operation of executing sticker tracking. The specific operation method can refer to the relevant operation description of the operation of setting the first target to be tracked on the first video screen and confirming the operation of executing sticker tracking, and the embodiment of the present application does not limit this.

[0232] In response to the user setting the second target to be tracked on the second video screen and confirming the operation of performing target tracking, the second target frame position is determined, and target tracking is performed on the second video segment in the video based on the second reference video frame, the second target frame position and the second sticker material, so that the second sticker material is added to the position of the second target on the second video segment, and the second video segment with the second sticker material added to the position of the second target is displayed.

[0233] The second target frame position is a target frame corresponding to the second target, and is used to indicate the position of the second target on the second reference video frame corresponding to the second video picture.

[0234] The second sticker material is a sticker material corresponding to the second target, that is, a sticker material to be added to the second target. The second sticker material can be a sticker material selected by the user, or a sticker material set by default, which is not limited in the embodiment of the present application. In addition, the second sticker material and the first sticker material can be the same sticker material, or different sticker materials, which is not limited in the embodiment of the present application.

[0235] The second video segment may be all or part of the video frames except the second reference video frame. For example, the second video segment includes a video segment after the second reference video frame and / or a video segment before the second reference video frame.

[0236] In addition, when the second sticker material is not added to the second reference video frame, the second video segment may also include the second reference video frame. For example, the second video segment may be all video frames of the video.

[0237] As an example, when the user specifies the tracking target in the first reference video frame and the second reference video frame respectively, the first video segment and the second video segment to be tracked may include the following situations:

[0238] In the first case, when the first sticker material is added to the first reference video frame and the second sticker material is added to the second reference video frame, the first video segment is all video frames in the video except the first reference video frame, and the second video segment is all video frames in the video except the second reference video frame. Alternatively, when the first sticker material is not added to the first reference video frame and the second sticker material is not added to the second reference video frame, the first video segment and the second video segment are all video frames.

[0239] In this way, the first sticker material can be added to the first target of all video frames of the video, and the second sticker material can be added to the second target of all video frames of the video, so that the video frame to which the first sticker material and the second sticker material are added at the same time can be displayed.

[0240] In the second case, the first video segment includes a video segment located at a first side of the first reference video frame, where the first side of the first reference video frame refers to a side that is not adjacent to the second reference video frame. The second video segment includes a video segment located at a second side of the second reference video frame, where the second side of the second reference video frame refers to a side that is not adjacent to the first reference video frame.

[0241] That is, for the video clip located on the first side of the first reference video frame (the video clip on the side not adjacent to the second reference video frame), tracking processing is performed based on the first reference video frame, the first target frame position and the first sticker material. For the video clip located on the second side of the second reference video frame (the video clip on the side not adjacent to the first reference video frame), tracking processing is performed based on the second reference video frame, the second target frame position and the second sticker material.

[0242] In addition, for the intermediate video segment (video frame of the adjacent area) between the first reference video frame and the second reference video frame, the intermediate video segment may be included in the first video segment or in the second video segment. That is, for the intermediate video segment, tracking processing may be performed based on the first reference video frame, the first target frame position and the first sticker material, that is, tracking processing is performed with the first target as the tracking target, or tracking processing may be performed based on the second reference video frame, the second target frame position and the second sticker material, that is, tracking processing is performed with the second target as the tracking target.

[0243] Please refer to Fig.17 , Fig.17 This is a schematic diagram of a video processing effect of setting a tracking target on multiple video frames provided by an embodiment of the present application. Please refer to Fig.17 In Figure (a), Fig.17 The video clip shown in (a) in FIG. 1 includes 5 video frames. The user sets the father's face as the first target to be tracked in the second video frame, and sets the current face as the second target to be tracked in the fourth video frame. In response to the user's setting operation, please refer to Fig.17 In Figure (b), for the video frame located on the first side (left side) of the second video frame, tracking processing can be performed based on the second video frame (first reference video frame), the first target frame position and the first sticker material (strawberry head), so that the first video frame has a "strawberry head" sticker added to the father's face. For the video frame located on the first side (right side) of the fourth video frame, tracking processing can be performed based on the fourth video frame (second reference video frame), the second target frame position and the second sticker material (glowing heart), so that the fifth video frame has a "glowing heart" sticker added to the mother's face. As for the middle video segment (third video frame) between the second and fourth video frames, tracking processing can be performed based on the second video frame (first reference video frame), the first target frame position and the first sticker material (strawberry head) according to the processing method of the second video frame, so that the third video frame has a "strawberry head" sticker added to the father's face. In addition, please refer to Fig.17 In Figure (c), for the middle video segment (the third video frame), tracking processing can also be performed based on the processing method of the fourth video frame, according to the fourth video frame (the second reference video frame), the second target frame position and the second sticker material (glowing heart), so that the third video frame has a "glowing heart" sticker added to the mother's face.

[0244] Furthermore, for the intermediate video segment between two reference video frames, the electronic device may also display first prompt information, the first prompt information being used to prompt the user to specify a tracking target for the intermediate video segment from the first target and the second target, that is, prompting the user on which side the tracking target is to be used for tracking processing. In this way, the flexibility of adding stickers can be further improved, meeting the user's diverse and personalized needs for adding stickers.

[0245] For example, in response to the user setting a second target to be tracked on the second video screen and confirming the operation of performing target tracking, the first prompt information is displayed. In response to the user confirming the operation of tracking the target, the intermediate video segment is tracked according to the reference video frame, the target frame position and the sticker material corresponding to the tracking target confirmed by the user.

[0246] In addition, in the embodiment of the present application, for a video frame on which a tracking target has been set, it is also supported to modify the tracking target set thereon, so that the electronic device re-performs tracking processing according to the modified tracking target.

[0247] In addition, in an embodiment of the present application, for video frames with tracking targets set, it is also supported to delete the tracking targets set thereon. If there are no video frames with tracking targets set after deletion (i.e., the tracking targets are set on only one video frame of the entire video), the intelligent tracking processing effect can be canceled, and the sticker material information of all video frames of the video can be restored to the initial default state, or consistent with the processing effect of the current video frame. In addition, if there are other video frames with tracking targets set after deletion, target tracking can be re-performed according to the tracking target of the nearest video frame with tracking targets set. If there are video frames with tracking targets set before and after the current one, target tracking can be performed according to the tracking target of any of the video frames.

[0248] For example, in response to the user canceling the sticker tracking operation, the sticker adding effect of the first video clip is canceled. Alternatively, the sticker adding effect of the first video clip is canceled, and the first sticker material is added to the preset position of each video frame in the first video clip, and the first video clip with the first sticker material added at the preset position is displayed. The preset position may be the center position of the video frame, etc., which is not limited in the embodiments of the present application.

[0249] In addition, during the above-mentioned operations (adding tracking targets, modifying tracking targets, and deleting tracking targets), you can also distinguish and manage the tracked parts and the untracked parts. If the part that is being tracked is corrected, the original tracking process must be terminated immediately. If the part that has not yet started tracking is corrected, the new parameters must be used directly for tracking.

[0250] Next, combine Figure 2The software architecture in is used to illustrate the forward tracking process in the synchronous tracking interface mode.

[0251] Please refer to Fig.18 , Fig.18 This is a schematic flow chart of a forward tracking process of a video in a synchronous tracking interface mode provided by an embodiment of the present application. The method can be executed by an electronic device, or by a processor or chip in the electronic device, and the embodiment of the present application does not make any limitation. For the convenience of description, the method is described in detail by taking an electronic device as an example. Fig.18 As shown, the process includes the following steps:

[0252] S201. The business layer receives the user's operation on the free creation option.

[0253] It should be understood that the business layer includes a free creation application. The free creation application is correspondingly presented as a free creation option on the interface. When the user clicks the free creation option, the business layer will enable the free creation application.

[0254] S202: In response to the user's operation on the free creation option, the business layer loads and displays images and videos.

[0255] Exemplarily, a free creation option is displayed in the UI interface of the gallery application of the electronic device. When the user clicks on the free creation option, multiple pictures and multiple videos are displayed in the gallery interface, and a confirmation control for the free creation (such as a check mark) is displayed.

[0256] S203: The business layer receives the image material selected by the user and receives confirmation from the user to perform the free creation operation.

[0257] In one example, after executing S202, the business layer can receive the image material selected by the user and receive the user's confirmation to perform the free creation operation. Figure 6 In the gallery interface shown in Figure (c), click the Free Creation option. Figure 6 After selecting one or more image materials (videos and pictures, one or more videos, or multiple pictures) in the gallery interface shown in (f) and then clicking the √ control to confirm the free creation operation, the business layer can receive the image materials selected by the user and receive the user's confirmation to perform the free creation operation.

[0258] In another example, the business layer may directly receive the image material selected by the user and receive the user's confirmation to perform the free creation operation without executing steps S201 and S202. Fig.11 Select one or more image materials in the gallery interface shown in (b) and click the Create option. Fig.11After clicking the free creation option in the gallery interface shown in (c), the business layer can receive the image material selected by the user and receive the user's confirmation to perform the free creation operation.

[0259] S204. In response to the user confirming to execute the free creation operation, the business layer sends the image material selected by the user to the basic function layer.

[0260] S205, the basic function layer initializes the editing preview pipeline and the special effects rendering pipeline.

[0261] It should be understood that after the user confirms to execute free creation, the modules at different layers interact to complete the algorithm initialization so as to use the initialized algorithm to crop the image material. The embodiment of the present application does not limit the method of algorithm initialization.

[0262] For example, the basic functional layer can create Figure 3 The editing preview pipeline and special effects rendering pipeline shown in the figure are then initialized to perform sticker rendering on each video frame of the video and preview the rendering results through the editing preview pipeline and the special effects rendering pipeline.

[0263] S206: The basic function layer preview displays the video generated by stitching the selected image materials.

[0264] After receiving the image material selected by the user sent by the basic layer, the basic function layer can splice the selected image material and preview the video generated after splicing.

[0265] For example, by editing the preview pipeline, you can preview and display the video generated by stitching selected image materials.

[0266] For example, if a user clicks Figure 6 After clicking the √ control in Figure (a), the graphical interface displays Figure 6 The video editing interface shown in Figure (b) includes a video preview window, in which the video generated after splicing can be previewed.

[0267] In another example, the business layer can directly receive the user's selection of a video from the gallery and the user's operation of performing video editing without performing steps S201-S205. In response to the user's selection of a video and the operation of performing video editing, the basic function layer sends the video to the basic function layer. The basic function layer initializes the editing preview pipeline and the special effects rendering pipeline, and previews and displays the video. For example, the user Fig.10 In (b) of the figure, after selecting a video, click the edit option. In response to the user selecting a video and clicking the edit option, the graphical interface displays Fig.10 The video editing interface shown in Figure (c) .

[0268] S207: The basic function layer receives an operation from the user to adjust the preview progress of the video.

[0269] The user can adjust the preview progress of the video so that the basic function layer previews and displays the video picture corresponding to any video frame of the video, so that the user can quickly preview the video picture of the target to be tracked.

[0270] Please refer to Figure 6 The video editing interface shown in Figure (b) also includes a preview progress bar. The user can adjust the preview progress of the video by dragging the preview progress bar so that the basic function layer previews and displays the video picture corresponding to any video frame of the video.

[0271] S208. In response to the user's operation of adjusting the preview progress of the video, the basic function layer displays the video screen 1 corresponding to the video frame 1 of the video.

[0272] In the embodiment of the present application, video frame 1 may be the first video frame of the video.

[0273] For example, the user can adjust the preview progress of the video to the preview time point corresponding to the video frame 1, so that the basic function layer displays the video screen 1 corresponding to the video frame 1.

[0274] S209 , the basic function layer receives an operation in which the user sets a target 1 to be tracked on the video screen 1 and confirms the execution of sticker tracking.

[0275] For example, after the user selects the sticker material to be added, the user drags the sticker material to the position of target 1 in video screen 1 to set target 1 to be tracked, that is, to set target 1 as the tracking target. Figure 6 In the user interface shown in FIG. (d) of FIG. 1 , a “strawberry head” sticker is selected. In response to the user selecting the “strawberry head” sticker, as shown in FIG. Figure 6 As shown in Figure (e) in the figure, add a "strawberry head" sticker to the center of the video screen 1. Then, Figure 6 As shown in Figure (f) in the figure, the user can drag the "Strawberry Head" sticker located in the center of the screen to the position of target 1 (father's face) to set target 1 to be tracked on video screen 1, and then click the "Smart Tracking" option to confirm the execution of sticker tracking.

[0276] In another example, the basic function layer may also directly receive the user's operation of playing the video without executing S207-S208, and start playing the video in response to the user's operation of playing the video. When playing to the video screen 1 corresponding to the video frame 1, the user sets the target 1 to be tracked on the video screen 1 and confirms the operation of performing sticker tracking. For example, the user can click Figure 6 In the (b) figure, the play button sent under the video preview window responds to the user clicking the play button to play the video in the video preview window.

[0277] S210. The business layer creates a target tracking algorithm instance for the target tracking module of the smart middle platform framework layer.

[0278] The business layer can create a target tracking algorithm instance for the target tracking module of the smart middle platform framework layer through the target tracking algorithm interface of the media middle platform framework layer to initialize the target tracking algorithm in the target tracking module.

[0279] In addition, you can also trigger the media platform framework layer to create Figure 3 The target tracking pipeline shown is used to implement the target tracking service.

[0280] S211 , the business layer determines target frame position 1 corresponding to target 1 .

[0281] Among them, target frame position 1 is the position of the target frame corresponding to target 1, which is used to indicate the position of target 1 on video frame 1 corresponding to video screen 1, for example, used to indicate the boundary of the image area corresponding to target 1 on video frame 1.

[0282] For example, the business layer can determine the target frame position 1 according to the position of the sticker material in the video frame 1. For example, determine the target 1 covered by the position of the sticker material in the video frame 1, and determine the target frame position 1 according to the image area corresponding to the target 1. For example, determine the rectangular frame that can contain the image area corresponding to the target 1 as the target frame corresponding to the target 1.

[0283] S212. The business layer sends video frame 1 and target frame position 1 to the target tracking algorithm module of the smart middle platform framework layer through the target tracking algorithm interface of the media middle platform framework.

[0284] The business layer may call the target tracking algorithm interface according to the video frame 1 and the target frame position 1. After the target tracking algorithm interface is called, the target tracking algorithm interface may send the video frame 1 and the target frame position 1 to the target tracking algorithm module.

[0285] S213. The basic functional layer decodes the video to obtain the next video frame (video frame 2) of video frame 1.

[0286] The basic functional layer may forward decode the video to sequentially decode and obtain video frames subsequent to video frame 1.

[0287] like Figure 2 As shown, the basic function layer includes an editing preview module, and the video can be decoded by the editing preview module. Figure 3 As shown, the basic functional layer includes an editing and previewing pipeline, and the video can be decoded by a decoding node of the editing and previewing pipeline.

[0288] In an embodiment of the present application, video frame 1 is the first video frame of the video, so the video frames after video frame 1 can be used as the target video segment to be tracked, and then the target video segment is decoded to obtain the next video frame of video frame 1, until the last video frame of the video or the target video segment is decoded.

[0289] In addition, after the last video frame of the video is decoded, a first mark may be added to the video frame, the first mark being used to indicate that the video frame is the last video frame to be processed in the video, or a blank frame carrying a second mark may be added after the video frame, the second mark being used to indicate that the previous video frame of the blank frame is the last video frame to be processed in the video. In this way, the completion of all video processing may be indicated by adding a marked video frame or a blank frame.

[0290] S214. The basic function layer updates the preview progress of the video to the preview time point corresponding to the current video frame 2.

[0291] By updating the preview progress of the video to the preview time point corresponding to the current video frame 2, the video track and the associated sticker track can be played forward synchronously.

[0292] S215 . The basic function layer performs texture conversion on the video frame 2 to obtain image texture 1 .

[0293] Among them, image texture 1 is the image texture of the image corresponding to video frame 2. Image texture refers to the visual texture effect presented in the image, which is composed of regular changes in pixel color and brightness.

[0294] like Figure 2 As shown, the basic function layer includes an editing preview module, which can perform texture conversion on the video frame 2 to obtain the image texture 1. Figure 3 As shown, the basic function layer includes an editing preview pipeline, and the texture conversion of the video frame 2 can be performed through the texture conversion node of the editing preview pipeline.

[0295] S216. The basic function layer sends image texture 1 to the resolution reduction module of the media middle platform framework layer.

[0296] S217 , the resolution reduction module performs resolution reduction processing on the image texture 1 to obtain the image texture 2 .

[0297] The resolution of image texture 2 is smaller than that of image texture 1. By reducing the resolution of image texture 1 to obtain image texture 2 with a smaller resolution, the efficiency of subsequent image processing can be improved.

[0298] S218. The resolution reduction module sends the image texture 2 to the texture to image frame module.

[0299] S219 , the texture-to-image-frame module converts the image texture 2 into video frame data of the video frame 2 .

[0300] For example, in the process of converting the GPU video memory into the internal memory, the GPU video memory data (image texture 2) may be converted into the internal memory data (video frame data).

[0301] S220: The texture-to-image frame module sends the video frame data to the target tracking algorithm interface.

[0302] S221. The target tracking algorithm interface sends the video frame data to the target tracking module of the smart middle platform framework layer.

[0303] The texture-to-image frame module can call the target tracking algorithm interface according to the video frame data of video frame 2 to send the video frame data to the target tracking algorithm interface. After the target tracking algorithm interface is called, the target tracking algorithm of the target tracking module of the smart middle platform framework layer can be called to perform target tracking on the video frame data of video frame 2 through the called target tracking algorithm.

[0304] S222, the target tracking module performs target tracking on the video frame data according to the video frame 1 and the target frame position 1, and obtains the target frame position of the video frame 2.

[0305] The target frame position of the video frame 2 refers to the target frame position corresponding to the target 1 in the video frame 2, and is used to indicate the position of the target 1 in the video frame 2 on the video frame 2.

[0306] After receiving the video frame data of video frame 2, the target tracking module can perform target tracking on the video frame data of video frame 2 according to the previously received video frame 1 and target frame position 1, thereby obtaining the target frame position of video frame 2.

[0307] In addition, after receiving video frame 1 and target frame position 1, the target tracking module can also determine the target type of target 1 in advance based on video frame 1 and target frame position 1, and determine a target tracking algorithm that matches the target type from at least one stored target tracking algorithm, so that when performing target tracking subsequently, according to video frame 1 and target frame position 1, the target tracking algorithm that matches the target type is used to track the video frame data of video frame 2.

[0308] Alternatively, in the above step S211, the business layer may also determine the target type of target 1, and in the above step S212, the business layer may send video frame 1, target box position 1 and target type to the target tracking algorithm module of the smart middle platform framework layer through the target tracking algorithm interface of the media middle platform framework. Accordingly, after receiving the video frame 1, target box position 1 and target type, the target tracking module may determine a target tracking algorithm that matches the target type from at least one stored target tracking algorithm according to the target type, so that when performing target tracking subsequently, according to the video frame 1 and the target box position 1, a target tracking algorithm that matches the target type is used to perform target tracking on the video frame data of the video frame 2.

[0309] S223. The target tracking algorithm module sends the target frame position of the video frame 2 to the business layer through the target tracking algorithm interface.

[0310] For example, the target tracking algorithm module sends the target frame position of video frame 2 to the target tracking algorithm interface, and then the target tracking algorithm interface sends it to the business layer, such as to the intelligent special effects tracking module of the business layer.

[0311] S224 . The business layer determines the sticker material information of the video frame 2 according to the target frame position of the video frame 2 . The sticker material information includes the material position and material size of the sticker material 1 .

[0312] Among them, sticker material 1 is the sticker material to be added on video frame 2, which can be a sticker material selected by the user. Please refer to Figure 6 The (d) figure in the figure may be the "Strawberry Head" sticker selected by the user when setting the target 1. The material position is used to indicate the adding position of the sticker material 1 to be added, and the material size is used to indicate the target size of the sticker material 1 to be added.

[0313] After the business layer (such as the special effects tracking module of the business layer) receives the target frame position of video frame 2, it can convert the target frame position of video frame 2 into the material position and material size of sticker material 1, so that sticker material 1 can be added to video frame 2 according to the material position and material size of sticker material 1.

[0314] The process of determining the sticker material information of video frame 2 according to the target frame position of video frame 2 by the business layer can refer to the above Fig.12 The relevant description of step A5 in this embodiment of the present application will not be repeated here.

[0315] In addition, after the business layer determines the sticker material information of video frame 2, it can also send the sticker material information of video frame 2 to the application function layer, which is cached by the application function layer. For example, the application function layer includes an engineering data model. After receiving the sticker material information of video frame 2, the application function layer can write it into the engineering data model, which is cached by the engineering data model.

[0316] In another embodiment, after the target tracking module tracks the video frame data of video frame 2 according to video frame 1 and target frame position 1, an abnormal flag may be obtained, which is used to indicate that the target of video frame 2 is lost or missing. In this case, the above steps S222-S224 can also be replaced by the following steps S225-S227.

[0317] S225. The target tracking module performs target tracking on the video frame data of the video frame 2 according to the video frame 1 and the target frame position 1 to obtain an abnormal identification.

[0318] S226. The target tracking algorithm module sends the abnormal identification of the video frame 2 to the business layer through the target tracking algorithm interface.

[0319] S227. The business layer determines the sticker material information of the previous video frame (video frame 1) of video frame 2 as the sticker material information of video frame 2 according to the abnormal flag of video frame 2.

[0320] In this way, it is convenient to subsequently use the sticker material information of the previous video frame to add sticker material 1 on video frame 2.

[0321] S228. The business layer sends sticker material information of video frame 2 to the basic function layer.

[0322] S229. The basic function layer adds sticker material 1 to video frame 2 according to the sticker material information of video frame 2.

[0323] S230 , the basic function layer displays the video frame 2 to which the sticker material 1 is added.

[0324] like Figure 2As shown, the basic function layer includes a special effect rendering module, and the business layer can send the sticker material information of the video frame 2 to the special effect rendering module of the basic function layer. After receiving the sticker material information of the video frame 2, the special effect rendering module can add the sticker material 1 to the video frame 2 according to the sticker material information of the video frame 2, and then send the video frame 2 with the added sticker material 1 for display. In this way, the video frame 2 with the added sticker material 1 can be displayed on the display screen, such as in the video preview window.

[0325] For example, Figure 3 As shown, the business layer can send the sticker material information of video frame 2 to the sticker rendering node in the special effect rendering pipeline of the basic function layer, and the sticker rendering node can render video frame 2 according to the sticker material information of video frame 2, and then send the rendered video frame to the display node, which will display it. In this way, the rendered video frame (i.e., video frame 2 with sticker material 1 added) can be displayed on the display screen, such as in the video preview window.

[0326] It should be noted that the above steps S213-S230 can be executed in a loop. That is, after step S230, the basic function layer can continue to decode the next video frame from the video, and track the next video frame according to the tracking processing method of video frame 2 in steps S213-S230, so as to generate and display the next video frame with sticker material 1 added. In this way, the loop is repeated until the tracking processing of the last video frame of the video is completed.

[0327] In addition, each time a preset number of video frames with sticker materials added are generated, the preset number of video frames with sticker materials added can be synthesized to generate a synthesized video. For example, when a preset number of video frames with sticker materials added are generated for the first time, the preset number of video frames with sticker materials added generated for the first time are synthesized; and then, when a preset number of video frames with sticker materials added are generated again, the preset number of video frames with sticker materials added generated again are synthesized with the synthesized video.

[0328] S231. After completing all tracking processing on the video, the basic function layer destroys the target tracking algorithm instance.

[0329] For example, the basic function layer may destroy the target tracking algorithm instance after completing the tracking process of the last video frame of the video.

[0330] As an example, the basic function layer may send a destruction instruction to the target tracking module through the target tracking algorithm interface, where the destruction instruction is used to instruct the destruction of the created target tracking algorithm instance.

[0331] It should be noted that Fig.18The embodiment is only explained by taking the user setting the tracking target on a video frame (video frame 1) of the video as an example. It should be understood that the embodiment of the present application also supports setting the tracking target on multiple video frames. For example, after the user sets the tracking target (target 1) on video frame 1, he can continue to set the tracking target (such as target 2) on other video frames (such as a video frame 3 after video frame 1). For the video frames after video frame 3, target 2 can be used as the tracking target for tracking processing. For the intermediate video frame between the two video frames, target 1 can be used as the tracking target for tracking processing, or target 2 can be used as the tracking target for tracking processing. The embodiment of the present application does not limit this. For example, the tracking target of the intermediate video frame between the two video frames can be specified by the user from target 1 and target 2.

[0332] Next, combine Figure 2 The software architecture in is used to illustrate the forward tracing process in the asynchronous tracing interface mode.

[0333] Please refer to Fig.19 , Fig.19 This is a schematic flow chart of a forward tracking process of a video in an asynchronous tracking interface mode provided by an embodiment of the present application. The method can be executed by an electronic device, or by a processor or chip in the electronic device, and the embodiment of the present application does not make any limitation. For the convenience of description, the method is described in detail by taking an electronic device as an example. Fig.18 As shown, the process includes the following steps:

[0334] S301. The business layer receives the user's operation on the free creation option.

[0335] S302: In response to the user's operation on the free creation option, the business layer loads and displays images and videos.

[0336] S303: The business layer receives the image material selected by the user and receives confirmation from the user to perform the free creation operation.

[0337] S304: In response to the user confirming to execute the free creation operation, the business layer sends the image material selected by the user to the basic function layer.

[0338] S305: The basic function layer initializes the editing preview pipeline and the special effects rendering pipeline.

[0339] S306: The basic function layer preview displays the video generated by stitching the selected image materials.

[0340] S307: The basic function layer receives an operation from the user to adjust the preview progress of the video.

[0341] S308. In response to the user's operation of adjusting the preview progress of the video, the basic function layer displays the video screen 1 corresponding to the video frame 1 of the video.

[0342] In the embodiment of the present application, video frame 1 may be the first video frame of the video.

[0343] S309 , the basic function layer receives the user's setting of the target 1 to be tracked on the video screen 1 and confirms the operation of executing sticker tracking.

[0344] S310. The business layer creates a target tracking algorithm instance for the target tracking module of the smart middle platform framework layer.

[0345] S311 , the business layer determines target frame position 1 corresponding to target 1.

[0346] S312. The business layer sends video frame 1 and target frame position 1 to the target tracking algorithm module of the smart middle platform framework layer through the target tracking algorithm interface of the media middle platform framework.

[0347] It should be understood that the specific implementation of the above steps S301-S312 is the same as the above Fig.18 Similarly, for steps S201-S212 in the embodiment, the specific implementation method can refer to the relevant description of the above steps S201-S212, and the embodiments of the present application will not be repeated here.

[0348] S313. The basic functional layer decodes the video to obtain the next video frame (video frame 2) of video frame 1.

[0349] The basic functional layer may forward decode the video to sequentially decode and obtain video frames subsequent to video frame 1.

[0350] like Figure 2 As shown, the basic function layer includes an editing preview module, and the video can be decoded by the editing preview module. Figure 3 As shown, the basic functional layer includes an editing and previewing pipeline, and the video can be decoded by a decoding node of the editing and previewing pipeline.

[0351] In an embodiment of the present application, video frame 1 is the first video frame of the video, so the video frames after video frame 1 can be used as the target video segment to be tracked, and then the target video segment is decoded to obtain the next video frame of video frame 1, until the last video frame of the video or the target video segment is decoded.

[0352] In addition, after the last video frame of the video is decoded, a first mark may be added to the video frame, the first mark is used to indicate that the video frame is the last video frame of the video, or a blank frame carrying a second mark is added after the video frame, the second mark is used to indicate that the previous video frame of the blank frame is the last video frame of the video. In this way, the completion of all video processing can be indicated by adding a marked video frame or a blank frame.

[0353] S314 . The basic function layer performs texture conversion on the video frame 2 to obtain the image texture 1 of the video frame 2 .

[0354] It should be noted that Fig.18 The difference between the embodiments is that the basic function layer does not decode the next video frame after the tracking process of a decoded video frame is completed, but can decode the video frames after video frame 1 in sequence, that is, after decoding a video frame, it can continue to decode the next video frame. Moreover, every time a video frame is decoded, the texture conversion of the video frame can be performed to obtain the image texture 1 of the video frame.

[0355] S315. The basic function layer sends the presentation time stamp (PTS) and image texture 1 of video frame 2 to the resolution reduction module of the media platform framework layer.

[0356] The PTS of the video frame 2 is used to indicate the display time of the video frame 2.

[0357] In the embodiment of the present application, the basic function layer can send the PTS and image texture of the decoded video frame to the media platform framework layer in sequence. In order to avoid the disorder of the image texture of the video frame sent in sequence, the PTS of the video frame can also be sent while sending the image texture of the video frame, so as to indicate the order of processing of the video frame through the PTS of the video frame.

[0358] S316 . The resolution reduction module performs resolution reduction processing on the image texture 1 of the video frame 2 to obtain the image texture 2 of the video frame 2 .

[0359] The resolution of image texture 2 is smaller than that of image texture 1. By reducing the resolution of image texture 1 to obtain image texture 2 with a smaller resolution, the efficiency of subsequent image processing can be improved.

[0360] S317. The resolution reduction module sends the PTS of the video frame 2 and the image texture 2 to the texture to image frame module.

[0361] S318 , the texture-to-image-frame module converts the image texture 2 of the video frame 2 into the video frame data of the video frame 2 .

[0362] For example, in the process of converting the GPU video memory into the internal memory, the GPU video memory data (image texture 2) may be converted into the internal memory data (video frame data).

[0363] S319, the texture-to-image frame module sends the PTS and video frame data of video frame 2 to the target tracking module of the smart middle platform framework layer through the target tracking algorithm interface of the media middle platform framework layer.

[0364] The texture-to-image-frame module may first send the PTS and video frame data of the video frame 2 to the target tracking algorithm interface, and then the target tracking algorithm interface sends them to the target tracking module.

[0365] For example, the texture-to-image frame module can call the target tracking algorithm interface according to the PTS and video frame data of video frame 2 to send the PTS and video frame data of video frame 2 to the target tracking module through the target tracking algorithm interface. After the target tracking algorithm interface is called, the target tracking algorithm of the target tracking module of the smart middle platform framework layer can be called to perform target tracking on the video frame data of video frame 2 through the called target tracking algorithm.

[0366] S320: The target tracking module stores the PTS and video frame data of the video frame 2 in the tracking queue.

[0367] The tracking queue is used to store video frame data of the video frames to be tracked.

[0368] It should be noted that Fig.18 The difference between the embodiments is that after receiving the video frame data sent by the basic function layer, the target tracking module does not directly track the received video frame data, but first stores the received video frame data in the tracking queue for queuing.

[0369] S321. The target tracking module obtains the latest video frame data of video frame B from the tracking queue.

[0370] The latest video frame B refers to the video frame to be processed that is at the front in the queue order of the tracking queue.

[0371] S322, the target tracking module performs target tracking on the video frame data of video frame B according to video frame 1 and target frame position 1 to obtain the target frame position of video frame B.

[0372] The target frame position of the video frame B refers to the target frame position corresponding to the target 1 in the video frame B, and is used to indicate the position of the target 1 in the video frame B on the video frame B.

[0373] In the embodiment of the present application, the target tracking module can sequentially obtain the video frame data to be processed from the tracking queue, and perform target tracking on the obtained video frame data. After the target tracking module obtains the video frame data of the latest video frame B, it can perform target tracking on the video frame data of video frame B according to the previously received video frame 1 and the target frame position 1, thereby obtaining the target frame position of video frame B.

[0374] As an example, the target tracking module may store the tracking result of the current video frame in a tracking result queue.

[0375] In addition, after the target tracking module performs target tracking on the video frame data of video frame B to obtain the tracking result, it can also return to step S321 to continue to obtain the video frame data of the next latest video frame from the tracking queue, and track the video frame data of the next latest video frame. It should be understood that the target tracking module can also continue to obtain the video frame data of the next latest video frame from the tracking queue after reporting the tracking result to the business layer, and track the video frame data of the next latest video frame. In this way, the video frames to be tracked can be tracked in sequence.

[0376] S323. The target tracking module sends a tracking completion notification to the business layer through the target tracking algorithm interface. The tracking completion notification carries the PTS of video frame B and the target frame position.

[0377] The tracking completion notification is used to indicate that the tracking of a video frame is completed. That is, each time the target tracking module completes tracking of a video frame and obtains the target frame position of the video frame, it can send a tracking completion notification to the business layer so that the business layer can perform sticker processing on the latest tracked video frame. Sticker processing means adding stickers to the video frame according to the target frame position and displaying the video frame after adding stickers.

[0378] In this way, the electronic device can track the target of the untracked video frame while performing sticker processing on the tracked video frame, without having to complete the tracking processing (target tracking + sticker processing) of a video frame before tracking the next video frame (target tracking + sticker processing), thereby improving video processing efficiency.

[0379] It should be understood that the target tracking module may also first send a tracking completion notification that does not carry the PTS and target frame position of the video frame B to the business layer, and after the business layer receives the tracking completion notification, it obtains the PTS and target frame position of the latest tracked video frame B from the target tracking module. For example, the business layer may obtain the PTS and target frame position of the latest tracked video frame B from the tracking result queue of the target tracking module. For example, the business layer may send an acquisition request to the target tracking module so that the target tracking module obtains the PTS and target frame position of the latest tracked video frame according to the acquisition request, and returns the acquired PTS and target frame position of the video frame to the business layer.

[0380] S324. When the business layer determines that video frame B is the video frame to be displayed according to the PTS of video frame B, the sticker material information of video frame B is determined according to the target frame position of video frame B. The sticker material information includes the material position and material size of sticker material 1.

[0381] Among them, sticker material 1 is the sticker material to be added to the video frame B, which can be a sticker material selected by the user. Please refer to Figure 6 The (d) figure in the figure may be the "Strawberry Head" sticker selected by the user when setting the target 1. The material position is used to indicate the adding position of the sticker material 1 to be added, and the material size is used to indicate the target size of the sticker material 1 to be added.

[0382] After the business layer (such as the special effects tracking module of the business layer) receives the target frame position of video frame B, it can convert the target frame position of video frame B into the material position and material size of sticker material 1, so that sticker material 1 can be added to video frame B according to the material position and material size of sticker material 1.

[0383] The process of determining the sticker material information of video frame B according to the target frame position of video frame B by the business layer can refer to the above Fig.12 The relevant description of step A5 in this embodiment of the present application will not be repeated here.

[0384] In another embodiment, after the target tracking module performs target tracking on the video frame data of the video frame B according to the video frame 1 and the target frame position 1, an abnormal flag may be obtained, and the abnormal flag is used to indicate that the target of the video frame B is lost or missing. In this case, the above steps S322-S324 can also be replaced by the following steps S325-S327.

[0385] S325 , performing target tracking on the video frame data of the video frame B according to the video frame 1 and the target frame position 1 to obtain an abnormal identification.

[0386] S326. The target tracking algorithm module sends the abnormal identification of the video frame B to the business layer through the target tracking algorithm interface.

[0387] S327. The business layer determines the sticker material information of the previous video frame of video frame B as the sticker material information of video frame B according to the abnormal flag of video frame B.

[0388] In this way, it is convenient to subsequently add sticker material to video frame B by using the sticker material information of the previous video frame.

[0389] S328. The business layer sends the sticker material information of video frame B to the basic function layer.

[0390] S329: The basic function layer updates the preview progress of the video to the preview time point corresponding to the current video frame B.

[0391] By updating the preview progress of the video to the preview time point corresponding to the current video frame B, the video track and the associated sticker track can be played forward synchronously.

[0392] S330. The basic function layer adds sticker material 1 to video frame B according to the sticker material information of video frame B.

[0393] In addition, each time a preset number of video frames with sticker materials added are generated, the preset number of video frames with sticker materials added can be synthesized to generate a synthesized video. For example, when a preset number of video frames with sticker materials added are generated for the first time, the preset number of video frames with sticker materials added generated for the first time are synthesized; and then, when a preset number of video frames with sticker materials added are generated again, the preset number of video frames with sticker materials added generated again are synthesized with the synthesized video.

[0394] S331. The basic function layer displays the video frame B to which the sticker material 1 is added.

[0395] like Figure 2 As shown, the basic function layer includes a special effect rendering module, and the business layer can send the sticker material information of the video frame B to the special effect rendering module of the basic function layer. After receiving the sticker material information of the video frame B, the special effect rendering module can add sticker material 1 to the video frame B according to the sticker material information of the video frame B, and then send the video frame B with the sticker material 1 added for display. In this way, the video frame B with the sticker material 1 added can be displayed on the display screen, such as in the video preview window.

[0396] For example, Figure 3As shown, the business layer can send the sticker material information of video frame B to the sticker rendering node in the special effect rendering pipeline of the basic function layer, and the sticker rendering node can render video frame B according to the sticker material information of video frame B, and then send the rendered video frame to the display node, which will display it. In this way, the rendered video frame (i.e., video frame B with sticker material 1 added) can be displayed on the display screen, such as in the video preview window.

[0397] It should be noted that after displaying the video frame B with the sticker material 1 added, the tracking completion notification sent by the target tracking module can continue to be received, so as to perform sticker processing on the next video frame according to the PTS and target frame position of the next video frame carried in the tracking completion notification. In this way, the video frames that have been tracked can be processed with stickers in turn until the last video frame is processed.

[0398] In the asynchronous tracking interface mode, the target can be tracked on the video frame to be tracked, and the sticker processing can be performed on the video frame that has been tracked. The two can be executed synchronously, which can improve the efficiency of video processing.

[0399] It should be noted that the buffer smoothing mode is also supported in the asynchronous tracking interface mode to ensure the smoothness of the tracking processing output through the buffer smoothing mode. For example, when the sticker processing speed is greater than the target tracking speed, the callback frequency is adjusted according to the target tracking speed. The callback frequency refers to the frequency of obtaining the tracking results of the latest video frame completed from the tracking results corresponding to the tracking queue, so that the input frequency of the sticker processing is adapted to the target tracking speed with a relatively satisfactory speed, thereby ensuring the continuous smoothness of the tracking processing output. For example, the frequency at which the target tracking module sends a tracking completion notification to the business layer can be adjusted according to the target tracking speed, or the frequency at which the business layer obtains the tracking results of the latest video frame completed from the tracking results of the tracking module can be adjusted. Alternatively, when the sticker processing speed is less than the target tracking speed, the callback frequency is adjusted according to the sticker processing speed so that the input frequency of the sticker processing is adapted to the slower sticker processing speed, thereby ensuring the continuous smoothness of the tracking processing output.

[0400] S332: After completing all tracking processing on the video, the basic function layer destroys the target tracking algorithm instance.

[0401] For example, the basic function layer may destroy the target tracking algorithm instance after completing the tracking process of the last video frame of the video.

[0402] As an example, the basic function layer may send a destruction instruction to the target tracking module through the target tracking algorithm interface, where the destruction instruction is used to instruct the destruction of the created target tracking algorithm instance.

[0403] It should be noted that Fig.19 The embodiment is only explained by taking the user setting the tracking target on a video frame (video frame 1) of the video as an example. It should be understood that the embodiment of the present application also supports setting the tracking target on multiple video frames. For example, after the user sets the tracking target (target 1) on video frame 1, he can continue to set the tracking target (for example, target 2) on other video frames (such as a video frame 3 after video frame 1). For the video frames after video frame 3, target 2 can be used as the tracking target for tracking processing. For the intermediate video frame between the two video frames, target 1 can be used as the tracking target for tracking processing, or target 2 can be used as the tracking target for tracking processing, and the embodiment of the present application does not limit this. For example, the tracking target of the intermediate video frame between the two video frames can be specified by the user from target 1 and target 2.

[0404] Next, combine Figure 2 The software architecture in is used to illustrate the forward tracking process in the synchronous tracking interface mode.

[0405] Please refer to Fig. 20 , Fig. 20 This is a schematic flow chart of a backward tracking process for a video clip in a synchronous tracking interface mode provided by an embodiment of the present application. The method can be executed by an electronic device, or by a processor or chip in the electronic device, and the embodiment of the present application does not make any limitation. For ease of description, the method is described in detail by taking an electronic device as an example.

[0406] like Fig.18 As shown, the process includes the following steps:

[0407] S401. The business layer receives the user's operation on the free creation option.

[0408] S402: In response to the user's operation on the free creation option, the business layer loads and displays images and videos.

[0409] S403: The business layer receives the image material selected by the user and receives confirmation from the user to perform the free creation operation.

[0410] S404: In response to the user confirming to execute the free creation operation, the business layer sends the image material selected by the user to the basic function layer.

[0411] S405: The basic function layer initializes the editing preview pipeline and the special effects rendering pipeline.

[0412] S406: The basic function layer preview displays the video generated by stitching the selected image materials.

[0413] S407: The basic function layer receives an operation of the user to adjust the preview progress of the video.

[0414] It should be understood that the specific implementation of the above steps S401-407 is the same as the above Fig.18 Similarly, steps S201-S207 in the above description may be referred to for the specific implementation method, and the embodiments of the present application will not be repeated here.

[0415] S408. In response to the user's operation of adjusting the preview progress of the video, the basic function layer displays the video screen 4 corresponding to the video frame 4 of the video.

[0416] In the embodiment of the present application, video frame 4 is an intermediate video frame of the video, that is, video frame 4 is neither the first video frame of the video nor the last video frame of the video.

[0417] The user can adjust the preview progress of the video to the preview time point corresponding to the video frame 4 so that the basic function layer previews and displays the video screen 4 corresponding to the video frame 4.

[0418] Please refer to Figure 6 The video editing interface shown in Figure (b) also includes a preview progress bar. The user can adjust the preview progress of the video by dragging the preview progress bar, such as dragging the preview progress condition backward to adjust the preview progress of the video to the preview time point corresponding to video frame 4.

[0419] S409: The basic function layer receives the user's setting of the target 3 to be tracked on the video screen 4 and confirms the operation of executing sticker tracking.

[0420] It should be understood that the method in which the user sets the target 3 to be tracked on the video screen 4 and confirms the operation of executing sticker tracking in the embodiment of the present application can refer to the above Fig.18 In step S209 of the embodiment, the user sets the target 1 to be tracked on the video screen 1 and confirms the operation method of executing sticker tracking, which will not be repeated here in the embodiment of the present application.

[0421] S410. The business layer creates a target tracking algorithm instance for the target tracking module of the smart middle platform framework layer.

[0422] S411 , the business layer determines target frame position 3 corresponding to target 3.

[0423] S412, the business layer sends the video frame 4 and the target frame position 3 to the target tracking algorithm module of the smart middle platform framework layer through the target tracking algorithm interface of the media middle platform framework.

[0424] S413: The basic function layer determines a video segment 1 to be backward tracked from the video according to the video frame 4.

[0425] For example, video segment 1 may be a video segment corresponding to a video from the first video frame of the video to the previous video frame 4. For video segment 1, the tracking processing method provided in the embodiment of the present application may be used to perform reverse tracking processing.

[0426] S414 , reversely decode the video segment 1 to obtain the previous video frame of the video frame 4 (video frame 5 ).

[0427] The basic function layer can perform reverse decoding on the video segment 1 to sequentially decode and obtain the video frames before the video frame 4. Figure 2 As shown, the basic function layer includes an editing preview module, and the video segment 1 can be decoded by the editing preview module. Figure 3 As shown, the basic function layer includes an editing preview pipeline, and the video segment 1 can be decoded by a decoding node of the editing preview pipeline.

[0428] In addition, after the last video frame of the video segment 1 is obtained by backward decoding (i.e., the first video frame in the forward direction), a first mark may be added to the video frame, the first mark is used to indicate that the video frame is the last video frame to be processed in the video segment 1, or a blank frame carrying a second mark is added after the video frame, the second mark is used to indicate that the previous video frame of the blank frame is the last video frame to be processed in the video segment 1. In this way, the completion of the processing of the video segment 1 can be indicated by adding a marked video frame or a blank frame.

[0429] S415. The basic function layer updates the preview progress of the video to the preview time point corresponding to the current video frame 5.

[0430] By updating the preview progress of the video to the preview time point corresponding to the current video frame 5, the video track and the associated sticker track can be played backward synchronously.

[0431] S416 . The basic function layer performs texture conversion on the video frame 5 to obtain the image texture 1 of the video frame 5 .

[0432] Image texture refers to the visual texture effect presented in the image, which is composed of regular changes in pixel color and brightness. Figure 2 As shown, the basic function layer includes an editing and previewing module, and the video frame 5 can be texture converted by the editing and previewing module to obtain the image texture 1 of the video frame 5. For example, Figure 3 As shown, the basic function layer includes an editing preview pipeline, and the texture conversion of the video frame 5 can be performed through the texture conversion node of the editing preview pipeline.

[0433] S417. The basic function layer sends the image texture 1 of the video frame 5 to the resolution reduction module of the media platform framework layer.

[0434] S418 . The resolution reduction module performs resolution reduction processing on the image texture 1 of the video frame 5 to obtain the image texture 2 of the video frame 5 .

[0435] The resolution of image texture 2 is smaller than that of image texture 1. By reducing the resolution of image texture 1 to obtain image texture 2 with a smaller resolution, the efficiency of subsequent image processing can be improved.

[0436] S419: The resolution reduction module sends the image texture 2 of the video frame 5 to the texture to image frame module.

[0437] S420 , the texture-to-image-frame module converts the image texture 2 of the video frame 5 into video frame data of the video frame 5 .

[0438] For example, in the process of converting the GPU video memory into the internal memory, the GPU video memory data (image texture 2) may be converted into the internal memory data (video frame data).

[0439] S421, the texture to image frame module sends the video frame data of video frame 5 to the target tracking module of the smart middle platform framework layer through the target tracking algorithm interface.

[0440] The texture-to-image frame module can call the target tracking algorithm interface according to the video frame data of video frame 5 to send the video frame data to the target tracking algorithm interface. After the target tracking algorithm interface is called, the target tracking algorithm of the target tracking module of the smart middle platform framework layer can be called to perform target tracking on the video frame data of video frame 5 through the called target tracking algorithm.

[0441] S422 , the target tracking module performs target tracking on the video frame data of the video frame 5 according to the video frame 4 and the target frame position 3 , and obtains the target frame position of the video frame 5 .

[0442] The target frame position of the video frame 5 refers to the target frame position corresponding to the target 3 in the video frame 5 , and is used to indicate the position of the target 3 in the video frame 5 on the video frame 5 .

[0443] After receiving the video frame data of video frame 5, the target tracking module can perform target tracking on the video frame data of video frame 5 according to the previously received video frame 4 and the target frame position 3, thereby obtaining the target frame position of video frame 5.

[0444] In addition, after receiving the video frame 4 and the target frame position 3, the target tracking module can also determine the target type of the target 3 in advance based on the video frame 4 and the target frame position 3, and determine the target tracking algorithm that matches the target type from at least one stored target tracking algorithm, so that when performing target tracking subsequently, according to the video frame 4 and the target frame position 3, the target tracking algorithm that matches the target type is used to track the video frame data of the video frame 5.

[0445] Alternatively, in the above step S411, the business layer can also determine the target type of target 3. In the above step S412, the business layer can send the video frame 4, the target box position 3 and the target type to the target tracking algorithm module of the smart middle platform framework layer through the target tracking algorithm interface of the media middle platform framework. Accordingly, after the target tracking module receives the video frame 4, the target box position 3 and the target type, it can determine the target tracking algorithm that matches the target type from at least one stored target tracking algorithm according to the target type, so that when performing target tracking subsequently, according to the video frame 4 and the target box position 3, the target tracking algorithm that matches the target type is used to perform target tracking on the video frame data of the video frame 5.

[0446] S423. The target tracking algorithm module sends the target frame position of the video frame 5 to the business layer through the target tracking algorithm interface.

[0447] For example, the target tracking algorithm module sends the target frame position of video frame 5 to the target tracking algorithm interface, and then the target tracking algorithm interface sends it to the business layer, such as to the intelligent special effects tracking module of the business layer.

[0448] S424 . The business layer determines the sticker material information of the video frame 5 according to the target frame position of the video frame 5 . The sticker material information includes the material position and material size of the sticker material 2 .

[0449] Among them, sticker material 2 is the sticker material to be added on video frame 5, which can be a sticker material selected by the user. Please refer to Figure 6 The (d) figure in the figure may be a "strawberry head" sticker selected by the user when setting the target 3 to be tracked on the video frame 4. The material position is used to indicate the adding position of the sticker material 2 to be added, and the material size is used to indicate the target size of the sticker material 2 to be added.

[0450] After the business layer (such as the special effects tracking module of the business layer) receives the target frame position of the video frame 5, the target frame position of the video frame 5 can be converted into the material position and material size of the sticker material 2, so that sticker material 2 can be added to the video frame 5 according to the material position and material size of the sticker material 2.

[0451] The process of determining the sticker material information of video frame 5 according to the target frame position of video frame 5 by the business layer can refer to the above Fig.12 The relevant description of step A5 in this embodiment of the present application will not be repeated here.

[0452] In addition, after the business layer determines the sticker material information of video frame 5, it can also send the sticker material information of video frame 5 to the application function layer, which is cached by the application function layer. For example, the application function layer includes an engineering data model. After receiving the sticker material information of video frame 5, the application function layer can write it into the engineering data model, which is cached by the engineering data model.

[0453] In another embodiment, after the target tracking module tracks the video frame data of the video frame 5 according to the video frame 4 and the target frame position 3, an abnormal flag may be obtained, which is used to indicate that the target of the video frame 5 is lost or missing. In this case, the above steps S422-S424 can also be replaced by the following steps S425-S427.

[0454] S425. The target tracking module performs target tracking on the video frame data of the video frame 5 according to the video frame 4 and the target frame position 3 to obtain an abnormal identification.

[0455] S426. The target tracking algorithm module sends the abnormal identification of video frame 5 to the business layer through the target tracking algorithm interface.

[0456] S427. The business layer determines the sticker material information of the next video frame (video frame 4) of video frame 5 as the sticker material information of video frame 5 according to the abnormal flag of video frame 5.

[0457] In this way, it is convenient to subsequently use the sticker material information of the previous video frame decoded backward to add sticker material 2 on the current video frame.

[0458] S428. The business layer sends the sticker material information of video frame 5 to the basic function layer.

[0459] S429. The basic function layer adds sticker material 2 to video frame 5 according to the sticker material information of video frame 5.

[0460] In addition, each time a preset number of video frames with sticker materials added are generated, the preset number of video frames with sticker materials added may be reversely synthesized to generate a synthesized video. For example, when a preset number of video frames with sticker materials added are generated for the first time, the preset number of video frames with sticker materials added generated for the first time are reversely synthesized; and then, when a preset number of video frames with sticker materials added are generated again, the preset number of video frames with sticker materials added generated again are reversely synthesized with the synthesized video.

[0461] S430 , the basic function layer displays the video frame 5 to which the sticker material 2 is added.

[0462] like Figure 2 As shown, the basic function layer includes a special effect rendering module, and the business layer can send the sticker material information of the video frame 5 to the special effect rendering module of the basic function layer. After receiving the sticker material information of the video frame 5, the special effect rendering module can add the sticker material 2 to the video frame 5 according to the sticker material information of the video frame 5, and then send the video frame 5 with the added sticker material 2 for display. In this way, the video frame 5 with the added sticker material 2 can be displayed on the display screen, such as in the video preview window.

[0463] For example, Figure 3 As shown, the business layer can send the sticker material information of video frame 5 to the sticker rendering node in the special effect rendering pipeline of the basic function layer, and the sticker rendering node can render video frame 5 according to the sticker material information of video frame 5, and then send the rendered video frame to the display node, which will display it. In this way, the rendered video frame (i.e., the video frame 5 with the sticker material 2 added) can be displayed on the display screen, such as in the video preview window.

[0464] It should be noted that the above steps S414-S430 can be executed in a loop. That is, after step S430, the basic function layer can continue to perform reverse decoding on the video segment 1 to decode the previous video frame of the current video frame (i.e., the next video frame of the reverse decoding), and perform tracking processing on the next video frame of the reverse decoding according to the tracking processing method of the video frame 5 in steps S414-S430 to generate and display the next video frame with the sticker material 2 added. In this way, the loop is repeated until the tracking processing of the last video frame of the video segment 1 (i.e., the first video frame in the forward direction) is completed.

[0465] In addition, the basic function layer can also determine the video segment 2 to be forward tracked from the video according to the video frame 4, according to the above Fig.18 The tracking processing method described in the embodiment performs forward tracking processing on the video segment 2. For example, the video segment 2 may be a video segment corresponding to the next video frame of the video frame 4 to the last video frame of the video.

[0466] It should be noted that the embodiment of the present application can first perform backward tracking processing on video segment 1 and then perform forward tracking processing on video segment 2, or can first perform forward tracking processing on video segment 2 and then perform backward tracking processing on video segment 1. The embodiment of the present application is not limited to this.

[0467] S231. After completing all tracking processing on the video, the basic function layer destroys the target tracking algorithm instance.

[0468] For example, the basic function layer may destroy the target tracking algorithm instance after completing the tracking processing of the video segment 1 and the video segment 2 of the video.

[0469] As an example, the basic function layer may send a destruction instruction to the target tracking module through the target tracking algorithm interface, where the destruction instruction is used to instruct the destruction of the created target tracking algorithm instance.

[0470] It should be noted that Fig. 20 The embodiment is only explained by taking the example of a user setting a tracking target on a video frame (video frame 4) of a video. It should be understood that the embodiment of the present application also supports setting tracking targets on multiple video frames. For example, after the user sets the tracking target (target 3) on video frame 4, he can continue to set tracking targets (such as target 4) on other video frames (such as a video frame 6 after video frame 4). For the video clip before video frame 4, target 3 can be used as the tracking target for tracking processing; for the video clip after video frame 6, target 4 can be used as the tracking target for tracking processing. For the video clip between the two video frames, target 3 can be used as the tracking target for tracking processing, or target 4 can be used as the tracking target for tracking processing, and the embodiment of the present application does not limit this. For example, the tracking target of the video clip between the two video frames can be specified by the user from target 3 and target 4.

[0471] Next, combine Figure 2 The software architecture in is used to illustrate the backward tracing process in the asynchronous tracing interface mode.

[0472] Please refer to Fig.21 , Fig.21 This is a schematic flow chart of a reverse tracking process for a video clip in an asynchronous tracking interface mode provided by an embodiment of the present application. The method can be executed by an electronic device, or by a processor or chip in the electronic device, and the embodiment of the present application does not make any limitation. For ease of description, the method is described in detail by taking an electronic device as an example.

[0473] like Fig.18 As shown, the process includes the following steps:

[0474] S501. The business layer receives the user's operation on the free creation option.

[0475] S502: In response to the user's operation on the free creation option, the business layer loads and displays images and videos.

[0476] S503: The business layer receives the image material selected by the user and receives confirmation from the user to perform the free creation operation.

[0477] S504: In response to the user confirming the operation of executing free creation, the business layer sends the image material selected by the user to the basic function layer.

[0478] S505: The basic function layer initializes the editing preview pipeline and the special effects rendering pipeline.

[0479] S506: The basic function layer preview displays the video generated by stitching the selected image materials.

[0480] S507: The basic function layer receives an operation of the user to adjust the preview progress of the video.

[0481] S508. In response to the user's operation of adjusting the preview progress of the video, the basic function layer displays the video screen 4 corresponding to the video frame 4 of the video.

[0482] In the embodiment of the present application, video frame 4 is an intermediate video frame of the video, that is, video frame 4 is neither the first video frame of the video nor the last video frame of the video.

[0483] The user can adjust the preview progress of the video to the preview time point corresponding to the video frame 4 so that the basic function layer previews and displays the video screen 4 corresponding to the video frame 4.

[0484] Please refer to Figure 6 The video editing interface shown in Figure (b) also includes a preview progress bar. The user can adjust the preview progress of the video by dragging the preview progress bar, such as dragging the preview progress condition backward to adjust the preview progress of the video to the preview time point corresponding to video frame 4.

[0485] S509 , the basic function layer receives the user's setting of the target 3 to be tracked on the video screen 1 and confirms the operation of executing sticker tracking.

[0486] S510, the business layer creates a target tracking algorithm instance for the target tracking module of the smart middle platform framework layer.

[0487] S511 , the business layer determines target frame position 3 corresponding to target 3.

[0488] S512, the business layer sends the video frame 4 and the target frame position 3 to the target tracking algorithm module of the smart middle platform framework layer through the target tracking algorithm interface of the media middle platform framework.

[0489] It should be understood that the specific implementation of the above steps S501-S512 is the same as the above Fig.18Similarly, for steps S201-S212 in the embodiment, the specific implementation method can refer to the relevant description of the above steps S201-S212, and the embodiments of the present application will not be repeated here.

[0490] S513: The basic function layer determines a video segment 1 to be backward tracked from the video according to the video frame 4.

[0491] S514 . The basic function layer decodes backward from the video segment 1 to obtain the previous video frame of the video frame 4 (video frame 5 ).

[0492] The basic function layer can perform reverse decoding on the video to sequentially decode and obtain the video frames before the video frame 4. Figure 2 As shown, the basic function layer includes an editing preview module, and the video can be decoded backwards through the editing preview module. Figure 3 As shown, the basic function layer includes an editing preview pipeline, and the video segment 1 can be decoded backwards through the decoding node of the editing preview pipeline.

[0493] In addition, after the last video frame of the video segment 1 is obtained by backward decoding (i.e., the first video frame in the forward direction), a first mark may be added to the video frame, the first mark is used to indicate that the video frame is the last video frame to be processed in the video segment 1, or a blank frame carrying a second mark is added after the video frame, the second mark is used to indicate that the previous video frame of the blank frame is the last video frame to be processed in the video segment 1. In this way, the completion of the processing of the video segment 1 can be indicated by adding a marked video frame or a blank frame.

[0494] S515 . The basic function layer performs texture conversion on the video frame 5 to obtain the image texture 1 of the video frame 5 .

[0495] It should be noted that Fig. 20 The difference between the embodiments is that the basic function layer does not decode the next video frame after the tracking process of a decoded video frame is completed, but can sequentially perform reverse decoding on the video frames before the video frame 4, that is, after decoding a video frame, it can also continue to decode the next video frame (the previous video frame of the current video frame) in reverse. Moreover, every time a video frame is decoded, the texture conversion of the video frame can be performed to obtain the image texture 1 of the video frame.

[0496] S516. The basic function layer sends the presentation time stamp (PTS) and image texture 1 of video frame 5 to the resolution reduction module of the media platform framework layer.

[0497] Among them, the PTS of video frame 5 is used to indicate the display time of video frame 5.

[0498] In the embodiment of the present application, the basic function layer can send the PTS and image texture of the decoded video frame to the media platform framework layer in sequence. In order to avoid the disorder of the image texture of the video frame sent in sequence, the PTS of the video frame can also be sent while sending the image texture of the video frame, so as to indicate the order of processing of the video frame through the PTS of the video frame.

[0499] S517 . The resolution reduction module performs resolution reduction processing on the image texture 1 of the video frame 5 to obtain the image texture 2 of the video frame 5 .

[0500] The resolution of image texture 2 is smaller than that of image texture 1. By reducing the resolution of image texture 1 to obtain image texture 2 with a smaller resolution, the efficiency of subsequent image processing can be improved.

[0501] S518. The resolution reduction module sends the PTS of the video frame 5 and the image texture 2 to the texture to image frame module.

[0502] S519 , the texture-to-image-frame module converts the image texture 2 of the video frame 5 into the video frame data of the video frame 5 .

[0503] For example, in the process of converting the GPU video memory into the internal memory, the GPU video memory data (image texture 2) may be converted into the internal memory data (video frame data).

[0504] S520, the texture-to-image frame module sends the PTS and video frame data of video frame 5 to the target tracking module of the smart middle platform framework layer through the target tracking algorithm interface of the media middle platform framework layer.

[0505] The texture-to-image frame module may first send the PTS and video frame data of the video frame 5 to the target tracking algorithm interface, and then the target tracking algorithm interface sends them to the target tracking module.

[0506] For example, the texture-to-image frame module can call the target tracking algorithm interface according to the PTS and video frame data of video frame 5, so as to send the PTS and video frame data of video frame 5 to the target tracking module through the target tracking algorithm interface. After the target tracking algorithm interface is called, the target tracking algorithm of the target tracking module of the smart middle platform framework layer can be called to perform target tracking on the video frame data of video frame 5 through the called target tracking algorithm.

[0507] S521. The target tracking module stores the PTS and video frame data of video frame 5 in the tracking queue.

[0508] The tracking queue is used to store video frame data of the video frames to be tracked.

[0509] It should be noted that Fig.18 The difference between the embodiments is that after receiving the video frame data sent by the basic function layer, the target tracking module does not directly track the received video frame data, but first stores the received video frame data in the tracking queue for queuing.

[0510] S522: The target tracking module obtains the latest video frame data of the video frame B from the tracking queue.

[0511] The latest video frame B refers to the video frame to be processed that is at the front in the queue order of the tracking queue.

[0512] S523 . The target tracking module performs target tracking on the video frame data of the video frame B according to the video frame 4 and the target frame position 3 to obtain the target frame position of the video frame B.

[0513] The target frame position of the video frame B refers to the target frame position corresponding to the target 3 in the video frame B, and is used to indicate the position of the target 3 in the video frame B on the video frame B.

[0514] In the embodiment of the present application, the target tracking module can sequentially obtain the video frame data to be processed from the tracking queue, and perform target tracking on the obtained video frame data. After the target tracking module obtains the video frame data of the latest video frame B, it can perform target tracking on the video frame data of video frame B according to the previously received video frame 4 and the target frame position 3, thereby obtaining the target frame position of video frame B.

[0515] As an example, the target tracking module may store the tracking result of the current video frame in a tracking result queue.

[0516] In addition, after the target tracking module performs target tracking on the video frame data of video frame B to obtain the tracking result, it can also return to step S522 to continue to obtain the video frame data of the next latest video frame from the tracking queue, and track the video frame data of the next latest video frame. It should be understood that the target tracking module can also continue to obtain the video frame data of the next latest video frame from the tracking queue after reporting the tracking result to the business layer, and track the video frame data of the next latest video frame. In this way, the video frames to be tracked can be tracked in sequence.

[0517] S524. The target tracking module sends a tracking completion notification to the business layer through the target tracking algorithm interface. The tracking completion notification carries the PTS of the video frame B and the target frame position.

[0518] The tracking completion notification is used to indicate that the tracking of a video frame is completed. That is, each time the target tracking module completes tracking of a video frame and obtains the target frame position of the video frame, it can send a tracking completion notification to the business layer so that the business layer can perform sticker processing on the latest tracked video frame. Sticker processing means adding stickers to the video frame according to the target frame position and displaying the video frame after adding stickers.

[0519] In this way, the electronic device can track the target of the untracked video frame while performing sticker processing on the tracked video frame, without having to complete the tracking processing (target tracking + sticker processing) of a video frame before tracking the next video frame (target tracking + sticker processing), thereby improving video processing efficiency.

[0520] It should be understood that the target tracking module may also first send a tracking completion notification that does not carry the PTS and target frame position of the video frame B to the business layer, and after the business layer receives the tracking completion notification, it obtains the PTS and target frame position of the latest tracked video frame B from the target tracking module. For example, the business layer may obtain the PTS and target frame position of the latest tracked video frame B from the tracking result queue of the target tracking module. For example, the business layer may send an acquisition request to the target tracking module so that the target tracking module obtains the PTS and target frame position of the latest tracked video frame according to the acquisition request, and returns the acquired PTS and target frame position of the video frame to the business layer.

[0521] S525. When the business layer determines that video frame B is the video frame to be displayed according to the PTS of video frame B, the sticker material information of video frame B is determined according to the target frame position of video frame B. The sticker material information includes the material position and material size of sticker material 2.

[0522] Among them, sticker material 2 is the sticker material to be added to video frame B, which can be a sticker material selected by the user. Please refer to Figure 6 The (d) figure in the figure may be the "Strawberry Head" sticker selected by the user when setting the target 3. The material position is used to indicate the adding position of the sticker material 2 to be added, and the material size is used to indicate the target size of the sticker material 2 to be added.

[0523] After the business layer (such as the special effects tracking module of the business layer) receives the target frame position of video frame B, it can convert the target frame position of video frame B into the material position and material size of sticker material 2, so that sticker material 2 can be added to video frame B according to the material position and material size of sticker material 2.

[0524] The process of determining the sticker material information of video frame B according to the target frame position of video frame B by the business layer can refer to the above Fig.12The relevant description of step A5 in this embodiment of the present application will not be repeated here.

[0525] In another embodiment, after the target tracking module performs target tracking on the video frame data of the video frame B according to the video frame 4 and the target frame position 3, an abnormal flag may be obtained, and the abnormal flag is used to indicate that the target of the video frame B is lost or missing. In this case, the above steps S522-S524 can also be replaced by the following steps S525-S527.

[0526] S526 , target tracking is performed on the video frame data of the video frame B according to the video frame 4 and the target frame position 3 to obtain an abnormality mark.

[0527] S527. The target tracking algorithm module sends the abnormal identification of the video frame B to the business layer through the target tracking algorithm interface.

[0528] S528. The business layer determines the sticker material information of the previous video frame of the reversely played video frame B as the sticker material information of the video frame B according to the abnormal flag of the video frame B.

[0529] The previous video frame of the video frame B played in the reverse direction is the next video frame of the video frame B played in the forward direction.

[0530] In this way, it is convenient to add sticker material to video frame B by using the sticker material information of the previous video frame played in reverse.

[0531] S529. The business layer sends sticker material information of video frame B to the basic function layer.

[0532] S530: The basic function layer updates the preview progress of the video to the preview time point corresponding to the current video frame B.

[0533] By updating the preview progress of the video to the preview time point corresponding to the current video frame B, the video track and the associated sticker track can be played backward synchronously.

[0534] S531. The basic function layer adds sticker material 2 to video frame B according to the sticker material information of video frame B.

[0535] In addition, each time a preset number of video frames with sticker materials added are generated, the preset number of video frames with sticker materials added may be reversely synthesized to generate a synthesized video. For example, when a preset number of video frames with sticker materials added are generated for the first time, the preset number of video frames with sticker materials added generated for the first time are reversely synthesized; and then, when a preset number of video frames with sticker materials added are generated again, the preset number of video frames with sticker materials added generated again are reversely synthesized with the synthesized video.

[0536] S532. The basic function layer displays the video frame B to which the sticker material 2 is added.

[0537] like Figure 2 As shown, the basic function layer includes a special effect rendering module, and the business layer can send the sticker material information of the video frame B to the special effect rendering module of the basic function layer. After receiving the sticker material information of the video frame B, the special effect rendering module can add sticker material 2 to the video frame B according to the sticker material information of the video frame B, and then send the video frame B with the sticker material 2 added to it for display. In this way, the video frame B with the sticker material 2 added can be displayed on the display screen, such as in the video preview window.

[0538] For example, Figure 3 As shown, the business layer can send the sticker material information of video frame B to the sticker rendering node in the special effect rendering pipeline of the basic function layer, and the sticker rendering node can render video frame B according to the sticker material information of video frame B, and then send the rendered video frame to the display node, which will display it. In this way, the rendered video frame (i.e., video frame B with sticker material 2 added) can be displayed on the display screen, such as in the video preview window.

[0539] It should be noted that after displaying the video frame B with the sticker material 2 added, the tracking completion notification sent by the target tracking module can continue to be received, so as to perform sticker processing on the next video frame according to the PTS and target frame position of the next video frame carried in the tracking completion notification. In this way, the video frames that have been tracked can be processed with stickers in sequence until the last video frame is processed.

[0540] In the asynchronous tracking interface mode, the target can be tracked on the video frame to be tracked, and the sticker processing can be performed on the video frame that has been tracked. The two can be executed synchronously, which can improve the efficiency of video processing.

[0541] It should be noted that the buffer smoothing mode is also supported in the asynchronous tracking interface mode to ensure the smoothness of the tracking processing output through the buffer smoothing mode. For example, when the sticker processing speed is greater than the target tracking speed, the callback frequency is adjusted according to the target tracking speed. The callback frequency refers to the frequency of obtaining the tracking results of the latest video frame completed from the tracking results corresponding to the tracking queue, so that the input frequency of the sticker processing is adapted to the target tracking speed with a relatively satisfactory speed, thereby ensuring the continuous smoothness of the tracking processing output. For example, the frequency at which the target tracking module sends a tracking completion notification to the business layer can be adjusted according to the target tracking speed, or the frequency at which the business layer obtains the tracking results of the latest video frame completed from the tracking results of the tracking module can be adjusted. Alternatively, when the sticker processing speed is less than the target tracking speed, the callback frequency is adjusted according to the sticker processing speed so that the input frequency of the sticker processing is adapted to the slower sticker processing speed, thereby ensuring the continuous smoothness of the tracking processing output.

[0542] In addition, the basic function layer can also determine the video segment 2 to be forward tracked from the video according to the video frame 4, according to the above Fig.19 The tracking processing method described in the embodiment performs forward tracking processing on the video segment 2. For example, the video segment 2 may be a video segment corresponding to the next video frame of the video frame 4 to the last video frame of the video.

[0543] It should be noted that the embodiment of the present application can first perform backward tracking processing on video segment 1 and then perform forward tracking processing on video segment 2, or can first perform forward tracking processing on video segment 2 and then perform backward tracking processing on video segment 1. The embodiment of the present application is not limited to this.

[0544] S352: After completing all tracking processing on the video, the basic function layer destroys the target tracking algorithm instance.

[0545] For example, the basic function layer may destroy the target tracking algorithm instance after completing the tracking processing of the video segment 1 and the video segment 2 of the video.

[0546] As an example, the basic function layer may send a destruction instruction to the target tracking module through the target tracking algorithm interface, where the destruction instruction is used to instruct the destruction of the created target tracking algorithm instance.

[0547] It should be noted that Fig.21The embodiment is only explained by taking the example of a user setting a tracking target on a video frame (video frame 4) of a video. It should be understood that the embodiment of the present application also supports setting tracking targets on multiple video frames. For example, after the user sets the tracking target (target 3) on video frame 4, he can continue to set tracking targets (such as target 4) on other video frames (such as a video frame 6 after video frame 4). For the video clip before video frame 4, target 3 can be used as the tracking target for tracking processing; for the video clip after video frame 6, target 4 can be used as the tracking target for tracking processing. For the video clip between the two video frames, target 3 can be used as the tracking target for tracking processing, or target 4 can be used as the tracking target for tracking processing, and the embodiment of the present application does not limit this. For example, the tracking target of the video clip between the two video frames can be specified by the user from target 3 and target 4.

[0548] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (such as: coaxial cable, optical fiber, data subscriber line (Digital Subscriber Line, DSL)) or wireless (such as: infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a digital versatile disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD)).

[0549] The above are optional embodiments provided for the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the technical scope disclosed in the present application shall be included in the protection scope of the present application.

Claims

1. A video processing method, characterized in that: The method comprises: Displaying a first video picture of a video, where the first video picture is a video picture corresponding to any video frame of the video; In response to the user setting a first target to be tracked on the first video screen and confirming an operation of performing sticker tracking, determining a first target frame position, where the first target frame position is used to indicate a position of the first target on a first reference video frame corresponding to the first video screen; According to the first reference video frame, the first target frame position and the first sticker material, a first video segment of the video is tracked so that the first sticker material is added to the position of the first target in the first video segment, and the first video segment includes a video segment after the first reference video frame and / or a video segment before the first reference video frame; The first video clip to which the first sticker material is added is displayed.

2. The method according to claim 1, characterized in that In response to the user setting a first target to be tracked on the first video screen and confirming an operation of performing target tracking, determining a first target frame position includes: In response to the user dragging the first sticker material to the position of the first target in the first video screen and triggering the smart tracking option, the first target frame position is determined.

3. The method according to claim 2, characterized in that In response to the user dragging the first sticker material to the position of the first target in the first video screen and triggering the smart tracking option, before determining the position of the first target frame, the method further includes: In response to a user triggering operation on a sticker option, displaying a first interface including a plurality of sticker materials; In response to a user selecting the first sticker material from the plurality of sticker materials, displaying the first sticker material at a preset position of the first video screen; In response to the user dragging the first sticker material to the position of the first target in the first video screen and triggering the smart tracking option, determining the position of the first target frame includes: In response to the user dragging the first sticker material from the preset position to the position of the first target on the first video screen and triggering the smart tracking option, the first target frame position is determined.

4. The method according to any one of claims 1 to 3, characterized in that: Before displaying the first video screen of the video, the method further includes: In response to the user's first operation, loading and displaying videos and pictures in the gallery; In response to the user selecting an image material in the gallery and confirming an operation of performing video editing, displaying a video editing interface, the video editing interface including a video preview window, the video preview window being used to preview and display a video screen of the video obtained according to the image material, the image material including a video, or including a video and a picture, or including a plurality of pictures; The first video screen of the displayed video includes: The first video frame is displayed in the video preview window.

5. The method according to any one of claims 1 to 4, characterized in that: The first video segment includes at least one video segment; The tracking process of the first video segment of the video according to the first reference video frame, the first target frame position and the first sticker material includes: sequentially acquiring target video frames to be processed from target video segments, wherein the target video segment is any one of the at least one video segment; According to the first reference video frame and the first target frame position, target tracking is performed on the acquired first video frame, where the first video frame is any target video frame in the target video segment; In a case where the target frame position of the first video frame is obtained by target tracking the first video frame, determining sticker material information of the first video frame according to the target frame position of the first video frame, wherein the sticker material information at least includes a material position of the first sticker material; The first sticker material is added to the first video frame according to the sticker material information of the first video frame to generate a second video frame, and the first sticker material is added to the position of the first target in the second video frame.

6. The method according to claim 5, characterized in that The step of sequentially acquiring target video frames to be processed from target video segments includes: Sequentially acquiring target video frames to be processed from the target video segments, and storing the acquired target video frames in a tracking queue, wherein the tracking queue is used to store the video frames to be tracked; The performing target tracking on the acquired first video frame according to the first reference video frame and the first target frame position includes: Sequentially acquiring video frames to be tracked from the tracking queue, and performing target tracking on the acquired video frames according to the first reference video frame and the first target frame position, wherein the acquired video frames include the first video frame; In the case where the target frame position of the first video frame is obtained by performing target tracking on the first video frame, determining the sticker material information of the first video frame according to the target frame position of the first video frame includes: The target frame position of the latest tracked video frame is obtained from the tracking results corresponding to the tracking queue in sequence, and the sticker material information of the video frame is determined according to the obtained target frame position of the video frame, and the video frame includes the first video frame.

7. The method according to claim 5 or 6, characterized in that The step of sequentially acquiring target video frames to be processed from the target video segments includes: When the target video segment is a video segment after the first reference video frame, sequentially acquiring target video frames to be processed from the target video segment in a forward video playback order; In the case that the target video segment is a video segment located before the first reference video frame, the target video frames to be processed are sequentially obtained from the target video segment in a reverse video playback order.

8. The method according to any one of claims 5 to 7, characterized in that: The target video frame is a video frame or a key frame in the target video segment.

9. The method according to any one of claims 5 to 7, characterized in that: The target video frame is a key frame, the first video frame is a first key frame, and the method further includes: When the target frame position of the first key frame is obtained after tracking the first key frame, linear interpolation is performed according to the target frame position of the first key frame and the target frame position of the third video frame to obtain the target frame position of an intermediate video frame, where the intermediate video frame is a video frame between the third video frame and the first key frame; Wherein, when the first key frame is the first key frame tracked in the first video segment, the third video frame is the first reference video frame; when the first key frame is not the first key frame, the third video frame is the most recently tracked key frame before the first key frame; Determining sticker material information of the intermediate video frame according to the target frame position of the intermediate video frame; The first sticker material is added to the intermediate video frame according to the sticker material information of the intermediate video frame to generate an intermediate video frame to which the first sticker material is added.

10. The method according to any one of claims 5 to 7, characterized in that: The target video frame is a key frame, the first video frame is a first key frame, and the method further includes: When the sticker material information of the first key frame is obtained, linear interpolation is performed according to the sticker material information of the first key frame and the sticker material information of the third video frame to obtain the sticker material information of the middle video frame; The first sticker material is added to the intermediate video frame according to the sticker material information of the intermediate video frame to generate an intermediate video frame to which the first sticker material is added.

11. The method according to claim 9 or 10, characterized in that The displaying the first video clip with the first sticker material added thereto includes: When the intermediate video frame with the first sticker material added thereto and the first key frame with the first sticker material added thereto are generated, the intermediate video frame with the first sticker material added thereto and the first key frame with the first sticker material added thereto are previewed and displayed.

12. The method according to any one of claims 5 to 8, characterized in that: The displaying of other video frames with the first sticker material added thereto includes: Whenever a second video frame with the first sticker material added thereto is generated, a preview of the second video frame is displayed.

13. The method according to any one of claims 5 to 12, characterized in that: The method further comprises: In a case where an abnormality mark is obtained by performing target tracking on the first video frame, determining the historical sticker material information as the sticker material information of the first video frame, wherein the abnormality mark is used to indicate that the first target is lost or missing in the first video frame; Among them, when there is a target video frame that has been tracked before the first video frame, the historical sticker material information is the sticker material information of the most recently tracked target video frame before the first video frame; when there is no target video frame that has been tracked before the first video frame, the historical sticker material information is the sticker material information of the first reference video frame.

14. The method according to any one of claims 1 to 13, characterized in that: In response to the user setting a first target to be tracked on the first video screen and confirming the operation of performing target tracking, after determining the position of the first target frame, the method further includes: Displaying a second video picture of the video, where the second video picture is a video picture corresponding to any video frame in the video except the first reference video frame; In response to the user setting a second target to be tracked on the second video screen and confirming an operation of performing target tracking, determining a second target frame position, where the second target frame position is used to indicate a position of the second target on a second reference video frame corresponding to the second video screen; According to the second reference video frame, the second target frame position and the second sticker material, target tracking is performed on a second video segment in the video, so that the second sticker material is added to the position of the second target on the second video segment, and the second video segment is all or part of the video frames in the video frame except the second reference video frame; The second video clip to which the second sticker material is added is displayed.

15. The method according to claim 14, characterized in that The first video segment includes a video segment located on a first side of the first reference video frame, and the second video segment includes a video segment located on a second side of the second reference video frame, the first side refers to a side that is not adjacent to the second reference video frame, and the second side refers to a side that is not adjacent to the first reference video frame.

16. The method according to claim 15, characterized in that The first video segment also includes an intermediate video segment between the first reference video frame and the second reference video frame, or the second video segment also includes the intermediate video segment.

17. The method according to claim 16, characterized in that In response to the user setting a second target to be tracked on the second video screen and confirming the operation of performing target tracking, after determining the position of the second target frame, the method further includes: displaying prompt information, wherein the prompt information is used to prompt the user to specify a tracking target from the first target and the second target for the middle video segment; If the tracking target specified by the user is the first target, the first video segment also includes the middle video segment; if the tracking target specified by the user is the second target, the second video segment also includes the middle video segment.

18. The method according to any one of claims 1 to 17, characterized in that: After tracking the first video segment of the video according to the first reference video frame, the first target frame position and the first sticker material, the method further includes: In response to the user canceling the operation of executing sticker tracking, adding the first sticker material at a preset position of each video frame in the first video clip; The first video clip with the first sticker material added at the preset position is displayed.

19. The method according to any one of claims 1 to 18, characterized in that: Before determining the first target frame position, the method further includes: Create an editing preview pipeline, a special effect rendering pipeline and a target tracking pipeline, wherein the editing preview pipeline includes a decoding node and a display sending node connected in sequence, the special effect rendering pipeline is located after the decoding node and before the display sending node, the special effect rendering node includes at least a sticker rendering node, the target tracking pipeline is located after the decoding node and before the sticker rendering node, and the target tracking pipeline includes at least a target tracking node; The tracking process of the first video segment of the video according to the first reference video frame, the first target frame position and the first sticker material includes: By running the editing preview pipeline, the special effects rendering pipeline and the target tracking pipeline, the first video segment of the video is tracked according to the first reference video frame, the first target frame position and the first sticker material.

20. The method of claim 19, wherein: The step of running the editing preview pipeline, the special effects rendering pipeline, and the target tracking pipeline to track the first video segment of the video according to the first reference video frame, the first target frame position, and the first sticker material includes: Decoding the first video segment by the decoding node, and sequentially acquiring target video frames to be processed from the decoding results; By means of the target tracking node, target tracking is performed on the acquired first video frame according to the first reference video frame and the first target frame position to obtain the target frame position of the first video frame, where the first video frame is any target video frame in the first video segment; The first sticker material is rendered on the first video frame according to the sticker material information of the first video frame through the sticker rendering node to generate a second video frame, wherein the first sticker material is added to the position of the first target in the second video frame, and the sticker material information of the first video frame is determined according to the target frame position of the first video frame, and the sticker material information of the first video frame at least includes the material position.

21. The method of claim 20, wherein: The displaying the first video clip with the first sticker material added thereto includes: The second video frame is displayed through the display node.

22. An electronic device, characterized in that: The electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 21 when executed by the processor.

23. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 21.

Images