Video processing method and apparatus, medium, and electronic device

Abstract

The present disclosure provides a video processing method and apparatus, a medium, and an electronic device. The video processing method comprises: determining a target area in a video to be processed (S11); detecting the target area in a video frame to be processed in said video to obtain a tracking bounding box for the target area (S12); smoothing the trajectory of the tracking bounding box (S13); and processing said video on the basis of the smoothed trajectory of the tracking bounding box to obtain a target video (S14).

Description

Video processing method, device, medium and electronic equipment

[0001] This application claims priority to Chinese patent application No. 202311816469.8 filed on December 26, 2023, and the contents of the above-mentioned Chinese patent application disclosure are hereby cited in their entirety as part of this application. Technical Field

[0002] The present disclosure relates to a video processing method, device, medium and electronic device. Background Art

[0003] In related technologies, the tracking frame directly output by the tracking algorithm may experience inter-frame jitter, thereby causing visual fatigue and poor video display quality. Summary of the Invention

[0004] This summary is provided to briefly introduce concepts that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] In a first aspect, the present disclosure provides a video processing method, comprising:

[0006] Determine the target area in the video to be processed;

[0007] Detecting the target area in the video frame to be processed in the video to be processed to obtain a tracking frame for the target area;

[0008] Smoothing the trajectory of the tracking frame;

[0009] The video to be processed is processed based on the trajectory of the smoothed tracking frame to obtain a target video.

[0010] In a second aspect, the present disclosure provides a video processing device, comprising:

[0011] A determination module, used to determine the target area in the video to be processed;

[0012] A detection module, configured to detect the target area in the video frame to be processed in the video to be processed, and obtain a tracking frame for the target area;

[0013] A smoothing module, configured to smooth the trajectory of the tracking frame;

[0014] The processing module is used to process the video to be processed based on the trajectory of the smoothed tracking frame to obtain a target video.

[0015] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of any one of the methods described in the first aspect.

[0016] In a fourth aspect, the present disclosure provides an electronic device, comprising:

[0017] a storage device having a computer program stored thereon;

[0018] A processing device is used to execute the computer program in the storage device to implement the steps of any one of the methods in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale. In the drawings:

[0020] FIG1 is a flowchart of a video processing method according to an embodiment of the present disclosure;

[0021] FIG2 shows a flow chart of determining a target smoothing coefficient based on a horizontal offset threshold, a vertical offset threshold, and a smoothing coefficient accuracy according to an embodiment of the present disclosure;

[0022] FIG3 is a schematic block diagram of a video processing apparatus according to an embodiment of the present disclosure; and

[0023] FIG4 shows a schematic structural diagram of an electronic device suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION

[0024] The following will be described with reference to the accompanying drawings in the embodiments of the present disclosure.

[0025] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0026] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0027] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Definitions of other terms are provided in the following description.

[0028] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0029] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0030] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0031] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0032] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0033] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0034] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0035] At the same time, it is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0036] Figure 1 is a flow chart of a video processing method according to an embodiment of the present disclosure. As shown in Figure 1 , the video processing method includes the following steps S11 to S14.

[0037] In step S11 , a target area in the video to be processed is determined.

[0038] Wherein, a specific area or a user-defined area in the video to be processed can be determined as the target area.

[0039] For example, the target area can be the head or body of a person in the video to be processed, or any other user-defined area, such as an animal area, a vehicle area, etc.

[0040] In step S12, the target area in the video frame to be processed in the video to be processed is detected to obtain a tracking frame for the target area.

[0041] Any detection method may be used to detect the target region in the to-be-processed video frame, such as a tracking algorithm, a target object detection method, etc. Furthermore, the target region in the to-be-processed video frame may be first identified, and then a minimum bounding box that can enclose the identified target region may be used as a tracking box for the target region.

[0042] The tracking frame can be represented by the coordinates of its vertices. For example, it can be represented by the coordinates of the top left corner and the bottom right corner of the tracking frame. For example, if the coordinates of the top left corner of the tracking frame are (X0, Y0) and the coordinates of the bottom right corner are (X1, Y1), then the width W of the tracking frame can be represented as: W = X1 - X0, and the height H of the tracking frame can be represented as: H = Y1 - Y0.

[0043] In step S13, the trajectory of the tracking frame is smoothed.

[0044] In step S14, the video to be processed is processed based on the trajectory of the smoothed tracking frame to obtain a target video.

[0045] By adopting the above technical solution, it is possible to determine the target area in the video to be processed, detect the target area in the video frame to be processed in the video to be processed, obtain a tracking frame for the target area, smooth the trajectory of the tracking frame, and process the video to be processed based on the trajectory of the smoothed tracking frame to obtain the target video. Therefore, through the smoothing of the trajectory of the tracking frame, the anti-shake effect of the tracking frame can be achieved in real time without producing a significant delay effect, eliminating the inter-frame jitter phenomenon of the tracking frame, and not causing visual fatigue, thereby improving the display effect of the video, improving the user experience, and enhancing the application value of the tracking frame information.

[0046] In some embodiments, the smoothing of the trajectory of the tracking frame described in step S13 may include: smoothing the trajectory of the tracking frame based on a target smoothing coefficient and the coordinates of the tracking frame. The coordinates of the tracking frame used for the smoothing process may be the vertex coordinates of the tracking frame, such as the coordinates of the upper left corner point, the coordinates of the lower right corner point, etc. of the tracking frame, or the coordinates of the center point of the tracking frame. The coordinates of the center point of the tracking frame can be obtained based on the vertex coordinates of the tracking frame. For example, the coordinates of the upper left corner point of the tracking frame are (X0, Y0) and the coordinates of the lower right corner point are (X1, Y1). Then, the coordinates of the center point of the tracking frame (Xc, Yc) are ((X0+X1) / 2, (Y0+Y1) / 2), the width W of the tracking frame is X1-X0, and the height H of the tracking frame is Y1-Y0.

[0047] In some embodiments, the smoothing of the trajectory of the tracking frame described in step S13 may include: smoothing the trajectory of the current tracking frame based on the center point coordinates of the previous smoothing frame, the center point coordinates of the current tracking frame, and the target smoothing coefficient, wherein the previous smoothing frame is a tracking frame obtained by smoothing the trajectory of the previous tracking frame.

[0048] For example, the center point coordinates of the previous smoothing frame and the center point coordinates of the current tracking frame may be weighted based on the target smoothing coefficient, thereby smoothing the trajectory of the current tracking frame.

[0049] Weighted processing can be achieved as follows: Xc2 = Xc1 * alpha 目标 +Xc*(1-alpha 目标 ) Yc2=Yc1*alpha 目标 +Yc*(1-alpha 目标 )

[0050] Among them, Xc2 and Yc2 represent the x-coordinate and y-coordinate of the center point of the current tracking frame after trajectory smoothing, Xc1 and Yc1 represent the x-coordinate and y-coordinate of the center point of the previous tracking frame after trajectory smoothing, Xc and Yc represent the x-coordinate and y-coordinate of the center point of the current tracking frame (that is, the center point before trajectory smoothing), alpha 目标 Represents the target smoothing coefficient.

[0051] Those skilled in the art should understand that the above weighted processing method is only an example and the present disclosure does not limit it.

[0052] By adopting the above technical solution, the trajectory of the current tracking frame can be smoothed based on the trajectory smoothing result of the previous tracking frame, so that the anti-shake effect of the tracking frame can be achieved in real time without producing a significant delay effect, eliminating the inter-frame jitter phenomenon of the tracking frame, and not causing visual fatigue, thereby improving the display effect of the video, improving the user experience, and enhancing the application value of the tracking frame information.

[0053] In some embodiments, if the current tracking frame is the tracking frame in the first video frame, smoothing of the tracking frame's trajectory may not be performed, and instead the vertex coordinates of the tracking frame (e.g., the coordinates of the upper left corner and the lower right corner) may be directly output. This eliminates the need to smooth the trajectory of the tracking frame in the first video frame, thereby improving the efficiency of trajectory smoothing.

[0054] In some embodiments, the target smoothing coefficient may be a preset fixed value, or may be determined based on a horizontal offset threshold, a vertical offset threshold, and a smoothing coefficient accuracy.

[0055] The horizontal offset threshold refers to the threshold for the horizontal offset of the tracking frame, and the vertical offset threshold refers to the threshold for the vertical offset of the tracking frame.

[0056] FIG2 shows a flowchart of determining a target smoothing coefficient based on a horizontal offset threshold, a vertical offset threshold, and a smoothing coefficient accuracy according to an embodiment of the present disclosure.

[0057] As shown in FIG2 , first, in step S21 , the current smoothing coefficient alpha1 is determined based on the initial upper smoothing coefficient high and the initial lower smoothing coefficient low.

[0058] The initial upper limit smoothing coefficient high is the initial value of the upper limit smoothing coefficient, and the initial lower limit smoothing coefficient low is the initial value of the lower limit smoothing coefficient. For example, the initial upper limit smoothing coefficient high can be set to 1 or other values, and the initial lower limit smoothing coefficient low can be set to 0 or other values.

[0059] The current smoothing coefficient alpha1 may be set to the average of the initial upper limit smoothing coefficient high and the initial lower limit smoothing coefficient low, that is, alpha1 = (low + high) / 2.

[0060] In step S22, based on the center point coordinates of the previous smoothing frame, the center point coordinates of the current tracking frame, and the current smoothing coefficient, the trajectory of the current tracking frame is smoothed to obtain the smoothed center point coordinates of the current tracking frame, wherein the previous smoothing frame is a tracking frame obtained by smoothing the trajectory of the previous tracking frame.

[0061] For example, the center point coordinates of the previous smoothing frame and the center point coordinates of the current tracking frame can be weighted based on the current smoothing coefficient alpha1. For example, the center point coordinates of the previous smoothing frame are (Xc1, Yc1), and the center point coordinates of the current tracking frame are (Xc, Yc). Then, the smooth center point coordinates of the current tracking frame obtained by weighted processing (Xc2, Yc2) are: Xc2 = Xc1*alpha1 + Xc*(1-alpha1), Yc2 = Yc1*alpha1 + Yc*(1-alpha1). It should be understood by those skilled in the art that this weighted processing method is only an example and is not limited in this disclosure.

[0062] In step S23 , a horizontal offset value and a vertical offset value are determined based on the smoothed center point coordinates of the current tracking frame and the center point coordinates of the current tracking frame.

[0063] Still taking the previous example, the horizontal offset value Dw is abs(Xc2-Xc), and the vertical offset value Dh is abs(Yc2-Yc), where abs represents absolute value operation.

[0064] In step S24, the initial upper limit smoothing coefficient and the initial lower limit smoothing coefficient are updated based on the horizontal offset threshold, the vertical offset threshold, the horizontal offset value, and the vertical offset value.

[0065] In some embodiments, if the horizontal offset value is less than the horizontal offset threshold and the vertical offset value is less than the vertical offset threshold, the initial lower limit smoothing coefficient is updated using the current smoothing coefficient. For example, the initial lower limit smoothing coefficient can be updated to the current smoothing coefficient, that is, setting low = alpha1.

[0066] In some embodiments, if the horizontal offset value is less than the horizontal offset threshold and the vertical offset value is less than the vertical offset threshold, the initial upper limit smoothing coefficient is updated using the current smoothing coefficient. For example, the initial upper limit smoothing coefficient can be updated to the current smoothing coefficient, that is, high = alpha1.

[0067] In step S25 , the target smoothing coefficient is determined based on the smoothing coefficient accuracy, the updated initial upper limit smoothing coefficient, and the updated initial lower limit smoothing coefficient.

[0068] For example, if the difference between the updated initial upper limit smoothing coefficient and the updated initial lower limit smoothing coefficient is less than the smoothing coefficient precision, it is indicated that the target smoothing coefficient has been found and the loop can be terminated. If the difference between the updated initial upper limit smoothing coefficient and the updated initial lower limit smoothing coefficient is not less than the smoothing coefficient precision, it is indicated that the target smoothing coefficient has not been found and the process returns to step S21 to continue searching for the target smoothing coefficient.

[0069] By adopting the above technical solution, the target smoothing coefficient can be determined based on the horizontal offset threshold, the vertical offset threshold, and the smoothing coefficient accuracy, so that the center point coordinate offset of the tracking frame after trajectory smoothing is just within the threshold range.

[0070] In some embodiments, the video processing method according to the embodiment of the present disclosure may further include: taking the minimum value of the preset upper limit target smoothing coefficient and the determined target smoothing coefficient as the final target smoothing coefficient. That is, the preset upper limit target smoothing coefficient can be compared with the target smoothing coefficient determined according to the process of Figure 2, and the minimum of the two can be taken as the final target smoothing coefficient. Then, the final target smoothing coefficient can be used to smooth the trajectory of the current tracking frame, for example, to obtain the center point coordinates (Xc2, Yc2) of the smoothed trajectory of the current tracking frame, as well as the width W2 and height H2: Xc2 = Xc1*alpha 目标 +Xc*(1-alpha 目标 ) Yc2=Yc1*alpha 目标 +Yc*(1-alpha 目标 ) W2=W1*alpha 目标 +W*(1-alpha 目标 ) H2=H1*alpha 目标 +H*(1-alpha 目标 )

[0071] Among them, Xc2 and Yc2 represent the x-coordinate and y-coordinate of the center point of the current tracking frame after trajectory smoothing, Xc1 and Yc1 represent the x-coordinate and y-coordinate of the center point of the previous tracking frame after trajectory smoothing, Xc and Yc represent the x-coordinate and y-coordinate of the center point of the current tracking frame (that is, the center point before trajectory smoothing), alpha 目标represents the target smoothing coefficient, W1 and H1 represent the width and height of the previous tracking frame after trajectory smoothing, W and H represent the width and height of the current tracking frame before trajectory smoothing, and W2 and H2 represent the width and height of the current tracking frame after trajectory smoothing.

[0072] By taking the minimum value of the preset upper limit target smoothing coefficient and the determined target smoothing coefficient as the final target smoothing coefficient, over-smoothing can be avoided.

[0073] In addition, after obtaining the center point coordinates (Xc2, Yc2), width W2, and height H2 of the current tracking frame after trajectory smoothing, the coordinates of the upper left corner point (Xc2-W2 / 2, Yc2-H2 / 2) and the lower right corner point coordinates (Xc2+W2 / 2, Yc2+H2 / 2) of the smoothed frame corresponding to the current tracking frame can be obtained. The coordinates of the tracking frame after trajectory smoothing can be used for visualization and as input to other algorithms, ensuring the availability of the tracking frame information. For example, the tracking frame after trajectory smoothing can be used in many scenarios, such as target locking, sticker tracking, smart cropping, motion focus, and so on.

[0074] In some embodiments, the processing of the to-be-processed video based on the smoothed trajectory of the tracking frame in step S14 to obtain the target video may include:

[0075] Locking the target area based on the coordinates of the smoothed tracking frame;

[0076] Based on the locking result of the target area, at least one of the following processing is performed on the video to be processed to obtain the target video: target area anti-shake, setting of follow elements, and focus tracking of the target area.

[0077] In this way, different target videos can be obtained according to different application scenarios. For example, the target video can be a target-locked video, a sticker-followed video, a motion-focused video, and so on.

[0078] By adopting the above technical solution, since the target video is obtained based on the smoothed tracking frame, the inter-frame jitter of the tracking frame can be eliminated, visual fatigue will not be caused, and the display effect of the target video is improved. For example, the anti-shake effect of target locking, the stability of sticker following, and the stability of motion focus can be improved, thereby improving the user experience.

[0079] Figure 3 is a schematic block diagram of a video processing device according to an embodiment of the present disclosure. As shown in Figure 3, the video processing device includes: a determination module 31 for determining a target region in a video to be processed; a detection module 32 for detecting the target region in a video frame to be processed in the video to be processed, and obtaining a tracking frame for the target region; a smoothing module 33 for smoothing the trajectory of the tracking frame; and a processing module 34 for processing the video to be processed based on the smoothed trajectory of the tracking frame to obtain the target video.

[0080] By adopting the above technical solution, it is possible to determine the target area in the video to be processed, detect the target area in the video frame to be processed in the video to be processed, obtain a tracking frame for the target area, smooth the trajectory of the tracking frame, and process the video to be processed based on the trajectory of the smoothed tracking frame to obtain the target video. Therefore, through the smoothing of the trajectory of the tracking frame, the anti-shake effect of the tracking frame can be achieved in real time without producing a significant delay effect, eliminating the inter-frame jitter phenomenon of the tracking frame, and not causing visual fatigue, thereby improving the display effect of the video, improving the user experience, and enhancing the application value of the tracking frame information.

[0081] Optionally, the determining module 31 determines the target area in the video to be processed, including: determining a specific area or a user-defined area in the video to be processed as the target area.

[0082] Optionally, the detection module 32 detects the target area in the video frame to be processed in the video to be processed to obtain a tracking frame for the target area, including: identifying the target area in the video frame to be processed in the video to be processed; and using the minimum enclosing frame that can enclose the identified target area as the tracking frame for the target area.

[0083] Optionally, the smoothing module 33 smoothes the trajectory of the tracking frame, including:

[0084] The trajectory of the current tracking frame is smoothed based on the center point coordinates of the previous smoothing frame, the center point coordinates of the current tracking frame, and a target smoothing coefficient, wherein the previous smoothing frame is a tracking frame obtained by smoothing the trajectory of the previous tracking frame.

[0085] Optionally, the video processing device further includes a target smoothing coefficient determination module, configured to:

[0086] Determining a current smoothing coefficient based on an initial upper smoothing coefficient and an initial lower smoothing coefficient;

[0087] Based on the center point coordinates of a previous smoothing frame, the center point coordinates of the current tracking frame, and the current smoothing coefficient, smoothing the trajectory of the current tracking frame to obtain the smoothed center point coordinates of the current tracking frame, wherein the previous smoothing frame is a tracking frame obtained by smoothing the trajectory of the previous tracking frame;

[0088] Determining a horizontal offset value and a vertical offset value based on the smoothed center point coordinates of the current tracking frame and the center point coordinates of the current tracking frame;

[0089] updating the initial upper limit smoothing coefficient and the initial lower limit smoothing coefficient based on a horizontal offset threshold, a vertical offset threshold, the horizontal offset value, and the vertical offset value;

[0090] The target smoothing coefficient is determined based on the smoothing coefficient accuracy, the updated initial upper limit smoothing coefficient, and the updated initial lower limit smoothing coefficient.

[0091] Optionally, the target smoothing coefficient determination module updates the initial upper limit smoothing coefficient and the initial lower limit smoothing coefficient based on a horizontal offset threshold, a vertical offset threshold, the horizontal offset value, and the vertical offset value, including:

[0092] If the horizontal offset value is less than the horizontal offset threshold and the vertical offset value is less than the vertical offset threshold, updating the initial lower limit smoothing coefficient using the current smoothing coefficient;

[0093] If the horizontal offset value is less than the horizontal offset threshold and the vertical offset value is less than the vertical offset threshold, the initial upper limit smoothing coefficient is updated using the current smoothing coefficient.

[0094] Optionally, the target smoothing coefficient determination module is further configured to: use the minimum value between a preset upper limit target smoothing coefficient and the determined target smoothing coefficient as the final target smoothing coefficient.

[0095] Optionally, the processing module 34 processes the video to be processed based on the smoothed trajectory of the tracking frame to obtain a target video, including:

[0096] Locking the target area based on the coordinates of the smoothed tracking frame;

[0097] Based on the locking result of the target area, at least one of the following processing is performed on the video to be processed to obtain the target video: target area anti-shake, setting of follow elements, and focus tracking of the target area.

[0098] The specific implementation of the operations performed by each module in the video processing device according to the embodiment of the present disclosure has been described in detail in the relevant method and will not be repeated here.

[0099] The present disclosure also provides a computer-readable medium having a computer program stored thereon, which implements the steps of any method described in the present disclosure when the program is executed by a processing device.

[0100] The present disclosure also provides an electronic device, comprising:

[0101] a storage device having a computer program stored thereon;

[0102] A processing device is used to execute the computer program in the storage device to implement the steps of any method described in the present disclosure.

[0103] Reference is now made to FIG4 , which illustrates a schematic diagram of the structure of an electronic device 600 suitable for implementing embodiments of the present disclosure. Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device illustrated in FIG4 is merely an example and should not limit the functionality or scope of use of embodiments of the present disclosure.

[0104] As shown in Figure 4, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0105] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although FIG4 shows the electronic device 600 with various devices, it should be understood that not all of the devices shown are required to be implemented or present. More or fewer devices may alternatively be implemented or present.

[0106] According to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0107] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0108] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0109] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0110] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: determines a target area in a video to be processed; detects the target area in a video frame to be processed in the video to be processed to obtain a tracking frame for the target area; smoothes the trajectory of the tracking frame; and processes the video to be processed based on the trajectory of the smoothed tracking frame to obtain a target video.

[0111] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0113] The modules described in the embodiments of the present disclosure may be implemented in software or hardware. In some cases, the name of a module does not necessarily limit the module itself. For example, a determination module may also be described as a "module for determining a target area in a video to be processed."

[0114] The functions described above in the present disclosure may be performed at least in part by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0115] In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems,

[0116] More specific examples of machine-readable storage media would include electrical connections based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0117] According to one or more embodiments of the present disclosure, Example 1 provides a video processing method, including:

[0118] Determine the target area in the video to be processed;

[0119] Detecting the target area in the video frame to be processed in the video to be processed to obtain a tracking frame for the target area;

[0120] Smoothing the trajectory of the tracking frame;

[0121] The video to be processed is processed based on the trajectory of the smoothed tracking frame to obtain a target video.

[0122] According to one or more embodiments of the present disclosure, Example 2 provides the video processing method of Example 1, wherein determining the target area in the video to be processed includes:

[0123] A specific area or a user-defined area in the video to be processed is determined as the target area.

[0124] According to one or more embodiments of the present disclosure, Example 3 provides the video processing method of Example 1, wherein detecting the target area in the to-be-processed video frame in the to-be-processed video to obtain a tracking frame for the target area includes:

[0125] Identifying the target area in the video frame to be processed in the video to be processed;

[0126] The smallest bounding box that can enclose the identified target area is used as a tracking box for the target area.

[0127] According to one or more embodiments of the present disclosure, Example 4 provides the video processing method of Example 1, wherein smoothing the trajectory of the tracking frame includes:

[0128] The trajectory of the current tracking frame is smoothed based on the center point coordinates of the previous smoothing frame, the center point coordinates of the current tracking frame, and a target smoothing coefficient, wherein the previous smoothing frame is a tracking frame obtained by smoothing the trajectory of the previous tracking frame.

[0129] According to one or more embodiments of the present disclosure, Example 5 provides the video processing method of Example 4, wherein the target smoothing coefficient is determined by:

[0130] Determining a current smoothing coefficient based on an initial upper smoothing coefficient and an initial lower smoothing coefficient;

[0131] Based on the center point coordinates of a previous smoothing frame, the center point coordinates of the current tracking frame, and the current smoothing coefficient, smoothing the trajectory of the current tracking frame to obtain the smoothed center point coordinates of the current tracking frame, wherein the previous smoothing frame is a tracking frame obtained by smoothing the trajectory of the previous tracking frame;

[0132] Determining a horizontal offset value and a vertical offset value based on the smoothed center point coordinates of the current tracking frame and the center point coordinates of the current tracking frame;

[0133] updating the initial upper limit smoothing coefficient and the initial lower limit smoothing coefficient based on a horizontal offset threshold, a vertical offset threshold, the horizontal offset value, and the vertical offset value;

[0134] The target smoothing coefficient is determined based on the smoothing coefficient accuracy, the updated initial upper limit smoothing coefficient, and the updated initial lower limit smoothing coefficient.

[0135] According to one or more embodiments of the present disclosure, Example 6 provides the video processing method of Example 5, wherein the updating of the initial upper limit smoothing coefficient and the initial lower limit smoothing coefficient based on the horizontal offset threshold, the vertical offset threshold, the horizontal offset value, and the vertical offset value includes:

[0136] If the horizontal offset value is less than the horizontal offset threshold and the vertical offset value is less than the vertical offset threshold, updating the initial lower limit smoothing coefficient using the current smoothing coefficient;

[0137] If the horizontal offset value is less than the horizontal offset threshold and the vertical offset value is less than the vertical offset threshold, the initial upper limit smoothing coefficient is updated using the current smoothing coefficient.

[0138] According to one or more embodiments of the present disclosure, Example 7 provides the video processing method of Example 5, wherein the video processing method further includes: using the minimum value of the preset upper limit target smoothing coefficient and the determined target smoothing coefficient as the final target smoothing coefficient.

[0139] According to one or more embodiments of the present disclosure, Example 8 provides the video processing method of Example 1, wherein the processing of the to-be-processed video based on the trajectory of the smoothed tracking frame to obtain the target video includes:

[0140] Locking the target area based on the coordinates of the smoothed tracking frame;

[0141] Based on the locking result of the target area, at least one of the following processing is performed on the video to be processed to obtain the target video: target area anti-shake, setting of follow elements, and focus tracking of the target area.

[0142] According to one or more embodiments of the present disclosure, Example 9 provides a video processing device, including:

[0143] A determination module, used to determine the target area in the video to be processed;

[0144] A detection module, configured to detect the target area in the video frame to be processed in the video to be processed, and obtain a tracking frame for the target area;

[0145] A smoothing module, configured to smooth the trajectory of the tracking frame;

[0146] The processing module is used to process the video to be processed based on the trajectory of the smoothed tracking frame to obtain a target video.

[0147] According to one or more embodiments of the present disclosure, Example 10 provides a computer-readable medium having a computer program stored thereon, which implements the steps of the method described in any one of Examples 1-8 when executed by a processing device.

[0148] According to one or more embodiments of the present disclosure, Example 11 provides an electronic device, including:

[0149] a storage device having a computer program stored thereon;

[0150] A processing device is used to execute the computer program in the storage device to implement the steps of the method described in any one of Examples 1-8.

[0151] The above description is merely an illustration of the technical principles employed in the embodiments of the present disclosure. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also encompass other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0152] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0153] Although the present disclosure has been described using language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims. Regarding the apparatus in the above-described embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method and will not be elaborated upon here.

Claims

1. A video processing method, comprising: Determining a target area in a video to be processed; Detecting the target area in a video frame to be processed in the video to be processed to obtain a tracking box for the target area; Smoothing the trajectory of the tracking box; Processing the video to be processed based on the trajectory of the smoothed tracking box to obtain a target video.

2. The video processing method according to claim 1, wherein, The determining the target area in the video to be processed includes: Determining a specific area or a user-defined area in the video to be processed as the target area.

3. The video processing method according to claim 1 or 2, wherein, The detecting the target area in the video frame to be processed in the video to be processed to obtain a tracking box for the target area includes: Identifying the target area in the video frame to be processed in the video to be processed; Taking the smallest bounding box that can enclose the identified target area as the tracking box for the target area.

4. The video processing method according to any one of claims 1-3, wherein the smoothing the trajectory of the tracking box includes: Smoothing the trajectory of the current tracking box based on the center point coordinates of the previous smoothed box, the center point coordinates of the current tracking box, and a target smoothing coefficient, wherein the previous smoothed box is a tracking box obtained by smoothing the trajectory of the previous tracking box.

5. The video processing method according to claim 4, wherein, The target smoothing coefficient is determined by the following method: Determining a current smoothing coefficient based on an initial upper limit smoothing coefficient and an initial lower limit smoothing coefficient; Smoothing the trajectory of the current tracking box based on the center point coordinates of the previous smoothed box, the center point coordinates of the current tracking box, and the current smoothing coefficient to obtain the smoothed center point coordinates of the current tracking box, wherein the previous smoothed box is a tracking box obtained by smoothing the trajectory of the previous tracking box; Determining a horizontal offset value and a vertical offset value based on the smoothed center point coordinates of the current tracking box and the center point coordinates of the current tracking box; Updating the initial upper limit smoothing coefficient and the initial lower limit smoothing coefficient based on a horizontal offset threshold, a vertical offset threshold, the horizontal offset value, and the vertical offset value; Determining the target smoothing coefficient based on a smoothing coefficient accuracy, the updated initial upper limit smoothing coefficient, and the updated initial lower limit smoothing coefficient.

6. The video processing method according to claim 5, wherein, The updating the initial upper limit smoothing coefficient and the initial lower limit smoothing coefficient based on a horizontal offset threshold, a vertical offset threshold, the horizontal offset value, and the vertical offset value includes: If the horizontal offset value is less than the horizontal offset threshold and the vertical offset value is less than the vertical offset threshold, updating the initial lower limit smoothing coefficient with the current smoothing coefficient; If the condition that the horizontal offset value is less than the horizontal offset threshold and the vertical offset value is less than the vertical offset threshold is not satisfied, updating the initial upper limit smoothing coefficient with the current smoothing coefficient.

7. The video processing method according to claim 5 further includes: Taking the minimum value between a preset upper limit target smoothing coefficient and the determined target smoothing coefficient as the final target smoothing coefficient.

8. The video processing method according to any one of claims 1-7, wherein, The processing the video to be processed based on the trajectory of the smoothed tracking box to obtain a target video includes: Lock the target area based on the coordinates of the smoothed tracking box; Based on the locking result of the target area, perform at least one of the following processes on the video to be processed to obtain the target video: anti-shake of the target area, setting of following elements, and focus tracking of the target area.

9. A video processing device, comprising: A determination module configured to determine a target area in a video to be processed; A detection module configured to detect the target area in a video frame to be processed in the video to be processed, and obtain a tracking box for the target area; A smoothing module configured to smooth the trajectory of the tracking box; A processing module configured to process the video to be processed based on the trajectory of the smoothed tracking box to obtain a target video.

10. A computer-readable medium stores a computer program, wherein, When the computer program is executed by the processing device, it implements the video processing method according to any one of claims 1-8.

11. An electronic device, comprising: A storage device configured to store a computer program; A processing device configured to execute the computer program in the storage device to implement the video processing method according to any one of claims 1-8.

Images