Video generation method, device, equipment and medium

Abstract

This disclosure provides a video generation method, apparatus, device, and medium, relating to the field of artificial intelligence technology, specifically computer vision, deep learning, and other technical fields, and applicable to AIGC and other scenarios. The specific implementation scheme is as follows: A set of target detection boxes is obtained from the original video through target detection; each target appearing in the target detection box set is tracked to obtain a set of target tracking boxes for each target in the original video, wherein the set of target tracking boxes represents the dynamic change information of the spatial position of each target in the temporal sequence of the original video; a selected target tracking box is obtained for each original video frame based on the target tracking box set, and a cropping box for each original video frame is determined based on the selected target tracking box; each original video frame is cropped according to the cropping box, and a target video is generated based on the cropped frame sequence. This disclosure can improve the quality of the generated video.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically computer vision, deep learning and other technical fields, and can be applied to scenarios such as AIGC, etc. Specifically, it relates to a video generation method, device, equipment and medium. Background Art

[0002] AIGC, or automated content generation, is a new and rapidly developing field of artificial intelligence that has received widespread and strong attention. With the advent of the 5G high-bandwidth network era, people are increasingly eager for digital content with more visual expressiveness. The efficiency of generating traditional digital content has become a bottleneck in the new era. As the next hotspot for exploration, AIGC has stimulated a large number of industry demands and also allowed us to see a new tipping point for artificial intelligence technology.

[0003] Since most of the existing video materials are 4:3 or 16:9 horizontal screen videos, the videos re-synthesized by AIGC are also horizontal screen, which is not a good user experience for mobile devices such as mobile phones. The video horizontal to vertical conversion technology can directly process the existing videos produced by AIGC and quickly convert them into vertical screen videos with a specified aspect ratio, which is convenient for playback on mobile devices such as mobile phones. Summary of the invention

[0004] The present disclosure provides a video generation method, apparatus, device and medium.

[0005] According to one aspect of the present disclosure, there is provided a video generation method, comprising:

[0006] Obtain the target detection frame set of the original video through target detection;

[0007] Tracking each target appearing in the target detection frame set to obtain a target tracking frame set of each target in the original video, wherein the target tracking frame set of each target is used to represent dynamic change information of the spatial position of each target in the time sequence of the original video;

[0008] Acquire a selected target tracking frame of each original video frame in the original video according to the target tracking frame set, and determine a cropping frame of each original video frame according to the selected target tracking frame;

[0009] Each original video frame is cropped according to the cropping frame, and a target video is generated according to a frame sequence obtained by the cropping.

[0010] According to another aspect of the present disclosure, there is provided a video generating apparatus, comprising:

[0011] An object detection module is used to obtain an object detection frame set of the original video through object detection;

[0012] A tracking module, used to track each target appearing in the target detection frame set, and obtain a target tracking frame set of each target in the original video, wherein the target tracking frame set of each target is used to represent the dynamic change information of the spatial position of each target in the time sequence of the original video;

[0013] a cropping frame determining module, configured to obtain a selected target tracking frame of each original video frame in the original video according to the target tracking frame set, and determine a cropping frame of each original video frame according to the selected target tracking frame;

[0014] The video generation module is used to crop each original video frame according to the cropping frame, and generate a target video according to the frame sequence obtained by the cropping.

[0015] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the video generating method described in any embodiment of the present disclosure.

[0019] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the video generating method described in any embodiment of the present disclosure.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is a flow chart of a video generation method according to an embodiment of the present disclosure;

[0023] Figure 2 is a schematic diagram of a process for performing target detection in a video generation method according to an embodiment of the present disclosure;

[0024] Figure 3 is a schematic diagram of a process of tracking a target in a video generation method according to an embodiment of the present disclosure;

[0025] Figure 4 is a schematic diagram of a process of obtaining a target tracking frame during a target tracking process according to an embodiment of the present disclosure;

[0026] Figure 5 is a schematic diagram of a process of determining a cropping frame in a video generation method according to an embodiment of the present disclosure;

[0027] Figure 6 is a flowchart of another video generation method according to an embodiment of the present disclosure;

[0028] Figure 7 is a flowchart of another video generation method according to an embodiment of the present disclosure;

[0029] Figure 8 is a structural schematic diagram of a video generating device according to an embodiment of the present disclosure;

[0030] Fig. 9 It is a block diagram of an electronic device used to implement the video generating method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] Figure 1 This is a flow chart of a video generation method according to an embodiment of the present disclosure. This embodiment can be applied to the case of converting a horizontal video to a vertical video, or vice versa. It involves the field of artificial intelligence technology, specifically computer vision, deep learning and other technical fields, and can be applied to AIGC and other scenarios. The method can be executed by a video generation device, which is implemented in software and / or hardware, and is preferably configured in an electronic device, such as a smart terminal, a computer device or a server. Figure 1 As shown, the method specifically includes the following:

[0033] S101. Obtain a set of target detection frames of an original video through target detection.

[0034] Among them, the original video can be a horizontal video or a vertical video. Through the video generation method of the embodiment of the present disclosure, the original horizontal video can be converted into a vertical target video, and the original vertical video can also be converted into a horizontal target video.

[0035] In one embodiment, any target detection algorithm in the prior art can be used to perform target detection on the original video frame in the original video to obtain a target detection frame set, such as the YOLO algorithm (a typical one-stage target detection algorithm). In another embodiment, a saliency detection algorithm can also be used to obtain a target detection frame set. Of course, in yet another embodiment, a target detection algorithm and a saliency detection algorithm can also be combined to obtain a target detection frame set. Through target detection, a target detection frame with clear semantics in the original video can be obtained.

[0036] The target detection frame set includes the targets and their detection frames that appear in the original video frame. The information of the detection frame may include the position of the detection frame, the category label of the target, and the confidence. Exemplarily, the detection frame can be expressed as {xmin, ymin, xmax, ymax, cls, s}, where xmin and xmax represent the minimum and maximum values ​​of the X-axis coordinates of the center of the detection frame, ymin and ymax represent the minimum and maximum values ​​of the Y-axis coordinates, cls represents the category label, and s represents the confidence. In specific implementation, the required detection frame with a specific category label and confidence can be selected as the target detection frame according to requirements. For example, you can select a detection frame with a category label of face and human body, or you can select a set number of detection frames with the highest confidence based on the number of detection frames. For example, you can select 5 face detection frames and 3 human body detection frames with the highest confidence. The disclosed embodiment does not impose any restrictions on how to filter the detection frames.

[0037] S102, track each target appearing in the target detection frame set to obtain a target tracking frame set of each target in the original video, wherein the target tracking frame set of each target is used to represent dynamic change information of the spatial position of each target in the temporal sequence of the original video.

[0038] S103 . Obtain a selected target tracking frame of each original video frame in the original video according to the target tracking frame set, and determine a cropping frame of each original video frame according to the selected target tracking frame.

[0039] S104: Crop each original video frame according to the cropping frame, and generate a target video according to a frame sequence obtained by cropping.

[0040] For example, target tracking can be implemented based on target tracking algorithms such as KCF (Kernel Correlation Filter). A target in a video usually appears in multiple video frames, and by tracking and obtaining a target tracking frame set of each target in the original video, each target tracking frame in the target tracking frame set can be used to represent the dynamic change information of the spatial position of each target in the temporal sequence of the original video, that is, the change of each target between frames can be obtained.

[0041] Then, the selected target tracking frame of each original video frame in the original video is obtained according to the target tracking frame set, and the cropping frame of each original video frame is determined according to the selected target tracking frame. Among them, the selected target tracking frame is the tracking frame of the optimal target among all targets in the original video frame. The judgment criteria of the optimal target can be flexibly defined according to actual needs. For example, it can be a specified optimal target, or it can be comprehensively defined according to factors such as the category of the target, the number of times the target appears in each video frame, the position and size of the target corresponding tracking frame, etc. The disclosed embodiment does not impose any restrictions on the criteria for defining the optimal target. After determining the cropping frame, each original video frame can be cropped according to the cropping frame, and the target video can be generated according to the cropped frame sequence.

[0042] In the application of generating a corresponding vertical video according to a horizontal video, since the horizontal video needs to be cropped, the main target in the video screen is often affected, such as the face, limbs or other foreground objects of the person being partially cut off, which greatly affects the viewing experience of the video. Therefore, it is necessary to consider the main target in the screen and crop it intelligently when cropping the video to ensure that the main target is either completely retained in the screen or completely cropped off, and partial truncation is not allowed. In the embodiment of the present disclosure, the main target with semantic information in the horizontal video can be obtained through target detection, and the corresponding target detection frame set is obtained. Then, the target tracking frame set of each target in the horizontal video is obtained by tracking, and the change of each target between frames is determined. Then, the selected target tracking frame is obtained from it, and the cropping frame is determined according to the selected target tracking frame. Finally, the video frame is cropped according to the cropping frame, and the vertical video can be generated according to the frame sequence obtained by cropping. On the one hand, it can ensure that the optimal main target is not truncated, and on the other hand, it can also accurately perform dynamic cropping between different frames to obtain the dynamic cropping frame of the moving target in different frames.

[0043] It can be seen that in the technical solution of the embodiment of the present disclosure, target tracking is used to obtain changes in the target between frames, thereby obtaining the dynamic changes in the spatial positions of various types of targets in time sequence. Therefore, a more accurate dynamic cropping frame can be obtained for targets moving in the video, reducing the possibility of moving targets being truncated, and improving the overall quality of the generated video.

[0044] Figure 2 FIG. 1 is a flow chart of target detection in a video generation method according to an embodiment of the present disclosure. Based on the above embodiment, this embodiment further optimizes the specific operation of target detection in S101. Figure 2 As shown, specifically including the following:

[0045] S201. Extract frames from original video frames of an original video according to a first interval frame number to obtain a first video frame set, and use the first video frame set as a video frame set to be detected.

[0046] S202 , performing target detection on each of the to-be-detected video frames in the to-be-detected video frame set to obtain a target detection frame set of the original video.

[0047] The first interval frame number can be flexibly set, and the disclosed embodiment does not impose any limitation on this. For example, it can be set to 6 frames, that is, starting from the first frame, frame extraction is performed every 6 frames. The set of video frames to be detected is obtained by frame extraction, and target detection is performed on each video frame to be detected. In this way, it can be ensured that the target appearing in the video is detected, and the number of frames of video processing can be reduced, thereby improving the efficiency of video generation.

[0048] In one implementation, the target detection algorithm and the saliency detection algorithm may be combined to obtain the target detection frame set of the original video. That is, target detection is performed on each of the to-be-detected video frames in the to-be-detected video frame set to obtain the target detection frame set of the original video, specifically including:

[0049] Obtaining a first detection frame set for each video frame to be detected using an object detection algorithm;

[0050] If there is any target to be detected video frame whose first detection frame set is empty, obtaining a second detection frame set of the target to be detected video frame by using a saliency detection algorithm;

[0051] A target detection frame set of the original video is obtained according to the first detection frame set and the second detection frame set.

[0052] Specifically, for each video frame to be detected, the target with clear semantics is first detected by the target detection algorithm, and then, for some frames where the target is not detected by the target detection algorithm, further detection is performed by the saliency detection algorithm. Finally, the set of detection frames detected in each video frame to be detected is used as the target detection frame set. Preferably, the first detection frame set includes a face detection frame and a human body detection frame. That is to say, through the target detection algorithm, a detection frame with a category label of face or human body is selected as the first detection frame. In this way, the main target in the video can be detected to a large extent. It should be noted that the area of ​​the first detection frame also needs to meet the condition of being greater than the first threshold, so as to facilitate the subsequent target tracking of these detection frames.

[0053] Furthermore, a saliency detection algorithm is used to obtain a second detection frame set of the target video frame to be detected, specifically including:

[0054] Obtain a saliency map of the target video frame to be detected using a saliency detection algorithm;

[0055] Binarizing the saliency map based on a second threshold to obtain a binary map of the saliency map;

[0056] Calculate the eight-neighborhood connected area of ​​the binary image, and calculate the rectangular bounding box for each connected area except the background area;

[0057] The rectangular bounding box is filtered according to the first condition to obtain a second detection box set.

[0058] Among them, the saliency detection algorithm can be implemented based on deep learning, for example, using the U2Net network model. The second threshold for binarizing the saliency map can be flexibly set, for example, to 0.5. The first condition includes: the height and width of the rectangular enclosing box are both greater than 0, and the area of ​​the rectangular enclosing box is greater than the third threshold, and the larger value of the height and width divided by the smaller value is less than the fourth threshold. The third threshold and the fourth threshold can be set according to actual conditions, and the present disclosure does not impose any restrictions on this. Exemplarily, the third threshold can be set to 64, and the fourth threshold can be set to 16. In this way, through the first condition, the long strip area with a large aspect ratio can be excluded from the rectangular enclosing box, and a second detection box set with better quality can be obtained.

[0059] The technical solution of the disclosed embodiment uses a target detection algorithm and a saliency detection algorithm to simultaneously detect initial targets of people and non-people. Compared with the existing solutions in the prior art that simply use a saliency detection algorithm or a target detection algorithm to detect targets, it can more accurately obtain the spatial positions and priorities of targets of different semantic categories. Therefore, when converting horizontal to vertical or vertical to horizontal, it can accurately judge the most important main target and calculate a more accurate cropping frame, thereby reducing the possibility of the main target being truncated and improving the availability of the generated video.

[0060] Figure 3 FIG. 1 is a flow chart of tracking a target in a video generation method according to an embodiment of the present disclosure. Based on the above embodiments, this embodiment further optimizes the specific operation of target tracking in S102. Figure 3 As shown, specifically including the following:

[0061] S301 . Extract frames from original video frames of an original video according to a second interval frame number to obtain a second video frame set, and use the second video frame set as a video frame set to be tracked.

[0062] S302 , in the set of video frames to be tracked, track each target appearing in the set of target detection frames, and obtain a target tracking frame of each target in each of the video frames to be tracked.

[0063] S303 , based on two adjacent target tracking frames of the same target, interpolate frames in other original video frames between the to-be-tracked video frames to which the two target tracking frames belong respectively, to obtain target tracking frames of the same target in other original video frames.

[0064] S304 , composing a target tracking frame set of each target in the original video according to the target tracking frame obtained through tracking and the target tracking frame obtained through interpolation.

[0065] Target tracking can be performed based on a set of video frames to be tracked obtained by frame extraction. Preferably, the first interval frame number can be greater than the second interval frame number, for example, the first interval frame number can be an integer multiple of the second interval frame number. For example, the number of video frames to be tracked can be twice the number of video frames to be detected.

[0066] Specifically, each video frame to be tracked in the set of video frames to be tracked is tracked frame by frame, and the target tracking frame in each video frame to be tracked is obtained. Then, since the video frame to be tracked is obtained by extracting frames at intervals of the second frame number, for two adjacent target tracking frames of the same target, that is, two target tracking frames tracked by the same target twice in different video frames to be tracked, interpolation is performed in other original video frames between the video frames to be tracked to which the two target tracking frames belong, and the target tracking frame of the same target in other original video frames is obtained. In specific implementation, uniform interpolation can be performed to simulate the shape and position changes of the tracking frame in the interval frame, and the maximum and minimum values ​​of the center position of the target tracking frame between different frames obtained by interpolation on the X-axis and the maximum and minimum values ​​on the Y-axis are distributed in an arithmetic progression. In this way, the target tracking frame obtained by tracking and the target tracking frame obtained by interpolation can be obtained, and they can be combined into a target tracking frame set of each target in the original video. In this way, by first tracking the target and then interpolating the frames to obtain all the target tracking frames, the overall processing efficiency of the target tracking process can be improved.

[0067] Figure 4 FIG. 1 is a flow chart of obtaining a target tracking frame during target tracking according to an embodiment of the present disclosure. Based on the above embodiments, this embodiment further optimizes how to obtain the target tracking frame in S302. Figure 4 As shown, specifically including the following:

[0068] S401 : taking a target detection frame corresponding to a target in a first frame of a set of video frames to be detected as a detection frame to be tracked.

[0069] In specific implementation, a target tracking set T (Track set) can be constructed according to the target detection frame in the first frame of the video frame to be detected. The information of each target detection frame can be expressed as d = {xmin, ymin xmax, ymax, cls, s}. In the Track set, each track can refer to the detection frame to be tracked, and record its related tracking information in the set. For example, it can be expressed as track t = {d t,newest_fidx,histroy}, where newest_fidx represents the frame number that made track t the latest update, that is, the frame number that track t was tracked the most recently, and histroy represents the array of tracking frames of track t in each frame tracked since the creation of the frame. t Indicates the target tracking box information of track t.

[0070] After creating the target tracking set T, the current initial detection frame to be tracked is the target detection frame corresponding to each target in the first frame of the video frame to be detected. Next, these target detection frames are tracked for each video frame to be tracked after the first frame to obtain the target tracking frame in each video frame to be tracked.

[0071] S402 , taking any video frame to be tracked in the set of video frames to be tracked as a current video frame to be tracked, and executing S403 - S406 on the current video frame to be tracked.

[0072] S403: using a target tracking algorithm to predict the position of the detection frame to be tracked in the current video frame to be tracked, and obtaining the predicted target tracking frame in the current video frame to be tracked according to the position.

[0073] The target tracking algorithm may be, for example, KCF (Kernel Correlation Filter). The KCF algorithm is used to predict the position of each detection frame t to be tracked in T in the current video frame to be tracked, including the center position and the size of the detection frame, and then the predicted target tracking frame in the current video frame to be tracked is obtained according to the position. At this time, the d attribute of each detection frame t to be tracked can be updated, the target tracking frame information can be recorded, the updated d' can be obtained, and the newest_fidx information of t can be updated to record the current tracked frame number.

[0074] S404: If target detection is performed on the current video frame to be tracked, then for each current target detection frame of the current video frame to be tracked, the matching degree between the current target detection frame and each current detection frame to be tracked is calculated.

[0075] S405. According to the matching degree, if there is a first current target detection frame that meets the matching condition, update the target tracking frame predicted by the detection frame to be tracked that matches the first current target detection frame in the current video frame to be tracked according to the first current target detection frame.

[0076] S406: For the second current target detection frame that does not meet the matching condition, use the second current target detection frame as a new detection frame to be tracked.

[0077] Since the first frame interval for extracting the set of video frames to be detected is different from the second frame interval for extracting the set of video frames to be tracked, some original video frames in the original video have both target detection and target tracking, while some original video frames have only target tracking. Therefore, after obtaining the predicted target tracking frame in the current video frame to be tracked, it is also necessary to determine whether the current video frame to be tracked has also been used as a video frame to be detected for target detection, and process them separately according to the situation.

[0078] If the current video frame to be tracked has not been detected, it means that there is no target detection frame in the current video frame to be tracked. Therefore, there is no need to match, and the predicted target tracking frame can be used as the target tracking frame of the current video frame to be tracked. If the current video frame to be tracked has been detected, it means that the current video frame to be tracked includes at least one target detection frame, so further matching is required to determine the final target tracking frame.

[0079] Specifically, if the current video frame to be tracked has been subjected to target detection, the matching degree between each current target detection frame of the current video frame to be tracked and each current detection frame to be tracked is calculated. That is, the matching degree between the target detection frame in the current video frame to be tracked and each current detection frame to be tracked is calculated one by one. According to the matching degree, if there is a first current target detection frame that meets the matching condition, the target tracking frame predicted by the detection frame to be tracked matching the first current target detection frame in the current video frame to be tracked is updated according to the first current target detection frame. That is, when the matching degree corresponding to any first current target detection frame meets the matching condition, the position of the first current target detection frame is used as the target tracking frame predicted by the detection frame to be tracked matching the first current target detection frame in the current video frame to be tracked. At the same time, the newest_fidx information of the detection frame to be tracked is updated, that is, the frame number of the current video frame to be tracked. For the second current target detection frame that does not meet the matching condition, it indicates that the second current target detection frame has not been tracked, so the second current target detection frame is taken as a new detection frame to be tracked and added to the target tracking set T.

[0080] In one embodiment, for each current target detection frame of the current video frame to be tracked, the matching degree between the current target detection frame and each current detection frame to be tracked is calculated, including: for each current target detection frame of the current video frame to be tracked, the first intersection-and-union ratio between the current target detection frame and each current detection frame to be tracked is calculated, and a corresponding bipartite graph is obtained according to the first intersection-and-union ratio; and the matching degree is obtained according to the bipartite graph using the Hungarian matching algorithm.

[0081] Among them, for each current target detection frame of the current video frame to be tracked, the first intersection and union ratio between it and each current detection frame to be tracked is calculated, including: obtaining at least one selected detection frame to be tracked from each current detection frame to be tracked, wherein the selected detection frame to be tracked satisfies the following conditions: the frame number of the selected detection frame to be tracked is greater than the difference between the frame number of the current video frame to be tracked and the set multiple (such as three times) of the first interval frame number; for each current target detection frame of the current video frame to be tracked, the first intersection and union ratio between it and each selected detection frame to be tracked is calculated. That is to say, when calculating the first intersection and union ratio, as a preferred embodiment, only the first intersection and union ratio between each current target detection frame and each selected detection frame to be tracked can be calculated, without calculating one by one with all detection frames to be tracked. By judging the above conditions, the detection frames to be tracked that have not appeared in many frames can be excluded, thereby improving the calculation efficiency.

[0082] For the matching condition appearing in S405, the process of judging whether the matching condition is met includes: obtaining the maximum matching detection frame to be tracked with each current target detection frame according to the value of the matching degree; respectively calculating the second intersection-and-union ratio of each current target detection frame and its corresponding maximum matching detection frame to be tracked; if the second intersection-and-union ratio is greater than the sixth threshold, it is judged that the matching condition is met, otherwise the matching condition is not met. In other words, the matching degree of the current target detection frame and each detection frame to be tracked can be obtained by calculation, and the detection frame with the largest value is selected as the maximum matching detection frame to be tracked, and then the second intersection-and-union ratio of the two is calculated, and the second intersection-and-union ratio is compared with the sixth threshold, and whether the matching condition is met is determined according to the comparison result. The sixth threshold can be flexibly set according to actual conditions.

[0083] It should be noted that the above is only an example of calculating the matching degree between two detection frames, and any method in the prior art can be used to implement it, and the embodiments of the present disclosure do not impose any limitation on this.

[0084] In addition, after matching and tracking the current video frame to be tracked, it is also necessary to maintain the history information in the target tracking set T. That is to say, the operation of S303 in the above embodiment is performed to insert frames for other original video frames that are not used as video frames to be tracked. It should be noted that the operation of inserting frames can be performed after the operation of S302, or after the target tracking frame of the current video frame to be tracked is obtained during the matching and tracking process for each current video frame to be tracked. When inserting frames, inserting frames can be performed for target tracking frames that are not currently newly created.

[0085] In another implementation, after taking the second current target detection frame as a new detection frame to be tracked, the operations performed on the current video frame to be tracked further include:

[0086] In all the current detection frames to be tracked, obtain the target frame number that was tracked most recently for each detection frame to be tracked, and calculate the difference between the frame number of the current video frame to be tracked and the target frame number;

[0087] If there is a target to be tracked detection frame whose corresponding sequence number difference is greater than the fifth threshold, the tracking of the target corresponding to the target to be tracked detection frame is stopped.

[0088] The sequence number difference indicates how many frames have not tracked the target detection frame when traversing to the current video frame to be tracked. If the sequence number difference is greater than the fifth threshold, it means that since the target detection frame has not been tracked for many frames, tracking of the target corresponding to the target detection frame can be stopped. The fifth threshold can be flexibly set according to actual conditions.

[0089] None of the methods in the prior art use the target tracking method to depict the relationship between the targets detected in each frame in time sequence, so for objects that move in time sequence, it is easy to cause the accuracy of the cropping frame to be reduced and the cropping frame to jump in time sequence and other unstable situations. On the one hand, it causes important targets to be easily truncated, and on the other hand, it also causes the user experience of the generated video to be poor. The technical solution of the embodiment of the present disclosure uses the target tracking algorithm to obtain the changes of the target between frames, thereby obtaining the dynamic changes of the spatial position of various targets in time sequence. Therefore, for the targets moving in the video, a more accurate dynamic cropping frame can be obtained, thereby improving the quality of video generation.

[0090] Figure 5 FIG. 1 is a flow chart of determining a cropping frame in a video generation method according to an embodiment of the present disclosure. Based on the above embodiments, this embodiment further optimizes how to determine the cropping frame in S103. Figure 5 As shown, specifically including the following:

[0091] S501. Calculate the score of each target according to the position of each target tracking frame in the original video frame to which it belongs, the weight and confidence of the type label of the corresponding target, and the number of original video frames tracked to each target.

[0092] For example, the score s can be calculated according to the following formula t :

[0093] s t =∑ d∈histroy d[s]*(d[ymax]-d[ymin])*d[xmax]-d[xmin])*score(d[cls])

[0094] Among them, score() represents the weight corresponding to each category label. For example, the weight of the face can be greater than the weight of other category labels. d represents the target tracking frame. In other words, the score of each target is to first calculate the area of ​​each target tracking frame, then multiply it by the confidence of the target tracking frame and the weight of the target category label, and finally add the results of each target tracking frame of the same target. It can be seen that the larger the area of ​​the target tracking frame, the higher the confidence, the greater the weight, and the more frames that track the target, the greater the score of the target, indicating that the target is more important in the video.

[0095] S502 : From the target tracking frames appearing in each original video frame, select a target tracking frame having the largest score corresponding to the target as a selected target tracking frame.

[0096] The selected target tracking box selected according to the maximum score is the main target in the original video frame. Cropping according to the main target can ensure that the main target of the video frame is not truncated.

[0097] Among them, for the first frame of the original video, if there is no target tracking frame with the largest score corresponding to the target, it means that there may be no target detection frame in the first frame of the original video. At this time, the first initialization frame can be used as the selected target tracking frame of the first frame of the original video. The first initialization frame is determined according to the width and height of the original video and the target video, and can be expressed as C0 = {W / 2, H / 2, targetW, targetH}, where W and H are the width and height of the original video, respectively, and targetW and targetH are the width and height of the target video, respectively.

[0098] S503: Determine a cropping frame for each original video frame according to the selected target tracking frame.

[0099] Specifically, the center position of the selected target tracking frame in each original video frame may be used as the center position of the cropping frame of each original video frame, and then the cropping frame of each original video frame is determined according to the center position and the width and height of the target video.

[0100] In one implementation, the center position of the selected target tracking frame in each original video frame is used as the center position of the cropping frame of each original video frame, including:

[0101] The target corresponding to the selected target tracking frame of the first frame of the original video frame is taken as the current optimal target;

[0102] For the first original video frame of the original video, the center position of the selected target tracking frame of the first original video frame is used as the center position of the cropping frame of the first original video frame;

[0103] Take any original video frame after the first original video frame as the current original video frame, and perform the following operations on the current original video frame:

[0104] Determine whether the target corresponding to the currently selected target tracking frame of the current original video frame is the current optimal target;

[0105] If it is the current optimal target, the momentum parameter of the smoothed cropping frame is used to calculate the center position of the cropping frame of the current original video frame based on the center position of the cropping frame of the previous frame of the current original video frame and the center position of the currently selected target tracking frame in the current original video frame;

[0106] If it is not the current optimal target, the center position of the currently selected target tracking frame in the current original video frame is used as the center position of the cropping frame of the current original video frame;

[0107] Update the current optimal target to the target corresponding to the currently selected target tracking frame.

[0108] That is, first, for the first frame of the original video, the center position of the selected target tracking box of the first frame of the original video is used as the center position of the cropping box of the first frame of the original video. At the same time, the target corresponding to the selected target tracking box is used as the current optimal target. For each subsequent frame of the original video, it is necessary to traverse frame by frame.

[0109] Specifically, for any current original video frame, first determine whether the target corresponding to the currently selected target tracking frame of the current original video frame is the current optimal target, so as to achieve the purpose of continuous tracking of the optimal target. If it is not the current optimal target, it means that the optimal target of the current original video frame has changed, then the center position of the currently selected target tracking frame in the current original video frame is directly used as the center position of the cropping frame of the current original video frame, and the current optimal target is updated to the target corresponding to the currently selected target tracking frame. If it is the current optimal target, it means that the optimal target of the current original video frame is the same as the previous frame, so the momentum parameter of the smoothed cropping frame is used, based on the center position of the cropping frame of the previous frame of the current original video frame, and the center position of the currently selected target tracking frame in the current original video frame, the center position of the cropping frame of the current original video frame is calculated. The formula can be expressed as:

[0110] C j [centerW]=C j-1 [centerW]*momentum+d mj [centerW]*(1-monentum)

[0111] Among them, momentum is the momentum parameter used to smooth the cropping box (such as 0.095), C j-1 [centerW] indicates the center position of the cropping frame of the previous frame, dmj [centerW] indicates the center position of the currently selected target tracking frame in the current original video frame. According to the above formula, the center C of the cropping frame of the current original video frame can be calculated. j [centerW]. By introducing the above momentum weighted average form to calculate the cropping frame, it is possible to ensure stable movement of the picture and avoid jitter when consecutive frames have the same optimal target, thereby improving the overall quality of the generated video.

[0112] In addition, if it is not the current optimal target, it can be further divided into two cases for processing. That is, the center position of the currently selected target tracking frame in the current original video frame is used as the center position of the cropping frame of the current original video frame, further including:

[0113] If it is not the current optimal target, then obtain the interval frame number between the current original video frame and the last cropping frame mutation;

[0114] If the interval frame number is less than the seventh threshold, the center position of the cropping frame of the current original video frame is calculated by using the momentum parameter of the smooth cropping frame based on the center position of the cropping frame of the previous frame of the current original video frame and the center position of the currently selected target tracking frame in the current original video frame;

[0115] If the interval frame number is greater than the seventh threshold, the center position of the currently selected target tracking frame in the current original video frame is used as the center position of the cropping frame of the current original video frame;

[0116] The cropping frame mutation means that if the X-axis distance between the target tracking frame of the optimal target corresponding to any frame and the center coordinates of the cropping frame of the previous frame of the arbitrary frame is greater than the eighth threshold, a cropping frame mutation occurs in the arbitrary frame.

[0117] That is to say, if the interval frame number between the frame where the cropping frame mutation occurred last time and the current frame does not reach the seventh threshold value (such as 30), it means that the optimal target has been frequently switched. At this time, the momentum weighted method is used to calculate the center position of the cropping frame of the current original video frame according to the center position of the cropping frame of the previous frame and the center position of the currently selected target tracking frame in the current original video frame, so that the target switching between different frames is smoother. Otherwise, the center position of the currently selected target tracking frame in the current original video frame is directly used as the center position of the cropping frame of the current original video frame. The purpose of this is to reduce the frequency of switching shots in the generated target video for scenes with large target motion and fast lens switching in the original video, avoid severe shaking and blurring in the target video, and improve the viewing effect of the video.

[0118] The technical solution of the disclosed embodiment not only uses the target tracking algorithm to obtain the target changes between frames, thereby obtaining the dynamic changes of the spatial positions of various targets in the time sequence, so as to obtain a more accurate dynamic cropping frame of the moving target in the video, reducing the possibility of the moving target being truncated. At the same time, by introducing the momentum weighted average form to calculate the cropping frame, it can ensure the stability of the picture, avoid jitter during movement, and improve the overall quality of the generated video.

[0119] In another implementation manner, in order to further ensure that the target is not truncated, before cropping each original video frame according to the cropping frame, the method of the embodiment of the present disclosure further includes:

[0120] For any current original video frame f j Perform the following operations, where j is a natural number:

[0121] Judgement j Whether the optimal target in is truncated;

[0122] If it is judged to be not truncated, traverse f one by one j , and judge whether any current target tracking frame satisfies the second condition, wherein the second condition is that the category label of the target corresponding to the current target tracking frame is human body, and f j The optimal target in is completely covered by the current target tracking frame;

[0123] If the second condition is met, continue the traversal;

[0124] If the second condition is not met, it is determined whether the coverage area of ​​the current target tracking frame and the optimal target tracking frame is greater than the eleventh threshold. If it is greater than the eleventh threshold, f is determined. j The target is truncated.

[0125] Specifically, before clipping, further determine whether there are still targets that are truncated. First, determine whether the optimal target is truncated. If it is not truncated, traverse f one by one j The target tracking frame in the video is used to determine whether other targets are truncated. Among them, when the optimal target is a human face, and there is a human body among other targets, and the target tracking frame of the human body completely covers the face tracking frame, it means that although the two are overlapped, they represent the same person, so no additional processing is required. For other target tracking frames that are not human bodies, the coverage area between them and the optimal target tracking frame is further determined. If it is greater than a certain threshold, it means that the target is truncated. If the total number of frames in which the target is truncated in all video frames exceeds a certain threshold, it means that the quality of the current target video does not meet the standard. The target video can be processed by not outputting or filtering it to ensure that the output generated video has a high video quality.

[0126] Among them, judging f j Whether the optimal objective in is truncated can include:

[0127] If any of the following formulas is satisfied, then f is judged j The optimal objective in is truncated:

[0128] C j [centerW]-(C j [width]) / 2)-d m [xmin]>10 or

[0129] C j [centerW]+(C j [width]) / 2)-d m [xmax]>10;

[0130] Among them, C j [width] indicates f j The width of the crop box, d m represents f j The optimal goal in d m [xmin] and d m [xmax] represents the minimum and maximum X-axis coordinates of the target tracking box corresponding to the optimal target.

[0131] It should be noted that the existing methods in the prior art do not judge the validity of the generated cropping results, and thus cannot determine the video quality after converting horizontal to vertical or vertical to horizontal. In the technical solution of the embodiments of the present disclosure, whether the video is valid is determined by judging whether all objects in each frame are truncated by the cropping frame, thereby filtering out videos in the generated results that have a poor user perception due to truncation of objects, thereby improving the quality of the output video in actual projects.

[0132] Figure 6 FIG. 1 is a flow chart of another video generation method according to an embodiment of the present disclosure. Based on the above embodiments, this embodiment further optimizes the implementation method based on scene division. Figure 6 As shown, the method specifically includes the following:

[0133] S601: Divide an input video into scenes to obtain at least one scene and its corresponding original video.

[0134] S602: For the original video of each scene, obtain a set of target detection frames of the original video of each scene through target detection.

[0135] S603. For each scene, track each target appearing in the target detection frame set to obtain a target tracking frame set of each target in the original video of each scene, wherein the target tracking frame set of each target is used to represent the dynamic change information of the spatial position of each target in the temporal sequence of the original video of each scene.

[0136] S604 , obtaining a selected target tracking frame of each original video frame in the original video of each scene according to the target tracking frame set, and determining a cropping frame of each original video frame of each scene according to the selected target tracking frame.

[0137] S605: Crop each original video frame of each scene according to the cropping frame.

[0138] S606: Acquire a frame sequence obtained by cropping the original video of each scene, and generate a target video according to the frame sequence.

[0139] Specifically, if the input video includes multiple different scenes, target detection and target tracking operations are performed for each scene separately, that is, the cropping frame is calculated independently between scenes, so that the method can be adapted to a wider range of multi-scene video types.

[0140] The scene division of the input video may be implemented by any method in the prior art. In one embodiment, the scene division of the input video to obtain at least one scene and its corresponding original video may include: extracting the input video frame by frame to obtain a frame set; calculating each frame f in the frame set i Compared with the previous frame f i-1 The pixel difference is , where i represents the frame number, and i is greater than or equal to 2; compare the pixel difference with the tenth threshold, and determine the frame f according to the comparison result. i Whether it is the first frame of a new scene. In other words, whether a scene switch occurs can be determined by comparing the pixel difference between two video frames.

[0141] Preferably, for each frame in the frame set, f i Compared with the previous frame f i-1 The pixel difference includes: calculating each frame f in the frame set i Compared with the previous frame f i-1 The first-order pixel difference; according to each frame f in the frame set i Compared with the previous frame f i-1 The first-order pixel difference of each pixel is used to calculate the second-order pixel difference; the smaller value between the first-order pixel difference and the second-order pixel difference is selected as the frame f i Correspondingly, comparing the pixel difference with the tenth threshold value includes: comparing the target pixel difference with the tenth threshold value.

[0142] Among them, the first-order pixel difference diffi Calculate according to the following formula: i =sum(S(Gray(f i ))-S(Gray(f i-1 )) / W*H*255*α;

[0143] The second-order pixel difference diff2 i Calculate according to the following formula: diff2 i =diff i -diff i-1 ;

[0144] Among them, Gray means converting the frame image in RGB format into grayscale, S means downsampling the image by a factor of two, sum means summing all pixels of the image, W and H are the width and height of the frame image respectively, and α is a constant coefficient.

[0145] Therefore, by further calculating the second-order pixel difference, and selecting the smaller value from the first-order pixel difference and the second-order pixel difference as the frame f i The target pixel difference can improve the accuracy of the difference comparison between video frames, thereby obtaining more accurate scene segmentation results.

[0146] For the original video frames of each scene after division, the method in the embodiment of the present disclosure further includes:

[0147] For any current scene other than the first scene in the input video, the cropping frame of the last frame in the previous scene of the current scene is used as the second initialization frame, and the second initialization frame is used as the cropping frame of the first original video frame in the current scene. In other words, although the cropping frames are calculated separately for different scenes, in any current scene other than the first scene, the cropping frame of the first original video frame in the current scene is initialized according to the cropping frame of the last frame in the previous scene of the current scene. In this way, when there is no optimal target in the first original video frame in the current scene, the initialized cropping frame can be used for cropping, and it plays a role in smooth connection with the last frame of the previous scene.

[0148] Furthermore, if there is a selected target tracking frame of the optimal target in the first frame of the original video frame in the current scene, the method of the embodiment of the present disclosure also includes: for the first frame of the original video frame of any current scene other than the first scene in the original video, calculating the X-axis distance between the center coordinates of the cropping frame determined based on the corresponding selected target tracking frame in the first frame of the original video frame of the current scene and the center coordinates of the cropping frame determined according to the second initialization frame; judging whether the X-axis distance is greater than a ninth threshold; if it is greater than the ninth threshold, updating the cropping frame determined according to the second initialization frame to the cropping frame determined based on the corresponding selected target tracking frame in the first frame of the original video frame of the current scene. That is to say, for the first frame of the original video frame in the current scene, it is necessary to compare the cropping frame determined based on the optimal target with the initialized cropping frame, specifically to calculate the X-axis distance between the center coordinates of the two frames. If the distance is greater than the ninth threshold, it means that the two are far apart, the optimal target in the current scene and the previous scene has changed, and the cropping frame has mutated. Therefore, the cropping frame determined by the second initialization frame is updated to the cropping frame determined in the first frame of the original video frame of the current scene based on the corresponding selected target tracking frame.

[0149] The technical solution of the disclosed embodiment adds a scene detection algorithm to calculate the dynamic cropping frame in each scene, and independently calculate the cropping frame between scenes, so it can adapt to a wider range of multi-scene video types. At the same time, when switching between different scenes, it can also ensure the smoothness of video conversion and improve the viewing effect.

[0150] Figure 7 FIG. 1 is a flow chart of another video generation method according to an embodiment of the present disclosure. Figure 7 As shown, the input video is first scene-segmented and then processed separately. For the original video of each scene, the frames are first extracted according to the first frame interval, and the target detection is performed on each extracted frame. Then the frames are extracted according to the second frame interval, and the target matching tracking is performed on each extracted frame, wherein the first frame interval is greater than the second frame interval. Therefore, some original video frames only perform target matching tracking, and some original video frames perform both target detection and target matching tracking. The target tracking frame of each original video frame is obtained through target matching tracking, and in the process of target tracking, the data of the current frame will also depend on the results of the previous frame tracking. Then, the target tracking frame of the optimal target is selected from the target tracking frame set based on the score sorting, and the cropping frame is determined according to the target tracking frame of the optimal target. Finally, the original video frame is cropped according to the cropping frame, and the cropped video frame sequence is encoded to obtain the output video.

[0151] Therefore, the present disclosure uses a target detection algorithm and a saliency detection algorithm to simultaneously detect initial targets of people and non-people. Compared with existing algorithms that simply use saliency detection or target detection, it can more accurately obtain the spatial position and priority of targets of different semantic categories. Therefore, when switching from horizontal to vertical, it can accurately judge the most important main target and calculate a more accurate cropping frame, reduce the possibility of the main target being truncated, and improve the availability of the generated video. At the same time, a target tracking algorithm is also used to obtain the change of the target between frames, so as to obtain the dynamic change of the spatial position of various types of targets in time sequence. Therefore, for targets moving in the video, a more accurate dynamic cropping frame can be obtained, reducing the possibility of the moving target being truncated. In addition, by introducing the momentum weighted average to calculate the cropping frame, the stable movement of the picture can be guaranteed to avoid jitter, and the overall quality of the generated video can be improved.

[0152] Figure 8 This is a schematic diagram of the structure of a video generation device according to an embodiment of the present disclosure. This embodiment can be applied to the case of converting a horizontal video to a vertical video, or vice versa. It involves the field of artificial intelligence technology, specifically computer vision, deep learning and other technical fields, and can be applied to AIGC and other scenarios. The device can implement the video generation method described in any embodiment of the present disclosure. Figure 8 As shown, the device 800 specifically includes:

[0153] The object detection module 801 is used to obtain an object detection frame set of the original video through object detection;

[0154] A tracking module 802 is used to track each target appearing in the target detection frame set, and obtain a target tracking frame set of each target in the original video, wherein the target tracking frame set of each target is used to represent the dynamic change information of the spatial position of each target in the time sequence of the original video;

[0155] A cropping frame determining module 803 is used to obtain a selected target tracking frame of each original video frame in the original video according to the target tracking frame set, and determine a cropping frame of each original video frame according to the selected target tracking frame;

[0156] The video generation module 804 is used to crop each of the original video frames according to the cropping frame, and generate a target video according to a frame sequence obtained by the cropping.

[0157] Optionally, the target detection module includes:

[0158] A first frame extraction submodule, configured to extract frames from original video frames of the original video according to a first frame interval, obtain a first video frame set, and use the first video frame set as a video frame set to be detected;

[0159] The target detection submodule is used to perform target detection on each of the to-be-detected video frames in the to-be-detected video frame set to obtain a target detection frame set of the original video.

[0160] Optionally, the target detection submodule includes:

[0161] A first detection frame set acquisition unit, configured to acquire a first detection frame set for each to-be-detected video frame by using a target detection algorithm;

[0162] A second detection frame set acquisition unit is used to acquire a second detection frame set of any target to be detected video frame by using a saliency detection algorithm if the first detection frame set is empty;

[0163] The target detection frame set acquisition unit is used to obtain the target detection frame set of the original video according to the first detection frame set and the second detection frame set.

[0164] Optionally, the first detection frame set includes a face detection frame and a body detection frame.

[0165] Optionally, an area of ​​each first detection box in the first detection box set is greater than a first threshold.

[0166] Optionally, the second detection frame set acquiring unit includes:

[0167] A saliency map acquisition subunit, used to acquire a saliency map of the target video frame to be detected by using a saliency detection algorithm;

[0168] A binary image acquisition subunit, configured to binarize the saliency map based on a second threshold value to obtain a binary image of the saliency map;

[0169] A rectangular bounding box calculation subunit, used to calculate the eight-neighborhood connected area of ​​the binary image, and calculate a rectangular bounding box for each connected area except the background area;

[0170] The second detection frame set acquisition subunit is used to filter according to the first condition in the rectangular bounding box to obtain the second detection frame set.

[0171] Optionally, the first condition includes:

[0172] The height and width of the rectangular enclosing frame are both greater than 0, and the area of ​​the rectangular enclosing frame is greater than a third threshold, and a product of the larger value of the height and the width divided by the smaller value is less than a fourth threshold.

[0173] Optionally, the tracking module includes:

[0174] A second frame extraction submodule is used to extract frames from original video frames of the original video according to a second interval frame number to obtain a second video frame set, and use the second video frame set as the video frame set to be tracked;

[0175] A tracking submodule, configured to track each target appearing in the target detection frame set in the set of video frames to be tracked, and obtain a target tracking frame of each target in each video frame to be tracked;

[0176] A frame insertion submodule is used to insert frames in other original video frames between the to-be-tracked video frames to which the two adjacent target tracking frames belong, according to the same target, to obtain the target tracking frames of the same target in the other original video frames;

[0177] The target tracking frame set acquisition submodule is used to form a target tracking frame set of each target in the original video according to the target tracking frame acquired through the tracking and the target tracking frame acquired through the interpolation.

[0178] Optionally, the first interval frame number is an integer multiple of the second interval frame number.

[0179] Optionally, the tracking submodule includes:

[0180] A detection frame acquisition unit to be tracked, used to take a target detection frame corresponding to a target in a first frame of the video frame set to be detected as a detection frame to be tracked;

[0181] The tracking unit is used to use any video frame to be tracked in the set of video frames to be tracked as a current video frame to be tracked, and perform the following operations on the current video frame to be tracked:

[0182] Predicting the position of the detection frame to be tracked in the current video frame to be tracked by using a target tracking algorithm, and obtaining the predicted target tracking frame in the current video frame to be tracked according to the position;

[0183] If the target detection is performed on the current video frame to be tracked, then for each current target detection frame of the current video frame to be tracked, the matching degree between the current target detection frame and each current detection frame to be tracked is calculated;

[0184] According to the matching degree, if there is a first current target detection frame that meets the matching condition, updating the target tracking frame predicted by the to-be-tracked detection frame matching the first current target detection frame in the current to-be-tracked video frame according to the first current target detection frame;

[0185] For a second current target detection frame that does not meet the matching condition, the second current target detection frame is used as a new detection frame to be tracked.

[0186] Optionally, after taking the second current target detection frame as a new detection frame to be tracked, the tracking unit is further configured to:

[0187] In all the current detection frames to be tracked, obtain the target frame number that was tracked most recently for each detection frame to be tracked, and calculate the difference between the frame number of the current video frame to be tracked and the target frame number;

[0188] If there is a target to be tracked detection frame whose corresponding sequence number difference is greater than the fifth threshold, the tracking of the target corresponding to the target to be tracked detection frame is stopped.

[0189] Optionally, the tracking unit calculates, for each current target detection frame of the current video frame to be tracked, a matching degree between the current target detection frame and each current detection frame to be tracked, specifically:

[0190] For each current target detection frame of the current video frame to be tracked, a first intersection-and-union ratio is calculated between the current target detection frame and each current detection frame to be tracked, and a corresponding bipartite graph is obtained according to the first intersection-and-union ratio;

[0191] According to the bipartite graph, the matching degree is obtained by using the Hungarian matching algorithm.

[0192] Optionally, the process of determining whether the matching condition is met includes:

[0193] According to the value of the matching degree, obtaining the maximum matching to-be-tracked detection frame with each current target detection frame;

[0194] Calculate the second intersection-over-union ratio of each current target detection frame and its corresponding maximum matching detection frame to be tracked respectively;

[0195] If the second intersection-over-union ratio is greater than a sixth threshold, it is determined that the matching condition is met.

[0196] Optionally, the tracking unit calculates, for each current target detection frame of the current video frame to be tracked, a first intersection-over-union ratio between the current target detection frame and each current detection frame to be tracked, specifically:

[0197] Obtain at least one selected detection frame to be tracked from each current detection frame to be tracked, wherein the selected detection frame to be tracked satisfies the following condition: the frame number of the selected detection frame to be tracked that was tracked last time is greater than the difference between the frame number of the current video frame to be tracked and the set multiple of the first interval frame number;

[0198] For each current target detection frame of the current video frame to be tracked, a first intersection-over-union ratio between the current target detection frame and each selected detection frame to be tracked is calculated.

[0199] Optionally, the cropping frame determination module includes:

[0200] A score calculation submodule, used to calculate the score of each target according to the position of each target tracking frame in the original video frame to which it belongs, the weight and confidence of the type label of the corresponding target, and the number of original video frames tracked to each target;

[0201] A selected target tracking frame acquisition submodule is used to select the target tracking frame with the largest score corresponding to the target from the target tracking frames appearing in each original video frame as the selected target tracking frame;

[0202] The cropping frame determination submodule is used to determine the cropping frame of each original video frame according to the selected target tracking frame.

[0203] Optionally, the selected target tracking frame acquisition submodule includes:

[0204] A first initialization unit is configured to, for a first original video frame of the original video, use the first initialization frame as the selected target tracking frame of the first original video frame if there is no target tracking frame with the largest score corresponding to the target;

[0205] The first initialization frame is determined according to the width and height of the original video and the target video respectively.

[0206] Optionally, the cropping frame determination submodule includes:

[0207] a center position determining unit, configured to set the center position of the selected target tracking frame in each of the original video frames as the center position of the cropping frame in each of the original video frames;

[0208] The cropping frame determining unit is used to determine the cropping frame of each original video frame according to the center position and the width and height of the target video.

[0209] Optionally, the center position determining unit includes:

[0210] A current optimal target determination subunit, used to take the target corresponding to the selected target tracking frame of the first original video frame as the current optimal target;

[0211] a center position determination subunit, configured to use, for a first original video frame of the original video, a center position of a selected target tracking frame of the first original video frame as a center position of a cropping frame of the first original video frame;

[0212] The current original video frame processing subunit is used to take any original video frame after the first original video frame as the current original video frame, and perform the following operations on the current original video frame:

[0213] Determine whether the target corresponding to the currently selected target tracking frame of the current original video frame is the current optimal target;

[0214] If it is the current optimal target, the center position of the cropping frame of the current original video frame is calculated using the momentum parameter of the smooth cropping frame based on the center position of the cropping frame of the previous frame of the current original video frame and the center position of the currently selected target tracking frame in the current original video frame;

[0215] If it is not the current optimal target, taking the center position of the currently selected target tracking frame of the current original video frame as the center position of the cropping frame of the current original video frame;

[0216] The current optimal target is updated to be the target corresponding to the currently selected target tracking frame.

[0217] Optionally, if it is not the current optimal target, the current original video frame processing subunit uses the center position of the currently selected target tracking frame in the current original video frame as the center position of the cropping frame of the current original video frame, specifically:

[0218] If it is not the current optimal target, then obtaining the interval frame number between the current original video frame and the last cropping frame mutation;

[0219] If the interval frame number is less than a seventh threshold, the center position of the cropping frame of the current original video frame is calculated by using the momentum parameter of the smooth cropping frame based on the center position of the cropping frame of the previous frame of the current original video frame and the center position of the currently selected target tracking frame in the current original video frame;

[0220] If the interval frame number is greater than a seventh threshold, taking the center position of the currently selected target tracking frame in the current original video frame as the center position of the cropping frame of the current original video frame;

[0221] The cropping frame mutation means that if the X-axis distance between the target tracking frame of the optimal target corresponding to any frame and the center coordinates of the cropping frame of the previous frame of the arbitrary frame is greater than the eighth threshold, a cropping frame mutation occurs in the arbitrary frame.

[0222] Optionally, the device further includes a scene division module, configured to:

[0223] Divide the input video into scenes to obtain at least one scene and its corresponding original video;

[0224] For each original video of the scene, the object detection module performs the operation of obtaining the object detection frame set of the original video through object detection;

[0225] Accordingly, the video generation module is specifically used for:

[0226] Cropping each of the original video frames according to the cropping frame;

[0227] A frame sequence is obtained by cropping the original video of each scene, and the target video is generated according to the frame sequence.

[0228] Optionally, the cropping frame determination submodule further includes:

[0229] The second initialization unit is used to use the cropping frame of the last frame in the previous scene of the current scene as the second initialization frame for any current scene other than the first scene in the input video, and use the second initialization frame as the cropping frame of the first frame of the original video frame in the current scene.

[0230] Optionally, the cropping frame determination submodule further includes a non-first scene first frame processing unit, which is specifically configured to:

[0231] For a first original video frame of any current scene other than the first scene in the original video, calculating an X-axis distance between a center coordinate of a cropping frame determined based on a corresponding selected target tracking frame in the first original video frame of the current scene and a center coordinate of a cropping frame determined based on the second initialization frame;

[0232] Determining whether the X-axis distance is greater than a ninth threshold;

[0233] If it is greater than the ninth threshold, the cropping frame determined according to the second initialization frame is updated to the cropping frame determined based on the corresponding selected target tracking frame in the first original video frame of the current scene.

[0234] Optionally, the scene division module includes:

[0235] A frame set acquisition submodule, used to extract the input video frame by frame to obtain a frame set;

[0236] The pixel difference calculation submodule is used to calculate the pixel difference value of each frame in the frame set. i Compared with the previous frame f i-1 , where i represents the frame number and i is greater than or equal to 2;

[0237] A comparison submodule is used to compare the pixel difference with a tenth threshold value, and determine the frame f according to the comparison result. i Whether this is the first frame of a new scene.

[0238] Optionally, the pixel difference calculation submodule includes:

[0239] A first-order pixel difference calculation unit is used to calculate each frame f in the frame set. i Compared with the previous frame f i-1 The first-order pixel difference of

[0240] A second-order pixel difference calculation unit is used to calculate the pixel difference value of each frame f in the frame set according to the pixel difference value of each frame f in the frame set. i Compared with the previous frame f i-1 The respective first-order pixel differences calculate the second-order pixel differences;

[0241] The target pixel difference determination unit is used to select the smaller value between the first-order pixel difference and the second-order pixel difference as the frame f i The target pixel difference;

[0242] Accordingly, the comparison submodule is specifically used for:

[0243] The target pixel difference value is compared with a tenth threshold value.

[0244] Optionally, the first-order pixel difference diff i Calculate according to the following formula: i =sum(S(Gray(f i ))-S(Gray(f i-1 )) / W*H*255*α;

[0245] The second-order pixel difference diff2 i Calculate according to the following formula: diff2 i =diff i -diff i-1 ;

[0246] Among them, Gray means converting the frame image in RGB format into grayscale, S means downsampling the image by a factor of two, sum means summing all pixels of the image, W and H are the width and height of the frame image respectively, and α is a constant coefficient.

[0247] Optionally, before the video generation module crops each original video frame according to the cropping frame, the device further includes a truncation determination module, which is specifically configured to:

[0248] For any current original video frame f j Perform the following operations, where j is a natural number:

[0249] Judge f j Whether the optimal target in is truncated;

[0250] If it is judged to be not truncated, traverse f one by one j, and judge whether any current target tracking frame satisfies the second condition, wherein the second condition is that the category label of the target corresponding to the current target tracking frame is human body, and f j The optimal target in is completely covered by the current target tracking frame;

[0251] If the second condition is met, continue the traversal;

[0252] If the second condition is not met, it is determined whether the coverage area of ​​the current target tracking frame and the tracking frame of the optimal target is greater than the eleventh threshold. If it is greater than the eleventh threshold, it is determined that f j The target is truncated.

[0253] Optionally, the judgment f j Whether the optimal objective in is truncated, including:

[0254] If any of the following formulas is satisfied, then f is judged j The optimal objective in is truncated:

[0255] C j [centerW]-(C j [width]) / 2)-d m [xmin]>10 or

[0256] C j [centerW]+(C j [width]) / 2)-d m [xmax]>10;

[0257] Among them, C j represents the cropping box of fj, C j [centerW] represents f j The X-axis coordinate of the center point of the cropping frame, C j [width] indicates f j The width of the crop box, d m represents f j The optimal goal in d m [xmin] and d m [xmax] represents the minimum and maximum X-axis coordinates of the target tracking box corresponding to the optimal target.

[0258] The above-mentioned product can execute the method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0259] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0260] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0261] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0262] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0263] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0264] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a video generation method. For example, in some embodiments, the video generation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the video generation method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the video generation method in any other appropriate manner (e.g., by means of firmware).

[0265] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0266] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0267] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0268] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0269] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0270] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services. The server may also be a server of a distributed system, or a server combined with a blockchain.

[0271] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning technology, big data processing technology, knowledge graph technology, and other major directions.

[0272] Cloud computing refers to a technology system that uses network access to elastically scalable shared physical or virtual resource pools. Resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for technical applications such as artificial intelligence and blockchain, as well as model training.

[0273] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution provided by this disclosure can be achieved, and this document does not limit this.

[0274] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A video generation method, comprising: Obtaining a target detection frame set of the original video through target detection; wherein the target detection frame set is detected and determined from a set of video frames to be detected obtained by extracting frames from the original video; In the set of video frames to be tracked, a target detection frame corresponding to a target in a first frame of the set of video frames to be detected is used as a detection frame to be tracked; wherein the set of video frames to be tracked is obtained by extracting frames from the original video; Any video frame to be tracked in the set of video frames to be tracked is used as the current video frame to be tracked, and the following operations are performed on the current video frame to be tracked: Predicting the position of the detection frame to be tracked in the current video frame to be tracked by using a target tracking algorithm, and obtaining the predicted target tracking frame in the current video frame to be tracked according to the position; If the target detection is performed on the current video frame to be tracked, then for each current target detection frame of the current video frame to be tracked, the matching degree between the current target detection frame and each current detection frame to be tracked is calculated; According to the matching degree, if there is a first current target detection frame that meets the matching condition, updating the target tracking frame predicted by the to-be-tracked detection frame matching the first current target detection frame in the current to-be-tracked video frame according to the first current target detection frame; For a second current target detection frame that does not meet the matching condition, taking the second current target detection frame as a new detection frame to be tracked; According to the target tracking frame set in the acquired video frame set to be tracked, a target tracking frame set of each target in the original video is obtained, wherein the target tracking frame set of each target is used to represent the dynamic change information of the spatial position of each target in the time sequence of the original video; Acquire a selected target tracking frame of each original video frame in the original video according to the target tracking frame set, and determine a cropping frame of each original video frame according to the selected target tracking frame; Each original video frame is cropped according to the cropping frame, and a target video is generated according to a frame sequence obtained by the cropping.

2. The method according to claim 1, wherein: The step of obtaining a target detection frame set of the original video through target detection includes: Extracting frames from original video frames of the original video according to a first interval frame number to obtain a first video frame set, and using the first video frame set as a video frame set to be detected; Perform object detection on each to-be-detected video frame in the to-be-detected video frame set to obtain an object detection frame set of the original video.

3. The method according to claim 2, wherein: The performing target detection on each to-be-detected video frame in the to-be-detected video frame set to obtain the target detection frame set of the original video includes: Obtaining a first detection frame set for each of the to-be-detected video frames using an object detection algorithm; If there is any target to be detected video frame whose first detection frame set is empty, obtaining a second detection frame set of the target to be detected video frame by using a saliency detection algorithm; The target detection frame set of the original video is obtained according to the first detection frame set and the second detection frame set.

4. The method according to claim 3, wherein: The first detection frame set includes a face detection frame and a body detection frame.

5. The method according to claim 3, wherein: An area of ​​each first detection frame in the first detection frame set is greater than a first threshold.

6. The method according to claim 3, wherein: The step of obtaining a second detection frame set of the target video frame to be detected by using a saliency detection algorithm includes: Obtaining a saliency map of the target video frame to be detected using a saliency detection algorithm; Binarizing the saliency map based on a second threshold value to obtain a binary map of the saliency map; Calculate the eight-neighborhood connected areas of the binary image, and calculate a rectangular bounding box for each connected area except the background area; The second detection box set is obtained by filtering the rectangular bounding box according to the first condition.

7. The method according to claim 6, wherein: The first condition includes: The height and width of the rectangular enclosing frame are both greater than 0, and the area of ​​the rectangular enclosing frame is greater than a third threshold, and a product of the larger value of the height and the width divided by the smaller value is less than a fourth threshold.

8. The method according to claim 2, wherein: After obtaining the target detection frame set of the original video through target detection, the method further includes: extracting frames from the original video frames of the original video according to a second interval frame number to obtain a second video frame set, and using the second video frame set as the video frame set to be tracked; Correspondingly, according to the acquired target tracking frame set in the to-be-tracked video frame set, acquiring the target tracking frame set of each target in the original video includes: According to two adjacent target tracking frames of the same target, interpolating frames in other original video frames between the to-be-tracked video frames to which the two target tracking frames belong respectively, to obtain the target tracking frames of the same target in the other original video frames; A target tracking frame set of each target in the original video is formed according to the target tracking frame obtained by the tracking and the target tracking frame obtained by the interpolation.

9. The method according to claim 8, wherein: The first interval frame number is an integer multiple of the second interval frame number.

10. The method according to claim 1, wherein: After taking the second current target detection frame as a new detection frame to be tracked, the operations performed on the current video frame to be tracked further include: In all the current detection frames to be tracked, obtain the target frame number that was tracked most recently for each detection frame to be tracked, and calculate the difference between the frame number of the current video frame to be tracked and the target frame number; If there is a target to be tracked detection frame whose corresponding sequence number difference is greater than the fifth threshold, the tracking of the target corresponding to the target to be tracked detection frame is stopped.

11. The method according to claim 1, wherein: The step of calculating the matching degree between each current target detection frame of the current video frame to be tracked and each current detection frame to be tracked includes: For each current target detection frame of the current video frame to be tracked, a first intersection-and-union ratio is calculated between the current target detection frame and each current detection frame to be tracked, and a corresponding bipartite graph is obtained according to the first intersection-and-union ratio; According to the bipartite graph, the matching degree is obtained by using the Hungarian matching algorithm.

12. The method according to claim 1, wherein: The process of determining whether the matching condition is met includes: According to the value of the matching degree, obtaining the maximum matching to-be-tracked detection frame with each current target detection frame; Calculate the second intersection-over-union ratio of each current target detection frame and its corresponding maximum matching detection frame to be tracked respectively; If the second intersection-over-union ratio is greater than a sixth threshold, it is determined that the matching condition is met.

13. The method according to claim 11, wherein: The step of calculating a first intersection-over-union ratio between each current target detection frame of the current video frame to be tracked and each current detection frame to be tracked includes: Obtain at least one selected detection frame to be tracked from each current detection frame to be tracked, wherein the selected detection frame to be tracked satisfies the following conditions: the frame number of the selected detection frame to be tracked that was tracked last time is greater than the difference between the frame number of the current video frame to be tracked and a set multiple of the first interval frame number; For each current target detection frame of the current video frame to be tracked, a first intersection-over-union ratio between the current target detection frame and each selected detection frame to be tracked is calculated.

14. The method according to claim 1, wherein: The step of obtaining a selected target tracking frame of each original video frame in the original video according to the target tracking frame set, and determining a cropping frame of each original video frame according to the selected target tracking frame includes: Calculate the score of each target according to the position of each target tracking frame in the original video frame to which it belongs, the weight and confidence of the type label of the corresponding target, and the number of original video frames tracked to each target; From the target tracking frames appearing in each original video frame, select the target tracking frame with the largest score corresponding to the target as the selected target tracking frame; A cropping frame of each original video frame is determined according to the selected target tracking frame.

15. The method according to claim 14, wherein: The step of selecting the target tracking frame with the largest score corresponding to the target from the target tracking frames appearing in each original video frame as the selected target tracking frame comprises: For the first original video frame of the original video, if there is no target tracking frame with the maximum score corresponding to the target, taking the first initialization frame as the selected target tracking frame of the first original video frame; The first initialization frame is determined according to the width and height of the original video and the target video respectively.

16. The method according to claim 14 or 15, wherein: The step of determining the cropping frame of each original video frame according to the selected target tracking frame includes: The center position of the selected target tracking frame in each of the original video frames is used as the center position of the cropping frame in each of the original video frames; The cropping frame of each original video frame is determined according to the center position and the width and height of the target video.

17. The method according to claim 16, wherein: The step of setting the center position of the selected target tracking frame of each original video frame as the center position of the cropping frame of each original video frame includes: The target corresponding to the selected target tracking frame of the first frame of the original video frame is taken as the current optimal target; For a first original video frame of the original video, taking the center position of the selected target tracking frame of the first original video frame as the center position of the cropping frame of the first original video frame; Any original video frame after the first original video frame is used as the current original video frame, and the following operations are performed on the current original video frame: Determine whether the target corresponding to the currently selected target tracking frame of the current original video frame is the current optimal target; If it is the current optimal target, the center position of the cropping frame of the current original video frame is calculated using the momentum parameter of the smooth cropping frame based on the center position of the cropping frame of the previous frame of the current original video frame and the center position of the currently selected target tracking frame in the current original video frame; If it is not the current optimal target, taking the center position of the currently selected target tracking frame of the current original video frame as the center position of the cropping frame of the current original video frame; The current optimal target is updated to be the target corresponding to the currently selected target tracking frame.

18. The method according to claim 17, wherein: If the target is not the current optimal target, taking the center position of the currently selected target tracking frame of the current original video frame as the center position of the cropping frame of the current original video frame, including: If it is not the current optimal target, then obtaining the interval frame number between the current original video frame and the last cropping frame mutation; If the interval frame number is less than a seventh threshold, the center position of the cropping frame of the current original video frame is calculated by using the momentum parameter of the smooth cropping frame based on the center position of the cropping frame of the previous frame of the current original video frame and the center position of the currently selected target tracking frame in the current original video frame; If the interval frame number is greater than a seventh threshold, taking the center position of the currently selected target tracking frame in the current original video frame as the center position of the cropping frame of the current original video frame; The cropping frame mutation means that if the X-axis distance between the target tracking frame of the optimal target corresponding to any frame and the center coordinates of the cropping frame of the previous frame of the arbitrary frame is greater than the eighth threshold, a cropping frame mutation occurs in the arbitrary frame.

19. The method according to claim 18, further comprising: Divide the input video into scenes to obtain at least one scene and its corresponding original video; For each original video of the scene, respectively performing the operation of obtaining a set of target detection frames of the original video through target detection; Accordingly, the step of cropping each original video frame according to the cropping frame and generating a target video according to a frame sequence obtained by the cropping includes: Cropping each of the original video frames according to the cropping frame; A frame sequence is obtained by cropping the original video of each scene, and the target video is generated according to the frame sequence.

20. The method according to claim 19, wherein: The step of determining the cropping frame of each original video frame according to the selected target tracking frame further includes: For any current scene other than the first scene in the input video, the cropping frame of the last frame in the previous scene of the current scene is used as the second initialization frame, and the second initialization frame is used as the cropping frame of the first frame of the original video frame in the current scene.

21. The method according to claim 20, wherein: The step of determining the cropping frame of each original video frame according to the selected target tracking frame further includes: For a first original video frame of any current scene other than the first scene in the original video, calculating an X-axis distance between a center coordinate of a cropping frame determined based on a corresponding selected target tracking frame in the first original video frame of the current scene and a center coordinate of a cropping frame determined based on the second initialization frame; Determining whether the X-axis distance is greater than a ninth threshold; If it is greater than the ninth threshold, the cropping frame determined according to the second initialization frame is updated to the cropping frame determined based on the corresponding selected target tracking frame in the first original video frame of the current scene.

22. The method according to claim 19, wherein: The step of dividing the input video into scenes to obtain at least one scene and its corresponding original video includes: Extracting the input video frame by frame to obtain a frame set; Calculate each frame f in the frame set i Compared with the previous frame f i-1 , where i represents the frame number and i is greater than or equal to 2; Compare the pixel difference with the tenth threshold value, and determine the frame f according to the comparison result. i Whether this is the first frame of a new scene.

23. The method according to claim 22, wherein: The calculation of each frame f in the frame set i Compared with the previous frame f i-1 The pixel difference includes: Calculate each frame f in the frame set i Compared with the previous frame f i-1 The first-order pixel difference of According to each frame f in the frame set i Compared with the previous frame f i-1 The respective first-order pixel differences calculate the second-order pixel differences; The smaller value between the first-order pixel difference and the second-order pixel difference is selected as the frame f i The target pixel difference; Accordingly, comparing the pixel difference with a tenth threshold value includes: The target pixel difference value is compared with a tenth threshold value.

24. The method according to claim 18, before cropping each of the original video frames according to the cropping frame, the method further comprises: For any current original video frame f j Perform the following operations, where j is a natural number: Judgement j Whether the optimal target in is truncated; If it is judged to be not truncated, traverse f one by one j , and judge whether any current target tracking frame satisfies the second condition, wherein the second condition is that the category label of the target corresponding to the current target tracking frame is human body, and f j The optimal target in is completely covered by the current target tracking frame; If the second condition is met, continue the traversal; If the second condition is not met, it is determined whether the coverage area of ​​the current target tracking frame and the tracking frame of the optimal target is greater than the eleventh threshold. If it is greater than the eleventh threshold, it is determined that f j The target is truncated.

25. A video generating device, comprising: An object detection module is used to obtain an object detection frame set of the original video through object detection; wherein the object detection frame set is detected and determined from a set of video frames to be detected obtained by extracting frames from the original video; A tracking module, used to obtain a target tracking frame set of each target in the original video according to the target tracking frame set in the acquired video frame set to be tracked, wherein the target tracking frame set of each target is used to represent the dynamic change information of the spatial position of each target in the time sequence of the original video; The tracking module includes: a detection frame acquisition unit to be tracked, which is used to use the target detection frame corresponding to the target in the first frame of the video frame set to be tracked as the detection frame to be tracked; a tracking unit, which is used to use any video frame to be tracked in the video frame set to be tracked as the current video frame to be tracked, and perform the following operations on the current video frame to be tracked: using the target tracking algorithm to predict the position of the detection frame to be tracked in the current video frame to be tracked, and according to the position, obtaining the predicted target tracking frame in the current video frame to be tracked; if the current video frame to be tracked is For the target detection, for each current target detection frame of the current video frame to be tracked, the matching degree between the current detection frame to be tracked and each detection frame to be tracked is calculated; according to the matching degree, if there is a first current target detection frame that meets the matching condition, the target tracking frame predicted in the current video frame to be tracked that matches the first current target detection frame is updated according to the first current target detection frame; for the second current target detection frame that does not meet the matching condition, the second current target detection frame is used as a new detection frame to be tracked; wherein the set of video frames to be tracked is obtained by extracting frames from the original video; a cropping frame determining module, configured to obtain a selected target tracking frame of each original video frame in the original video according to the target tracking frame set, and determine a cropping frame of each original video frame according to the selected target tracking frame; The video generation module is used to crop each original video frame according to the cropping frame, and generate a target video according to the frame sequence obtained by the cropping.

26. The device according to claim 25, wherein The target detection module comprises: A first frame extraction submodule, configured to extract frames from original video frames of the original video according to a first frame interval, obtain a first video frame set, and use the first video frame set as a video frame set to be detected; The target detection submodule is used to perform target detection on each of the to-be-detected video frames in the to-be-detected video frame set to obtain a target detection frame set of the original video.

27. The device according to claim 26, wherein: The target detection submodule includes: A first detection frame set acquisition unit, configured to acquire a first detection frame set for each to-be-detected video frame by using a target detection algorithm; A second detection frame set acquisition unit is used to acquire a second detection frame set of any target to be detected video frame by using a saliency detection algorithm if the first detection frame set is empty; The target detection frame set acquisition unit is used to obtain the target detection frame set of the original video according to the first detection frame set and the second detection frame set.

28. The device according to claim 26, wherein The tracking module also includes: A second frame extraction submodule is used to extract frames from original video frames of the original video according to a second interval frame number to obtain a second video frame set, and use the second video frame set as the video frame set to be tracked; A frame insertion submodule is used to insert frames in other original video frames between the to-be-tracked video frames to which the two adjacent target tracking frames belong, according to the same target, to obtain the target tracking frames of the same target in the other original video frames; The target tracking frame set acquisition submodule is used to form a target tracking frame set of each target in the original video according to the target tracking frame acquired through the tracking and the target tracking frame acquired through the interpolation.

29. The device according to claim 25, wherein: The cropping frame determination module includes: A score calculation submodule, used to calculate the score of each target according to the position of each target tracking frame in the original video frame to which it belongs, the weight and confidence of the type label of the corresponding target, and the number of original video frames tracked to each target; A selected target tracking frame acquisition submodule is used to select the target tracking frame with the largest score corresponding to the target from the target tracking frames appearing in each original video frame as the selected target tracking frame; The cropping frame determination submodule is used to determine the cropping frame of each original video frame according to the selected target tracking frame.

30. The device according to claim 29, wherein: The cropping frame determination submodule includes: a center position determining unit, configured to set the center position of the selected target tracking frame in each of the original video frames as the center position of the cropping frame in each of the original video frames; The cropping frame determining unit is used to determine the cropping frame of each original video frame according to the center position and the width and height of the target video.

31. The device according to claim 30, wherein The center position determining unit comprises: A current optimal target determination subunit, used to take the target corresponding to the selected target tracking frame of the first frame of the original video frame as the current optimal target; a center position determination subunit, configured to use, for a first original video frame of the original video, a center position of a selected target tracking frame of the first original video frame as a center position of a cropping frame of the first original video frame; The current original video frame processing subunit is used to take any original video frame after the first original video frame as the current original video frame, and perform the following operations on the current original video frame: Determine whether the target corresponding to the currently selected target tracking frame of the current original video frame is the current optimal target; If it is the current optimal target, the center position of the cropping frame of the current original video frame is calculated using the momentum parameter of the smooth cropping frame based on the center position of the cropping frame of the previous frame of the current original video frame and the center position of the currently selected target tracking frame in the current original video frame; If it is not the current optimal target, taking the center position of the currently selected target tracking frame of the current original video frame as the center position of the cropping frame of the current original video frame; The current optimal target is updated to be the target corresponding to the currently selected target tracking frame.

32. The device according to claim 31, wherein If it is not the current optimal target, the current original video frame processing subunit uses the center position of the currently selected target tracking frame in the current original video frame as the center position of the cropping frame of the current original video frame, specifically: If it is not the current optimal target, then obtaining the interval frame number between the current original video frame and the last cropping frame mutation; If the interval frame number is less than a seventh threshold, the center position of the cropping frame of the current original video frame is calculated by using the momentum parameter of the smooth cropping frame based on the center position of the cropping frame of the previous frame of the current original video frame and the center position of the currently selected target tracking frame in the current original video frame; If the interval frame number is greater than a seventh threshold, taking the center position of the currently selected target tracking frame in the current original video frame as the center position of the cropping frame of the current original video frame; The cropping frame mutation means that if the X-axis distance between the target tracking frame of the optimal target corresponding to any frame and the center coordinates of the cropping frame of the previous frame of the arbitrary frame is greater than the eighth threshold, a cropping frame mutation occurs in the arbitrary frame.

33. The apparatus of claim 32, further comprising: The truncation judgment module is used to perform the following operations before the video generation module crops each original video frame according to the cropping frame: For any current original video frame f j Perform the following operations, where j is a natural number: Judgement j Whether the optimal target in is truncated; If it is judged to be not truncated, traverse f one by one j , and judge whether any current target tracking frame satisfies the second condition, wherein the second condition is that the category label of the target corresponding to the current target tracking frame is human body, and f j The optimal target in is completely covered by the current target tracking frame; If the second condition is met, continue the traversal; If the second condition is not met, it is determined whether the coverage area of ​​the current target tracking frame and the tracking frame of the optimal target is greater than the eleventh threshold. If it is greater than the eleventh threshold, it is determined that f j The target is truncated.

34. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the video generation method according to any one of claims 1 to 24.

35. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the video generation method according to any one of claims 1-24.

36. A computer program product comprising a computer program / instructions which, when executed by a processor, implement the video generation method according to any one of claims 1-24.

Images