Video generation method, device, computer equipment and storage medium

By tracking and detecting human targets on panoramic videos and adjusting the field of view angle to generate flat videos, the complex problem of panoramic video processing in the existing technology is solved, and the slow-motion video with simple shooting and generating dynamic special effects is realized, improving the user experience.

CN118509542BActive Publication Date: 2024-11-29LABPANO TECHNOLOGY (CHANGZHOU) CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410962785.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-29
Estimated Expiration
2044-07-18

AI Technical Summary

Technical Problem

The method of generating special effects videos including moving objects in the prior art is complicated and cumbersome, and users need to have high professional video processing capabilities, especially the processing of panoramic videos is more complicated.

Method used

By obtaining panoramic video, tracking and detecting human objects, drawing rectangular renderings, adjusting the field of view angle to highlight the body area of ​​interest, and generating flat and slow-motion videos.

Benefits of technology

It realizes slow-motion flat videos that can generate dynamic special effects with simple shooting, which reduces the user's video processing complexity, adds the gameplay of panoramic videos, and improves user playability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118509542B_ABST
    Figure CN118509542B_ABST
Patent Text Reader

Abstract

The present application provides a video generation method, device, computer equipment and storage medium, the method comprising: obtaining a panoramic video, tracking a human target for each video frame, obtaining a target bounding box of the human target, drawing a rectangular rendering screen according to the target bounding box so that the target bounding box is located in the middle area of ​​the rectangular rendering screen, performing human body detection on the target bounding box to obtain a body region of interest, adjusting the field of view of the rendering screen so that the body region of interest occupies a set proportion of the rendering screen, generating a plane video about the human target according to the adjusted rendering screen, adjusting the frame rate of the plane video, and generating a slow-motion video of a target frame rate. The present application can automatically generate a special-effect plane video of a dynamic moment without the user having to perform complex video editing operations.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video processing technology, and specifically to a video generation method, apparatus, computer equipment, and storage medium. Background Art

[0002] With the development of video processing technology, people are becoming more and more interested in various special video processing effects, especially image processing involving moving objects. For example, the processing method for close-up shots of moving objects during motion. At present, because the processing and generation of videos involving moving objects are more complicated than those of ordinary videos, the user's shooting skills are required to be high, and the user's video editing technology for post-processing videos to generate special effects videos is required to be high, especially for panoramic videos. Since panoramic videos have a 360° panoramic field of view and more video content, the processing is more complicated and cumbersome. Therefore, there is an urgent need for a method for generating dynamic special effects videos that is simple in shooting and does not require users to perform complex video editing operations in the later stage. Summary of the invention

[0003] The embodiments of the present application provide a video generation method, apparatus, computer device and storage medium. The technical problem to be solved is that, in view of the defects of the prior art, the existing special effects video generation method containing moving objects is complicated and cumbersome, and requires the user to have high professional video processing capabilities.

[0004] In order to solve the above technical problems, a technical solution adopted by the present application is: to provide a video generation method, apparatus, computer equipment and storage medium, wherein the video generation method comprises:

[0005] Get panoramic video;

[0006] Tracking a human target in each video frame of the panoramic video to obtain a target bounding box of the human target, and drawing a rectangular rendering picture according to the target bounding box so that the target bounding box is located in a middle area of ​​the rectangular rendering picture;

[0007] Performing human body detection on the target bounding box to obtain a body region of interest;

[0008] Adjusting the field of view of the rendering image so that the body region of interest occupies a set proportion of the rendering image;

[0009] Generating a planar video about the human target according to the adjusted rendering picture;

[0010] The frame rate of the planar video is adjusted to generate a slow-motion video with a target frame rate.

[0011] In a specific implementation, performing human body detection on the target bounding box to obtain a body region of interest includes:

[0012] Performing human key point detection on the target bounding box to obtain N human skeleton key points of the human target, where N is a natural number and 10≤N≤20;

[0013] Calculating the displacement of the N human skeleton key points of the human target in the plurality of video frames;

[0014] Determine a target human skeleton key point set according to the displacements of the N human skeleton key points in the plurality of video frames;

[0015] According to the mapping relationship between the target human skeleton key point set and the human body, the human body part corresponding to the target human skeleton key point set is determined to be the body region of interest.

[0016] In a specific implementation, the detecting of human key points on the target bounding box to obtain N human skeleton key points of the human target includes:

[0017] The target bounding box is output to a human skeleton key point detection model, and the N human skeleton key points in the target bounding box of each video frame are detected by the human skeleton key point detection model to obtain a coordinate sequence of the N human skeleton key points.

[0018] In a specific implementation, the calculating the displacement of the N human skeleton key points of the human target in the plurality of video frames includes:

[0019] Set the displacement of the human skeleton key point in the i-1th frame to the next frame i as , wherein the is the coordinate sequence of the human skeleton key point in the i-1th frame, norm is the norm calculation, k is a natural number, defined as the kth human skeleton key point, 0≤k≤N;

[0020] The displacement of the N human skeleton key points of the human target in the M consecutive video frames starting from the iMth frame is the displacement of the N human skeleton key points of the human target in multiple video frames, and the displacement is , 0≤k≤N.

[0021] In a specific implementation, determining a target human skeleton key point set according to the displacements of the N human skeleton key points in the plurality of video frames includes:

[0022] According to the displacement of each of the human skeleton key points in the multiple video frames, the L human skeleton key points with the largest displacement are selected as the target human skeleton key points, and the target human skeleton key point set is determined based on the target human skeleton key points.

[0023] In a specific implementation, determining the human body part corresponding to the target human body skeleton key point set as the body region of interest according to the mapping relationship between the target human body skeleton key point set and the human body, includes:

[0024] Based on the human skeleton key point detection model, the human body parts corresponding to the target human skeleton key point set are mapped, and the corresponding human body parts are determined to be the body region of interest.

[0025] In a specific implementation, N is 18, L is 2, and the body region of interest occupies a set proportion of 5%-10% of the rendering image.

[0026] In a specific implementation, a video generating device is also included, the device comprising:

[0027] An acquisition module, wherein the acquisition module is used to acquire a panoramic video;

[0028] A tracking module, wherein the tracking module is used to track a human target in each video frame of the panoramic video to obtain a target bounding box of the human target, and draw a rectangular rendering picture according to the target bounding box so that the target bounding box is located in a middle area of ​​the rectangular rendering picture;

[0029] A human body detection module, the human body detection module is used to perform human body detection on the target bounding box to obtain a body region of interest;

[0030] A viewing angle adjustment module, the viewing angle adjustment module is used to adjust the viewing angle of the rendering picture so that the body region of interest occupies a set proportion of the rendering picture;

[0031] A plane video generation module, the plane video generation module is used to generate a plane video about the human target according to the adjusted rendering picture;

[0032] A frame rate adjustment module is used to adjust the frame rate of the planar video to generate a slow motion video with a target frame rate.

[0033] In a specific embodiment, a computer device is also included, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the video generation method described in any of the above specific embodiments are implemented.

[0034] In a specific implementation, it also includes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the video generation method described in any of the above specific implementations are implemented.

[0035] The beneficial effects of the present application are as follows: Different from the prior art, the embodiments of the present application provide a video generation method, apparatus, computer device and storage medium, wherein the video generation method includes:

[0036] Get panoramic video;

[0037] Tracking a human target in each video frame of the panoramic video to obtain a target bounding box of the human target, and drawing a rectangular rendering picture according to the target bounding box so that the target bounding box is located in a middle area of ​​the rectangular rendering picture;

[0038] Performing human body detection on the target bounding box to obtain a body region of interest;

[0039] Adjusting the field of view of the rendering image so that the body region of interest occupies a set proportion of the rendering image;

[0040] Generating a planar video about the human target according to the adjusted rendering picture;

[0041] The frame rate of the planar video is adjusted to generate a slow-motion video with a target frame rate.

[0042] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0043] The present application obtains a target bounding box about a human target through a target tracking algorithm, and then inputs the target bounding box into a human skeleton key point model to obtain N human skeleton key points of the human target. Finally, a body region of interest is obtained by analyzing the N human skeleton key points, and then key protruding parts of the human target are determined based on the body region of interest to obtain a slow-motion planar video with dynamic special effects.

[0044] The video generation method of dynamic special effects of the present application does not require complicated shooting techniques and subsequent complicated video editing operations. Only simple shooting is required to quickly and efficiently generate slow-motion planar videos with dynamic special effects through the video generation method of the present application, adding a new way to play panoramic videos and improving user playability. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 This is a schematic diagram of the implementation process of a video generation method provided in Example 1 of the present application;

[0047] Figure 2 It is a schematic diagram of key points of human skeleton in a video generation method provided in Example 1 of the present application;

[0048] Figure 3 This is a schematic diagram of the module composition of a video generating device provided in Embodiment 2 of the present application;

[0049] Figure 4 It is a schematic diagram of the internal structure of a computer device provided in Example 3 of the present application. DETAILED DESCRIPTION

[0050] The present application is further described in detail below in conjunction with the accompanying drawings and implementation methods. It is particularly noted that the following implementation methods are only used to illustrate the present application, but do not limit the scope of the present application. Similarly, the following implementation methods are only some implementation methods of the present application rather than all implementation methods. All other implementation methods obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0051] The terms "including" and "having" and any variations thereof in this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or modules is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other steps or modules inherent to these processes, methods, products or devices.

[0052] The above terms are only for the convenience of description and should not be construed as limitations on the present technical solution.

[0053] Reference to "embodiment" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0054] The specific implementation of this application is described in detail below in conjunction with specific embodiments:

[0055] Embodiment 1:

[0056] Figure 1 The implementation process of a video generation method provided in Example 1 of the present application is shown. For the convenience of explanation, only the part related to the embodiment of the present application is shown, which is described in detail as follows.

[0057] Get panoramic video;

[0058] It should be noted that the embodiments of the present application are applicable to video image processing devices such as video display and acquisition. The panoramic video is a video spliced ​​by panoramic cameras.

[0059] Tracking a human target in each video frame of the panoramic video to obtain a target bounding box of the human target, and drawing a rectangular rendering picture according to the target bounding box so that the target bounding box is located in a middle area of ​​the rectangular rendering picture;

[0060] It should be noted that the target detection and target tracking algorithms used in performing human target tracking on each video frame of the panoramic video.

[0061] The target detection algorithm may be a common detection method, which may be one or more detection methods based on manual features, including template matching, key point matching, and key feature methods; the common detection method may also be one or more detection methods based on convolutional neural network technology, and the convolutional neural network technology may adopt one or more models of YOLO (You Only Look Once), SSD (Single Shot MultiBox), R-CNN (Region-based Convolutional Neural Networks), or MaskR-CNN (MaskRegion-based Convolutional Neural Networks). Exemplarily, the YOLO model may be one or more models of versions such as YOLOv1-YOLOv5, YOLOR, and YOLOX.

[0062] When the human target is detected by the target detection algorithm, a target bounding box of the human target is obtained. The target bounding box is a rectangular frame diagram containing all the features of the human target. According to the rectangular frame diagram, a rectangular rendering screen is drawn. The rectangular rendering screen is a video screen of the size of a display screen. The rectangular frame diagram is located in the middle area of ​​the rectangular rendering screen to highlight the position of the human target in the rectangular rendering screen.

[0063] When performing the human target tracking, the video frame in which the human target is initially detected is used as the first video screen, and the target is tracked using a preset target tracking algorithm to obtain a target rectangular frame of the first video screen. It can be a tracker of Discriminative Correlation Filter (DCF) and other related filter types, a Siamese Region Proposal Network (SiamRPN) and other trackers based on CNN technology, or other trackers.

[0064] Performing human body detection on the target bounding box to obtain a body region of interest;

[0065] It should be noted that the body region of interest is the body part of the human target, which needs to be highlighted in the rectangular rendering image, such as the arm, thigh and other body parts with a large range of movement to form a dynamic moment special effect.

[0066] One specific implementation is that performing human body detection on the target bounding box to obtain a body region of interest includes:

[0067] Performing human key point detection on the target bounding box to obtain N human skeleton key points of the human target, where N is a natural number and 10≤N≤20;

[0068] Calculating the displacement of the N human skeleton key points of the human target in the plurality of video frames;

[0069] Determine a target human skeleton key point set according to the displacements of the N human skeleton key points in the plurality of video frames;

[0070] According to the mapping relationship between the target human skeleton key point set and the human body, the human body part corresponding to the target human skeleton key point set is determined to be the body region of interest.

[0071] It should be noted that human keypoints detection, also known as human posture estimation, is a relatively basic task in computer vision and a prerequisite for human motion recognition, behavior analysis, human-computer interaction, etc. In general, human keypoints detection can be subdivided into single / multi-person keypoints detection and 2D / 3D keypoints detection. At the same time, some algorithms will track keypoints after completing keypoint detection, which is also called human posture tracking.

[0072] Human key point detection algorithms include but are not limited to OpenPose, AlphaPose, etc.

[0073] The target human skeleton key point set, that is, the human skeleton key points with the largest displacement amplitude among the N human skeleton key points, can be calculated to determine which part of the target human body has the largest movement amplitude, thereby determining it as the body area of ​​interest.

[0074] One specific implementation method is that the human body key point detection is performed on the target bounding box to obtain N human body skeleton key points of the human target, including:

[0075] The target bounding box is output to a human skeleton key point detection model, and the N human skeleton key points in the target bounding box of each video frame are detected by the human skeleton key point detection model to obtain a coordinate sequence of the N human skeleton key points.

[0076] It should be noted that the human skeleton key point detection model is an image dataset containing various human posture features. Including but not limited to MPII (Max Planck Institute Informatik) and MS COCO. MPII is a dataset used by the Max Planck Institute for Informatik for human posture estimation. It has about 25,000 images and contains more than 40,000 human bodies with annotated key points. Its data is mainly based on multiple people. There are verification and test sets for single-frame single-person posture, single-frame multi-person posture and video multi-person posture. Most methods mainly use single-frame multi-person posture test sets. It annotates up to 16 possible key points of the whole body, and the test set also includes annotations of body part occlusion, 3D torso, and head direction; the MSCOCO dataset is a large and rich dataset for object detection, segmentation, subtitles and human key points. Among them, the human key point dataset part of the MS COCO dataset is the mainstream dataset for multi-person posture estimation. It contains more than 200,000 images and 250,000 key point-marked person instances, and annotates up to 17 possible key points of the whole body, such as Figure 2As shown in the schematic diagram of the human skeleton key point model, the 17 key points are the nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles. On average, there are 2 people in one image, and at most 13 people.

[0077] One specific implementation method is that the calculation of the displacement of the N human skeleton key points of the human target in the plurality of video frames includes:

[0078] Set the displacement of the human skeleton key point in the i-1th frame to the next frame i as , wherein the is the coordinate sequence of the human skeleton key point in the i-1th frame, norm is the norm calculation, k is a natural number, defined as the kth human skeleton key point, 0≤k≤N;

[0079] The displacement of the N human skeleton key points of the human target in the M consecutive video frames starting from the iMth frame is the displacement of the N human skeleton key points of the human target in multiple video frames, and the displacement is , 0≤k≤N.

[0080] It should be noted that the norm is a function with the concept of "length" and can be used to measure the length or size of each vector in a vector space (or matrix). In the embodiment of the present application, the norm calculation adopts the L1 norm or the L2 norm, the L1 norm is to calculate the sum of the displacements of each of the human skeleton key points of the human target in multiple video frames, and the L2 norm is to calculate the Euclidean distance of each of the human skeleton key points in multiple video frames.

[0081] One specific implementation method is that determining a target human skeleton key point set according to the displacement of the N human skeleton key points in the plurality of video frames includes:

[0082] According to the displacement of each of the human skeleton key points in the multiple video frames, the L human skeleton key points with the largest displacement are selected as the target human skeleton key points, and the target human skeleton key point set is determined based on the target human skeleton key points.

[0083] It should be noted that, the larger the displacement of the human skeleton key point, the larger the movement amplitude of the corresponding human body part. Therefore, the L human skeleton key points with the largest displacement are taken as the target human skeleton key points, and it can be confirmed that the human body part corresponding to these L human skeleton key points is the body area of ​​interest, and L is an integer.

[0084] One specific implementation method is that, according to the mapping relationship between the target human skeleton key point set and the human body, determining the human body part corresponding to the target human skeleton key point set as the body region of interest includes:

[0085] Based on the human skeleton key point detection model, the human body parts corresponding to the target human skeleton key point set are mapped, and the corresponding human body parts are determined to be the body region of interest.

[0086] It should be noted that each of the human skeleton key points or several of the human skeleton key points together can map out the corresponding human body parts, such as Figure 2 As shown, the human skeleton key point 13 maps the human head, 0 maps the human neck, 1 maps the human right shoulder, 4 maps the human left shoulder, 1, 2, 3 sets map the human right hand, 4, 5, 6 map the human left hand, 7, 8, 9 map the human right leg, 10, 11, 12 map the human left leg, according to the target human skeleton key point set includes Figure 2 By determining which human skeleton key points in the human skeleton key point model are present, it is possible to determine that the human body part corresponding to the target human skeleton key point set is the body region of interest. If the target human skeleton key point set includes the human skeleton key points 7 and 8, the right thigh part is determined as the body region of interest.

[0087] Adjusting the field of view of the rendering image so that the body region of interest occupies a set proportion of the rendering image;

[0088] In one specific implementation manner, N is 18, L is 2, and the body region of interest occupies a set proportion of 5%-10% of the rendering image.

[0089] It should be noted that, in the human skeleton key point detection, 18 human skeleton key points are usually detected, which can include various key point parts of the human body to the greatest extent. The target human skeleton key point set is based on a large amount of experimental experience, and 2 human skeleton key points are selected to confirm the body region of interest, so that the body region of interest can be detected with maximum efficiency and excessive calculation amount can be avoided.

[0090] After the body region of interest is determined, the proportion of the body region of interest in the rendering image is adjusted to highlight the body region of interest, and the set proportion is 5%-10%, preferably 5%.

[0091] Another feasible implementation is to achieve the purpose of highlighting the body region of interest by adjusting the viewing angle of the body region of interest in the entire rendering image, such as making the body region of interest appear in the middle area of ​​the rendering image as much as possible.

[0092] Generating a planar video about the human target according to the adjusted rendering picture;

[0093] It should be noted that, in the two-dimensional video about the human target, the human target is located in the middle area of ​​the display screen of the two-dimensional video.

[0094] The frame rate of the planar video is adjusted to generate a slow-motion video with a target frame rate.

[0095] By performing slow motion processing on the human target in motion, a two-dimensional video with dynamic moment special effects of the human target in the body area of ​​interest can be obtained.

[0096] Embodiment 2:

[0097] Figure 3 A schematic diagram of the module composition of a video generating device provided in the second embodiment of the present application is shown. For the sake of convenience of explanation, only the parts related to the embodiment of the present application are shown.

[0098] A video generation device according to an embodiment of the present application includes:

[0099] An acquisition module, wherein the acquisition module is used to acquire a panoramic video;

[0100] A tracking module, wherein the tracking module is used to track a human target in each video frame of the panoramic video to obtain a target bounding box of the human target, and draw a rectangular rendering picture according to the target bounding box so that the target bounding box is located in a middle area of ​​the rectangular rendering picture;

[0101] A human body detection module, the human body detection module is used to perform human body detection on the target bounding box to obtain a body region of interest;

[0102] A viewing angle adjustment module, which is used to adjust the viewing angle of the rendering image so that the body area of ​​interest occupies a set proportion of the rendering image.

[0103] A plane video generation module, the plane video generation module is used to generate a plane video about the human target according to the adjusted rendering picture;

[0104] A frame rate adjustment module is used to adjust the frame rate of the planar video to generate a slow motion video with a target frame rate.

[0105] A video generation device in an embodiment of the present application is used to implement a video generation method provided in any specific implementation manner of the first embodiment of the present application:

[0106] Get panoramic video;

[0107] Tracking a human target in each video frame of the panoramic video to obtain a target bounding box of the human target, and drawing a rectangular rendering picture according to the target bounding box so that the target bounding box is located in a middle area of ​​the rectangular rendering picture;

[0108] Performing human body detection on the target bounding box to obtain a body region of interest;

[0109] Adjusting the field of view of the rendering image so that the body region of interest occupies a set proportion of the rendering image;

[0110] Generating a planar video about the human target according to the adjusted rendering picture;

[0111] The frame rate of the planar video is adjusted to generate a slow-motion video with a target frame rate.

[0112] It should be noted that each module in the above video generation device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0113] It should be noted that for technical details that are not described in detail in this embodiment, reference can be made to the video generation method provided in the above-mentioned embodiment 1, and will not be repeated here.

[0114] Embodiment three:

[0115] In an embodiment of the present application, a computer device is provided. Figure 4 A schematic diagram of the internal structure of a computer device.

[0116] The computer device includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the video generation method in any specific implementation manner provided in the first embodiment of the present application, specifically including:

[0117] Get panoramic video;

[0118] Tracking a human target in each video frame of the panoramic video to obtain a target bounding box of the human target, and drawing a rectangular rendering picture according to the target bounding box so that the target bounding box is located in a middle area of ​​the rectangular rendering picture;

[0119] Performing human body detection on the target bounding box to obtain a body region of interest;

[0120] Adjusting the field of view of the rendering image so that the body region of interest occupies a set proportion of the rendering image;

[0121] Generating a planar video about the human target according to the adjusted rendering picture;

[0122] The frame rate of the planar video is adjusted to generate a slow-motion video with a target frame rate.

[0123] It should be noted that the computer device may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a video generation method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0124] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0125] It should be noted that for technical details that are not described in detail in this embodiment, reference can be made to the video generation method provided in the above-mentioned embodiment 1, and will not be repeated here.

[0126] Embodiment 4:

[0127] In an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the video generation method in any specific implementation manner provided in the first embodiment of the present application is implemented, specifically including:

[0128] Get panoramic video;

[0129] Tracking a human target in each video frame of the panoramic video to obtain a target bounding box of the human target, and drawing a rectangular rendering picture according to the target bounding box so that the target bounding box is located in a middle area of ​​the rectangular rendering picture;

[0130] Performing human body detection on the target bounding box to obtain a body region of interest;

[0131] Adjusting the field of view of the rendering image so that the body region of interest occupies a set proportion of the rendering image;

[0132] Generating a planar video about the human target according to the adjusted rendering picture;

[0133] The frame rate of the planar video is adjusted to generate a slow-motion video with a target frame rate.

[0134] It should be noted that for technical details that are not described in detail in this embodiment, reference can be made to the video generation method provided in the above-mentioned embodiment 1, and will not be repeated here.

[0135] Different from the prior art, the embodiments of the present application provide a video generation method, apparatus, computer device and storage medium, wherein the video generation method includes:

[0136] Get panoramic video;

[0137] Tracking a human target in each video frame of the panoramic video to obtain a target bounding box of the human target, and drawing a rectangular rendering picture according to the target bounding box so that the target bounding box is located in a middle area of ​​the rectangular rendering picture;

[0138] Performing human body detection on the target bounding box to obtain a body region of interest;

[0139] Adjusting the field of view of the rendering image so that the body region of interest occupies a set proportion of the rendering image;

[0140] Generating a planar video about the human target according to the adjusted rendering picture;

[0141] The frame rate of the planar video is adjusted to generate a slow-motion video with a target frame rate.

[0142] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0143] The present application obtains a target bounding box about a human target through a target tracking algorithm, and then inputs the target bounding box into a human skeleton key point model to obtain N human skeleton key points of the human target. Finally, a body region of interest is obtained by analyzing the N human skeleton key points, and then key protruding parts of the human target are determined based on the body region of interest to obtain a slow-motion planar video with dynamic special effects.

[0144] The video generation method of dynamic special effects of the present application does not require complicated shooting techniques and subsequent complicated video editing operations. Only simple shooting is required to quickly and efficiently generate slow-motion planar videos with dynamic special effects through the video generation method of the present application, adding a new way to play panoramic videos and improving user playability.

[0145] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0146] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0147] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a terminal device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0148] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video generation method, It is characterized in that The method comprises: Get panoramic video; Tracking a human target in each video frame of the panoramic video to obtain a target bounding box of the human target, and drawing a rectangular rendering picture according to the target bounding box so that the target bounding box is located in a middle area of ​​the rectangular rendering picture; Performing human body detection on the target bounding box to obtain a body region of interest; Adjusting the field of view of the rendering image so that the body region of interest occupies a set proportion of the rendering image; Generating a planar video about the human target according to the adjusted rendering picture; Adjusting the frame rate of the planar video to generate a slow-motion video with a target frame rate; The performing human body detection on the target bounding box to obtain a body region of interest includes: Performing human key point detection on the target bounding box to obtain N human skeleton key points of the human target, where N is a natural number and 10≤N≤20; Calculating the displacement of the N human skeleton key points of the human target in the plurality of video frames; Determine a target human skeleton key point set according to the displacements of the N human skeleton key points in the plurality of video frames; According to the mapping relationship between the target human skeleton key point set and the human body, the human body part corresponding to the target human skeleton key point set is determined to be the body region of interest.

2. A video generation method according to claim 1, It is characterized in that The detecting of human key points on the target bounding box to obtain N human skeleton key points of the human target includes: The target bounding box is output to a human skeleton key point detection model, and the N human skeleton key points in the target bounding box of each video frame are detected by the human skeleton key point detection model to obtain a coordinate sequence of the N human skeleton key points.

3. A video generation method according to claim 2, It is characterized in that The calculating of the displacement of the N human skeleton key points of the human target in the plurality of video frames comprises: Set the displacement of the human skeleton key point in the i-1th frame to the next frame i as MF i_k =norm(P i-1_k ,P i_k ), wherein the P i-1_k is the coordinate sequence of the human skeleton key point in the i-1th frame, norm is the norm calculation, k is a natural number, defined as the kth human skeleton key point, 0≤k≤N; The displacement of the N human skeleton key points of the human target in the M consecutive video frames starting from the iMth frame is the displacement of the N human skeleton key points of the human target in multiple video frames, and the displacement is MM i_k =∑ i-M≤j<i MF i_k , 0≤k≤N.

4. A video generation method according to claim 3, It is characterized in that Determining a target human skeleton key point set according to the displacements of the N human skeleton key points in the plurality of video frames comprises: According to the displacement of each of the human skeleton key points in the multiple video frames, the L human skeleton key points with the largest displacement are selected as the target human skeleton key points, and the target human skeleton key point set is determined based on the target human skeleton key points.

5. A video generation method according to claim 4, It is characterized in that The determining, according to the mapping relationship between the target human skeleton key point set and the human body, the human body part corresponding to the target human skeleton key point set as the body region of interest comprises: Based on the human skeleton key point detection model, the human body parts corresponding to the target human skeleton key point set are mapped, and the corresponding human body parts are determined to be the body region of interest.

6. A video generation method according to claim 4 or 5, It is characterized in that The N is 18, the L is 2, and the set proportion of the body region of interest in the rendering image is 5%-10%.

7. A video generating device, It is characterized in that The device comprises: An acquisition module, wherein the acquisition module is used to acquire a panoramic video; A tracking module, wherein the tracking module is used to track a human target in each video frame of the panoramic video to obtain a target bounding box of the human target, and draw a rectangular rendering picture according to the target bounding box so that the target bounding box is located in a middle area of ​​the rectangular rendering picture; A human body detection module, the human body detection module is used to perform human body detection on the target bounding box to obtain a body region of interest; The performing human body detection on the target bounding box to obtain a body region of interest includes: Performing human key point detection on the target bounding box to obtain N human skeleton key points of the human target, where N is a natural number and 10≤N≤20; Calculating the displacement of the N human skeleton key points of the human target in the plurality of video frames; Determine a target human skeleton key point set according to the displacements of the N human skeleton key points in the plurality of video frames; According to the mapping relationship between the target human skeleton key point set and the human body, determining the human body part corresponding to the target human skeleton key point set as the body region of interest; A field of view angle adjustment module, the field of view angle adjustment module is used to adjust the field of view angle of the rendering picture so that the body region of interest occupies a set proportion of the rendering picture; A plane video generation module, the plane video generation module is used to generate a plane video about the human target according to the adjusted rendering picture; A frame rate adjustment module is used to adjust the frame rate of the planar video to generate a slow motion video with a target frame rate.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the video generation method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the video generation method according to any one of claims 1 to 8 are implemented.

Citation Information

Patent Citations

  • Multi-target tracking and displaying method in panoramic video

    CN116506585A

  • Person-of-interest centric timelapse video with ai input on home security camera to protect privacy

    US20220058394A1