A panoramic video editing method, device, storage medium and equipment

Abstract

The present invention is applicable to the field of panoramic video technology, and provides a panoramic video editing method, device, storage medium and equipment. The method includes: acquiring a panoramic video captured by a panoramic camera, and recording the forward direction view angle of the panoramic camera during moving shooting; performing a frame extraction operation on the acquired panoramic video to obtain corresponding panoramic video frames and / or fisheye image frames; identifying and acquiring salient objects, symmetric objects and sky objects according to the panoramic video frames and / or fisheye image frames; using a preset object tracking algorithm to track the salient objects, symmetric objects and sky objects to obtain the view angles where the salient objects are located, the view angles where the symmetric objects are located and the view angles where the sky objects are located; and editing the panoramic video according to the view angles to generate a target video corresponding to the panoramic video, realizing automatic editing of the panoramic video, and at the same time ensuring the smoothness of transitions, as well as the effectiveness and interestingness of the content in the target video.

Description

Technical Field

[0001] The present invention belongs to the technical field of video processing, and particularly relates to a panoramic video editing method, apparatus, storage medium, and device. Background Art

[0002] A panoramic video is a video obtained by using a panoramic camera to perform an all-round 360-degree shooting. A user can arbitrarily view a dynamic video within the shooting angle range of the panoramic camera. When watching a panoramic video, since a flat display can only display an image of one perspective of the panoramic video at a certain moment, when the user wants to view a certain significant target object during a certain period of the panoramic video playback, it may be necessary to continuously control the display to rotate the perspective because the target disappears from the current perspective. Therefore, the operation is rather troublesome and the viewing experience is also affected. Summary of the Invention

[0003] Embodiments of the present invention provide a panoramic video editing method, apparatus, storage medium, and device, which are used to solve the problem that the smoothness of the panoramic video during output is not high because the prior art cannot provide an effective panoramic video editing method.

[0004] On the one hand, embodiments of the present invention provide a panoramic video editing method, and the method includes the following steps:

[0005] Obtain a panoramic video shot by a panoramic camera, and record the forward direction perspective of the panoramic camera during shooting;

[0006] Perform a frame extraction operation on the obtained panoramic video to obtain corresponding panoramic video frames and / or fisheye image frames;

[0007] According to the panoramic video frames and / or fisheye image frames, identify and obtain significant targets, symmetric targets, and sky targets;

[0008] Adopt a preset target tracking algorithm to track the significant targets, symmetric targets, and sky targets, and obtain the perspectives where the significant targets are located, the perspectives where the symmetric targets are located, and the perspectives where the sky targets are located;

[0009] Traverse the panoramic video, and edit the panoramic video according to the forward direction perspective, the perspectives where the significant targets are located, the perspectives where the symmetric targets are located, and the perspectives where the sky targets are located, to generate a target video corresponding to the panoramic video.

[0010] Combined with the first aspect, in a possible implementation manner, obtaining a panoramic video shot by a panoramic camera and recording the forward direction perspective of the panoramic camera during shooting specifically includes:

[0011] Obtain the forward direction when the panoramic camera is moving and shooting according to the panoramic video, and obtain the lens image corresponding to the forward direction view;

[0012] Obtain the forward direction view of the panoramic camera during shooting according to the lens image;

[0013] Wherein, the panoramic video is an original spherical video.

[0014] Combined with the first aspect, in a possible implementation manner, according to the panoramic video frame and / or the dual-fisheye image frame, identify and obtain significant objects, symmetric objects, and sky objects, specifically including:

[0015] Use a preset significant object detection and recognition algorithm to detect the panoramic video frame and / or the dual-fisheye image frame to obtain significant objects.

[0016] Combined with the first aspect, in a possible implementation manner, detect the panoramic video frame and / or the dual-fisheye image frame to obtain significant objects, specifically including:

[0017] When a preset type of significant object is detected in the currently detected panoramic video frame and / or dual-fisheye image frame, set the object with the maximum significance value as the current significant object of the currently detected panoramic video frame and / or dual-fisheye image frame.

[0018] Combined with the first aspect, in a possible implementation manner, according to the panoramic video frame and / or the dual-fisheye image frame, identify and obtain significant objects, symmetric objects, and sky objects, specifically including:

[0019] According to the panoramic video frame and / or the dual-fisheye image frame, obtain the timestamp of the panoramic video frame and / or the dual-fisheye image frame;

[0020] According to the timestamp of the panoramic video frame and / or the dual-fisheye image frame, obtain the first rotation matrix of the panoramic video frame and / or the dual-fisheye image frame;

[0021] Render the panoramic video frame and / or the dual-fisheye image frame according to the first rotation matrix to obtain the image in the vertically upward direction of the panoramic video frame and / or the dual-fisheye image frame;

[0022] Use a preset symmetric object detection and recognition algorithm to detect the image in the vertically upward direction of the panoramic video frame and / or the dual-fisheye image frame to obtain symmetric objects.

[0023] Combined with the first aspect, in a possible implementation manner, according to the panoramic video frame and / or the dual-fisheye image frame, identify and obtain significant objects, symmetric objects, and sky objects, specifically including:

[0024] Obtain the timestamps of the panoramic video frame and / or the dual fisheye image frame based on the panoramic video frame and / or the dual fisheye image frame;

[0025] Obtain the first rotation matrix of the panoramic video frame and / or the dual fisheye image frame according to the timestamps of the panoramic video frame and / or the dual fisheye image frame;

[0026] Decompose the travel direction matrix of the panoramic video frame and / or the dual fisheye image frame into the first Euler angles (Yaw, Pitch, Roll respectively), and set the Pitch angle to PI / 2 to obtain the second Euler angles;

[0027] Convert the second Euler angles to obtain the second rotation matrix;

[0028] Render the panoramic video frame and / or the dual fisheye image frame according to the second rotation matrix to obtain the image in the vertically upward direction of the panoramic video frame and / or the dual fisheye image frame;

[0029] Adopt a preset sky target detection and recognition algorithm to detect the image in the vertically upward direction of the panoramic video frame and / or the dual fisheye image frame, and obtain sky targets.

[0030] Combined with the first aspect, in a possible implementation manner, adopt a preset target tracking algorithm to track the salient target, the symmetry target, and the sky target, and obtain the viewing angles where the salient target, the symmetry target, and the sky target are located, specifically including:

[0031] When the current salient target, the current symmetry target, and the current sky target are detected in the current detected panoramic video frame and / or dual fisheye image frame, adopt a preset target tracking algorithm to sequentially track the current salient target, the current symmetry target, and the current sky target in subsequent panoramic video frames and / or dual fisheye image frames, and obtain the viewing angles where the current salient target, the current symmetry target, and the current sky target are located.

[0032] Combined with the first aspect, in a possible implementation manner, after obtaining the viewing angles where the current salient target, the current symmetry target, and the current sky target are located, it further includes:

[0033] Detect the stop tracking events of the current salient target, the current symmetry target, and the current sky target respectively. When the stop tracking events of the current salient target, the current symmetry target, and the current sky target are detected, respectively jump to the step of performing frame extraction on the panoramic video, and continue to respectively identify and obtain the salient target, the symmetry target, and the sky target.

[0034] In combination with the first aspect, in a possible implementation manner, after using a preset target tracking algorithm to sequentially track the current saliency target, current symmetry target, and current sky target in subsequent panoramic video frames and / or dual fisheye image frames, and obtaining the viewing angles of the current saliency target, current symmetry target, and current sky target, it specifically further includes:

[0035] Respectively obtain the center coordinates of the tracking frames of the current saliency target, current symmetry target, and current sky target in the current detected panoramic video frame and / or dual fisheye image frame, and calculate the spherical viewpoint coordinates of the current saliency target, current symmetry target, and current sky target according to the center coordinates;

[0036] According to the spherical viewpoint coordinates, respectively obtain the lens images corresponding to the viewing angles of the saliency target, symmetry target, and sky target;

[0037] According to the lens images corresponding to the viewing angles of the saliency target, symmetry target, and sky target, respectively generate a saliency target view segment, a symmetry target view segment, and a sky target view segment.

[0038] In combination with the first aspect, in a possible implementation manner, the stop tracking event is losing the current saliency target, current symmetry target, and current sky target or the area of the tracking frame is less than a preset area.

[0039] In combination with the first aspect, in a possible implementation manner, according to the forward direction viewing angle, the viewing angle of the saliency target, the viewing angle of the symmetry target, and the viewing angle of the sky target, clip the panoramic video to generate the target video corresponding to the panoramic video, specifically including:

[0040] According to the duration of the panoramic video;

[0041] And / or, the number and corresponding durations of the appearances of the saliency target view segment and / or sky target view segment and / or symmetry target view segment;

[0042] And / or, the relationship between the viewing angle of the saliency target and / or the viewing angle of the symmetry target and / or the viewing angle of the sky target and the forward direction viewing angle;

[0043] Set the playback numbers and corresponding playback speeds of the saliency target view segment and / or sky target view segment and / or symmetry target view segment, and automatically clip the saliency target view segment and / or sky target view segment and / or symmetry target view segment to generate the target video corresponding to the panoramic video;

[0044] Among them, the target video is a single - perspective video or a planar video.

[0045] Combined with the first aspect, in a possible implementation manner, before setting the playback quantity and corresponding playback speed of the significant - target perspective segments and / or sky - target perspective segments and / or symmetry - target perspective segments, it further includes:

[0046] Set the camera rotation mode and / or the corresponding playback speed according to the panoramic video duration, the quantity and duration of the significant - target perspective segments and / or sky - target perspective segments and / or symmetry - target perspective segments.

[0047] In a second aspect, the present invention provides a panoramic video editing device, characterized in that the device includes:

[0048] An acquisition module: used to acquire the panoramic video captured by a panoramic camera and record the forward - direction perspective of the panoramic camera during shooting;

[0049] A frame - extraction module: used to perform frame - extraction operations on the acquired panoramic video to obtain corresponding panoramic video frames and / or fisheye - image frames;

[0050] An identification module: used to identify and acquire significant targets, symmetry targets, and sky targets according to the panoramic video frames and / or fisheye - image frames;

[0051] A tracking module: used to adopt a preset target - tracking algorithm to track the significant targets, symmetry targets, and sky targets to obtain the perspectives where the significant targets are located, the perspectives where the symmetry targets are located, and the perspectives where the sky targets are located;

[0052] A processing module: used to traverse the panoramic video and edit the panoramic video according to the forward - direction perspective, the perspectives where the significant targets are located, the perspectives where the symmetry targets are located, and the perspectives where the sky targets are located to generate a target video corresponding to the panoramic video.

[0053] In a third aspect, the present invention provides a computer - readable storage medium, the computer - readable storage medium stores a computer program, characterized in that when the computer program is executed by a processor, it implements the steps of the method described in the first aspect.

[0054] In a fourth aspect, the present invention provides a panoramic video editing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that when the processor executes the computer program, it implements the steps of the method described in the first aspect.

[0055] The present invention provides a panoramic video editing method, which obtains a panoramic video captured by a panoramic camera and records the forward direction view of the panoramic camera during shooting; performs frame extraction on the obtained panoramic video to obtain corresponding panoramic video frames and / or fisheye image frames; identifies and obtains significant objects, symmetric objects, and sky objects according to the panoramic video frames and / or fisheye image frames; uses a preset object tracking algorithm to track the significant objects, symmetric objects, and sky objects to obtain the views where the significant objects are located, the views where the symmetric objects are located, and the views where the sky objects are located; edits the panoramic video according to the forward direction view, the views where the significant objects are located, the views where the symmetric objects are located, and the views where the sky objects are located, and generates a target video corresponding to the panoramic video, realizing automatic editing of the panoramic video, and at the same time ensuring the smoothness of transitions in the target video, as well as the effectiveness and interest of the content. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flowchart of the implementation of the panoramic video editing method provided in Embodiment 1 of the present invention;

[0057] Figure 2 is a schematic structural diagram of the panoramic video editing device provided in Embodiment 2 of the present invention;

[0058] Figure 3 is a schematic structural diagram of the panoramic video editing device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0059] In order to make the objectives, technical solutions, and beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0060] The following describes the specific implementation of the present invention in detail with reference to specific embodiments:

[0061] Example 1:

[0062] Figure 1 shows the implementation process of the panoramic video editing method provided in Embodiment 1 of the present invention. The panoramic video editing method provided in the embodiments of the present invention can be applied to a computing device, where the computing device can be, but is not limited to, various personal computers, laptop computers, smartphones, tablet computers, and panoramic cameras.

[0063] For the sake of convenience of description, Figure 1 only the parts related to the embodiments of the present invention are shown and are described in detail as follows:

[0064] S101. Obtain the panoramic video captured by the panoramic camera, and record the forward direction view of the panoramic camera during shooting;

[0065] The embodiments of the present invention are applicable to panoramic video editing for editing panoramic videos. The panoramic video is obtained by shooting with a panoramic camera composed of two or more fisheye lenses, and the obtained panoramic video is an original spherical video. When recording the forward direction view of the panoramic camera during shooting, specifically, the forward direction of the panoramic camera during moving shooting is obtained according to the panoramic video, and then the lens image corresponding to the forward direction view is obtained, so as to record the forward direction view of the panoramic camera through the lens image corresponding to the forward direction view.

[0066] Specifically, obtaining the forward direction of the panoramic camera during moving shooting according to the panoramic video, and then obtaining the lens image corresponding to the forward direction view can be further specifically:

[0067] Obtain the rotation amount of the panoramic camera relative to the world coordinate system when shooting the current video frame and the

[0068] multichannel fisheye images corresponding to the current video frame and the previous video frame;

[0069] Extract the corner points of the multichannel fisheye images corresponding to the previous video frame of the panoramic video respectively to obtain the sequence of corner points to be tracked;

[0070] Track the sequence of corner points to be tracked respectively to obtain the matching point pairs to be tracked in the fisheye images corresponding to the current video frame and the previous video frame;

[0071] Optimize the displacement amount of the current video frame of the panoramic camera relative to the previous video frame according to the matching point pairs to obtain the optimized displacement amount;

[0072] Take the optimized displacement amount as the forward direction of the virtual camera, calculate the rotation matrix of the current virtual camera, and perform transition rendering on the current video frame of the panoramic video by using the rotation amount of the panoramic camera relative to the world coordinate system when shooting the current video frame and the rotation matrix of the current virtual camera.

[0073] Perform frame extraction on the obtained panoramic video to obtain the corresponding panoramic video frames and / or double fisheye image frames;

[0074] In the embodiments of the present invention, frame extraction is performed on the obtained panoramic video at a preset time interval to obtain the corresponding panoramic video frames and / or double fisheye image frames.

[0075] Identify and obtain significant targets, symmetric targets and sky targets according to the panoramic video frames and / or double fisheye image frames;

[0076] In the first embodiment of the present invention, the panoramic video frame and / or the fisheye image frame are sequentially detected to identify and obtain significant targets, symmetric targets, and sky targets, which are achieved through the following steps:

[0077] (1) In the embodiment of the present invention, when performing significant target detection on the panoramic video frame and / or the fisheye image frame, when a significant target under the preset target category is detected in the currently detected panoramic video frame and / or fisheye image frame, the target with the largest significance value is set as the current significant target of the currently detected panoramic video frame and / or fisheye image frame, so as to accurately obtain the significant target to be tracked in the currently detected panoramic video frame and / or fisheye image frame when multiple significant targets are detected. When no significant target under the target category is detected in the currently detected panoramic video frame and / or fisheye image frame, it is confirmed that there is no significant target in the currently detected panoramic video frame and / or fisheye image frame. The target category can be set according to the preset shooting scene of the panoramic camera to further improve the accuracy of target detection. For example, the target category can be sculptures, stone tablets, flower beds, and landmark buildings. The significant targets under the sculpture category can include animal sculptures, plant sculptures, human body sculptures, etc.

[0078] (2) In the first embodiment of the present invention, performing symmetric target detection on the panoramic video frame and / or the fisheye image frame specifically includes:

[0079] Obtaining the timestamp of the panoramic video frame and / or the fisheye image frame according to the panoramic video frame and / or the fisheye image frame;

[0080] Obtaining the first rotation matrix of the panoramic video frame and / or the fisheye image frame according to the timestamp of the panoramic video frame and / or the fisheye image frame;

[0081] Rendering the panoramic video frame and / or the fisheye image frame according to the first rotation matrix to obtain the image in the vertically upward direction of the panoramic video frame and / or the fisheye image frame;

[0082] Using a preset symmetric target detection and recognition algorithm to detect the image in the vertically upward direction of the panoramic video frame and / or the fisheye image frame to obtain symmetric targets.

[0083] (3) In the first embodiment of the present invention, performing sky target detection on the panoramic video frame and / or the fisheye image frame specifically includes:

[0084] Obtaining the timestamp of the panoramic video frame and / or the fisheye image frame according to the panoramic video frame and / or the fisheye image frame;

[0085] Obtain a first rotation matrix of the panoramic video frame and / or the dual fisheye image frame according to the timestamps of the panoramic video frame and / or the dual fisheye image frame;

[0086] Decompose the travel direction matrix of the panoramic video frame and / or the dual fisheye image frame into first Euler angles (Yaw, Pitch, Roll respectively), and set the Pitch angle to PI / 2 to obtain second Euler angles;

[0087] Convert the second Euler angles to obtain a second rotation matrix;

[0088] Render the panoramic video frame and / or the dual fisheye image frame according to the second rotation matrix to obtain an image in the vertically upward direction of the panoramic video frame and / or the dual fisheye image frame;

[0089] Use a preset sky target detection and recognition algorithm to detect the image in the vertically upward direction of the panoramic video frame and / or the dual fisheye image frame to obtain sky targets.

[0090] (4) In the first embodiment of the present invention, the panoramic video frame and / or the dual fisheye image frame are sequentially detected to identify and obtain salient targets, symmetry targets, and sky targets. Specifically, when performing target detection and recognition on the panoramic video frame and / or the dual fisheye image frame, algorithms including but not limited to the FT (Frequency-tuned Salient Region Detection) algorithm or the superpixel convolutional neural network (for example, A Superpixelwise Convolutional Neural Network for Salient Object Detection) can be used to improve the accuracy of target detection while ensuring the stability of target detection.

[0091] Use a preset target tracking algorithm to track the salient target, symmetry target, and sky target to obtain the viewing angles where the salient target is located, the viewing angle where the symmetry target is located, and the viewing angle where the sky target is located;

[0092] In the first embodiment of the present invention, when a current salient target, a current symmetry target, and a current sky target are detected in the currently detected panoramic video frame and / or the dual fisheye image frame, a preset target tracking algorithm is used to sequentially track the current salient target, the current symmetry target, and the current sky target in subsequent panoramic video frames and / or dual fisheye image frames, and obtain the viewing angles where the current salient target is located, the viewing angle where the current symmetry target is located, and the viewing angle where the current sky target is located.

[0093] Specifically, the preset target tracking algorithm may adopt, including but not limited to, the KCF (High-speed Tracking with Kernelized Correlation filters) algorithm or the DSST (Accurate Scale Estimation for Robust Visual Tracking) algorithm, etc.

[0094] In the first embodiment of the present invention, after obtaining the perspectives of the current saliency target, the current symmetry target, and the current sky target, it further includes:

[0095] Detect the stop tracking events of the current saliency target, the current symmetry target, and the current sky target respectively. When the stop tracking events of the current saliency target, the current symmetry target, and the current sky target are detected, jump to the step of extracting frames from the panoramic video respectively, and continue to identify and obtain the saliency target, the symmetry target, and the sky target respectively.

[0096] In the first embodiment of the present invention, after using the preset target tracking algorithm to sequentially track the current saliency target, the current symmetry target, and the current sky target in subsequent panoramic video frames and / or fisheye image frames, and obtaining the perspectives of the current saliency target, the current symmetry target, and the current sky target, it specifically further includes:

[0097] Obtain the center coordinates of the tracking frames of the current saliency target, the current symmetry target, and the current sky target in the currently detected panoramic video frame and / or fisheye image frame respectively, and calculate the spherical viewpoint coordinates of the current saliency target, the current symmetry target, and the current sky target respectively according to the center coordinates;

[0098] According to the spherical viewpoint coordinates, obtain the lens images corresponding to the perspectives of the saliency target, the symmetry target, and the sky target respectively;

[0099] Generate a saliency target perspective segment, a symmetry target perspective segment, and a sky target perspective segment respectively according to the lens images corresponding to the perspectives of the saliency target, the symmetry target, and the sky target.

[0100] In the first embodiment of the present invention, the stop tracking event is losing the current saliency target, the current symmetry target, and the current sky target or the area of the tracking frame is less than the preset area.

[0101] Traverse the panoramic video, and clip the panoramic video according to the forward direction view, the view where the salient object is located, the view where the symmetric object is located, and the view where the sky object is located, to generate a target video corresponding to the panoramic video.

[0102] In the first embodiment of the present invention, clipping the panoramic video according to the forward direction view, the view where the salient object is located, the view where the symmetric object is located, and the view where the sky object is located, to generate a target video corresponding to the panoramic video, specifically includes:

[0103] According to the duration of the panoramic video;

[0104] And / or, the number and corresponding duration of the appearance of the salient object view segment and / or the sky object view segment and / or the symmetric object view segment;

[0105] And / or, the relationship between the view where the salient object is located and / or the view where the symmetric object is located and / or the view where the sky object is located and the forward direction view;

[0106] Set the playback quantity and corresponding playback speed of the salient object view segment and / or the sky object view segment and / or the symmetric object view segment, and automatically clip the salient object view segment and / or the sky object view segment and / or the symmetric object view segment to generate a target video corresponding to the panoramic video;

[0107] Wherein, the target video is a single-view video or a planar video.

[0108] In the first embodiment of the present invention, before setting the playback quantity and corresponding playback speed of the salient object view segment and / or the sky object view segment and / or the symmetric object view segment, it further includes:

[0109] According to the duration of the panoramic video, the number and duration of the appearance of the salient object view segment and / or the sky object view segment and / or the symmetric object view segment, set the lens rotation mode and / or the corresponding playback speed.

[0110] In the first embodiment of the present invention, the above steps can be further specifically:

[0111] Traverse the panoramic video frames and / or the dual-fisheye image frames. When there is no salient object or symmetric object or sky object in the traversed panoramic video frames and / or dual-fisheye image frames, clip the lens image corresponding to the forward direction view in the panoramic video frames and / or dual-fisheye image frames, and set the playback speed of the lens image;

[0112] When there are significant objects in the panoramic video frame and / or the dual fisheye image frame, according to the duration of the perspective segment of the significant object and the time interval between two adjacent significant segments, clip the lens image corresponding to the perspective of the significant object in the panoramic video frame and / or the dual fisheye image frame, and set the playing speed of the lens image;

[0113] When there are symmetric objects in the panoramic video frame and / or the dual fisheye image frame, according to the duration of the perspective segment of the symmetric object and the duration of the panoramic video, determine to clip the lens image corresponding to the perspective of the symmetric object in the panoramic video frame and / or the dual fisheye image frame, and set the playing speed of the lens image;

[0114] When there are sky objects in the panoramic video frame and / or the dual fisheye image frame, according to the preset rules, determine to clip the lens image corresponding to the perspective of the sky object in the panoramic video frame and / or the dual fisheye image frame, and set the playing speed of the lens image;

[0115] Generate the target video corresponding to the panoramic video according to the clipped lens images and the set corresponding playing speeds;

[0116] Wherein, the target video is a single-perspective video or a planar video.

[0117] In the first embodiment of the present invention, according to the duration of the perspective segment of the significant object and the time interval between two adjacent significant segments, clipping the lens image corresponding to the perspective of the significant object in the panoramic video frame and / or the dual fisheye image frame, and setting the playing speed of the lens image can be further specifically:

[0118] When the duration of the significant target perspective segment is less than a preset first threshold, discard the significant target perspective segment; when the duration of the significant target perspective segment is greater than the preset first threshold and less than the preset second threshold, perform corresponding expansion on the significant target perspective segment according to the preset expansion rule. If the expanded significant target perspective segment is still less than the preset second threshold, discard the significant target perspective segment; when the duration of the significant target perspective segment is greater than the preset first threshold and less than the preset second threshold, perform corresponding expansion on the significant target perspective segment according to the preset expansion rule. If the expanded significant target perspective segment is greater than the preset second threshold, retain the expanded significant target perspective segment and set the playback speed of the lens image to the preset first speed; when the duration of the significant target perspective segment is greater than the preset second threshold and the time interval between the significant target perspective segment and the previous significant target perspective segment is less than the preset threshold, set the playback speed of the lens image to the preset first speed; when the duration of the significant target perspective segment is greater than the preset second threshold and the time interval between the significant target perspective segment and the previous significant target perspective segment is greater than the preset threshold, set the playback speed of the lens images in the first half of the significant perspective segment to the preset first speed, set the playback speed of the lens images in the second half of the significant perspective segment to the preset second speed, and the first speed can be greater than or less than the second speed, and the duration of the first half of the significant perspective segment can be greater than or less than the duration of the second half of the significant perspective segment;

[0119] In the first embodiment of the present invention, according to the duration of the symmetry target perspective segment and the duration of the panoramic video, determine the lens images corresponding to the perspectives where the symmetry targets are located in the clipped panoramic video frames and / or the fisheye image frames, and set the playback speed of the lens images. Specifically, it can be further:

[0120] When the duration of the symmetry target perspective segment is less than the preset threshold, discard the symmetry target perspective segment; when the duration of the panoramic video is greater than the preset threshold and the duration of the symmetry target perspective segment is also greater than the preset threshold, set the playback speed of the lens image to the preset third speed; when the duration of the panoramic video is less than the preset threshold and the duration of the symmetry target perspective segment is greater than the preset threshold, set the playback speed of the lens image to the preset fourth speed, where the preset third speed can be greater than or less than the preset fourth speed.

[0121] Preferably, in the first embodiment of the present invention, according to the duration of the panoramic video, the number and duration of the significant target perspective segments and / or the sky target perspective segments and / or the symmetry target perspective segments, set the lens rotation mode and / or the corresponding playback speed. Specifically, it can be further:

[0122] Calculate the rotation direction from the perspective of the forward direction to the perspective of the saliency target. If the rotation direction is greater than a preset threshold, based on the rotation direction, first rotate the camera image corresponding to the perspective of the saliency target clockwise, and then rotate it counterclockwise; if the rotation direction is less than the preset threshold, based on the rotation direction, first rotate the camera image corresponding to the perspective of the saliency target counterclockwise, and then rotate it clockwise;

[0123] Among them, the rotation direction can be the rotation angle from the perspective of the forward direction to the perspective of the saliency target, specifically referring to the angle of rotation in the Yaw direction (the axis perpendicular to the ground) from the perspective of the forward direction to the perspective of the saliency target.

[0124] Generate a saliency target segment corresponding to the perspective of the saliency target according to the rotation result;

[0125] Preferably, in the first embodiment of the present invention, automatically edit the saliency target perspective segment and / or the sky target perspective segment and / or the symmetry target perspective segment to generate the target video corresponding to the panoramic video. Further specifically, it can be:

[0126] If the interval time between the first saliency target segment and the second saliency target segment is less than the preset threshold, directly switch to the camera image corresponding to the panoramic video frame and / or the forward direction perspective of the fisheye image frame after the end of the first saliency target segment, and then enter the second saliency target segment;

[0127] If the interval time between the first saliency target segment and the second saliency target segment is greater than the preset threshold, insert a preset symmetry target segment and / or sky target segment between the first saliency target segment and the second saliency target segment.

[0128] Preferably, in the first embodiment of the present invention, automatically edit the saliency target perspective segment and / or the sky target perspective segment and / or the symmetry target perspective segment to generate the target video corresponding to the panoramic video. Further specifically, it can be:

[0129] Select the key points of the saliency target perspective segment and / or the sky target perspective segment and / or the symmetry target perspective segment, combine the key points into a smooth curve according to a preset rule, and then interpolate any point between the key points according to a preset rule. Through the above operations, smooth switching between multiple target perspectives can be ensured;

[0130] Among them, the key points can select time as the abscissa and the relative rotation angle as the ordinate.

[0131] Example 2:

[0132] Figure 2 The structure of the panoramic video editing device provided in the second embodiment of the present invention is shown. For ease of description, only the parts related to the embodiments of the present invention are shown.

[0133] In the embodiments of the present invention, the panoramic video editing device includes an acquisition module 21, a frame extraction module 22, an identification module 23, a tracking module 24, and a processing module 25, where:

[0134] The acquisition module 21: is configured to acquire a panoramic video captured by a panoramic camera and record the forward direction view angle of the panoramic camera during shooting;

[0135] The frame extraction module 22: is configured to perform a frame extraction operation on the acquired panoramic video to obtain corresponding panoramic video frames and / or dual fisheye image frames;

[0136] The identification module 23: is configured to identify and acquire significant targets, symmetric targets, and sky targets according to the panoramic video frames and / or dual fisheye image frames;

[0137] The tracking module 24: is configured to use a preset target tracking algorithm to track the significant targets, symmetric targets, and sky targets to obtain the view angles where the significant targets are located, the view angles where the symmetric targets are located, and the view angles where the sky targets are located;

[0138] The processing module 25: is configured to traverse the panoramic video and edit the panoramic video according to the forward direction view angle, the view angles where the significant targets are located, the view angles where the symmetric targets are located, and the view angles where the sky targets are located to generate a target video corresponding to the panoramic video.

[0139] In the embodiments of the present invention, each module of the panoramic video editing device can be implemented by corresponding hardware or software units. Each module can be an independent software or hardware unit, or can be integrated into a software or hardware unit, which is not used to limit the present invention here.

[0140] Example 3:

[0141] In the embodiments of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps in the embodiments of the above-mentioned panoramic video editing method are implemented. For example, Figure 1 the steps S101 to S105 shown. Or, when the computer program is executed by a processor, the functions of each unit in the above-mentioned device embodiments are implemented. For example Figure 2 the functions of the units 21 to 25 shown.

[0142] The computer-readable storage medium of the embodiments of the present invention may include any entity or device, recording medium that can carry computer program code, for example, memories such as ROM / RAM, magnetic disks, optical disks, flash memories, etc.

[0143] Example 4:

[0144] Figure 3 The structure of the panoramic video editing device provided in the third embodiment of the present invention is shown. For the sake of convenience of description, only the parts related to the embodiments of the present invention are shown.

[0145] The panoramic video editing device 3 of the embodiments of the present invention includes a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30. When the processor 30 executes the computer program 32, the steps in the above-mentioned method embodiments of the panoramic video editing method are implemented, for example Figure 1 The steps S101 to S105 shown. Alternatively, when the processor 30 executes the computer program 32, the functions of each module in the above-mentioned device embodiments are implemented, for example Figure 2 The functions of the modules 21 to 25 shown.

[0146] The panoramic video editing device of the embodiments of the present invention may be a smart phone, a personal computer, or the panoramic camera itself, etc. The steps implemented when the processor 30 in the panoramic video editing device 3 executes the computer program 32 to implement the panoramic video editing method can refer to the description of the foregoing method embodiments and will not be repeated here.

[0147] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A panoramic video automatic editing method, characterized in that, The method includes: Obtain a panoramic video captured by a panoramic camera, and record the forward direction view of the panoramic camera during shooting; Perform a frame extraction operation on the obtained panoramic video to obtain corresponding panoramic video frames and / or fisheye image frames; Identify and obtain salient objects, symmetric objects, and sky objects based on the panoramic video frames and / or fisheye image frames; Use a preset object tracking algorithm to track the salient objects, symmetric objects, and sky objects, and obtain the views of the salient objects, symmetric objects, and sky objects; Traverse the panoramic video, and clip the panoramic video according to the forward direction view, the view of the salient object, the view of the symmetric object, and the view of the sky object to generate a target video corresponding to the panoramic video, including: select key points of the salient object view segment, sky object view segment, and symmetric object view segment, combine the key points into a smooth curve according to a preset rule, and then interpolate any point between the key points according to a preset rule according to the curve to obtain the target video.

2. The method according to claim 1, wherein Obtain a panoramic video captured by a panoramic camera, and record the forward direction view of the panoramic camera during shooting, specifically including: Obtain the forward direction of the panoramic camera during moving shooting according to the panoramic video, and obtain a lens image corresponding to the forward direction view; Obtain the forward direction view of the panoramic camera during shooting according to the lens image; Wherein, the panoramic video is an original spherical video.

3. The method according to claim 1, wherein Identify and obtain salient objects, symmetric objects, and sky objects based on the panoramic video frames and / or fisheye image frames, specifically including: Use a preset salient object detection and recognition algorithm to detect the panoramic video frames and / or fisheye image frames to obtain salient objects.

4. The method according to claim 3, wherein Detect the panoramic video frames and / or fisheye image frames to obtain salient objects, and specifically further include: When a preset type of salient object is detected in the currently detected panoramic video frame and / or fisheye image frame, set the object with the maximum saliency value as the current salient object of the currently detected panoramic video frame and / or fisheye image frame.

5. The method according to claim 1, characterized in that, Identify and obtain salient objects, symmetric objects, and sky objects based on the panoramic video frames and / or fisheye image frames, specifically including: Obtain the timestamps of the panoramic video frames and / or fisheye image frames according to the panoramic video frames and / or fisheye image frames; Obtain the first rotation matrix of the panoramic video frames and / or fisheye image frames according to the timestamps of the panoramic video frames and / or fisheye image frames; Render the panoramic video frames and / or fisheye image frames according to the first rotation matrix to obtain an image in the vertically upward direction of the panoramic video frames and / or fisheye image frames; Use a preset symmetric object detection and recognition algorithm to detect the image in the vertically upward direction of the panoramic video frames and / or fisheye image frames to obtain symmetric objects.

6. The method according to claim 1, wherein Identify and obtain salient objects, symmetric objects, and sky objects based on the panoramic video frames and / or fisheye image frames, specifically including: Obtain the timestamps of the panoramic video frame and / or the dual fisheye image frame based on the panoramic video frame and / or the dual fisheye image frame; Obtain the first rotation matrix of the panoramic video frame and / or the dual fisheye image frame according to the timestamps of the panoramic video frame and / or the dual fisheye image frame; Decompose the travel direction matrix of the panoramic video frame and / or the dual fisheye image frame into the first Euler angles (Yaw, Pitch, Roll respectively), and set the Pitch angle to PI / 2 to obtain the second Euler angles; Convert the second Euler angles to obtain the second rotation matrix; Render the panoramic video frame and / or the dual fisheye image frame according to the second rotation matrix to obtain the image in the vertically upward direction of the panoramic video frame and / or the dual fisheye image frame; Use a preset sky target detection and recognition algorithm to detect the image in the vertically upward direction of the panoramic video frame and / or the dual fisheye image frame to obtain sky targets.

7. The method according to claim 1, characterized in that, Use a preset target tracking algorithm to track the salient target, the symmetry target, and the sky target to obtain the perspectives where the salient target, the symmetry target, and the sky target are located, specifically including: When the current salient target, the current symmetry target, and the current sky target are detected in the currently detected panoramic video frame and / or dual fisheye image frame, use a preset target tracking algorithm to sequentially track the current salient target, the current symmetry target, and the current sky target in subsequent panoramic video frames and / or dual fisheye image frames, and obtain the perspectives where the current salient target, the current symmetry target, and the current sky target are located.

8. The method according to claim 7, wherein After obtaining the perspectives where the current salient target, the current symmetry target, and the current sky target are located, it further includes: Detect the stop tracking events of the current salient target, the current symmetry target, and the current sky target respectively. When the stop tracking events of the current salient target, the current symmetry target, and the current sky target are detected, respectively jump to the step of frame extraction operation on the panoramic video, and continue to respectively identify and obtain the salient target, the symmetry target, and the sky target.

9. The method according to claim 7, wherein After using a preset target tracking algorithm to sequentially track the current salient target, the current symmetry target, and the current sky target in subsequent panoramic video frames and / or dual fisheye image frames and obtaining the perspectives where the current salient target, the current symmetry target, and the current sky target are located, it specifically further includes: Respectively obtain the center coordinates of the tracking frames of the current salient target, the current symmetry target, and the current sky target in the currently detected panoramic video frame and / or dual fisheye image frame, and calculate the spherical viewpoint coordinates of the current salient target, the current symmetry target, and the current sky target according to the center coordinates; According to the spherical viewpoint coordinates, respectively obtain the lens images corresponding to the perspectives where the salient target, the symmetry target, and the sky target are located; Generate a significant object perspective segment, a symmetry object perspective segment, and a sky object perspective segment respectively according to the lens images corresponding to the perspectives where the significant object, the symmetry object, and the sky object are located.

10. The method according to claim 8, characterized in that, The stop tracking event is the loss of the current significant object, the current symmetry object, and the current sky object, or the area of the tracking frame is less than a preset area.

11. The method according to any one of claims 1 to 10, characterized in that, Clip the panoramic video according to the forward direction perspective, the perspective where the significant object is located, the perspective where the symmetry object is located, and the perspective where the sky object is located to generate a target video corresponding to the panoramic video, specifically including: According to the duration of the panoramic video; And / or, the number and corresponding duration of the appearances of the significant object perspective segment and / or the sky object perspective segment and / or the symmetry object perspective segment; And / or, the relationship between the perspective where the significant object is located and / or the perspective where the symmetry object is located and / or the perspective where the sky object is located and the forward direction perspective; Set the playback number and corresponding playback speed of the significant object perspective segment and / or the sky object perspective segment and / or the symmetry object perspective segment, and automatically clip the significant object perspective segment and / or the sky object perspective segment and / or the symmetry object perspective segment to generate a target video corresponding to the panoramic video; Wherein, the target video is a single perspective video or a planar video.

12. The method according to claim 11, characterized in that, Before setting the playback number and corresponding playback speed of the significant object perspective segment and / or the sky object perspective segment and / or the symmetry object perspective segment, it further includes: Set the lens rotation mode and / or corresponding playback speed according to the duration of the panoramic video, the number and duration of the appearances of the significant object perspective segment and / or the sky object perspective segment and / or the symmetry object perspective segment.

13. A panoramic video editing device, characterized in that, The device includes: An acquisition module: used to acquire the panoramic video captured by the panoramic camera and record the forward direction perspective of the panoramic camera during shooting; A frame extraction module: used to perform frame extraction on the acquired panoramic video to obtain corresponding panoramic video frames and / or fisheye image frames; An identification module: used to identify and acquire significant objects, symmetry objects, and sky objects according to the panoramic video frames and / or fisheye image frames; A tracking module: used to track the significant object, the symmetry object, and the sky object by using a preset target tracking algorithm to obtain the perspectives where the significant object, the symmetry object, and the sky object are located; A processing module: used to traverse the panoramic video and clip the panoramic video according to the forward direction perspective, the perspective where the significant object is located, the perspective where the symmetry object is located, and the perspective where the sky object is located to generate a target video corresponding to the panoramic video, including: selecting key points of the significant object perspective segment, the sky object perspective segment, and the symmetry object perspective segment, combining the key points into a smooth curve according to a preset rule, and then interpolating any point between the key points according to a preset rule according to the curve to obtain the target video.

14. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

15. A panoramic video editing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 12 are implemented.

Images