Time slicing system and method for single-view-angle dynamic video

Abstract

The invention relates to a time slicing system and method for a single-view dynamic video, and the system comprises an athlete recognition and segmentation module which carries out the tracking and recognition of an athlete, and segmenting the pixels of an area related to the motion of the athlete in each frame of image from a video; the background panorama construction module automatically stitches the scene moved by the camera, and reconstructs a pure background panorama without athletes in combination with the result of the athlete identification and segmentation module; the past action positioning module performs feature matching on each frame in the video and the background panorama, and determines the accurate position of the frame in the panorama and the position of the past action in the frame; and the alpha-mixing module is used for fusing the past actions of the athlete into the current frame through alpha-mixing according to the position information output by the past action positioning module to obtain a new video with key action ghost shadows reserved. Compared with the prior art, high-efficiency and high-precision time slice processing can be realized under the condition of a dynamic mirror moving camera.

Description

Technical Field

[0001] The present invention relates to the technical field of video data processing, and in particular to a time slicing system and method for single-view dynamic video. Background Art

[0002] Time slicing is an innovative technique that records individual moments in a time period and synthesizes these moments into a still frame. This process is not just simple photography or video recording, but a highly precise technique that captures the subtle changes in time and allows these changes to be clearly presented to the audience. Time slicing can be applied to many different scenes, such as city streets, sports games, buildings, natural scenery, etc. This technology allows viewers to see many different scenes in one photo, thus gaining a new perspective.

[0003] For example, in sports competitions, in order to better showcase the wonderful performances of athletes, especially when explaining and analyzing the game, it is particularly important to reproduce the athletes' movements at different time points in the same video. This technology allows viewers to see the details of the athletes' movements and technical points more intuitively, helping them to understand the process of the game and the athletes' performance more deeply. Whether in the broadcasting of sports events or in the production of special effects and teaching videos, this technology plays a vital role.

[0004] Existing methods usually rely on fixed cameras to synchronously shoot the athletes' movements, capturing the athletes' postures and movements at different times. Subsequently, professional editing software is used to finely edit these images at different time points. Specifically, the editor will accurately edit the outlines of the athletes at each moment, and paste these outlines one by one into the same frame, so as to achieve the effect of superimposing the athletes' movements at different moments into one image. Although this method can achieve the desired visual effect, it usually takes a lot of time and manpower, especially when dealing with complex movements or requiring high precision. In addition, since the action outlines at each moment need to be manually adjusted and aligned, there may be certain errors in the whole process, affecting the final display effect. Therefore, this traditional method often requires a high level of technology and rich experience in actual operation.

[0005] In summary, the existing slicing technology is manually completed, time-consuming and labor-intensive, and often requires 1 day or even several days to complete. In addition, the existing slicing technology relies on a fixed camera and cannot slice videos with moving cameras, nor can it achieve panoramic display. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a time slicing system and method for single-view dynamic video, which can realize efficient and high-precision time slicing processing under the condition of a dynamic camera.

[0007] The object of the present invention can be achieved by the following technical scheme: a time slicing system for single-view dynamic video, comprising a player identification and segmentation module, a background panorama construction module, a past action positioning module and an α-blending module, wherein the player identification and segmentation module is used to track and identify the player, and segment the pixels of the area related to the player's action in each frame of the video;

[0008] The background panorama construction module is used to automatically stitch the scene where the camera has moved, and combine the results of the athlete recognition and segmentation module to reconstruct a panorama of the area seen by the dynamic camera without athletes and pure background;

[0009] The past action positioning module is used to perform feature matching between each frame in the video and the background panorama, and determine the exact position of the frame in the panorama, and the position of the past action in the frame;

[0010] The α-blending module integrates the athlete's past actions into the current frame through α-blending according to the position information output by the past action positioning module, thereby obtaining a new video that retains the residual image of the key action.

[0011] Furthermore, the athlete recognition and segmentation module specifically uses a SAM (Segment Anything) model to perform fully automatic athlete contour detection, tracking and segmentation, so as to extract character masks corresponding to the athletes in all frames from the video.

[0012] Furthermore, the working process of the background panorama construction module includes:

[0013] Filter multiple key frames from the video;

[0014] Detect the feature points of the key frames, perform feature matching between every two key frames, and determine the key frame camera parameters;

[0015] Combining the character mask and key frame camera parameters, the panorama is stitched to obtain a continuous panorama without athletes.

[0016] Furthermore, the background panorama construction module specifically uses a SIFT (Scale-Invariant Feature Transform) algorithm to detect feature points of key frames.

[0017] Furthermore, the background panorama construction module specifically uses OpenCV to estimate homography transformation and equivalent rotation and camera parameters to perform Bundle Adjustment camera parameter optimization;

[0018] The human mask is then used to remove the human area, and then OpenCV is used to detect seams and stitch them into a continuous panoramic image without athletes.

[0019] Furthermore, the working process of the past action positioning module includes:

[0020] Extract the feature points of each frame during the camera movement, and then match the feature points in the panorama with the feature points in each frame.

[0021] Using the feature points matched between the panorama and each frame, calculate the homography transformation between the frame and the panorama;

[0022] For any past frame, its homography is used to map it to the panorama, and then the inverse of the homography of the current frame is used to map the panorama to the current frame, that is, the mapping of the past frame to the current frame in the dynamic scene is obtained.

[0023] Furthermore, the past action positioning module specifically uses SIFT or SuperPoint method to extract feature points of each frame of the image during the camera movement process.

[0024] Furthermore, the past action positioning module specifically uses a nearest neighbor matching method or a LightGlue matching method to match feature points in the panoramic image with feature points in each frame of the image.

[0025] Further, the working process of the alpha-blending module includes: using the mapping of all frames to the background panorama, projecting the athlete in the past key frame into the current frame, blending the image using a specified opacity so that the athlete's actions in the past key frame rate are displayed in the current frame;

[0026] Project the key actions into the panorama, connect it to the end of the original video, and display the panorama by gradually zooming in the perspective.

[0027] A time slicing method for a single-view dynamic video comprises the following steps:

[0028] S1. For single-view dynamic videos, on the one hand, through athlete recognition segmentation, the character mask corresponding to the athlete in each frame image is segmented from the video;

[0029] On the other hand, key frames are selected from the video, and key frame feature detection and matching are performed to obtain key frame camera parameters;

[0030] S2, combining the person mask and the key frame camera parameters, stitching the panorama to obtain a continuous background panorama without the athlete;

[0031] S3, performing feature detection on each frame of the video, and matching feature points in the background panorama with feature points in each frame of the video, and using the matched feature points to calculate the homography between the frame of the video and the panorama;

[0032] S4, for any past frame, use its homography transformation to map it to the panorama, and then use the inverse of the homography transformation of the current frame to map the panorama to the current frame, so as to obtain the mapping of the past frame to the current frame in the dynamic scene;

[0033] S5. Using the mapping obtained in step S4, the athlete in the past key frame is projected into the current frame, the image is blended using the specified opacity, and the key frame action is projected into the panorama, so that the athlete's action in the past key frame rate is displayed in the current frame, and a panorama containing the key frame action is obtained.

[0034] Compared with the prior art, the present invention has the following advantages:

[0035] The present invention sets an athlete recognition segmentation module, a background panorama construction module, a past action positioning module and an α-blending module, uses the athlete recognition segmentation module to track and identify the athlete, and segments the pixels of the area related to the athlete's action in each frame of the video; uses the background panorama construction module to automatically stitch the scene moved by the camera, and combines the result of the athlete recognition segmentation module to reconstruct a dynamic camera panorama of the area seen by the pure background without the athlete; uses the past action positioning module to match the features of each frame in the video with the background panorama, determines the exact position of the frame in the panorama, and the position of the past action in the frame; uses the α-blending module to integrate the athlete's past action into the current frame through α-blending according to the position information output by the past action positioning module, and obtains a new video that retains the residual image of the key action. Thus, a time slicing scheme under the dynamic camera movement is realized, which can recover the athlete's action at the past moment from the moving lens and reproduce it in the current picture, and can construct a panorama containing the key action, without manual cutting, and without fixing the camera position, and can perform efficient and high-precision time slicing on the dynamic camera, and can also realize panoramic playback.

[0036] In the present invention, the athlete recognition and segmentation module performs athlete contour detection, tracking and segmentation on the video to obtain the character masks corresponding to the athletes in all frames in the video, and then uses the background panoramic image construction module to perform feature detection and matching on the key frames in the video to obtain the key frame camera parameters, and then combines the character masks to construct a panoramic background image without athletes.

[0037] In the present invention, the past action positioning module extracts the feature points of each frame in the video and matches the feature points of each frame with the feature points of the panoramic image, so as to accurately obtain the mapping position of each frame in the panoramic image and the position of the past action in the frame.

[0038] In the present invention, the α-blending module uses the mapping output by the past action positioning module to project the athletes in the past key frames into the current frame, and uses the specified opacity to blend the images so that the current frame shows the athletes' actions in the past key frames, and also projects these key actions into the panoramic image, which is connected to the end of the original video and is displayed in a panoramic manner by gradually zooming in the viewing angle. In this way, any past action of the athlete can be repeatedly exposed in the current frame from the video, and an edited video containing the athlete's key actions can be automatically obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the system structure of the present invention;

[0040] Figure 2 It is a schematic diagram of the method flow of the present invention;

[0041] Figure 3 It is a schematic diagram of the application process of the embodiment;

[0042] Explanation of the marks in the figure: 1. Athlete recognition and segmentation module, 2. Background panorama construction module, 3. Past action positioning module, 4. α-blending module. DETAILED DESCRIPTION

[0043] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] Example

[0045] like Figure 1 As shown, a time slicing system for a single-view dynamic video includes a player identification and segmentation module 1, a background panorama construction module 2, a past action positioning module 3 and an α-blending module 4, wherein the player identification and segmentation module 1 is used to track and identify the player, and segment the pixels of the area related to the player's action in each frame of the video;

[0046] The background panorama construction module 2 is used to automatically stitch the scene where the camera has moved, and combine the results of the athlete recognition and segmentation module 1 to reconstruct a panorama of the area seen by the dynamic camera without athletes and pure background;

[0047] The past action location module 3 is used to match the features of each frame in the video with the background panorama, and determine the exact position of the frame in the panorama, as well as the position of the past action in the frame;

[0048] The α-blending module 4 integrates the athlete's past actions into the current frame through α-blending according to the position information output by the past action positioning module 3, and obtains a new video that retains the residual image of the key action.

[0049] Based on the above system, a time slicing method for single-view dynamic video is implemented, such as Figure 2 As shown, including:

[0050] S1. For single-view dynamic videos, on the one hand, through athlete recognition segmentation, the character mask corresponding to the athlete in each frame image is segmented from the video;

[0051] On the other hand, key frames are selected from the video, and key frame feature detection and matching are performed to obtain key frame camera parameters;

[0052] S2, combining the person mask and the key frame camera parameters, stitching the panorama to obtain a continuous background panorama without the athlete;

[0053] S3, performing feature detection on each frame of the video, and matching feature points in the background panorama with feature points in each frame of the video, and using the matched feature points to calculate the homography between the frame of the video and the panorama;

[0054] S4, for any past frame, use its homography transformation to map it to the panorama, and then use the inverse of the homography transformation of the current frame to map the panorama to the current frame, so as to obtain the mapping of the past frame to the current frame in the dynamic scene;

[0055] S5. Using the mapping obtained in step S4, the athlete in the past key frame is projected into the current frame, the image is blended using the specified opacity, and the key frame action is projected into the panorama, so that the athlete's action in the past key frame rate is displayed in the current frame, and a panorama containing the key frame action is obtained.

[0056] This embodiment applies the above solution. First, the athlete recognition and segmentation module, the background panorama construction module, the past action positioning module and the α-blending module are built. The collaborative working process of each module is as follows: Figure 3 As shown:

[0057] 1. Athlete Recognition and Segmentation Module

[0058] This module uses SAM (Segment Anything) to perform fully automatic athlete contour detection, tracking and segmentation. The user clicks on the athlete of interest in the first frame, and the SAM model is used to perform subsequent tracking and automatic segmentation to obtain the character mask of the athlete in all subsequent frames.

[0059] 2. Background Panorama Construction Module

[0060] First, multiple key frames are uniformly generated or manually selected from the video, and the SIFT (Scale-Invariant Feature Transform) algorithm is used to detect the feature points of these key frames and perform pairwise matching between every two frames.

[0061] Then, OpenCV is used to estimate the homography transformation and equivalent rotation and camera parameters, and the Bundle Adjustment camera parameter optimization is performed. After the athlete mask is used to remove the human area, OpenCV is used to detect seams and stitch them into a continuous panoramic image without the athletes.

[0062] 3. Past Action Localization Module

[0063] This module first uses SIFT (Scale-Invariant Feature Transform) or SuperPoint method to extract the feature points of each frame during the camera movement, and then uses the nearest neighbor matching method or LightGlue matching method to match the feature points in the panorama with the feature points in each frame.

[0064] The feature points matched between the panorama and each frame are used to calculate the homography between the frame and the panorama. For any past frame, its homography is used to map it to the panorama, and then the inverse of the homography of the current frame is used to map the panorama to the current frame. In this way, the mapping of the past frame to the current frame in a dynamic scene can be obtained.

[0065] 4. α-Blending Module

[0066] This module uses the mapping output by the past action positioning module to project the athletes in the past key frames into the current frame, and mixes the images with the specified opacity so that the current frame shows the athletes' actions in the past key frames. Finally, these key actions are projected into the panorama, which is connected to the end of the original video to display the panorama in a way that the camera gradually zooms in on the view.

[0067] It should be noted that, when the present solution is actually applied, other existing feature detection and matching methods, as well as human body recognition and segmentation methods may be used.

[0068] In summary, this solution is based on a single-view dynamic camera video, and can obtain a reconstructed panoramic image containing any past moment of the athlete. By performing multiple exposures and time slicing on the athlete, any past action of the athlete is repeatedly exposed in the current frame from the video, and an edited video containing the athlete's key actions is automatically obtained. This solution is applied to sports event broadcasting technology, which can support dynamic camera movements, does not require the camera to be fixed, can combine character masks to construct a panoramic background image without athletes, realize time slicing under dynamic camera conditions, and obtain a panoramic image containing the athlete's key actions. It has the ability to accurately reconstruct sports panoramas, segment player actions, and locate key positions, and can mix key actions into different moments of the original video for display.

Claims

1. A time slicing system for single-view dynamic video, characterized in that: The system comprises an athlete recognition and segmentation module (1), a background panorama construction module (2), a past action positioning module (3) and an α-blending module (4), wherein the athlete recognition and segmentation module (1) is used to track and identify athletes and segment pixels in each frame of the video that are related to the athlete's action. The background panorama construction module (2) is used to automatically stitch the scene where the camera has moved, and combine the results of the athlete recognition and segmentation module (1) to reconstruct a panorama of the area seen by the dynamic camera without athletes and pure background; The past action positioning module (3) is used to perform feature matching between each frame in the video and the background panorama, and determine the exact position of the frame in the panorama, and the position of the past action in the frame; The α-blending module (4) integrates the athlete's past movements into the current frame through α-blending according to the position information output by the past movement positioning module (3), thereby obtaining a new video that retains the residual image of the key movement.

2. The time slicing system for single-view dynamic video according to claim 1, characterized in that: The athlete recognition and segmentation module (1) specifically uses the SAM model to perform fully automatic athlete contour detection, tracking and segmentation, so as to extract character masks corresponding to the athletes in all frames from the video.

3. The time slicing system for single-view dynamic video according to claim 2, characterized in that: The working process of the background panorama construction module (2) includes: Filter multiple key frames from the video; Detect the feature points of the key frames, perform feature matching between every two key frames, and determine the key frame camera parameters; Combining the character mask and key frame camera parameters, the panorama is stitched to obtain a continuous panorama without athletes.

4. The time slicing system for single-view dynamic video according to claim 3, characterized in that: The background panorama construction module (2) specifically uses the SIFT algorithm to detect feature points of key frames.

5. The time slicing system for single-view dynamic video according to claim 3, characterized in that: The background panorama construction module (2) specifically uses OpenCV to estimate homography transformation and equivalent rotation and camera parameters to perform Bundle Adjustment camera parameter optimization; The human mask is then used to remove the human area, and then OpenCV is used to detect seams and stitch them into a continuous panoramic image without athletes.

6. The time slicing system for single-view dynamic video according to claim 1, characterized in that: The working process of the past action positioning module (3) includes: Extract the feature points of each frame during the camera movement, and then match the feature points in the panorama with the feature points in each frame. Using the feature points matched between the panorama and each frame, calculate the homography transformation between the frame and the panorama; For any past frame, its homography is used to map it to the panorama, and then the inverse of the homography of the current frame is used to map the panorama to the current frame, that is, the mapping of the past frame to the current frame in the dynamic scene is obtained.

7. The time slicing system for single-view dynamic video according to claim 6, characterized in that: The past action positioning module (3) specifically uses SIFT or SuperPoint method to extract feature points of each frame of the image during the camera movement process.

8. The time slicing system for single-view dynamic video according to claim 6, characterized in that: The past action positioning module (3) specifically uses the nearest neighbor matching method or the LightGlue matching method to match the feature points in the panoramic image with the feature points in each frame of the image.

9. The time slicing system for single-view dynamic video according to claim 6, characterized in that: The working process of the alpha-blending module (4) includes: using the mapping of all frames and the background panorama, projecting the athlete of the past key frame into the current frame, blending the image using a specified opacity, so that the current frame shows the athlete's action in the past key frame rate; Project the key actions into the panorama, connect it to the end of the original video, and display the panorama by gradually zooming in the perspective.

10. A time slicing method for a single-view dynamic video, applied to a time slicing system for a single-view dynamic video as claimed in claim 1, characterized in that: The following steps are involved: S1. For single-view dynamic videos, on the one hand, through athlete recognition segmentation, the character mask corresponding to the athlete in each frame image is segmented from the video; On the other hand, key frames are selected from the video, and key frame feature detection and matching are performed to obtain key frame camera parameters; S2, combining the person mask and the key frame camera parameters, stitching the panorama to obtain a continuous background panorama without the athlete; S3, performing feature detection on each frame of the video, and matching feature points in the background panorama with feature points in each frame of the video, and using the matched feature points to calculate the homography between the frame of the video and the panorama; S4. For any past frame, use its homography transformation to map it to the panorama, and then use the inverse of the homography transformation of the current frame to map the panorama to the current frame, so as to obtain the mapping of the past frame to the current frame in the dynamic scene; S5. Using the mapping obtained in step S4, the athlete in the past key frame is projected into the current frame, the image is blended using a specified opacity, and the key frame action is projected into the panorama, so that the athlete's action in the past key frame rate is displayed in the current frame, and a panorama containing the key frame action is obtained.

Images