Video special effect adding method and device and electronic equipment

Abstract

The invention provides a video special effect adding method and device and electronic equipment, and is applied to the technical field of video processing, and the method comprises the steps: constructing a free view angle model corresponding to a free view angle video; determining the position of a target object to which a special effect needs to be added in the free view angle video in the free view angle model, and adding the special effect to the position corresponding to the target object; performing full-view rendering on the free-view video in the free-view model; and generating a free view angle video added with the special effect based on the rendered video data of each view angle. According to the method, full-view rendering is carried out on the special effect in the free-view video, so that the rendered free-view video can ensure that the special effect presents a stable and natural effect in each view.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a method, device and electronic device for adding video special effects. Background Art

[0002] Video effects in related technologies include single-sided, double-sided, and stereoscopic effects. Single-sided effects apply special effects to a specific area or object in a video. These effects can include filters, dynamic effects, text effects, and more. Single-sided effects are typically used to modify and optimize a specific part of a video, making the content more vivid and engaging. Double-sided effects can simultaneously apply special effects to multiple areas or objects in a video, making the entire video more colorful. Stereoscopic effects can add visual depth and dynamics to a video, making the viewer feel as if they are actually there.

[0003] These special effects are typically applied to fixed-angle videos. These effects are typically applied to static frames, allowing creators to more easily control the placement, size, and movement of the effects. However, in free-angle videos, changes in the video angle can affect the presentation of the effects. Therefore, existing methods for adding special effects to fixed-angle videos are less effective for free-angle videos. Summary of the Invention

[0004] Embodiments of the present application provide a method, device, and electronic device for adding special effects to a video, so as to solve the problem of poor special effects adding effect in existing methods for adding special effects to a video.

[0005] In order to solve the above technical problems, this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for adding special effects to a video, the method comprising:

[0007] Constructing a free viewpoint model corresponding to the free viewpoint video;

[0008] Determining a position of a target object to which special effects need to be added in the free-viewpoint video in the free-viewpoint model, and adding the special effects to a position corresponding to the target object;

[0009] Performing full-view rendering on the free-view video in the free-view model;

[0010] A free-viewpoint video with the special effects added thereto is generated based on the rendered video data of each viewpoint.

[0011] Optionally, generating the free-viewpoint video with the special effects added based on the rendered video data of each perspective includes:

[0012] Match and fuse the rendered video data from each perspective to obtain the processed free-viewpoint video;

[0013] Projecting the processed free-viewpoint video onto a display screen;

[0014] receiving a viewing angle adjustment instruction input by a user, and performing frame insertion processing on the free viewing angle video in the display screen;

[0015] A free-viewpoint video with the special effects added thereto is obtained based on the free-viewpoint video after the frame insertion processing.

[0016] Optionally, the receiving of the viewing angle adjustment instruction input by the user and performing frame insertion processing on the free viewing angle video in the display screen includes:

[0017] Receive a viewing angle adjustment instruction input by a user;

[0018] Based on the perspective adjustment instruction, determining a plurality of video frame images in the free perspective video in the display screen whose perspectives need to be switched;

[0019] Based on the multiple video frame images, generating multiple intermediate frame images, wherein each intermediate frame image is generated based on every two adjacent video frame images in the multiple video frame images;

[0020] Calculating an insertion angle of each intermediate frame image in a plurality of intermediate frame images according to an angle between every two adjacent video frame images in the plurality of video frame images;

[0021] The plurality of intermediate frame images are synthesized with the plurality of video frame images based on an insertion angle of each of the plurality of intermediate frame images.

[0022] Optionally, the multiple video frame images include adjacent first video frame images and second video frame images, and the multiple intermediate frame images include target intermediate frame images generated based on the first video frame images and the second video frame images;

[0023] The step of generating a plurality of intermediate frame images based on the plurality of video frame images comprises:

[0024] Calculating a first eigenvalue of the special effect in the first video frame image and a second eigenvalue of the special effect in the second video frame image;

[0025] Obtaining an average eigenvalue based on the first eigenvalue and the second eigenvalue;

[0026] Based on the average feature value, a target intermediate frame image is generated.

[0027] Optionally, the multiple video frame images include adjacent first video frame images and second video frame images, and the multiple intermediate frame images include target intermediate frame images generated based on the first video frame images and the second video frame images;

[0028] The step of generating a plurality of intermediate frame images based on the plurality of video frame images comprises:

[0029] Determining a first position of the special effect in the first video frame image and a second position of the special effect in the second video frame image;

[0030] predicting a motion trajectory of the special effect according to the first position and the second position;

[0031] The target intermediate frame image is generated based on the motion trajectory of the special effect.

[0032] Optionally, determining a position of a target object to which special effects need to be added in the free-viewpoint video in the free-viewpoint model includes:

[0033] Obtaining positions of multiple cameras that shoot the free-viewpoint video;

[0034] Acquire video images captured by multiple cameras at the same time;

[0035] Performing pixel matching on the video images captured by each of the multiple cameras to calculate disparity values of corresponding pixels in the multiple video images;

[0036] Obtaining the distance between the target object and each camera in the free viewpoint video;

[0037] According to the positions of the multiple cameras, the disparity values of corresponding pixels in the multiple video images and the distance of the target object from each camera in the free perspective video, the position of the target object to which special effects need to be added in the free perspective video in the free perspective model is determined.

[0038] Optionally, performing full-view rendering on the free-view video in the free-view model includes:

[0039] The current viewing angle of the free viewing angle video is rendered with full computing power, the nearby viewing angles of the free viewing angle video are rendered with half computing power, and the distant viewing angle of the free viewing angle video is not rendered with computing power.

[0040] In a second aspect, an embodiment of the present application further provides a video special effects adding device, the video special effects adding device comprising:

[0041] A first construction module is used to construct a free viewpoint model corresponding to the free viewpoint video;

[0042] A first processing module is configured to determine a position of a target object in the free-viewpoint video to which special effects need to be added in the free-viewpoint model, and to add the special effects to a position corresponding to the target object;

[0043] A first rendering module, configured to perform full-view rendering on the free-view video in the free-view model;

[0044] The first generating module is used to generate a free-viewpoint video with the special effects added thereto based on the rendered video data of each viewpoint.

[0045] Optionally, the first generating module includes:

[0046] The first processing unit is configured to match and fuse the rendered video data of each perspective to obtain a processed free-perspective video;

[0047] A first projection unit, configured to project the processed free-viewpoint video onto a display screen;

[0048] a second processing unit, configured to receive a viewing angle adjustment instruction input by a user and perform frame insertion processing on the free viewing angle video in the display screen;

[0049] The first acquisition unit is configured to obtain the free-viewpoint video with the special effect added thereto based on the free-viewpoint video after the frame insertion process.

[0050] Optionally, the second processing unit includes:

[0051] A first receiving subunit, configured to receive a viewing angle adjustment instruction input by a user;

[0052] A first determining subunit is configured to determine, based on the viewing angle adjustment instruction, a plurality of video frame images in the free viewing angle video in the display screen whose viewing angles need to be switched;

[0053] A first generating subunit is configured to generate a plurality of intermediate frame images based on the plurality of video frame images, wherein each intermediate frame image is generated based on every two adjacent video frame images among the plurality of video frame images;

[0054] A first calculation subunit is configured to calculate an insertion angle of each of the plurality of intermediate frame images according to an angle between every two adjacent video frame images in the plurality of video frame images;

[0055] The first synthesis subunit is configured to synthesize the plurality of intermediate frame images with the plurality of video frame images based on an insertion angle of each of the plurality of intermediate frame images.

[0056] Optionally, the multiple video frame images include adjacent first video frame images and second video frame images, and the multiple intermediate frame images include target intermediate frame images generated based on the first video frame images and the second video frame images;

[0057] The first generating subunit is specifically configured to:

[0058] Calculating a first eigenvalue of the special effect in the first video frame image and a second eigenvalue of the special effect in the second video frame image;

[0059] Obtaining an average eigenvalue based on the first eigenvalue and the second eigenvalue;

[0060] Based on the average feature value, a target intermediate frame image is generated.

[0061] Optionally, the multiple video frame images include adjacent first video frame images and second video frame images, and the multiple intermediate frame images include target intermediate frame images generated based on the first video frame images and the second video frame images;

[0062] The first generating subunit is specifically configured to:

[0063] Determining a first position of the special effect in the first video frame image and a second position of the special effect in the second video frame image;

[0064] predicting a motion trajectory of the special effect according to the first position and the second position;

[0065] The target intermediate frame image is generated based on the motion trajectory of the special effect.

[0066] Optionally, the first processing module includes:

[0067] A first acquisition unit is used to acquire positions of multiple cameras shooting the free-viewpoint video;

[0068] A second acquisition unit is used to acquire video images shot by multiple cameras at the same time;

[0069] a first calculation unit, configured to perform pixel matching on the video images captured by each of the plurality of cameras, and calculate disparity values of corresponding pixels in the plurality of video images;

[0070] A first acquiring unit, configured to acquire the distance between the target object and each camera in the free-viewpoint video;

[0071] The first determination unit is used to determine the position of the target object to which special effects need to be added in the free perspective video in the free perspective model based on the positions of the multiple cameras, the disparity values of corresponding pixels in the multiple video images, and the distance of the target object from each camera in the free perspective video.

[0072] Optionally, performing full-view rendering on the free-view video in the free-view model includes:

[0073] The current viewing angle of the free viewing angle video is rendered with full computing power, the nearby viewing angles of the free viewing angle video are rendered with half computing power, and the distant viewing angle of the free viewing angle video is not rendered with computing power.

[0074] In a third aspect, an embodiment of the present application further provides an electronic device comprising a transceiver, a processor, a memory, and a computer program stored on the memory and executable on the processor, wherein the computer program implements the steps of the above-mentioned method for adding video special effects when executed by the processor.

[0075] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for adding video special effects are implemented.

[0076] In a fifth aspect, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the method for adding video special effects as described in the first aspect.

[0077] The method for adding special effects to a video in an embodiment of the present application includes constructing a free-viewpoint model corresponding to a free-viewpoint video; determining the position of a target object in the free-viewpoint video to which a special effect is to be added within the free-viewpoint model, and adding the special effect to the position corresponding to the target object; performing full-view rendering of the free-viewpoint video in the free-viewpoint model; and generating a free-viewpoint video with the special effect added based on the rendered video data from each viewpoint. By performing full-view rendering of the special effects in the free-viewpoint video, the method ensures that the rendered free-viewpoint video presents a stable and natural effect from all viewpoints. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0079] Figure 1 This is a flow chart of a method for adding special effects to a video provided by an embodiment of the present application;

[0080] Figure 2 is a schematic diagram of a free viewing angle model provided in an embodiment of the present application;

[0081] Figure 3 This is one of the display screens of the video special effects provided in the embodiment of the present application;

[0082] Figure 4 This is the second display screen of the video special effects provided in the embodiment of the present application;

[0083] Figure 5 This is the third display screen of the video special effects provided in the embodiment of the present application;

[0084] Figure 6 This is a structural diagram of a video special effects adding device provided in one embodiment of the present application;

[0085] Figure 7 This is a structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0086] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0087] The embodiment of the present application provides a method for adding special effects to a video. Figure 1 , Figure 1 This is a flow chart of the video adding method provided by the embodiment of the present application. Figure 1 As shown, the following steps are included:

[0088] Step 101: construct a free viewpoint model corresponding to the free viewpoint video;

[0089] In this step, the free viewpoint video is a video shot by multiple cameras, and the user can select the angle to watch the video through interactive operations, such as changing the viewpoint, rotating the camera, etc. Figure 2 , abstract the free viewpoint video into a spherical model (the free viewpoint model mentioned above), the coordinates of the sphere center are (x, y, z), and the plane of the sphere center is B. In this way, the video content can be presented on a spherical surface while keeping the position of the sphere center unchanged. Spherical vector The opposite direction is the main viewing angle of the two-dimensional video watched by the user. The opposite direction becomes When the direction is opposite to the main perspective, the presentation of the special effects (u, v, w) in the video will also change.

[0090] Step 102: Determine the position of the target object in the free viewpoint video to which special effects need to be added in the free viewpoint model, and add the special effects to the position corresponding to the target object;

[0091] In this step, the distance between the target object and the camera measured by binocular cameras, structured light and other technologies can be used to determine the position of the target object in the free perspective model, or a computer vision algorithm can be used to determine the position of the target object in the free perspective model. After determining the position of the target object in the free perspective model, the special effect is added to the position corresponding to the target object. If the special effect needs to move with the target object in the video, tracking and masking technology can be used to ensure that the special effect can accurately follow the motion trajectory of the target object to move, so as to keep the center point of the special effect and the center point of the free perspective model aligned, so that the special effect is integrated with the video content.

[0092] Step 103: Perform full-view rendering on the free-view video in the free-view model;

[0093] In ordinary videos, considering that viewers can only watch videos from fixed angles, the special effects in the video only need to be rendered and optimized in some perspectives. In free-perspective videos, the special effects need to be rendered and optimized in all perspectives, so that the special effects can remain consistent in all perspectives, allowing viewers to get a more real and realistic experience when watching videos from different angles.

[0094] Step 104: Generate a free-viewpoint video with the special effects added thereto based on the rendered video data of each viewpoint.

[0095] In this step, the video data of different perspectives are matched and fused. Taking into account the consistency between the video data of different perspectives, the video data of each rendered video are synthesized to obtain a free-perspective video with special effects added.

[0096] In one implementation, a free-viewpoint model corresponding to the free-viewpoint video is first constructed. The position of the target object to which special effects are to be added is then determined within the free-viewpoint model. After the special effects are added to the corresponding position of the target object, the free-viewpoint video of the free-viewpoint model is fully rendered. Taking into account the consistency of video data from different perspectives, the rendered video data from each perspective is matched and fused to produce the free-viewpoint video with the added special effects. This implementation ensures that the rendered free-viewpoint video, by fully rendering the special effects in the free-viewpoint video, presents a stable and natural effect from all perspectives.

[0097] Optionally, generating the free-viewpoint video with the special effects added based on the rendered video data of each perspective includes:

[0098] Match and fuse the rendered video data from each perspective to obtain the processed free-viewpoint video;

[0099] Projecting the processed free-viewpoint video onto a display screen;

[0100] receiving a viewing angle adjustment instruction input by a user, and performing frame insertion processing on the free viewing angle video in the display screen;

[0101] A free-viewpoint video with the special effects added thereto is obtained based on the free-viewpoint video after the frame insertion processing.

[0102] In one implementation, the projection and synthesis of special effects in free-viewpoint videos requires consideration of the matching and fusion of multi-view data, as well as consistency across different perspectives, to ensure a stable and natural effect across all viewing angles. In contrast, adding special effects to regular videos is relatively simple and can be performed directly on the video's two-dimensional image, without the need for multi-viewpoint data fusion and matching.

[0103] The projection of special effects refers to the projection of special effects on the two-dimensional display interface viewed by users. The up, down, left, and right movements during free-viewing angles will cause the content of the two-dimensional display interface to switch between different cameras, resulting in non-linear switching in the projection and synthesis of special effects. Incorporating the non-linear switching generated by switching between different cameras into the projection and synthesis of special effects is the key to ensuring that special effects present stable and natural effects from all perspectives.

[0104] Because free-viewpoint video uses a circumferential camera system, users can freely choose different viewing angles to watch the video later. When the user needs to adjust the viewing angle, the system receives input from the user to adjust the viewing angle. For example, if 24 cameras are shooting a video in a circle, the video will experience a 10-15 degree single-frame perspective change when switching perspectives. To make the perspective switching process smoother, interpolation is required for the free-viewpoint video.

[0105] In terms of interpolation methods, standard video interpolation primarily involves inserting new frames into the original video footage to improve animation smoothness and continuity. This is typically done by interpolating existing 2D images to generate new frames. However, interpolation for videos with special effects often requires processing of the special effects, considering their interaction and integration. In terms of interpolation complexity, standard video interpolation is relatively simple, typically using linear interpolation or other simple interpolation algorithms when processing 2D images. Because free-viewpoint videos with special effects often involve more complex 3D scenes and animations, interpolation requires more computing resources and algorithmic support to ensure a natural and realistic effect. In terms of interpolation effects, standard video interpolation primarily aims to enhance animation smoothness and continuity, with greater emphasis placed on frame-to-frame transitions when processing motion transitions and detailed animations. When interpolating free-viewpoint videos with special effects, factors such as perspective changes and lighting effects must be considered to ensure consistency between the interpolated effect and the original.

[0106] In this embodiment, when the viewing angle of the free-viewpoint video with special effects is switched, interpolation processing is performed on the free-viewpoint video, which can make the perspective switching smoother, reduce sudden jumps, and improve the viewing experience.

[0107] Optionally, the receiving of the viewing angle adjustment instruction input by the user and performing frame insertion processing on the free viewing angle video in the display screen includes:

[0108] Receive a viewing angle adjustment instruction input by a user;

[0109] Based on the perspective adjustment instruction, determining a plurality of video frame images in the free perspective video in the display screen whose perspectives need to be switched;

[0110] Based on the multiple video frame images, generating multiple intermediate frame images, wherein each intermediate frame image is generated based on every two adjacent video frame images in the multiple video frame images;

[0111] Calculating an insertion angle of each intermediate frame image in a plurality of intermediate frame images according to an angle between every two adjacent video frame images in the plurality of video frame images;

[0112] The plurality of intermediate frame images are synthesized with the plurality of video frame images based on an insertion angle of each of the plurality of intermediate frame images.

[0113] In one embodiment, the specific process for interpolating frames in a free-viewpoint video displayed on a screen is as follows: First, based on the user's perspective adjustment instruction, multiple video frames in the free-viewpoint video are determined for which perspectives to switch. These video frames are typically located at the position in the free-viewpoint video where the user selects the perspective to switch. An intermediate frame is generated for every two adjacent video frames in the multiple video frames. These intermediate frames smoothly transition the user's perspective, making the switching more natural.

[0114] An insertion angle for each of the multiple intermediate frames is calculated based on the angle between each two adjacent video frames in the multiple video frames. For example, when the angles between the two adjacent video frames are 20° and 40°, respectively, the insertion angle of the generated intermediate frame is 30°. After determining the insertion angle for each of the multiple intermediate frames, the multiple intermediate frames are combined with the original multiple video frames.

[0115] In this embodiment, interpolation can make the switching process of existing multi-view images smoother. For example, in a video shooting method with 24 cameras in a circle, the minimum angle for user switching is 15 degrees. When the interpolation method of inserting one frame is applied, the user's switching angle will become 7.5 degrees; when the interpolation method of inserting two frames is applied, the user's switching angle will become 5 degrees.

[0116] Optionally, the multiple video frame images include adjacent first video frame images and second video frame images, and the multiple intermediate frame images include target intermediate frame images generated based on the first video frame images and the second video frame images;

[0117] The step of generating a plurality of intermediate frame images based on the plurality of video frame images comprises:

[0118] Calculating a first eigenvalue of the special effect in the first video frame image and a second eigenvalue of the special effect in the second video frame image;

[0119] Obtaining an average eigenvalue based on the first eigenvalue and the second eigenvalue;

[0120] Based on the average feature value, a target intermediate frame image is generated.

[0121] In one embodiment, for video interpolation including special effects, the generated intermediate frame image needs to take into account the interaction and combination with the special effects. Taking the adjacent first video frame image and the second video frame image in multiple video frame images as an example, the edge feature extraction algorithm can be used to extract the key features of the special effects in the first video frame image, calculate the first eigenvalue of the special effects in the first video frame image, and extract the key features of the special effects in the second video frame image, and calculate the second eigenvalue of the special effects in the second video frame image. Furthermore, the average eigenvalue of the first eigenvalue and the second eigenvalue is calculated to generate a target intermediate frame image, thereby ensuring that the special effects in the target intermediate frame image are consistent with the special effects in the first video frame image and the second video frame image.

[0122] Optionally, the multiple video frame images include adjacent first video frame images and second video frame images, and the multiple intermediate frame images include target intermediate frame images generated based on the first video frame images and the second video frame images;

[0123] The step of generating a plurality of intermediate frame images based on the plurality of video frame images comprises:

[0124] Determining a first position of the special effect in the first video frame image and a second position of the special effect in the second video frame image;

[0125] predicting a motion trajectory of the special effect according to the first position and the second position;

[0126] The target intermediate frame image is generated based on the motion trajectory of the special effect.

[0127] In one embodiment, for video interpolation including special effects, the generated intermediate frame images need to take into account the interaction and integration with the special effects. Taking the first and second adjacent video frame images in multiple video frame images as an example, a dynamic motion model of the special effect can be constructed. Based on the first position of the special effect in the first video frame and the second position in the second video frame, the motion trajectory of the special effect is predicted. Based on the predicted motion trajectory of the special effect, the position of the special effect in the target intermediate frame image is determined, and the target intermediate frame image is generated, thereby ensuring the stability and naturalness of the presentation of the special effect when the perspective is switched in the display screen of the free-viewpoint video.

[0128] Optionally, determining a position of a target object to which special effects need to be added in the free-viewpoint video in the free-viewpoint model includes:

[0129] Obtaining positions of multiple cameras that shoot the free-viewpoint video;

[0130] Acquire video images captured by multiple cameras at the same time;

[0131] Performing pixel matching on the video images captured by each of the multiple cameras to calculate disparity values of corresponding pixels in the multiple video images;

[0132] Obtaining the distance between the target object and each camera in the free viewpoint video;

[0133] According to the positions of the multiple cameras, the disparity values of corresponding pixels in the multiple video images and the distance of the target object from each camera in the free perspective video, the position of the target object to which special effects need to be added in the free perspective video in the free perspective model is determined.

[0134] In one embodiment, in order to ensure that special effects can be accurately added to the target object, the position of the target object needs to be accurately determined. A method for obtaining stereoscopic depth information of multi-view video can be used. Specifically: first, camera calibration is performed to obtain the camera's internal and external parameters to ensure the accuracy of subsequent calculations; video images taken at the same time by multiple cameras shooting free-view videos are obtained, the video images are preprocessed, feature points, edges and other information are extracted, and pixel-level disparity matching is performed to calculate the disparity values of corresponding pixels in multiple video images; based on the position of the target object in each camera's view and the known camera position information, the distance between the target object and each camera can be calculated, which can be achieved through methods such as triangulation; the depth information of the target object in the current frame and nearby frames of multiple viewpoints is obtained to assist in completing the positioning of the target object.

[0135] Then, based on the positions of multiple cameras, the angle differences between different cameras, the disparity values of corresponding pixels in multiple video images, and the distance of the target object from each camera in the free-viewpoint video, the position of the target object in the free-viewpoint model is determined so that special effects can be accurately added to the target object.

[0136] See also Figure 3 , which shows the special effects display. The little person in the picture is the target object, and there are three special effects types in the upper right corner of the picture. After determining the position of the target object, you can accurately add, move, and delete special effects. Specifically:

[0137] (1) The process of adding special effects: see Figure 4 , after the user clicks on the target object to which special effects need to be added in the video, he drags the special effects onto the target object in the special effects editing screen to complete the special effects addition;

[0138] (2) Special effects modification process: see Figure 5 When the user feels that there is a problem with the position of the special effect added, the user clicks on the target object, and the relevant special effect can be highlighted and moved. The user can move and scale the special effect from different perspectives to confirm the position of the special effect;

[0139] (3) Special effects deletion process: When the user needs to delete a special effect, the user needs to select the target object in the video and click the special effects deletion button.

[0140] Optionally, performing full-view rendering on the free-view video in the free-view model includes:

[0141] The current viewing angle of the free viewing angle video is rendered with full computing power, the nearby viewing angles of the free viewing angle video are rendered with half computing power, and the distant viewing angle of the free viewing angle video is not rendered with computing power.

[0142] In one embodiment, since full-perspective special effects rendering and optimization consume a lot of computing power, the rendering processing strength of each perspective in the free-perspective video needs to be differentiated. Specifically: full-computing-power rendering processing is performed on the part that can be seen from the user's perspective, that is, all computing resources and algorithms are used for high-quality rendering to ensure the image quality and details of this perspective; half-computing-power rendering processing is performed on the part that the user's perspective will see, that is, partial computing resources and algorithms are used for rendering to reduce computing costs while maintaining a certain image quality; no computing-power rendering processing is performed on the part that cannot be seen from the user's perspective, that is, low-resolution or simplified rendering methods may be used directly to save computing resources and speed up processing.

[0143] In this implementation, computing resources can be allocated based on the importance and distance of the viewpoint to improve efficiency while ensuring video quality. By adopting different rendering processing methods for different viewpoints, more flexible and efficient rendering management can be achieved in free-viewpoint videos.

[0144] See also Figure 6 , Figure 6 This is a structural diagram of a video adding device provided by an embodiment of the present application. Figure 6 As shown, the video adding device 600 includes:

[0145] A first construction module 601 is used to construct a free viewpoint model corresponding to the free viewpoint video;

[0146] A first processing module 602 is configured to determine a position of a target object in the free-viewpoint video to which special effects need to be added in the free-viewpoint model, and to add the special effects to a position corresponding to the target object;

[0147] A first rendering module 603 is configured to perform full-view rendering on the free-view video in the free-view model;

[0148] The first generating module 604 is configured to generate a free-viewpoint video with the special effects added thereto based on the rendered video data of each viewpoint.

[0149] Optionally, the first generating module includes:

[0150] The first processing unit is configured to match and fuse the rendered video data of each perspective to obtain a processed free-perspective video;

[0151] A first projection unit, configured to project the processed free-viewpoint video onto a display screen;

[0152] a second processing unit, configured to receive a viewing angle adjustment instruction input by a user and perform frame insertion processing on the free viewing angle video in the display screen;

[0153] The first acquisition unit is configured to obtain the free-viewpoint video with the special effect added thereto based on the free-viewpoint video after the frame insertion process.

[0154] Optionally, the second processing unit includes:

[0155] A first receiving subunit, configured to receive a viewing angle adjustment instruction input by a user;

[0156] A first determining subunit is configured to determine, based on the viewing angle adjustment instruction, a plurality of video frame images in the free viewing angle video in the display screen whose viewing angles need to be switched;

[0157] A first generating subunit is configured to generate a plurality of intermediate frame images based on the plurality of video frame images, wherein each intermediate frame image is generated based on every two adjacent video frame images among the plurality of video frame images;

[0158] A first calculation subunit is configured to calculate an insertion angle of each of the plurality of intermediate frame images according to an angle between every two adjacent video frame images in the plurality of video frame images;

[0159] The first synthesis subunit is configured to synthesize the plurality of intermediate frame images with the plurality of video frame images based on an insertion angle of each of the plurality of intermediate frame images.

[0160] Optionally, the multiple video frame images include adjacent first video frame images and second video frame images, and the multiple intermediate frame images include target intermediate frame images generated based on the first video frame images and the second video frame images;

[0161] The first generating subunit is specifically configured to:

[0162] Calculating a first eigenvalue of the special effect in the first video frame image and a second eigenvalue of the special effect in the second video frame image;

[0163] Obtaining an average eigenvalue based on the first eigenvalue and the second eigenvalue;

[0164] Based on the average feature value, a target intermediate frame image is generated.

[0165] Optionally, the multiple video frame images include adjacent first video frame images and second video frame images, and the multiple intermediate frame images include target intermediate frame images generated based on the first video frame images and the second video frame images;

[0166] The first generating subunit is specifically configured to:

[0167] Determining a first position of the special effect in the first video frame image and a second position of the special effect in the second video frame image;

[0168] predicting a motion trajectory of the special effect according to the first position and the second position;

[0169] The target intermediate frame image is generated based on the motion trajectory of the special effect.

[0170] Optionally, the first processing module includes:

[0171] A first acquisition unit is used to acquire positions of multiple cameras shooting the free-viewpoint video;

[0172] A second acquisition unit is used to acquire video images shot by multiple cameras at the same time;

[0173] a first calculation unit, configured to perform pixel matching on the video images captured by each of the plurality of cameras, and calculate disparity values of corresponding pixels in the plurality of video images;

[0174] A first acquiring unit, configured to acquire the distance between the target object and each camera in the free-viewpoint video;

[0175] The first determination unit is used to determine the position of the target object to which special effects need to be added in the free perspective video in the free perspective model based on the positions of the multiple cameras, the disparity values of corresponding pixels in the multiple video images, and the distance of the target object from each camera in the free perspective video.

[0176] Optionally, performing full-view rendering on the free-view video in the free-view model includes:

[0177] The current viewing angle of the free viewing angle video is rendered with full computing power, the nearby viewing angles of the free viewing angle video are rendered with half computing power, and the distant viewing angle of the free viewing angle video is not rendered with computing power.

[0178] An embodiment of the present application also provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, the various processes of the above-mentioned embodiment of the method for adding video special effects to an electronic device are implemented, and the same technical effect can be achieved. To avoid repetition, they will not be described here.

[0179] For details, see Figure 7 As shown, an embodiment of the present application further provides an electronic device, including a bus 701, a transceiver 702, an antenna 703, a bus interface 704, a processor 705 and a memory 706.

[0180] The processor 705 is configured to:

[0181] Constructing a free viewpoint model corresponding to the free viewpoint video;

[0182] Determining a position of a target object to which special effects need to be added in the free-viewpoint video in the free-viewpoint model, and adding the special effects to a position corresponding to the target object;

[0183] Performing full-view rendering on the free-view video in the free-view model;

[0184] A free-viewpoint video with the special effects added thereto is generated based on the rendered video data of each viewpoint.

[0185] Optionally, the processor 705 is specifically configured to:

[0186] Match and fuse the rendered video data from each perspective to obtain the processed free-viewpoint video;

[0187] Projecting the processed free-viewpoint video onto a display screen;

[0188] The transceiver 702 is used for:

[0189] receiving a viewing angle adjustment instruction input by a user, and performing frame insertion processing on the free viewing angle video in the display screen;

[0190] The processor 705 is specifically configured to:

[0191] A free-viewpoint video with the special effects added thereto is obtained based on the free-viewpoint video after the frame insertion processing.

[0192] Optionally, the transceiver 702 is configured to:

[0193] Receive a viewing angle adjustment instruction input by a user;

[0194] The processor 705 is specifically configured to:

[0195] Based on the perspective adjustment instruction, determining a plurality of video frame images in the free perspective video in the display screen whose perspectives need to be switched;

[0196] Based on the multiple video frame images, generating multiple intermediate frame images, wherein each intermediate frame image is generated based on every two adjacent video frame images in the multiple video frame images;

[0197] Calculating an insertion angle of each intermediate frame image in a plurality of intermediate frame images according to an angle between every two adjacent video frame images in the plurality of video frame images;

[0198] The plurality of intermediate frame images are synthesized with the plurality of video frame images based on an insertion angle of each of the plurality of intermediate frame images.

[0199] Optionally, the multiple video frame images include adjacent first video frame images and second video frame images, and the multiple intermediate frame images include target intermediate frame images generated based on the first video frame images and the second video frame images;

[0200] The processor 705 is specifically configured to:

[0201] Calculating a first eigenvalue of the special effect in the first video frame image and a second eigenvalue of the special effect in the second video frame image;

[0202] Obtaining an average eigenvalue based on the first eigenvalue and the second eigenvalue;

[0203] Based on the average feature value, a target intermediate frame image is generated.

[0204] Optionally, the multiple video frame images include adjacent first video frame images and second video frame images, and the multiple intermediate frame images include target intermediate frame images generated based on the first video frame images and the second video frame images;

[0205] The processor 705 is specifically configured to:

[0206] Determining a first position of the special effect in the first video frame image and a second position of the special effect in the second video frame image;

[0207] predicting a motion trajectory of the special effect according to the first position and the second position;

[0208] The target intermediate frame image is generated based on the motion trajectory of the special effect.

[0209] Optionally, the processor 705 is specifically configured to:

[0210] Obtaining positions of multiple cameras that shoot the free-viewpoint video;

[0211] Acquire video images captured by multiple cameras at the same time;

[0212] Performing pixel matching on the video images captured by each of the multiple cameras to calculate disparity values of corresponding pixels in the multiple video images;

[0213] Obtaining the distance between the target object and each camera in the free viewpoint video;

[0214] According to the positions of the multiple cameras, the disparity values of corresponding pixels in the multiple video images and the distance of the target object from each camera in the free perspective video, the position of the target object to which special effects need to be added in the free perspective video in the free perspective model is determined.

[0215] Optionally, the processor 705 is specifically configured to:

[0216] The current viewing angle of the free viewing angle video is rendered with full computing power, the nearby viewing angles of the free viewing angle video are rendered with half computing power, and the distant viewing angle of the free viewing angle video is not rendered with computing power.

[0217] exist Figure 7 In the embodiment, the bus architecture (represented by bus 701) is shown. Bus 701 may include any number of interconnected buses and bridges. Bus 701 links together various circuits including one or more processors represented by processor 705 and memory represented by memory 706. Bus 701 may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and are therefore not described further herein. Bus interface 704 provides an interface between bus 701 and transceiver 702. Transceiver 702 may be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. Data processed by processor 705 is transmitted on a wireless medium via antenna 703. Furthermore, antenna 703 receives data and transmits the data to processor 705.

[0218] The processor 705 is responsible for managing the bus 701 and general processing, and may also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. The memory 706 may be used to store data used by the processor 705 when performing operations.

[0219] Optionally, the processor 705 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0220] The present application also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements the various processes of the above-described video adding method embodiment and achieves the same technical effect. To avoid repetition, the details are not described here. The computer-readable storage medium may be, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0221] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and to avoid repetition, they will not be described here.

[0222] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0223] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0224] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A method for adding special effects to a video, characterized in that: The method comprises: Constructing a free viewpoint model corresponding to the free viewpoint video; Determining a position of a target object to which special effects need to be added in the free-viewpoint video in the free-viewpoint model, and adding the special effects to a position corresponding to the target object; Performing full-view rendering on the free-view video in the free-view model; A free-viewpoint video with the special effects added thereto is generated based on the rendered video data of each viewpoint.

2. The method for adding video special effects according to claim 1, wherein: The generating of the free-viewpoint video with the special effects added thereto based on the rendered video data of each viewpoint includes: Match and fuse the rendered video data from each perspective to obtain the processed free-viewpoint video; Projecting the processed free-viewpoint video onto a display screen; receiving a viewing angle adjustment instruction input by a user, and performing frame insertion processing on the free viewing angle video in the display screen; A free-viewpoint video with the special effects added thereto is obtained based on the free-viewpoint video after the frame insertion processing.

3. The method for adding video special effects according to claim 2, wherein: The receiving of the viewing angle adjustment instruction input by the user and performing frame insertion processing on the free viewing angle video in the display screen includes: Receive a viewing angle adjustment instruction input by a user; Based on the perspective adjustment instruction, determining a plurality of video frame images in the free perspective video in the display screen whose perspectives need to be switched; Based on the multiple video frame images, generating multiple intermediate frame images, wherein each intermediate frame image is generated based on every two adjacent video frame images in the multiple video frame images; Calculating an insertion angle of each intermediate frame image in a plurality of intermediate frame images according to an angle between every two adjacent video frame images in the plurality of video frame images; The plurality of intermediate frame images are synthesized with the plurality of video frame images based on an insertion angle of each of the plurality of intermediate frame images.

4. The method for adding video special effects according to claim 3, wherein: The multiple video frame images include a first video frame image and a second adjacent video frame image, and the multiple intermediate frame images include a target intermediate frame image generated based on the first video frame image and the second video frame image; The step of generating a plurality of intermediate frame images based on the plurality of video frame images comprises: Calculating a first eigenvalue of the special effect in the first video frame image and a second eigenvalue of the special effect in the second video frame image; Obtaining an average eigenvalue based on the first eigenvalue and the second eigenvalue; Based on the average feature value, a target intermediate frame image is generated.

5. The method for adding video special effects according to claim 3, wherein: The multiple video frame images include a first video frame image and a second adjacent video frame image, and the multiple intermediate frame images include a target intermediate frame image generated based on the first video frame image and the second video frame image; The step of generating a plurality of intermediate frame images based on the plurality of video frame images comprises: Determining a first position of the special effect in the first video frame image and a second position of the special effect in the second video frame image; predicting a motion trajectory of the special effect according to the first position and the second position; The target intermediate frame image is generated based on the motion trajectory of the special effect.

6. The method for adding video special effects according to claim 1, wherein: Determining the position of the target object to which special effects need to be added in the free-viewpoint video in the free-viewpoint model includes: Obtaining positions of multiple cameras that shoot the free-viewpoint video; Acquire video images captured by multiple cameras at the same time; Performing pixel matching on the video images captured by each of the multiple cameras to calculate disparity values of corresponding pixels in the multiple video images; Obtaining the distance between the target object and each camera in the free viewpoint video; According to the positions of the multiple cameras, the disparity values of corresponding pixels in the multiple video images and the distance of the target object from each camera in the free perspective video, the position of the target object to which special effects need to be added in the free perspective video in the free perspective model is determined.

7. A video special effects adding device, characterized in that: The device comprises: A first construction module is used to construct a free viewpoint model corresponding to the free viewpoint video; A first processing module is configured to determine a position of a target object in the free-viewpoint video to which special effects need to be added in the free-viewpoint model, and to add the special effects to a position corresponding to the target object; A first rendering module, configured to perform full-view rendering on the free-view video in the free-view model; The first generating module is used to generate a free-viewpoint video with the special effects added thereto based on the rendered video data of each viewpoint.

8. An electronic device, characterized in that: The method comprises a transceiver, a processor, a memory and a computer program stored in the memory and runnable on the processor, wherein when the computer program is executed by the processor, the steps of the method for adding video special effects as claimed in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for adding video special effects according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method for adding video special effects as described in any one of claims 1 to 6.

Images