Video editing method and device, medium, terminal and equipment

Abstract

Embodiments of the present application provide a video editing method and device based on multi-angle free-view data, a medium, a terminal and equipment. The method comprises: displaying multi-angle free-view video; receiving user operation on the multi-angle free-view video; determining information of at least two key frames according to the operation, the information of the key frames comprising frame time of the key frames and frame image of the key frames, the viewpoint of the frame image being selected from a virtual path; determining a viewpoint path between the at least two key frames according to the viewpoint of the frame image of the at least two key frames and the virtual path; and editing the multi-angle free-view data based on the viewpoint path to obtain video between the frame time of the at least two key frames. The technical solution in the embodiments of the present application supports video editing of multi-angle free-view data.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of integrated circuits, and in particular to a video editing method and device based on multi-angle free-view data, a medium, a terminal and an apparatus. BACKGROUND

[0002] In the field of video processing, video data can be received and played to a user. Such video playing is usually based on a fixed view angle, and editing of such video is also based on a fixed view angle. However, in many application scenarios, post-production needs to edit multi-angle materials.

[0003] One way to provide a video with multiple view angles is to capture multiple images with different view angles by using multiple cameras, and to reconstruct images based on the multiple images to meet the requirement of adjusting the view angle of a to-be-viewed picture. The data basis for image reconstruction is referred to as multi-angle free-view data.

[0004] However, there is still a lack of video editing methods for multi-angle free-view data. SUMMARY

[0005] Therefore, in embodiments of the present application, a video editing method based on multi-angle free-view data is provided, which includes the following steps.

[0006] displaying a multi-angle free-view video, the multi-angle free-view video being generated based on multi-angle free-view data, the multi-angle free-view data supporting viewing of a to-be-viewed area from a viewpoint included in a virtual path;

[0007] receiving an operation of a user on the multi-angle free-view video;

[0008] determining information of at least two key frames according to the operation, the information of the key frames including frame time of the key frames and frame images of the key frames, the viewpoints of the frame images being selected from the virtual path;

[0009] determining a viewpoint path between the at least two key frames according to the viewpoints of the frame images of the at least two key frames and the virtual path;

[0010] editing the multi-angle free-view data based on the viewpoint path to obtain a video between the frame time of the at least two key frames.

[0011] Optionally, the receiving of the operation of the user includes:

[0012] determining selection of the user on the frame time;

[0013] determining indication of the user on a selected area in the frame image corresponding to the selected frame time;

[0014] receiving an indication of a user's confirmation of the key frame.

[0015] Optionally, the presenting the multi-angle free-view video comprises presenting a preselected frame image, the preselected frame image being a frame image corresponding to a frame time selected by the user.

[0016] Optionally, the information of the at least two key frames determined according to the operation comprises:

[0017] In response to the indication of the user's confirmation of the key frame, determining that the selected frame time is a frame time of the key frame, and determining that the selected region in the preselected frame image is a frame image of the key frame.

[0018] Optionally, the determining the indication of the user's selection of the region in the preselected frame image comprises:

[0019] presenting a selected frame on the preselected frame image;

[0020] receiving at least one of the following operations of the user on the selected frame: a drag operation and a zoom operation.

[0021] Optionally, the determining the indication of the user's selection of the region in the preselected frame image comprises receiving an operation of a viewpoint transformation of the preselected frame image.

[0022] Optionally, the determining the indication of the user's selection of the frame time comprises receiving a selection of a frame time by the user on a time axis.

[0023] Optionally, the receiving the indication of the user's confirmation of the key frame is performed by a key frame confirmation component.

[0024] Optionally, the receiving the operation of the user on the multi-angle free-view video comprises receiving a viewpoint transformation operation of the presented frame image by the user.

[0025] The presenting the frame time and the frame image corresponding to the frame time comprises updating the presentation of the frame image according to the transformed viewpoint.

[0026] Optionally, the viewpoints in the viewpoint path between the at least two key frames are selected from the virtual path.

[0027] Optionally, the viewpoint path between the at least two key frames is a smooth viewpoint path.

[0028] Optionally, the virtual path comprises a base path, the base path being based on viewpoints of a plurality of images synchronized in the multi-angle free-view data, and the receiving the operation of the user on the multi-angle free-view video comprises receiving an angle transformation indication consistent with an angle of a viewpoint in the base path.

[0029] The embodiment of the application further provides a video editing method based on multi-angle free-view data, comprising:

[0030] displaying a multi-angle free-view video, a viewpoint of a frame image in the multi-angle free-view video being selected from a virtual path, the virtual path being a set of viewpoints for viewing a to-be-viewed region;

[0031] receiving an operation of a user on the multi-angle free-view video;

[0032] determining information of at least two key frames according to the operation, the information of the key frames comprising a frame time of the key frames and a frame image of the key frames;

[0033] sending the information of the key frames to a cloud;

[0034] receiving a video between frame times of the at least two key frames from the cloud, wherein the video between the frame times of the at least two key frames is obtained by editing the multi-angle free-view data based on a viewpoint path, the viewpoint path being determined according to the viewpoint of the frame image of the key frames and the virtual path.

[0035] The embodiment of the application further provides a video editing method based on multi-angle free-view data, comprising:

[0036] receiving information of at least two key frames, the information of the key frames being determined according to an operation of a user on a multi-angle free-view video, the multi-angle free-view video supporting viewing of a to-be-viewed region from viewpoints included in a virtual path;

[0037] determining a viewpoint path between the at least two key frames according to a viewpoint of a frame image of the at least two key frames and the virtual path;

[0038] editing the multi-angle free-view data based on the viewpoint path to obtain a video between frame times of the at least two key frames;

[0039] sending the video between the frame times of the at least two key frames.

[0040] The embodiment of the application further provides a video editing system based on multi-angle free-view data, comprising:

[0041] A terminal device is adapted to display a multi-angle free-view video, a viewpoint of a frame image in the multi-angle free-view video is selected from a virtual path, the virtual path is a set of viewpoints for viewing a to-be-viewed region; receive an operation of a user on the multi-angle free-view video; determine information of at least two key frames according to the operation, the information of the key frames includes frame time of the key frames and frame images of the key frames; send the information of the key frames to a cloud device; receive a video between the frame time of the at least two key frames from the cloud device;

[0042] A cloud device is adapted to receive information of at least two key frames from the terminal device; determine a viewpoint path between the at least two key frames according to a viewpoint of a frame image of the at least two key frames and the virtual path; edit multi-angle free-view data based on the viewpoint path to obtain a video between the frame time of the at least two key frames; and send the video between the frame time of the at least two key frames to the terminal device.

[0043] Embodiments of the present application further provide a computer device, comprising a memory and a processor, the memory has stored thereon a computer program capable of running on the processor, and the processor executes the video editing method when running the computer program.

[0044] Embodiments of the present application further provide a computer readable storage medium having stored thereon a computer program, and the computer program runs the video editing method.

[0045] Embodiments of the present application further provide a video editing apparatus based on multi-angle free-view data, comprising:

[0046] A display unit displays a multi-angle free-view video, the multi-angle free-view video is generated based on multi-angle free-view data, the multi-angle free-view data supports viewing a to-be-viewed region from viewpoints included in a virtual path;

[0047] The virtual path is a set of viewpoints for viewing the to-be-viewed region;

[0048] An operation receiving unit receives an operation of a user on the multi-angle free-view video;

[0049] A key frame information generating unit is adapted to determine information of at least two key frames according to the operation, the information of the key frames includes frame time of the key frames and frame images of the key frames, and a viewpoint of the frame image of the key frames is selected from the virtual path;

[0050] A path determining unit is adapted to determine a viewpoint path between the at least two key frames according to a viewpoint of a frame image of the at least two key frames and the virtual path;

[0051] a video generation unit, configured to edit the multi-angle free perspective data based on the viewpoint path to obtain a video between frame moments of the at least two key frames.

[0052] The embodiment of the present application further provides a video editing device based on multi-angle free perspective data, comprising:

[0053] a display unit, configured to display a multi-angle free perspective video, a viewpoint of a frame image in the multi-angle free perspective video being selected from a virtual path, the virtual path being a set of viewpoints for viewing a to-be-viewed region;

[0054] an operation receiving unit, configured to receive an operation of a user on the multi-angle free perspective video;

[0055] a key frame information generation unit, configured to determine information of at least two key frames according to the operation, the information of the key frames comprising frame moments of the key frames and frame images of the key frames;

[0056] send the information of the key frames to a cloud;

[0057] an edited video receiving unit, configured to receive a video between frame moments of the at least two key frames from the cloud, wherein the video between frame moments of the at least two key frames is obtained by editing the multi-angle free perspective data based on a viewpoint path, and the viewpoint path is determined according to the viewpoints of the frame images of the key frames and the virtual path.

[0058] The embodiment of the present application further provides a video editing device based on multi-angle free perspective data, comprising:

[0059] an operation receiving unit, configured to receive information of at least two key frames, the information of the key frames being determined according to an operation of a user on a multi-angle free perspective video, the multi-angle free perspective video supporting viewing of a to-be-viewed region from viewpoints included in a virtual path;

[0060] a path determination unit, configured to determine a viewpoint path between the at least two key frames according to viewpoints of frame images of the at least two key frames and the virtual path;

[0061] a video generation unit, configured to edit the multi-angle free perspective data based on the viewpoint path to obtain a video between frame moments of the at least two key frames;

[0062] an edited video sending unit, configured to send the video between frame moments of the at least two key frames.

[0063] The technical solution in the embodiments of the present application determines information of at least two key frames by receiving an operation of a user, edits multi-angle free perspective data based on a viewpoint path, and obtains a video between frame moments of the at least two key frames, so as to provide a method for editing multi-angle free perspective data for the user and meet the needs of the user for video editing in different scenes. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of the provided drawings.

[0065] Figure 1 FIG. 1 is a schematic diagram of a multi-angle free perspective display system;

[0066] Figure 2a FIG. 2 is a schematic diagram of two devices displaying; Figure 2b

[0067] Figure 3 FIG. 4 is a schematic diagram of a flow of a video editing method in the embodiments of the present application;

[0068] Figure 4 FIG. 5 is a schematic diagram of a timeline in the embodiments of the present application;

[0069] Figure 5a FIG. 6 is a schematic diagram of an interface at a frame moment in the embodiments of the present application;

[0070] Figure 5b FIG. 7 is a schematic diagram of an editing interface at another frame moment in the embodiments of the present application;

[0071] Figure 5c FIG. 8 is a schematic diagram of another editing interface at another frame moment in the embodiments of the present application;

[0072] Figure 6 FIG. 9 is a schematic diagram of a flow of receiving an operation of a user in the embodiments of the present application;

[0073] Figure 7 FIG. 10 is a schematic diagram of display of an editing interface in the embodiments of the present application;

[0074] Figure 8 FIG. 11 is a schematic diagram of another editing interface in the embodiments of the present application;

[0075] Figures 9 to 11 FIG. 12 is a schematic diagram of a part of frame images of display of an edited video of an editing interface in the embodiments of the present application;

[0076] ​Figure 12 FIG. 6 is a schematic diagram of another editing interface in an embodiment of the present application;

[0077] Figure 13 FIG. 7 is a schematic diagram of another editing interface in an embodiment of the present application;

[0078] Figure 14 、 Figure 14a 、 Figure 14b FIG. 8 is a schematic diagram of part of frame images of a displayed video after editing in an embodiment of the present application;

[0079] Figure 15 FIG. 9 is a schematic diagram of a flow of a method for generating multi-angle free-view data in an embodiment of the present application;

[0080] Figure 16 FIG. 10 is a schematic diagram of a flow of a specific implementation of step S153 shown in FIG. 9 in an embodiment of the present application; Figure 15

[0081] FIG. 11 is a schematic diagram of mapping a viewpoint position of an image to a viewpoint position in a virtual path in an embodiment of the present application; Figure 17

[0082] FIG. 12 is a schematic diagram of a flow of a specific implementation of step S163 shown in FIG. 9 in an embodiment of the present application; Figure 18 Figure 16 FIG. 13 is a schematic diagram of part of a flow of another specific implementation of step S153 shown in FIG. 9 in an embodiment of the present application;

[0083] Figure 19 Figure 15 FIG. 14 is a schematic of a texture map of a to-be-adjusted image before interpolation in an embodiment of the present application;

[0084] Figure 20 FIG. 15 is a schematic of a texture map of a to-be-adjusted image after interpolation in an embodiment of the present application;

[0085] Figure 21 FIG. 16 is a schematic of a hole region after interpolation in an embodiment of the present application;

[0086] Figure 22 FIG. 17 is another schematic of a hole region after interpolation in an embodiment of the present application;

[0087] Figure 23 FIG. 18 is another schematic of a hole region after interpolation in an embodiment of the present application;

[0088] Figure 24 FIG. 19 is another schematic of a hole region after interpolation in an embodiment of the present application;

[0089] Figure 25 FIG. 20 is another schematic of a hole region after interpolation in an embodiment of the present application;

[0090] Figure 26 ​​Another schematic diagram of the intersection of the non-hole regions in one embodiment of the present application;

[0091] Figure 27 A schematic diagram of the intersection of the non-hole regions in one embodiment of the present application Figures 22 to 26 A schematic diagram of the intersection of the non-hole regions in one embodiment of the present application;

[0092] Figure 28 A schematic diagram of the intersection of the non-hole regions in one embodiment of the present application Figure 27 A schematic diagram of the intersection of the non-hole regions in one embodiment of the present application;

[0093] Figure 29 A schematic diagram of the intersection of the non-hole regions in one embodiment of the present application DETAILED DESCRIPTION

[0094] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0095] Multi-angle free-view data is data that can support users to switch views for watching.

[0096] The multi-angle free-view data can include texture maps collected by multiple cameras, all camera parameters, and depth maps of each camera. The generation process of the multi-angle free-view data can generally include multi-camera video capturing, camera internal and external parameter calculation, and depth map calculation.

[0097] Based on the multi-angle free-view data, in combination with the indication of the watching position and the watching angle by the user, the user end can use the depth map rendering (DIBR, Depth Image-Based Rendering) method to perform image reconstruction, to obtain a reconstructed image corresponding to the indication of the user, so as to meet the watching demand of the user at different positions and angles.

[0098] The viewing position and the viewing angle, which can also be referred to as a viewpoint, can be represented by 6 parameters, 3 of which represent the viewing position or the viewpoint position, and 3 of which represent the viewing angle or the viewpoint angle. The shooting position and the shooting angle (corresponding to the camera pose) in the camera parameters of the shooting device can also represent the 6 degrees of freedom, and a pair of images that have been shot, the shooting position and the shooting angle of the corresponding shooting device of which can also be referred to as the viewpoint of the images. Similarly, the viewing position and the viewing angle indicated by the user can be referred to as a virtual viewpoint, and can also be represented by the 6 dimensions. From this perspective, the multi-angle free-view video data described in the foregoing can also be referred to as 6 degrees of freedom video data (6DoF video data).

[0099] Figure 1 FIG. 1 is a schematic diagram of a multi-angle free-view display system.

[0100] With reference to Figure 1 In embodiments of the present application, a collection system 11 including a plurality of collection devices can be used to collect data of a region to be viewed, where the collection devices can be cameras CJ1 to CJ6. The collected synchronized multiple images can be processed by the collection system 11 or by a server 12 to generate multi-angle free-view video data that can support virtual viewpoint switching by a terminal device 13. The terminal device 13 can display reconstructed images generated based on the multi-angle free-view video data, and the reconstructed images correspond to virtual viewpoints. According to user instructions, the terminal device 13 can display reconstructed images corresponding to different virtual viewpoints to switch the viewing position and the viewing angle. For example, with reference to Figure 2a and Figure 2b The terminal device 13 can also display images viewed from different angles.

[0101] With reference to Figure 1 In specific implementations, the process of image reconstruction to obtain reconstructed images can be implemented by the terminal device 13, by a device located in a content delivery network (CDN) in an edge computing manner, or by the server 12. It can be understood that Figure 1 are merely examples and are not intended to limit the collection system, the server, the terminal device, and the specific implementation manner.

[0102] In embodiments of the present application, the terminal device 13 is used to edit video based on multi-angle free-view video data. The terminal device 13 can be a desktop computer, a notebook computer, a tablet computer, a smart phone, or any device that can receive user instructions. The application scenarios of video editing can be various, such as competitive sports and stage scenes, without limitation.

[0103] With reference toFigure 3 The embodiment of the present application provides a video editing method, which can comprise the following steps:

[0104] In step S31, a multi-angle free perspective video is displayed, wherein the view point of a frame image in the multi-angle free perspective video is selected from a virtual path, and the virtual path is a set of view points for viewing a to-be-viewed area;

[0105] In step S32, a user operation on the multi-angle free perspective video is received.

[0106] In step S33, information of at least two key frames is determined according to the operation, wherein the information of the key frames comprises a frame time of the key frames and a frame image of the key frames.

[0107] In step S34, a view point path between the at least two key frames is determined based on the view point of the frame image of the at least two key frames and the virtual path.

[0108] In step S35, the multi-angle free perspective data is edited based on the view point path, to obtain a video between the frame times of the at least two key frames.

[0109] In the embodiment of the present application, the information of the at least two key frames is determined by receiving the user operation, and the multi-angle free perspective data is edited based on the view point path, to obtain the video between the frame times of the at least two key frames, so that a method for editing the multi-angle free perspective data by the user can be provided, and the demand of the user for video editing in different scenes can be met.

[0110] In addition, by selecting the view point of the image of the key frame from the virtual path, the image quality of the video clip can be higher.

[0111] In a specific implementation, the multi-angle free perspective video can comprise a plurality of images, which can be referred to as frame images or video frames; and the time position of the frame images can be referred to as a frame time.

[0112] In a specific implementation, the multi-angle free perspective video can be displayed in various ways, for example, the frame time can be displayed on a time axis, and refer to Figure 4 The component 42 on the time axis 41 can display the frame image before the frame time.

[0113] In a specific implementation, the frame image displayed can be a frame image in various scenes, for example, a frame image in a stage scene or a frame image in a competitive sports scene. Further, the area for displaying the frame image can be in the same interface as the time axis, for example, refer to Figures 5a to 5c In the editing interface, the frame image of different frame times is displayed. It can be understood that, Figures 5a to 5cThe above is only for illustration and is not limited to the interface layout and display content. In addition, the frame time can also be displayed in other manners, which are not limited herein.

[0114] In combination with reference Figure 6 In a specific implementation, a manner of displaying and receiving user operation can include the following steps:

[0115] Step S61: determining user selection of a frame time;

[0116] Step S62: displaying a preselected frame image corresponding to the frame time selected by the user;

[0117] Step S63: determining user indication of a selected region in the preselected frame image;

[0118] Step S64: receiving user confirmation indication of the key frame.

[0119] In a specific implementation, the manner of user selection of a frame time can be various, for example, in combination with reference Figure 4 The position of the frame time to be selected can be determined by the position of the component 42 on the time axis 41, and the position of the component 42 at the current time can be confirmed as the selected frame time through a confirmation operation.

[0120] For example, in combination with reference Figures 5a to 5c The user can select a frame time by dragging a component on the time axis. Alternatively, the user can also select a frame time in a direct input manner.

[0121] Further, in the interface of the editing software, a confirmation component can be included to confirm the selection of the frame time. The specific position of the confirmation component can be various, for example, it can be located below the time axis. In combination with reference Figures 5a to 5c In the interface of the editing software, a plurality of components are shown below the time axis, which can include a confirmation component. Alternatively, the selected frame time can also be confirmed through a shortcut key.

[0122] In a specific implementation, a frame image corresponding to the frame time selected by the user can be displayed. For example, in combination with reference Figures 5a to 5c When the user drags a component on the time axis, an image corresponding to each frame time in the user dragging process can be displayed. Alternatively, the user can directly click a position corresponding to a frame time to select, and then a frame image corresponding to the selected frame time, i.e., a preselected frame image, can be displayed.

[0123] In an embodiment, the user's confirmation indication of the key frame can include the aforementioned confirmation of the frame time and the confirmation of the selected region. The confirmation of both can be done by the same confirmation indication. For example, after the frame time has been selected and the user has received the adjustment of the selected region, a component can mark the key frame, confirming the currently selected frame time as the frame time of the key frame and the currently selected region as the frame image of the key frame. That is, the information of the key frame can be generated in response to the user's confirmation indication.

[0124] In an embodiment, the user's indication of the selected region in the preselected frame image can be received. The indication can be received in various ways. For example, with reference to Figure 7 The selected region can be selected by the selected box 71. The user can drag or scale the selected box to determine the selected region. By providing the selected box, the user can be more intuitively supported to perform the selection operation, improving the efficiency of the editing process.

[0125] In addition, by providing the preselected box, the process of realizing the lens cropping in the post-production by editing can be supported, and the unnecessary part can be cropped, so that the lens effect of directly shooting a person can be produced, or the effect of cutting off the surrounding irrelevant region can be produced

[0126] With reference to Figure 7 and Figure 8 In an embodiment, the user can determine the positions of different selected boxes at different frame times to confirm the display regions corresponding to different frame times.

[0127] The frame times determined by the user are two key frame times, respectively, as shown in Figure 7 and Figure 8 For example, the regions in the selected boxes 71 shown in Figure 7 and Figure 8 are selected regions, and the video between the two key frame times can be generated after editing. The edited video can include, for example, the frame image 91 in Figure 9 , the frame images 101 to Figure 10 , and the frame image 111 shown in Figure 11 . It can be understood that the edited video can also be displayed through other interfaces.

[0128] In the example shown in Figures 7 to 11 , the size of the editing box does not change, and through this editing method, the effect of live straight shooting can be obtained in the post-production. In another embodiment, the user can also adjust the size of the editing box to pursue more post-production effects.

[0129] In specific implementations, the user can adjust the viewpoint of the frame image displayed in the editing interface. The manner of displaying the viewpoint of the frame image can be diverse, for example, the user can swipe left or right on the screen, drag the viewpoint through a mouse, or adjust the viewpoint through a shortcut key.

[0130] The user's indication of the frame image can include information of the viewpoint. For example, referring to Figure 13 , the user can indicate the selected area through the selected box 71 after adjusting the displayed frame image to the viewpoint position shown in the frame image in Figure 12 .

[0131] In combination with the foregoing, the size of the selected box can be diverse, in combination with reference to Figure 8 and Figure 13 , the frame time shown in the above two figures is the frame time of the two key frames, and the selected area of the selected box 71 in the above two figures is the frame image of the two key frames, respectively. Then a video between the two frame times can be generated. The video can include the frame images as shown in Figure 14 , Figure 14a , Figure 14b , Figure 17 .

[0132] By adjusting the size of the selected box, the effect of zooming in or out of the live shot can be achieved during editing, and a more distant or closer viewing effect can be supported; by receiving the user's indication of the viewpoint adjustment, in the case where the viewpoints of the two key frames are different, the effect of adjusting the live shot position can be achieved through post-editing. By using the selected box, the shot can be cropped.

[0133] In the above manner, the user performing editing can achieve planning or cutting of the virtual shot while playing, and obtain more diverse editing effects.

[0134] In addition, during the process of receiving the user's operation on the multi-angle free-view video, the user can cancel the confirmation indication of the key frame, reselect the key frame, and thus obtain a satisfactory editing effect.

[0135] In specific implementations, the multi-angle free-view data based on which the display and editing are performed corresponds to a virtual path, and the virtual path is a pre-determined virtual path, which can include a basic path, and the basic path can be obtained based on the viewpoints of the multiple images synchronized in the multi-angle free-view data, and specifically can serve as a basis for the user to perform the view change operation.

[0136] In a specific implementation, the display of the frame images at different frame moments can be based on the perspective supported by the base path. Further, the received operation of the user can be performed on the displayed images. For example, the user performing the editing can switch the perspective of the viewpoint, i.e., the viewpoint perspective, based on the base path.

[0137] In this way, the image quality of each frame image in the edited video can be guaranteed, and thus the quality of the edited video can be guaranteed.

[0138] The multi-angle free-perspective editing method described above in the embodiments of the present application can be used in a terminal device or can be completed by the terminal device and the cloud together.

[0139] In a specific implementation, the display and the reception of the indication of the user can be implemented by the terminal device, and the calculation of the viewpoint path between the at least two key frames and the generation of the video between the frame moments of the at least two key frames can be implemented by the cloud. In this way, the computing pressure of the terminal device can be reduced, and the efficiency of the editing method can be improved. The cloud can be a device that is wirelessly connected to the terminal device. For example, in combination with reference to Figure 1 The server 12 or the CDN in the foregoing can be used.

[0140] The embodiments of the present application also provide a video editing method based on multi-angle free-perspective data, which can include the following steps:

[0141] displaying a multi-angle free-perspective video, a viewpoint of a frame image in the multi-angle free-perspective video being selected from a virtual path, the virtual path being a set of viewpoints for viewing a to-be-viewed region;

[0142] receiving an operation of a user on the multi-angle free-perspective video;

[0143] determining information of at least two key frames according to the operation, the information of the key frames including frame moments of the key frames and frame images of the key frames;

[0144] sending the information of the key frames to a cloud;

[0145] receiving a video between the frame moments of the at least two key frames from the cloud, wherein the video between the frame moments of the at least two key frames is obtained by editing the multi-angle free-perspective data based on a viewpoint path, and the viewpoint path is determined according to the viewpoints of the frame images of the key frames and the virtual path.

[0146] In specific implementations, the video received from the cloud can be a video generated based on high-definition multi-angle free-view video data. Before sending the information of the key frames to the cloud, a video between frame moments of the at least two key frames can also be generated based on low-definition multi-angle free-view video. The definition of the low-definition multi-angle free-view video is lower than that of the high-definition multi-angle free-view data.

[0147] In this way, the computing pressure of the terminal device can be reduced, and the user who performs editing can preview the video effect after editing the selected key frames. By processing the high-definition multi-angle free-view video data by the cloud device, the corresponding high-definition multi-angle free-view video can be obtained for subsequent display for the user who performs viewing, thereby providing better visual effects.

[0148] The multi-angle free-view video can be a video generated based on multi-angle free-view data, and the generation manner can be various. For example, the multi-angle free-view video can be generated by the following manner: obtaining a plurality of videos that are frame-synchronized and have different shooting angles; performing analysis on each video to obtain an image combination of a plurality of frame moments, the image combination including a plurality of frame images that are frame-synchronized; determining depth data of each frame image in the image combination based on the image combination of each frame moment; generating a stitching image corresponding to each frame moment, the stitching image including a first field storing pixel data of each frame image in the image combination and a second field storing depth data of each frame image in the image combination; and generating video data based on a plurality of the stitching images.

[0149] In specific implementations, the display is based on low-definition multi-angle free-view video. Similar to the beneficial effects described above, since the user who performs editing performs editing based on the display of the frame images, visual experience is not necessary for the user, and thus the data amount of the terminal device that performs editing can be reduced.

[0150] The video editing method described above can be used in the terminal device described above, and other specific implementations and beneficial effects can be referred to the descriptions above, which will not be described herein.

[0151] The embodiments of the present application also provide a video editing method based on multi-angle free-view data, including:

[0152] receiving information of at least two key frames, the information of the key frames being determined according to user operations on multi-angle free-view video, the multi-angle free-view video supporting viewing of a to-be-viewed area from viewpoints included in a virtual path;

[0153] determining a viewpoint path between the at least two key frames according to the viewpoints of the frame images of the at least two key frames and the virtual path;

[0154] editing the multi-angle free perspective data based on the viewpoint path to obtain a video between frame moments of the at least two key frames;

[0155] sending the video between frame moments of the at least two key frames.

[0156] The video editing method can be used in the cloud as described above, and the specific implementation and beneficial effects can be referred to the foregoing description.

[0157] The multi-angle free perspective data used in the embodiments of the present application can be generated in the following manner.

[0158] With reference to Figure 1 The multi-angle free perspective data is collected by the collection system 11, and the images can be frame images in a video. The collection system 11 needs to be provided with an array of collection devices, such as an array of cameras or video cameras.

[0159] The existing multi-angle viewing based on multi-angle free perspective data needs to improve the image quality, and the cost of the collection system is high.

[0160] The applicant has further found that the positions and angles of the collection devices in the collection system have a great influence on the quality of the reconstructed images. If the consistency of the positions and angles of the virtual viewpoints corresponding to the reconstructed images and the collection devices in the collection system is poor, the reconstructed images will have a hole effect.

[0161] The embodiments of the present application further provide a multi-angle free perspective data generation method, which can include the following steps in combination with FIG. 15:

[0162] Step S151, acquiring a plurality of synchronized images, the viewpoints of the plurality of images being different

[0163] Step S152, determining a virtual path, the virtual path being a set of viewpoints for viewing a viewing area

[0164] Step S153, pre-processing the plurality of images according to the virtual path and the viewpoint of each image in the plurality of images to obtain pre-processed images conforming to the viewpoints in the virtual path;

[0165] Step S154, generating multi-angle free perspective data based on the pre-processed images.

[0166] Step S154, generating multi-angle free perspective data based on the pre-processed images.

[0167] The plurality of synchronized images can be images collected by a collection device, specifically can be synchronized frame images collected by a plurality of camera devices, or can be images collected by a plurality of cameras at the same time. The plurality of synchronized images are images obtained by photographing the viewing area, which can be various and can be located in various scenes, such as competitive sports scenes, stage performance scenes, etc. The viewing point of the viewing area can be the viewing point of the user on the terminal, which can be referred to as the virtual viewing point.

[0168] In the embodiment of the application, the plurality of synchronized images are preprocessed to obtain a set of preprocessed images consistent with the virtual path, and the multi-angle free-view data is further obtained based on the set of preprocessed images, which can improve the consistency of the multi-angle free-view data and the viewing point, and further improve the image quality when the image is reconstructed based on the multi-angle free-view data.

[0169] In specific implementation, the virtual path can be various. The virtual path can be pre-set according to specified viewing requirements or historical data. Alternatively, the virtual path can be obtained according to the fitting result of the camera parameters corresponding to the plurality of synchronized images. For example, the virtual path can be obtained according to the fitting result of the camera parameters.

[0170] The camera parameters can include internal parameter data and external parameter data. The internal parameter data includes attribute data of the photographing device of the image, such as the optical center, focal length, and attribute information of the photographing device. The external parameter data can include the photographing position and photographing angle.

[0171] In specific implementation, the basic path can be obtained by fitting the external parameters. As described above, the external parameters of the camera can be represented by six parameters, here three parameters representing the camera position (corresponding to the viewing point position of the image) are denoted as (x, y, z), and three parameters representing the camera pose (corresponding to the viewing angle of the image) are denoted as (θ, ), which represent the rotation angles of the camera around the x, y, and z axes. The virtual path includes the motion trajectory (x, y, z) of the virtual camera in space, and the three rotation directions (θ, ) of each point on the trajectory, thereby including the six degrees of freedom parameters of each virtual position.

[0172] According to the basic path, the virtual path can be further determined. The virtual path can include the basic path and the viewing point supported by the result of the focal length transformation based on the basic path.

[0173] By determining the virtual path according to the result of fitting the camera parameters of the plurality of images, the consistency of the virtual path with the plurality of images can be better, the utilization of the plurality of images can be improved, and the reconstructed image quality when viewing the to-be-viewed region can be further improved.

[0174] In specific implementations, reference is made to Figure 15 and Figure 16 , Figure 15 In step S153, the plurality of images can be preprocessed, which can further include the following steps:

[0175] As described above, the viewpoint includes a viewpoint position and a viewpoint angle. The virtual path is a set of viewpoints for viewing the to-be-viewed region, that is, includes a plurality of virtual viewpoints. Further, the virtual path can include the viewpoint positions of the plurality of virtual viewpoints and the corresponding viewpoint angles of the viewpoint positions. Reference is made to Figure 15 and FIG. 16, Figure 15 In specific implementations, step S153 can further include the following steps:

[0176] In step S161, the viewpoint position of each image in the plurality of images is mapped to the viewpoint position in the virtual path.

[0177] In step S162, according to the correspondence between the viewpoint position and the viewpoint angle in the virtual path, the target angle corresponding to the viewpoint position of each image in the plurality of images is determined.

[0178] In step S163, according to the viewpoint angle and the target angle of each image in the plurality of images, an image consistent with the corresponding target angle is obtained.

[0179] In specific implementations, the viewpoint position of each image in the plurality of images can be mapped to the viewpoint position of the base path, or the viewpoint position of the base path after focal length transformation.

[0180] In specific implementations, the mapping of the viewpoint position of each image in the plurality of images to the virtual path can be various. For example, when the viewpoint position of the virtual viewpoint and the viewpoint position of the image are both expressed in the 6-DOF manner as described above, the viewpoint position of the image can be mapped to the nearest viewpoint position on the virtual path according to the coordinate positions of the two. Alternatively, the two can be unified to the same coordinate system before mapping. Alternatively, the mapping can be performed in other ways.

[0181] For example, reference is made to Figure 17, the arcuate dashed line in the figure shows the basic path to the viewing area 51. The dashed triangle in the figure shows a part of the virtual viewpoint on the virtual path. The black solid triangle in the figure shows the viewpoint 52 indicated by the camera parameters of an image in the synchronized multiple images, the position of the viewpoint is represented as (x1, y1, z1), the viewing angle of the viewpoint is represented as (θ1, ) By mapping to the virtual path, the position of the viewpoint (x1, y1, z1) of the image is mapped to the position of the virtual viewpoint 53 (x2, y2, z2) shown by the dashed triangle in the figure which partially coincides with the virtual viewpoint 53, the viewing angle of the virtual viewpoint 53 is represented as (θ2, ) The viewing angle (θ2, ) of the virtual viewpoint 53 can be taken as the target viewing angle of the viewpoint 52 indicated by the camera parameters of the image.

[0182] It can be understood that, although Figure 17 a basic basic path is shown, the complete virtual path is not shown. It can be understood that the virtual path includes the viewpoints on the basic path, and includes the viewpoints at the same viewing angle which are closer to or farther from the viewing area by zooming in or zooming out of the image, for example, as shown in Figure 17 the viewpoint 54 obtained by zooming out of the viewpoint 55. By changing the focal length, the viewing effect of being closer to or farther from the stage can be supported.

[0183] In addition, it can be understood that Figure 17 is only for illustration, and is not a limitation on the specific implementation of the virtual path, the mapping method, etc.

[0184] In specific implementation, in combination with reference to Figure 18 , the image consistent with the corresponding target viewing angle can be obtained by the following steps:

[0185] Step S181, determining a to-be-adjusted image in the multiple images whose viewing angle of the viewpoint of the image is different from the target viewing angle;

[0186] Step S182, interpolating the to-be-adjusted image according to the corresponding target viewing angle.

[0187] In specific implementation, the determination method of the target viewing angle of each to-be-adjusted image in the multiple images is described above, and will not be repeated here. The algorithm of interpolating the to-be-adjusted image according to the corresponding target viewing angle can be various, and the algorithm of interpolating each to-be-adjusted image in the multiple images can be the same or different, which is not limited here.

[0188] In a specific implementation, the texture map and the depth map of the image to be adjusted can be interpolated respectively. The interpolation operation herein is used to warp the image so as to make the adjusted image consistent with the target view angle.

[0189] Further, when interpolating the texture map of the image to be adjusted, an interpolation algorithm with a filter template can be selected to ensure the smoothness of the corrected texture map. When interpolating the depth map of the image to be adjusted, a nearest-neighbor interpolation algorithm can be selected to ensure that the depth value will not change in the correction process.

[0190] With reference to Figure 16 and Figure 19 In a specific implementation, after obtaining the images consistent with the target view angles in step S163, the following steps can be further included:

[0191] In step S191, a clipping range is determined according to the intersection of the valid pixel ranges in each of the images consistent with the target view angles, and the valid pixel range is a non-hole region.

[0192] In step S192, the images consistent with the target view angles are clipped according to the clipping range.

[0193] With reference to Figure 20 and Figure 21 Taking a stage scene as an example, Figure 20 a texture map of the image to be adjusted before interpolation in the stage scene is shown, Figure 21 and a corresponding texture map after interpolation is shown. As can be seen, after the interpolation corresponding to the view angle transformation, a hole is generated at the edge of the image, that is, Figure 21 the black region at the edge of the image.

[0194] It can be understood that, Figure 20 and Figure 21 are only for illustration, and not a limitation on the actual shape transformation before and after the difference. Moreover, the scenarios applicable to the embodiments of the present application are diverse, and are not limited to the stage scene.

[0195] In addition, Figure 20 and Figure 21 To illustrate more clearly, a larger hole region is shown. In an actual application scenario, the range of the hole region can be smaller. In an actual application scenario, the range of the hole region can be diverse. For example, see Figures 22 to 26 where the black region illustrates a part of the hole region after the difference processing. It can be understood that, Figures 20 to 26 are only for illustration, and not a limitation on the specific implementation manner or presentation form in the embodiments of the present application.

[0196] Since the hollow region does not contain valid pixels. By cutting off the part that does not contain valid pixels, the resources of subsequent processing, storage and transmission can be saved.

[0197] In specific implementations, the cutting range can be determined according to the intersection of the hollow-free regions in the respective difference images. For example Figures 22 to 26 The intersection of the white regions in the difference images, as shown in FIG. 6B, is Figure 27 The intersection of the white regions in the difference images, as shown in FIG. 6B, is

[0198] In specific implementations, the cutting region can be determined in combination with the above intersection around the center of the image. For example, in combination with reference to Figure 28 The range schematically shown by the dashed box can be taken as the cutting region. In this way, the center of the image can be kept consistent before and after cutting, and the visual effect can be guaranteed.

[0199] It can be understood that Figures 22 to 28 This is only a schematic and is not a limitation on the number of synchronized images, the shape of the hollow region, the specific cutting method, etc.

[0200] In specific implementations, the synchronized multiple images can be images corresponding to the same frame time in video data collected by multiple collection devices; generating multi-angle free-view data based on the preprocessed image set can include storing the depth map and the texture map corresponding to the processed image in frame time order. The generated multi-angle free-view data can be multi-angle free-view video data.

[0201] The specific implementation of generating multi-angle free-view data is not limited herein, and the specific implementation modes such as adjusting the resolution of images in the preprocessed image set, downsampling, splicing images, encapsulating the spliced data, and other implementation modes that can be implemented by those skilled in the art can be used in combination with other steps in this application.

[0202] In specific implementations, the multi-angle free-view data is usually compressed and transmitted to the user. By determining the virtual path before compressing the multi-angle free-view data, and preprocessing the synchronized multiple images based on the virtual path, the texture and depth images of each view of the multi-angle free-view data entering the compression link can have a higher consistency with the virtual path, thereby significantly reducing the hollow effect of the free-view data at the image edge when the user switches the view, and the camera debugging requirements of the live setup can be reduced to a practical level, so that a higher free-view experience can be achieved in a low-cost scene without electric gimbals and other camera position adjustment devices.

[0203] Those skilled in the art can understand that the descriptions of "in a specific implementation", "in an embodiment", "for example" and the like in the present application mean that the specific features, structures or characteristics described in combination with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples without contradiction.

[0204] In addition, any process or method descriptions in flow charts or otherwise described herein in the foregoing embodiments can be understood as representing code modules, segments, or portions of code that include one or more executable instructions for implementing specific logic functions or steps in the process. And the scope of the preferred embodiments of the present application includes additional implementation in which the functions may be performed in an order different from that shown or discussed, including functions performed in substantially simultaneous, or in reverse order, according to the functions involved.

[0205] The embodiments of the present application also provide a video editing system based on multi-angle free perspective data, comprising:

[0206] The terminal device is adapted to display a multi-angle free perspective video, a viewpoint of a frame image in the multi-angle free perspective video is selected from a virtual path, and the virtual path is a set of viewpoints for watching a to-be-watched area. The terminal device receives a user operation on the multi-angle free perspective video, determines information of at least two key frames according to the operation, the information of the key frames includes a frame time of the key frame and a frame image of the key frame, and sends the information of the key frames to a cloud. The terminal device receives a video between the frame times of the at least two key frames from the cloud.

[0207] The cloud device is adapted to receive information of at least two key frames from the terminal device, determine a viewpoint path between the at least two key frames according to a viewpoint of a frame image of the at least two key frames and the virtual path, edit the multi-angle free perspective data based on the viewpoint path to obtain a video between the frame times of the at least two key frames, and send the video between the frame times of the at least two key frames to the terminal device.

[0208] In a specific implementation, the video received by the terminal device from the cloud can be a video generated based on high-definition multi-angle free perspective data;

[0209] Before sending the information of the key frames to the cloud, the terminal device is further adapted to generate a video between frame moments of the at least two key frames based on the low-definition multi-angle free-view video, wherein the definition of the low-definition multi-angle free-view video is lower than that of the high-definition multi-angle free-view data.

[0210] Further, the terminal device is adapted to display based on the low-definition multi-angle free-view video.

[0211] In this way, the computing pressure of the terminal device can be reduced, and the user who performs editing can preview the video effect after editing of the selected key frames. By processing the high-definition multi-angle free-view data by the cloud device, the corresponding high-definition multi-angle free-view video can be obtained, which is used for subsequent display for the user who performs watching, and better visual effect is provided.

[0212] In addition, since the user who performs editing performs editing based on the display of the frame image, the visual experience is not necessary requirement of the user, and thus the data amount of the terminal device which performs editing can be reduced.

[0213] The cloud device is the cloud device described above. Other specific implementation methods and beneficial effects of the video editing system based on multi-angle free-view data can be referred to the description above, and will not be described here.

[0214] In specific implementation, the implementation manner of receiving the operation of the user can be various, which is not limited here. For example, the operation of the user on the touch screen of the terminal device can be received, or the operation of the user on the shortcut key can be received, or the operation of the user can be received through other sensors loaded on the terminal device, such as a gravity sensing sensor and a sound sensor, or the action instruction of the user can also be received through a wearable device.

[0215] In specific implementation, the manner of image display based on the multi-angle free-view data according to the operation of the user can be various. For example, the virtual viewpoint can be determined according to the operation of the user, and display based on the virtual viewpoint is performed.

[0216] In specific implementation, the information indicated by the operation of the user can be various, which can include the virtual viewpoint described above, or can include the indication of the display range of the picture, or can include the selection of the target object in the image displayed based on the multi-angle free-view data.

[0217] Correspondingly, the image display based on the multi-angle free-view video data according to the user operation can correspond to the information indicated by the user operation. For example, when the information indicated by the user operation includes selection of a target object, the image display on the image can include: obtaining a virtual information image generated based on augmented reality special effect input data of the target object; and synthesizing and displaying the virtual information image and the corresponding image.

[0218] For example, in combination with reference 1, Figure 29 The target object can be Figure 29 For example, in combination with reference 1,

[0219] It can be understood that the target object can be various in different scenes, and the virtual image information can also be various information matched with the scene, which is not limited herein.

[0220] In a specific implementation, the user operation can be received by a terminal device, and the image display can be performed on a display device, and a screen of the display device is larger than a screen of the terminal device. The terminal device can also display the corresponding image to facilitate the user operation, or can not display the image. In this way, better visual experience can be brought to the user for editing, and the user can better complete the editing operation.

[0221] Further, the image displayed by the terminal device can be a low-definition image, so as to improve the operation experience of the user and the visual experience when watching.

[0222] It can be understood that the foregoing image can be a frame image in a multi-angle free-view video, which can also be referred to as a video frame. The multi-angle free-view video is video data generated according to multi-angle free-view video data, and supports multi-view watching.

[0223] In the specific implementation of the present application, the foregoing editing interaction method can also be used for various live broadcasts. For example, in combination with reference 1, the method can be used for live broadcast of a sports event. The sports event is photographed by a collection system 11, a display device 13 can watch the sports event in real time, and editing can be performed during the watching process. Further, during the image display process, the user can adjust the viewpoint, display a virtual information image on a watching interface, or switch a background, etc.

[0224] It can be understood that the live content can be diverse, which is not limited herein. For example, it can also be a live broadcast of an education classroom scene, and a student user watching can adjust the view point, display a virtual information image on the viewing interface, or also switch the background, etc. In this scenario, the virtual information image displayed can be related to the classroom content.

[0225] The embodiments of the present application also provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program capable of running on the processor, and the processor executes the video editing method when running the computer program.

[0226] The computer device includes but is not limited to a server, a desktop computer, a smart phone, a notebook computer, a tablet computer, a smart bracelet, a smart watch, other smart devices, or a distributed processing system formed by communication connection of multiple devices of any one or more of the above.

[0227] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program capable of running the video editing method.

[0228] That is, the video editing method in the embodiments of the present application can be realized as software or computer code that can be stored in a recording medium such as a CDROM, RAM, floppy disk, hard disk or magneto-optical disk, or by computer code originally stored in a remote recording medium or non-transitory machine readable medium and downloaded through a network and to be stored in a local recording medium, so that the method described herein can be processed by such software stored on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware such as an ASIC or FPGA. It can be understood that the computer, processor, microprocessor controller or programmable hardware includes a storage component (for example, RAM, ROM, flash memory, etc.) that can store or receive software or computer code, which, when accessed and executed by the computer, processor or hardware, implements the video editing method described herein. In addition, when a general-purpose computer accesses code for implementing the video editing method shown herein, the execution of the code will convert the general-purpose computer into a special-purpose computer for executing the video editing method shown herein.

[0229] The embodiments of the present application also provide a video editing device based on multi-angle free perspective data, which can include the following units:

[0230] The display unit displays a multi-angle free perspective video, wherein the multi-angle free perspective video is generated based on multi-angle free perspective data, and the multi-angle free perspective data supports viewing of a to-be-viewed area from a view point included in a virtual path;

[0231] An operation receiving unit is configured to receive an operation of a user on the multi-angle free-view video;

[0232] A key frame information generating unit is configured to determine information of at least two key frames according to the operation, the information of the key frames including frame time of the key frames and frame image of the key frames, the viewpoint of the frame image of the key frames being selected from the virtual path;

[0233] A path determining unit is configured to determine a viewpoint path between the at least two key frames according to the viewpoint of the frame image of the key frames and the virtual path;

[0234] A video generating unit is configured to edit the multi-angle free-view data based on the viewpoint path to obtain a video between the frame time of the at least two key frames.

[0235] The specific implementation and beneficial effects of the video editing apparatus can be referred to the foregoing description, and will not be repeated here.

[0236] The embodiment of the present application further provides another video editing apparatus based on multi-angle free-view data, comprising:

[0237] A display unit is configured to display a multi-angle free-view video, the viewpoint of a frame image in the multi-angle free-view video being selected from a virtual path, the virtual path being a set of viewpoints for viewing a to-be-viewed region;

[0238] An operation receiving unit is configured to receive an operation of a user on the multi-angle free-view video;

[0239] A key frame information generating unit is configured to determine information of at least two key frames according to the operation, the information of the key frames including frame time of the key frames and frame image of the key frames;

[0240] The information of the key frames is sent to a cloud;

[0241] An edited video receiving unit is configured to receive a video between the frame time of the at least two key frames from the cloud, wherein the video between the frame time of the at least two key frames is obtained by editing the multi-angle free-view data based on a viewpoint path, the viewpoint path being determined according to the viewpoint of the frame image of the key frames and the virtual path.

[0242] The video editing apparatus can be used in a terminal device, and the specific implementation and beneficial effects can be referred to the foregoing description, and will not be repeated here.

[0243] The embodiment of the present application further provides a video editing apparatus based on multi-angle free-view data, comprising:

[0244] The operation receiving unit is adapted to receive information of at least two key frames, the information of the key frames being determined according to user operation on the multi-angle free perspective video, the multi-angle free perspective video supporting viewing of a to-be-viewed area from a viewpoint included in a virtual path;

[0245] The path determining unit is adapted to determine a viewpoint path between the at least two key frames according to the viewpoints of frame images of the at least two key frames and the virtual path;

[0246] The video generating unit is adapted to edit the multi-angle free perspective data based on the viewpoint path, to obtain a video between frame moments of the at least two key frames.

[0247] The edited video sending unit is adapted to send the video between frame moments of the at least two key frames.

[0248] The video editing apparatus can be used in the cloud, and specific implementation and beneficial effects thereof can be referred to the foregoing, and will not be described herein again.

[0249] The various units described in the video editing apparatus in the embodiments of the present application can be realized by software, hardware, firmware or any combination thereof, in whole or in part. When realized by software, the computer program can be realized in the form of a computer program product in whole or in part. The computer program product includes one or more computer programs. When the computer program instruction is loaded and executed on the computer, the flow or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer program can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another.

[0250] Moreover, the various functional modules described above can be integrated in one processing component, or each module can be physically present separately, or two or more functional modules can be integrated in one component. The above integrated component can be realized in the form of hardware or software functional modules. When the above integrated component is realized in the form of software functional modules and sold or used as an independent product, it can also be stored in a computer readable storage medium. The storage medium can be a read-only memory, a magnetic disk or an optical disk.

[0251] Although the embodiments of the present application are disclosed as above, the present application is not limited thereto. Any person skilled in the art can make various modifications and changes without departing from the spirit and scope of the embodiments of the present application, and therefore the protection scope of the present application should be subject to the scope defined by the claims.

Claims

1. A method for video editing based on multi-angle free-view data, comprising: displaying a multi-angle free-view video, the multi-angle free-view video being generated based on multi-angle free-view data, the multi-angle free-view data supporting viewing of a to-be-viewed area from viewpoints included in a virtual path; receiving an operation of a user on the multi-angle free-view video, including: determining selection of a frame time by the user; determining an indication of a selected area in a frame image corresponding to the selected frame time by the user; receiving a confirmation indication of a key frame by the user; wherein the selected area is selected by a selection box, and the user determines a position of the selection box at different frame times to confirm a display area corresponding to different frame times; determining information of at least two key frames according to the operation, the information of the key frames including frame times of the key frames and frame images of the key frames, and a viewpoint of the frame image of the key frame being selected from the virtual path; determining a viewpoint path between the at least two key frames according to the viewpoints of the frame images of the at least two key frames and the virtual path; editing the multi-angle free-view data based on the viewpoint path to obtain a video between the frame times of the at least two key frames.

2. The video editing method of claim 1, wherein, The displaying of the multi-angle free-view video includes: displaying a preselected frame image, the preselected frame image being a frame image corresponding to the selected frame time by the user.

3. The video editing method of claim 2, wherein, The determining of the information of the at least two key frames according to the operation includes: in response to the confirmation indication of the key frame by the user, determining that the selected frame time is a frame time of the key frame, and determining that the selected area in the preselected frame image is a frame image of the key frame.

4. The video editing method of claim 2, wherein, The determining of the indication of the selected area in the preselected frame image by the user includes: displaying a selection box on the preselected frame image; receiving at least one of the following operations of the user on the selection box: a drag operation and a zoom operation.

5. The video editing method of claim 2, wherein, The determining of the indication of the selected area in the preselected frame image by the user includes: receiving an operation of viewpoint transformation of the preselected frame image.

6. The video editing method of claim 1, wherein, The determining of the selection of the frame time by the user includes: receiving selection of a frame time by the user on a time axis.

7. The video editing method of claim 1, wherein, The receiving of the confirmation indication of the key frame by the user is through a key frame confirmation component.

8. The video editing method of claim 1, wherein, The receiving of the operation of the user on the multi-angle free-view video includes: receiving an operation of viewpoint transformation of the displayed frame image by the user; The displaying of the multi-angle free-view video includes: updating the display of the frame image according to the transformed viewpoint. 9.The method for video editing based on multi-angle free-view data according to claim 1, wherein a viewpoint in the viewpoint path between the at least two key frames is selected from the virtual path. 10.The method of claim 1, wherein, The viewpoint path between the at least two key frames is a smooth viewpoint path.

11. The method of claim 1, wherein, The virtual path includes a base path, the base path being obtained based on viewpoints of a plurality of images synchronized in the multi-angle free-view data, and the receiving of the operation of the user on the multi-angle free-view video includes: receiving an indication of angle transformation consistent with an angle of a viewpoint in the base path.

12. A video editing method based on multi-angle free-view data, characterized in that, including: displaying a multi-angle free-view video, a viewpoint of a frame image in the multi-angle free-view video being selected from a virtual path, the virtual path being a set of viewpoints for viewing a to-be-viewed region; receiving an operation of a user on the multi-angle free-view video, including: determining selection of a frame time by the user; determining indication of a selected region in a frame image corresponding to the selected frame time by the user; receiving confirmation indication of a key frame by the user; wherein the selected region is selected by a selection box, and the user determines a position of the selection box at different frame times to confirm a display region corresponding to different frame times; determining information of at least two key frames according to the operation, the information of the key frames including frame times of the key frames and frame images of the key frames; sending the information of the key frames to a cloud; receiving a video between the frame times of the at least two key frames from the cloud; wherein the video between the frame times of the at least two key frames is obtained based on a viewpoint path editing the multi-angle free-view data, the viewpoint path being determined according to the viewpoints of the frame images of the key frames and the virtual path.

13. The editing method of claim 12, wherein, The video received from the cloud is a video generated based on high-definition multi-angle free-view data; Before sending the information of the key frames to the cloud, the video between the frame times of the at least two key frames is generated based on low-definition multi-angle free-view video; The definition of the low-definition multi-angle free-view video is lower than that of the high-definition multi-angle free-view data.

14. The editing method of claim 12, wherein, The multi-angle free-view video displayed includes a low-definition multi-angle free-view video.

15. A video editing method based on multi-angle free-view data, characterized in that, including: receiving information of at least two key frames, the information of the key frames being determined according to an operation of a user on a multi-angle free-view video, the multi-angle free-view video supporting viewing of a to-be-viewed region from viewpoints included in a virtual path; The operation of the user on the multi-angle free-view video includes: determining selection of a frame time by the user; determining indication of a selected region in a frame image corresponding to the selected frame time by the user; receiving confirmation indication of a key frame by the user; wherein the selected region is selected by a selection box, and the user determines a position of the selection box at different frame times to confirm a display region corresponding to different frame times; determining a viewpoint path between the at least two key frames according to the viewpoints of the frame images of the at least two key frames and the virtual path; editing the multi-angle free-view data based on the viewpoint path to obtain a video between the frame times of the at least two key frames; sending the video between the frame times of the at least two key frames.

16. A video editing system based on multi-angle free-view data, characterized by, including: a terminal device, adapted to display a multi-angle free-view video, a viewpoint of a frame image in the multi-angle free-view video being selected from a virtual path, the virtual path being a set of viewpoints for viewing a to-be-viewed region; receive an operation of a user on the multi-angle free-view video; determine information of at least two key frames according to the operation, the information of the key frames including frame times of the key frames and frame images of the key frames; send the information of the key frames to a cloud; receive, from the cloud, a video between frame time instants of the at least two key frames; the operation of the user on the multi-angle free-view video includes: determining selection of frame time instants by the user; determining indication of selected regions in frame images corresponding to the selected frame time instants by the user; receiving confirmation indication of the key frames by the user; wherein the selected regions are selected by selection boxes, and the user determines positions of different selection boxes at different frame time instants to confirm display regions corresponding to different frame time instants; the cloud device is adapted to receive information of at least two key frames from the terminal device; determine a viewpoint path between the at least two key frames according to viewpoints of frame images of the at least two key frames and the virtual path; edit the multi-angle free-view data based on the viewpoint path to obtain a video between frame time instants of the at least two key frames; and send the video between frame time instants of the at least two key frames to the terminal device.

17. The multi-angle free-view video editing system according to claim 16, wherein, the video received by the terminal device from the cloud is a video generated based on high-definition multi-angle free-view data; before sending the information of the key frames to the cloud, the terminal device is further adapted to: generate a video between frame time instants of the at least two key frames based on low-definition multi-angle free-view video; and the definition of the low-definition multi-angle free-view video is lower than that of the high-definition multi-angle free-view data.

18. The multi-angle free-view video editing system according to claim 16, wherein, the terminal device is adapted to display based on low-definition multi-angle free-view video. 19.A computer device, comprising a memory and a processor, wherein a computer program is stored in the memory and is capable of running on the processor, and the computer device is characterized in that, the processor executes the computer program to perform the video editing method according to any one of claims 1 to 11 or the video editing method according to claim 12, or the video editing method according to claim 15.

20. A computer readable storage medium having stored thereon a computer program, characterized in that, the computer program performs the video editing method according to any one of claims 1 to 11 or the video editing method according to claim 12, or the video editing method according to claim 15.

21. A video editing apparatus based on multi-angle free-view data, characterized by comprising: comprise: a display unit adapted to display a multi-angle free-view video, the multi-angle free-view video being generated based on multi-angle free-view data, the multi-angle free-view data supporting viewing of a to-be-viewed region from viewpoints included in a virtual path; an operation receiving unit adapted to receive operation of the user on the multi-angle free-view video, the operation including: determining selection of frame time instants by the user; determining indication of selected regions in frame images corresponding to the selected frame time instants by the user; and receiving confirmation indication of key frames by the user; wherein the selected regions are selected by selection boxes, and the user determines positions of different selection boxes at different frame time instants to confirm display regions corresponding to different frame time instants; a key frame information generating unit adapted to determine information of at least two key frames according to the operation, the information of the key frames including frame time instants of the key frames and frame images of the key frames, and viewpoints of the frame images of the key frames being selected from the virtual path; a path determining unit adapted to determine a viewpoint path between the at least two key frames according to viewpoints of frame images of the at least two key frames and the virtual path; and The video generation unit is adapted to edit the multi-angle free perspective data based on the viewpoint path to obtain a video between frame moments of the at least two key frames.

22. A video editing device based on multi-angle free-viewpoint data, characterized in that, The video generation unit is adapted to edit the multi-angle free perspective data based on the viewpoint path to obtain a video between frame moments of the at least two key frames. The display unit is adapted to display a multi-angle free perspective video, a viewpoint of a frame image of the multi-angle free perspective video being selected from a virtual path, the virtual path being a set of viewpoints for viewing a to-be-viewed region. The operation receiving unit is adapted to receive user operations on the multi-angle free perspective video, including: determining user selection of a frame moment; determining user indication of a selected region in a frame image corresponding to the selected frame moment; receiving user confirmation indication of a key frame; wherein the selected region is selected by a selection box, and the user determines positions of different selection boxes at different frame moments to confirm display regions corresponding to different frame moments. The key frame information generation unit is adapted to determine information of at least two key frames according to the operations, the information of the key frames including frame moments of the key frames and frame images of the key frames. The key frame information generation unit is adapted to determine information of at least two key frames according to the operations, the information of the key frames including frame moments of the key frames and frame images of the key frames. The edited video receiving unit is adapted to receive a video between frame moments of the at least two key frames from the cloud, wherein the video between frame moments of the at least two key frames is obtained by editing the multi-angle free perspective data based on a viewpoint path, and the viewpoint path is determined according to viewpoints of the frame images of the key frames and the virtual path.

23. A video editing device based on multi-angle free-viewpoint data, characterized in that, The operation receiving unit is adapted to receive information of at least two key frames, the information of the key frames being determined according to user operations on a multi-angle free perspective video, the multi-angle free perspective video supporting viewing of a to-be-viewed region from viewpoints included in a virtual path. The operation receiving unit is adapted to receive information of at least two key frames, the information of the key frames being determined according to user operations on a multi-angle free perspective video, the multi-angle free perspective video supporting viewing of a to-be-viewed region from viewpoints included in a virtual path. The path determination unit is adapted to determine a viewpoint path between the at least two key frames according to viewpoints of frame images of the at least two key frames and the virtual path. The video generation unit is adapted to edit the multi-angle free perspective data based on the viewpoint path to obtain a video between frame moments of the at least two key frames. The edited video sending unit is adapted to send the video between frame moments of the at least two key frames. ​

Images