Video editing method and device, equipment and storage medium

Through neural video field coding and optimized video editing algorithms, the problem of hardware devices limiting video editing length is solved, and long video editing is efficiently edited and video quality is improved.

CN118921529AInactive Publication Date: 2024-11-08PEKING UNIV SHENZHEN GRADUATE SCHOOL
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410577183.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-05-10
Publication Date
2024-11-08
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In the prior art, the video memory demand of video editing algorithms increases rapidly with the increase of the number of frames to be edited, resulting in limited hardware devices limiting their editable video length and being unable to edit long videos.

Method used

The video to be edited is encoded into neural network parameters in the neural video field through the neural video field, and the neural network parameters are optimized based on user text instructions and target video frame images to obtain the optimized neural video field to obtain the edited video.

Benefits of technology

It realizes efficient editing of long videos under limited video memory conditions, reduces inter-frame inconsistency, improves video editing quality and editable video length.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118921529A_ABST
    Figure CN118921529A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of multimedia, and discloses a video editing method, device and equipment and a storage medium, and the method comprises the steps: coding a to-be-edited video into neural network parameters in a neural video field through the neural video field; determining a target video frame image corresponding to the target frame number in the to-be-edited video; optimizing the neural network parameters based on the user text instruction and the target video frame image through a preset text image model to obtain an optimized neural video field; and obtaining an edited video based on the optimized neural video field. According to the method, the to-be-edited video is encoded into the neural network parameters in the neural video field, and the neural network parameters are optimized based on the user text instruction and the target video frame image in the to-be-edited video, so that the edited video is obtained based on the optimized neural video field; the technical problem that a long video cannot be edited due to the fact that limited hardware equipment limits the length of a video editable by a video editing algorithm in the prior art is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of multimedia technology, and in particular to a video editing method, device, equipment and storage medium. Background Art

[0002] As electronic devices become more and more intelligent, the function of electronic devices to shoot videos and pictures makes it convenient for people to record scenery and people at any time, which can meet the daily needs of users. In order to optimize the captured pictures, video editing can usually be used to adjust specific content of the video (for example, the color of the video content, the spatial position and appearance of the object, etc.) to meet the visual needs of users.

[0003] Currently, video editing is usually achieved through video editing algorithms. However, the video memory requirements of these video editing algorithms will increase rapidly as the number of frames to be edited increases. At this time, limited hardware devices will limit the length of the editable video, resulting in the inability to edit long videos.

[0004] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of the present invention is to provide a video editing method, device, equipment and storage medium, aiming to solve the technical problem in the prior art that limited hardware equipment will limit the length of videos that can be edited by the video editing algorithm, resulting in the inability to edit long videos.

[0006] To achieve the above object, the present invention provides a video editing method, which comprises:

[0007] Encoding the video to be edited into neural network parameters in the neural video field through the neural video field;

[0008] Determine a target video frame image corresponding to a target frame sequence number in the video to be edited;

[0009] Optimizing the neural network parameters based on the user text instruction and the target video frame image through a preset text image model to obtain an optimized neural video field;

[0010] Obtain an edited video based on the optimized neural video field.

[0011] Optionally, the step of encoding the video to be edited into neural network parameters in the neural video field through the neural video field comprises:

[0012] Encoding the video to be edited based on the input space coordinates through the neural video field to obtain a first pixel value in the rendered video corresponding to the video to be edited;

[0013] Determine a target pixel value corresponding to the input spatial coordinate in the video to be edited;

[0014] The video to be edited and the rendered video are fitted based on the first pixel value and the target pixel value to encode the video to be edited into neural network parameters in the neural video field.

[0015] Optionally, the neural video field includes: an explicit part and an implicit part, the implicit part is composed of a multi-layer perceptron; the step of encoding the video to be edited based on the input space coordinates through the neural video field to obtain a first pixel value in the rendered video corresponding to the video to be edited includes:

[0016] Encoding the video to be edited based on the input space coordinates through the explicit part to obtain video encoding features;

[0017] The video coding feature is decoded through the implicit part to generate a first pixel value in the rendered video corresponding to the video to be edited.

[0018] Optionally, the explicit part is composed of three planes; and the step of encoding the video to be edited based on the input space coordinates through the explicit part to obtain the video encoding features comprises:

[0019] Encoding the video to be edited based on the input space coordinates through the explicit part to obtain a first plane feature, a second plane feature and a third plane feature corresponding to the video to be edited;

[0020] The first plane feature, the second plane feature and the third plane feature are combined to obtain a video coding feature.

[0021] Optionally, the step of optimizing the neural network parameters based on the user text instruction and the target video frame image by using a preset text image model to obtain an optimized neural video field includes:

[0022] Rendering the target video frame image through the neural video field to obtain a rendered video frame image;

[0023] Editing the rendered video frame based on a user text instruction through a preset text image model to obtain an edited frame image;

[0024] The neural network parameters are optimized based on the rendered video frame image and the edited frame image to obtain an optimized neural video field.

[0025] Optionally, the step of rendering the target video frame image through the neural video field to obtain a rendered video frame image includes:

[0026] Determine the coordinate information corresponding to all pixels in the target video frame image;

[0027] Determine a coordinate set based on the target frame sequence number and the coordinate information;

[0028] The target video frame image is rendered based on the coordinate set through the neural video field to obtain a rendered video frame image.

[0029] Optionally, the step of optimizing the neural network parameters based on the rendered video frame image and the edited frame image to obtain an optimized neural video field includes:

[0030] Using the edited frame image as a pseudo label, training the rendered video frame image and the edited frame image through an objective function to optimize the neural network parameters and obtain an optimized neural video field;

[0031] Wherein, the objective function is:

[0032]

[0033] In the formula, is the objective function, is the neural network parameter, F t For the rendered video frame image, is the editing frame image.

[0034] In addition, to achieve the above object, the present invention also provides a video editing device, the device comprising:

[0035] A video encoding module, used for encoding the video to be edited into neural network parameters in the neural video field through the neural video field;

[0036] An image determination module, used to determine a target video frame image corresponding to a target frame sequence number in the video to be edited;

[0037] A parameter optimization module, used to optimize the neural network parameters based on the user text instruction and the target video frame image through a preset text image model to obtain an optimized neural video field;

[0038] A video acquisition module is used to acquire the edited video based on the optimized neural video field.

[0039] In addition, to achieve the above objectives, the present invention also proposes a video editing device, which includes: a memory, a processor, and a video editing program stored in the memory and executable on the processor, wherein the video editing program is configured to implement the steps of the video editing method described above.

[0040] In addition, to achieve the above-mentioned purpose, the present invention further proposes a storage medium, on which a video editing program is stored, and when the video editing program is executed by a processor, the steps of the video editing method described above are implemented.

[0041] The present invention discloses encoding a video to be edited into neural network parameters in a neural video field through a neural video field; determining a target video frame image corresponding to a target frame number in the video to be edited; optimizing the neural network parameters based on user text instructions and target video frame images through a preset text image model to obtain an optimized neural video field; acquiring an edited video based on the optimized neural video field; compared with the prior art, the video memory requirement of the video editing algorithm will increase rapidly with the increase in the number of frames to be edited, but the limited hardware equipment will limit the editable video length. Since the present invention encodes the video to be edited into neural network parameters in the neural video field through a neural video field, and optimizes the neural network parameters based on user text instructions and the target video frame images in the video to be edited, so as to acquire the edited video based on the optimized neural video field, the technical problem in the prior art that the limited hardware equipment will limit the editable video length of the video editing algorithm, resulting in the inability to edit long videos, is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of the structure of a video editing device in a hardware operating environment involved in an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of a flow chart of a first embodiment of a video editing method of the present invention;

[0044] Figure 3 A schematic diagram of the video editing process in the video editing method of the present invention;

[0045] Figure 4 A schematic diagram of a flow chart of a second embodiment of a video editing method of the present invention;

[0046] Figure 5 A detailed flow chart of video editing in the video editing method of the present invention;

[0047] Figure 6 A schematic diagram of a flow chart of a third embodiment of a video editing method of the present invention;

[0048] Figure 7 A diagram showing the effect of video editing performed by the video editing method of the present invention;

[0049] Figure 8 It is a structural block diagram of the first embodiment of the video editing device of the present invention.

[0050] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0052] Reference Figure 1 , Figure 1 The present invention is a schematic diagram of the structure of a video editing device in a hardware operating environment according to an embodiment of the present invention.

[0053] like Figure 1 As shown, the video editing device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0054] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the video editing device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0055] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a video editing program.

[0056] exist Figure 1In the video editing device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the video editing device of the present invention can be set in the video editing device, and the video editing device calls the video editing program stored in the memory 1005 through the processor 1001, and executes the video editing method provided by the embodiment of the present invention.

[0057] The embodiment of the present invention provides a video editing method, referring to Figure 2 , Figure 2 FIG. 4 is a flow chart of a first embodiment of a video editing method according to the present invention.

[0058] In this embodiment, the video editing method includes the following steps:

[0059] Step S10: Encode the video to be edited into neural network parameters in the neural video field through the neural video field.

[0060] It should be noted that the execution subject of the method of this embodiment can be a video editing device for adjusting the content of any video, such as a mobile phone, a tablet computer, a personal computer, etc., or other video editing systems that can achieve the same or similar functions and include the video editing device. Here, the video editing method provided in this embodiment and the following embodiments is specifically described using a video editing system (hereinafter referred to as the system).

[0061] It should be noted that the above-mentioned neural video field is a memory-efficient video signal representation method, and this scheme can use this efficient video editing framework to perform long video editing tasks.

[0062] It is understandable that the video to be edited may be any video that needs to have its content adjusted, wherein the color, spatial position and appearance of the object in the video to be edited may be adjusted in this solution.

[0063] In this embodiment, the system can encode the video to be edited through the neural video field, so that the video to be edited can be represented as the network parameters of the neural video field, that is, the video signal can be effectively encoded using the neural video field memory, and the editing effect can be injected into the video in the form of optimizing the network parameters of the neural video field, thereby realizing long video editing of hundreds or thousands of frames. At the same time, the system encodes the original video by utilizing the fitting ability of the neural video field, records its content, timing, and motion priors, so that the encoded video field can render the video with high fidelity and show content consistent with the original video, so that the video can be edited with high quality and timing consistency after the field optimization stage, reducing the inconsistency between frames, thereby improving the quality of video editing.

[0064] Step S20: determining a target video frame image corresponding to a target frame sequence number in the video to be edited.

[0065] It should be noted that the target frame number can be a randomly selected frame number, and this embodiment does not limit the specific number. Correspondingly, the target video frame image can be the image corresponding to the target frame number in the video frame to be edited. In practical applications, a frame number t can be randomly selected as the target frame number, and the target video frame image can be the t-th frame image V in the video to be edited. t .

[0066] Step S30: Optimizing the neural network parameters based on the user text instructions and the target video frame image through a preset text image model to obtain an optimized neural video field.

[0067] It should be noted that the above-mentioned preset text image model may be a model for implementing video signal editing through a user's text instruction. In this embodiment, the preset text image model includes but is not limited to a T2I model (text-to-image diffusion model).

[0068] It should be understood that the above-mentioned user text instruction may be an instruction input by the user for guiding the editing of the video signal, and this embodiment does not limit the specific text content input by the user.

[0069] In practical applications, the system can introduce user text instructions to adjust the network parameters of the neural video field so that it can be updated in the editing direction of the text description. In order to provide the supervision information required for updating the parameters, this solution can use the trained T2I model to edit the image frame by frame based on the frame (image) of the video to be edited and the user text instruction (text), and use the edited image as a pseudo label to optimize the neural network parameters of the neural video field. At this time, the optimized neural video field can be obtained.

[0070] Step S40: Obtaining an edited video based on the optimized neural video field.

[0071] In this embodiment, after the system optimizes the network parameters of the neural video field, it can directly render an edited video that conforms to the user's instruction description and retains the semantic layout of the original video through the optimized neural video field.

[0072] In the specific implementation, refer to Figure 3 , Figure 3 FIG. 4 is a flow chart of video editing in the video editing method of the present invention. Figure 3 As shown, first the system can select the video to be edited and the editing instructions to build a neural video field network And determine the pre-trained T2I model ε θ The system can then encourage the neural video field Rendered frame t image F t With the tth frame V of the original video t Consistent, so that the neural video field network Fit the video to be edited and encode it into the neural network parameters of the neural video field Then we enter the optimization phase. In the optimization phase, we can first randomly select a frame number t and select the tth frame V from the original video. t (i.e., target video frame image) and let Render the tth frame image F t . Then, using the user text instruction (e.g., “turn it into a brown bear”) as the text condition, V t For image conditions, using the T2I model ε θ Edit the rendering result F under the guidance of these two conditions t , get the edited image Afterwards, continue with is a pseudo label, encouraging rendering results F t and Consistent, optimized neural video field Parameters After the optimization is completed, The edited video can be directly rendered to meet the user's instruction description and retain the semantic layout of the original video.

[0073] The present embodiment discloses encoding a video to be edited into neural network parameters in a neural video field through a neural video field; determining a target video frame image corresponding to a target frame number in the video to be edited; optimizing the neural network parameters based on user text instructions and target video frame images through a preset text image model to obtain an optimized neural video field; obtaining an edited video based on the optimized neural video field; compared with the prior art, the video memory requirement of the video editing algorithm will increase rapidly with the increase in the number of frames to be edited, but limited hardware equipment will limit the length of the video that can be edited. Since the present embodiment encodes the video to be edited into neural network parameters in the neural video field through a neural video field, and optimizes the neural network parameters based on user text instructions and target video frame images in the video to be edited, so as to obtain the edited video based on the optimized neural video field, the technical problem in the prior art that limited hardware equipment will limit the length of the video that can be edited by the video editing algorithm, resulting in the inability to edit long videos, is solved.

[0074] refer to Figure 4 , Figure 4 FIG. 4 is a flow chart of a second embodiment of a video editing method according to the present invention.

[0075] Based on the first embodiment above, in order to efficiently encode a long video signal with limited video memory, in this embodiment, step S10 includes:

[0076] Step S101: encoding a video to be edited based on input spatial coordinates through a neural video field to obtain a first pixel value in a rendered video corresponding to the video to be edited.

[0077] It should be noted that the above-mentioned input spatial coordinates may be any spatial coordinates input into the system. For example, in this embodiment, the input spatial coordinates may be expressed as (x, y, t).

[0078] It should be noted that the first pixel value may be a pixel value at an input space coordinate position in the rendered video of the video to be edited.

[0079] Furthermore, the neural video field includes: an explicit part and an implicit part, and the implicit part is composed of a multi-layer perceptron; the step S101 includes:

[0080] Step S101a: Encode the video to be edited based on the input space coordinates through the explicit part to obtain video encoding features.

[0081] It should be noted that this solution can pre-build an explicit-implicit hybrid architecture as a neural video field And use it to encode the video information to be edited, that is, the neural video field in this embodiment includes an explicit part and an implicit part.

[0082] It can be understood that the above-mentioned video encoding features may be features obtained after encoding the video to be edited.

[0083] Furthermore, the explicit part is composed of three planes; the step S101a includes: encoding the video to be edited based on the input spatial coordinates through the explicit part to obtain the first plane feature, the second plane feature and the third plane feature corresponding to the video to be edited; merging the first plane feature, the second plane feature and the third plane feature to obtain the video encoding feature.

[0084] It should be noted that the explicit part of the neural video field in this embodiment can be composed of three planes, namely, three feature maps composed of learnable parameters. Figure 5 , Figure 5 FIG. 4 is a detailed flow chart of video editing in the video editing method of the present invention. Figure 5 As shown, the explicit part of the neural video field can be denoted as θ ex The three planes constituting the explicit part may be the xy plane, the xt plane and the yt plane respectively. Accordingly, the first plane feature, the second plane feature and the third plane feature may be the features corresponding to the input space coordinate position of the video to be edited in these three planes.

[0085] In practical applications, if the input spatial coordinates are (x, y, t), the system can obtain the first plane features (x, y), the second plane features (x, t) and the third plane features (y, t) corresponding to the position (x, y, t) in the video to be edited in the xy plane, the xt plane and the yt plane respectively, and then the first plane features, the second plane features and the third plane features can be merged to obtain the video coding features, such as Figure 5 The cube in Figure 2 is shown.

[0086] Step S101b: decoding the video coding feature through the implicit part to generate a first pixel value in the rendered video corresponding to the video to be edited.

[0087] It should be understood that the implicit part in this embodiment can be composed of a multi-layer perceptron, such as Figure 5 θ im As shown in FIG. 1 , a multilayer perceptron (MLP) is a feedforward artificial neural network model that can map multiple input data sets to a single output data set.

[0088] In practical applications, the system can input the video coding features into a multi-layer perceptron, and the multi-layer perceptron can output a pixel value (R, G, B) after decoding it, that is, the first pixel value mentioned above.

[0089] Step S102: Determine the target pixel value corresponding to the input space coordinate in the video to be edited.

[0090] It can be understood that the above target pixel value may be a pixel value at an input space coordinate position in the video to be edited.

[0091] Step S103: fitting the video to be edited and the rendered video based on the first pixel value and the target pixel value to encode the video to be edited into neural network parameters in the neural video field.

[0092] In this embodiment, the system can encourage the decoded pixel (i.e., the first pixel value) to be consistent with the pixel value at the position (x, y, t) of the original video (i.e., the target pixel value). Parameters Among them, the objective function is as follows:

[0093]

[0094] It should be noted that during the video fitting process, the video editing effects are injected into Parameters In, and because The motion and timing priors of the original video have been learned in the video fitting stage, so the final rendered edited video not only shows an editing effect consistent with the text, but also has good timing consistency, thereby alleviating the problem of inconsistency between frames and improving the quality of video editing.

[0095] This embodiment discloses encoding a video to be edited based on input spatial coordinates through a neural video field to obtain a first pixel value in a rendered video corresponding to the video to be edited; determining a target pixel value corresponding to the input spatial coordinates in the video to be edited; fitting the video to be edited and the rendered video based on the first pixel value and the target pixel value to encode the video to be edited into neural network parameters in the neural video field. Since the neural video field in this embodiment is based on explicit-implicit construction, it is possible to efficiently encode long video signals with limited video memory and represent the video as parameters of a neural network. At the same time, this embodiment constructs an explicit-implicit neural video field by combining three planes and a multi-layer perceptron, thereby reducing the video memory overhead required for encoding the video while ensuring the fitting accuracy, providing support for the subsequent implementation of long video editing of hundreds or thousands of frames.

[0096] refer to Figure 6 , Figure 6 FIG. 4 is a flow chart of a third embodiment of a video editing method according to the present invention.

[0097] Based on the above embodiments, in this embodiment, step S30 includes:

[0098] Step S301: Rendering the target video frame image through the neural video field to obtain a rendered video frame image.

[0099] It should be understood that the above-mentioned rendered video frame image may be an image obtained after the neural video field renders the target video frame image.

[0100] Furthermore, the step S301 includes: determining the coordinate information corresponding to all pixel points in the target video frame image; determining a coordinate set based on the target frame sequence number and the coordinate information; rendering the target video frame image based on the coordinate set through the neural video field to obtain a rendered video frame image.

[0101] It should be noted that the above coordinate information can be the horizontal coordinates and vertical coordinates of all pixels in the target video frame image. Correspondingly, the above coordinate set can be a set consisting of the horizontal coordinates, vertical coordinates and target frame number of the pixel points. For example, if the target video frame image is the t-th frame image in the video to be edited, that is, the target frame number is t, then the horizontal coordinate set of all pixels on the target video frame image can be represented as X, and the coordinate set of the vertical coordinates of all pixels on the target video frame image can be represented as Y. At this time, the above coordinate set can be represented as (X, Y, t).

[0102] In practical applications, neural video fields The image of the tth frame can be rendered according to the input coordinate set (X, Y, t), that is, the above-mentioned rendered video frame image F t . Among them, the rendering function is constructed as follows:

[0103]

[0104] Step S302: Editing the rendered video frame based on a user text instruction using a preset text image model to obtain an edited frame image.

[0105] It should be noted that in this embodiment, the system can use the T2I model to edit and render the video frame image F based on the user text instruction. t , get the edit frame image

[0106] Step S303: Optimizing the neural network parameters based on the rendered video frame image and the edited frame image to obtain an optimized neural video field.

[0107] Further, the step S303 includes: using the edited frame image as a pseudo label, training the rendered video frame image and the edited frame image through an objective function to optimize the neural network parameters and obtain an optimized neural video field;

[0108] Wherein, the objective function is:

[0109]

[0110] In the formula, is the objective function, is the neural network parameter, F t For the rendered video frame image, is the editing frame image.

[0111] In this embodiment, by introducing a trained T2I model to optimize the neural video field and inject editing effects, any image editing and processing algorithm can be applied to image editing and processing, so that video editing can benefit from the development of image editing methods and even complete video-related downstream tasks.

[0112] In the specific implementation, Figure 5 As shown, in the on-site editing stage, the system can first determine the pre-trained T2I model ε θ and user instructions to start optimizing the neural video field trained in the previous stage In this process, a frame number t is randomly selected in each iteration to retrieve the tth frame V in the video to be edited. t , where V t and user instructions as image conditions and text conditions, respectively, for editing the T2I model. Then, Render the image F of frame t according to the input coordinate set (X, Y, t) t , and edit F with the T2I model t , get the edit frame Afterwards, the system can generate edit frames using the T2I model. For pseudo labels, encourage The rendered frame F t and Consistent, optimized The parameters of the rendered frame gradually show the editing effect that conforms to the user's instructions. In this process, the editing effect is injected into Parameters In, and because The motion and timing priors of the original video have been learned in the video fitting stage, so the final rendered edited video not only shows the editing effect consistent with the text, but also has good timing consistency, which reduces the problem of inconsistency between frames. The entire video can be rendered at the given coordinates as the result of video editing. Figure 7 , Figure 7 The figure shows the effect of video editing performed by the video editing method of the present invention.

[0113] This embodiment discloses rendering the target video frame image through the neural video field to obtain the rendered video frame image; editing the rendered video frame based on the user text instruction through the preset text image model to obtain the edited frame image; optimizing the neural network parameters based on the rendered video frame image and the edited frame image to obtain the optimized neural video field. Since the trained T2I model can be introduced in this embodiment to optimize the neural video field and inject the editing effect, any image editing and processing algorithm can be applied to image editing and processing, thereby improving the flexibility and extensibility of video editing.

[0114] In addition, an embodiment of the present invention further provides a storage medium, on which a video editing program is stored. When the video editing program is executed by a processor, the steps of the video editing method described above are implemented.

[0115] Reference Figure 8 , Figure 8 It is a structural block diagram of the first embodiment of the video editing device of the present invention.

[0116] like Figure 8 As shown, the video editing device proposed in the embodiment of the present invention includes:

[0117] A video encoding module 801, used to encode the video to be edited into neural network parameters in the neural video field through the neural video field;

[0118] An image determination module 802 is used to determine a target video frame image corresponding to a target frame sequence number in the video to be edited;

[0119] A parameter optimization module 803 is used to optimize the neural network parameters based on the user text instruction and the target video frame image through a preset text image model to obtain an optimized neural video field;

[0120] The video acquisition module 804 is used to acquire the edited video based on the optimized neural video field.

[0121] The video editing device of the present embodiment discloses encoding the video to be edited into the neural network parameters in the neural video field through the neural video field; determining the target video frame image corresponding to the target frame number in the video to be edited; optimizing the neural network parameters based on the user text instructions and the target video frame image through a preset text image model to obtain the optimized neural video field; obtaining the edited video based on the optimized neural video field; compared with the video editing algorithm in the prior art, the video memory requirement will increase rapidly with the increase in the number of frames to be edited, but the limited hardware equipment will limit the editable video length. Since the present embodiment encodes the video to be edited into the neural network parameters in the neural video field through the neural video field, and optimizes the neural network parameters based on the user text instructions and the target video frame image in the video to be edited, so as to obtain the edited video based on the optimized neural video field, the technical problem in the prior art that the limited hardware equipment will limit the editable video length of the video editing algorithm, resulting in the inability to edit long videos, is solved.

[0122] Based on the above-mentioned first embodiment of the video editing device of the present invention, a second embodiment of the video editing device of the present invention is proposed.

[0123] In this embodiment, the video encoding module 801 is also used to encode the video to be edited based on the input spatial coordinates through the neural video field to obtain a first pixel value in the rendered video corresponding to the video to be edited; determine the target pixel value corresponding to the input spatial coordinates in the video to be edited; and fit the video to be edited and the rendered video based on the first pixel value and the target pixel value to encode the video to be edited into the neural network parameters in the neural video field.

[0124] Furthermore, the neural video field includes: an explicit part and an implicit part, the implicit part is composed of a multi-layer perceptron; the video encoding module 801 is also used to encode the video to be edited based on the input spatial coordinates through the explicit part to obtain video encoding features; and decode the video encoding features through the implicit part to generate a first pixel value in the rendered video corresponding to the video to be edited.

[0125] Furthermore, the explicit part is composed of three planes; the video encoding module 801 is also used to encode the video to be edited based on the input spatial coordinates through the explicit part to obtain the first plane feature, the second plane feature and the third plane feature corresponding to the video to be edited; the first plane feature, the second plane feature and the third plane feature are merged to obtain the video encoding feature.

[0126] This embodiment discloses encoding a video to be edited based on input spatial coordinates through a neural video field to obtain a first pixel value in a rendered video corresponding to the video to be edited; determining a target pixel value corresponding to the input spatial coordinates in the video to be edited; fitting the video to be edited and the rendered video based on the first pixel value and the target pixel value to encode the video to be edited into neural network parameters in the neural video field. Since the neural video field in this embodiment is based on explicit-implicit construction, it is possible to efficiently encode long video signals with limited video memory and represent the video as parameters of a neural network. At the same time, this embodiment constructs an explicit-implicit neural video field by combining three planes and a multi-layer perceptron, thereby reducing the video memory overhead required for encoding the video while ensuring the fitting accuracy, providing support for the subsequent implementation of long video editing of hundreds or thousands of frames.

[0127] Based on the above-mentioned device embodiments, a third embodiment of the video editing device of the present invention is proposed.

[0128] In this embodiment, the parameter optimization module 803 is also used to render the target video frame image through the neural video field to obtain a rendered video frame image; edit the rendered video frame based on user text instructions through a preset text image model to obtain an edited frame image; optimize the neural network parameters based on the rendered video frame image and the edited frame image to obtain an optimized neural video field.

[0129] Furthermore, the parameter optimization module 803 is also used to determine the coordinate information corresponding to all pixel points in the target video frame image; determine a coordinate set based on the target frame sequence number and the coordinate information; render the target video frame image based on the coordinate set through the neural video field to obtain a rendered video frame image.

[0130] Furthermore, the parameter optimization module 803 is further used to use the edited frame image as a pseudo label, and train the rendered video frame image and the edited frame image through an objective function to optimize the neural network parameters and obtain an optimized neural video field;

[0131] Wherein, the objective function is:

[0132]

[0133] In the formula, is the objective function, is the neural network parameter, F t For the rendered video frame image, is the editing frame image.

[0134] This embodiment discloses rendering the target video frame image through the neural video field to obtain the rendered video frame image; editing the rendered video frame based on the user text instruction through the preset text image model to obtain the edited frame image; optimizing the neural network parameters based on the rendered video frame image and the edited frame image to obtain the optimized neural video field. Since the trained T2I model can be introduced in this embodiment to optimize the neural video field and inject the editing effect, any image editing and processing algorithm can be applied to image editing and processing, thereby improving the flexibility and extensibility of video editing.

[0135] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0136] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0137] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0138] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A video editing method, characterized in that: The video editing method comprises: Encoding the video to be edited into neural network parameters in the neural video field through the neural video field; Determine a target video frame image corresponding to a target frame sequence number in the video to be edited; Optimizing the neural network parameters based on the user text instruction and the target video frame image through a preset text image model to obtain an optimized neural video field; Obtain an edited video based on the optimized neural video field.

2. The video editing method according to claim 1, characterized in that: The step of encoding the video to be edited into neural network parameters in the neural video field through the neural video field comprises: Encoding the video to be edited based on the input space coordinates through the neural video field to obtain a first pixel value in the rendered video corresponding to the video to be edited; Determine a target pixel value corresponding to the input spatial coordinate in the video to be edited; The video to be edited and the rendered video are fitted based on the first pixel value and the target pixel value to encode the video to be edited into neural network parameters in the neural video field.

3. The video editing method according to claim 2, characterized in that: The neural video field includes: an explicit part and an implicit part, wherein the implicit part is composed of a multi-layer perceptron; the step of encoding the video to be edited based on the input space coordinates through the neural video field to obtain a first pixel value in the rendered video corresponding to the video to be edited includes: Encoding the video to be edited based on the input space coordinates through the explicit part to obtain video encoding features; The video coding feature is decoded through the implicit part to generate a first pixel value in the rendered video corresponding to the video to be edited.

4. The video editing method according to claim 3, characterized in that: The explicit part is composed of three planes; the step of encoding the video to be edited based on the input space coordinates through the explicit part to obtain the video encoding features includes: Encoding the video to be edited based on the input space coordinates through the explicit part to obtain a first plane feature, a second plane feature and a third plane feature corresponding to the video to be edited; The first plane feature, the second plane feature and the third plane feature are combined to obtain a video coding feature.

5. The video editing method according to claim 1, characterized in that: The step of optimizing the neural network parameters based on the user text instruction and the target video frame image through a preset text image model to obtain an optimized neural video field includes: Rendering the target video frame image through the neural video field to obtain a rendered video frame image; Editing the rendered video frame based on a user text instruction through a preset text image model to obtain an edited frame image; The neural network parameters are optimized based on the rendered video frame image and the edited frame image to obtain an optimized neural video field.

6. The video editing method according to claim 5, characterized in that: The step of rendering the target video frame image by using the neural video field to obtain a rendered video frame image comprises: Determine the coordinate information corresponding to all pixels in the target video frame image; Determine a coordinate set based on the target frame sequence number and the coordinate information; The target video frame image is rendered based on the coordinate set through the neural video field to obtain a rendered video frame image.

7. The video editing method according to claim 5, characterized in that: The step of optimizing the neural network parameters based on the rendered video frame image and the edited frame image to obtain an optimized neural video field comprises: Using the edited frame image as a pseudo label, training the rendered video frame image and the edited frame image through an objective function to optimize the neural network parameters and obtain an optimized neural video field; Wherein, the objective function is: In the formula, is the objective function, is the neural network parameter, F t For the rendered video frame image, is the editing frame image.

8. A video editing device, characterized in that: The device comprises: A video encoding module, used for encoding the video to be edited into neural network parameters in the neural video field through the neural video field; An image determination module, used to determine a target video frame image corresponding to a target frame sequence number in the video to be edited; A parameter optimization module, used to optimize the neural network parameters based on the user text instruction and the target video frame image through a preset text image model to obtain an optimized neural video field; A video acquisition module is used to acquire the edited video based on the optimized neural video field.

9. A video editing device, characterized in that: The device comprises: a memory, a processor, and a video editing program stored in the memory and executable on the processor, wherein the video editing program is configured to implement the steps of the video editing method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a video editing program, and when the video editing program is executed by the processor, the steps of the video editing method according to any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Blind watermark implantation method and device, blind watermark detection method and device, equipment and storage medium

    CN111932432A