Video generation method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present disclosure provide a video generation method and apparatus, an electronic device, and a storage medium. The method comprises: obtaining an initial camera movement sequence on the basis of an initial camera movement template, wherein the initial camera movement sequence comprises at least one camera movement node corresponding to an initial video, and the camera movement node is used for representing a camera movement feature of a shot picture during video playback; acquiring a video content feature of a video to be processed; obtaining an optimization camera movement sequence on the basis of the video content feature and the initial camera movement sequence, wherein the optimization camera movement sequence comprises at least one optimization camera movement node, and a camera movement feature corresponding to the optimization camera movement node matches the video content feature; and processing said video on the basis of the optimization camera movement sequence, and generating an output video, wherein a shot picture of the output video is provided with the camera movement feature corresponding to the optimization camera movement node. The output video is generated on the basis of the optimization camera movement sequence and said video, so as to solve the problem of the mismatch between a camera movement special effect generated by the initial camera movement template and the content of said video, thereby improving the visual impression of the camera movement special effect.

Description

Video generation method, device, electronic device and storage medium

[0001] This application claims priority to the Chinese invention patent application entitled “Video Generation Method, Device, Electronic Device and Storage Medium” and application number 2023107702269, filed on June 27, 2023. The entire contents of that application are incorporated herein by reference. Technical Field

[0002] The embodiments of the present disclosure relate to the field of Internet technology, and in particular to a video generation method, device, electronic device, and storage medium. Background Art

[0003] Currently, video platforms and video processing applications offer users a variety of special effects templates to improve the efficiency and quality of their own video creation. Among them, camera templates process videos to add simulated camera effects, enhancing the visual experience.

[0004] However, in the actual application of the camera movement template, there is a problem that the camera movement effect provided by the camera movement template does not match the content of the video to be processed actually shot by the user, which affects the visual perception of the camera movement special effects.

[0005] Summary of the Invention

[0006] In a first aspect, an embodiment of the present disclosure provides a video generation method, comprising: obtaining an initial camera movement sequence based on an initial camera movement template, the initial camera movement sequence including at least one camera movement node corresponding to the initial video, the camera movement node being used to characterize the camera movement characteristics of the lens picture when the video is played; obtaining video content characteristics of the video to be processed, and obtaining an optimized camera movement sequence based on the video content characteristics and the initial camera movement sequence, the optimized camera movement sequence including at least one optimized camera movement node, the camera movement characteristics corresponding to the optimized camera movement node matching the video content characteristics; processing the video to be processed based on the optimized camera movement sequence to generate an output video, the lens picture of the output video having the camera movement characteristics corresponding to the optimized camera movement node.

[0007] In a second aspect, an embodiment of the present disclosure provides a video generation device, comprising: a loading module, for obtaining an initial camera movement sequence based on an initial camera movement template, wherein the initial camera movement sequence includes at least one camera movement node corresponding to the initial video, and the camera movement node is used to characterize the camera movement characteristics of the lens picture when the video is played; a processing module, for obtaining video content characteristics of the video to be processed, and obtaining an optimized camera movement sequence based on the video content characteristics and the initial camera movement sequence, wherein the optimized camera movement sequence includes at least one optimized camera movement node, and the camera movement characteristics corresponding to the optimized camera movement node match the video content characteristics; an output module, for processing the video to be processed based on the optimized camera movement sequence to generate an output video, and the lens picture of the output video has the camera movement characteristics corresponding to the optimized camera movement node.

[0008] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the video generation method described in the first aspect and various possible designs of the first aspect.

[0009] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the video generation method described in the first aspect and various possible designs of the first aspect is implemented.

[0010] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the video generation method described in the first aspect and various possible designs of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments or current solutions of the present disclosure, a brief introduction will be given below to the drawings required for use in the description of the embodiments or current solutions. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0012] FIG1 is a diagram of an application scenario of the video generation method provided by an embodiment of the present disclosure;

[0013] FIG2 is a flow chart of a video generation method according to an embodiment of the present disclosure;

[0014] FIG3 is a flowchart of a specific implementation of step S102 in the embodiment shown in FIG2 ;

[0015] FIG4 is a schematic diagram of a process for determining video content features according to an embodiment of the present disclosure;

[0016] FIG5 is a flow chart of a possible implementation of step S103 in the embodiment shown in FIG2 ;

[0017] FIG6 is a schematic diagram of an optimized camera movement node generated based on a first content feature according to an embodiment of the present disclosure;

[0018] FIG7 is a flow chart of another possible implementation of step S103 in the embodiment shown in FIG2 ;

[0019] FIG8 is a schematic diagram of an optimized camera movement node generated based on a second content feature according to an embodiment of the present disclosure;

[0020] FIG9 is a flow chart of another possible implementation of step S103 in the embodiment shown in FIG2 ;

[0021] FIG10 is a flowchart of a specific implementation of step S1037 in the embodiment shown in FIG9 ;

[0022] FIG11 is a second flow chart of a video generation method according to an embodiment of the present disclosure;

[0023] FIG12 is a flowchart of a specific implementation of step S205 in the embodiment shown in FIG11 ;

[0024] FIG13 is a flowchart of a specific implementation of step S206 in the embodiment shown in FIG11 ;

[0025] FIG14 is a structural block diagram of a video generating apparatus provided by an embodiment of the present disclosure;

[0026] FIG15 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure;

[0027] FIG16 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] The embodiments of the present disclosure provide a video generation method, device, electronic device, and storage medium to overcome the problem that the camera movement effect provided by the camera movement template does not match the content of the video to be processed actually shot by the user.

[0029] The video generation method, device, electronic device and storage medium provided in this embodiment obtain an initial camera movement sequence based on an initial camera movement template, wherein the initial camera movement sequence includes at least one camera movement node corresponding to the initial video, and the camera movement node is used to represent the camera movement characteristics of the camera lens during video playback; obtain video content characteristics of the video to be processed, and obtain an optimized camera movement sequence based on the video content characteristics and the initial camera movement sequence, wherein the optimized camera movement sequence includes at least one optimized camera movement node, and the camera movement characteristics corresponding to the optimized camera movement node match the video content characteristics; process the video to be processed based on the optimized camera movement sequence to generate an output video, and the camera lens of the output video has the camera movement characteristics corresponding to the optimized camera movement node. By utilizing the video content features of the video to be processed, the initial camera movement template is set to obtain an optimized camera movement sequence composed of optimized camera movement nodes, and an output video with camera movement effects is generated based on the optimized camera movement sequence. Since the camera movement features corresponding to the optimized camera movement nodes in the optimized camera movement sequence match the video content features of the video to be processed, the output video is generated based on the optimized camera movement sequence and the video to be processed, thereby avoiding the problem of mismatch between the camera movement effects generated by the initial camera movement template and the content of the video to be processed, and improving the visual perception of the camera movement effects.

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0031] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0032] The following explains the application scenarios of the embodiments of the present disclosure:

[0033] FIG1 is an application scenario diagram of the video generation method provided by an embodiment of the present disclosure. The video generation method provided by an embodiment of the present disclosure can be applied to an application scenario of adding camera movement effects to a video. Specifically, as shown in FIG1 , the method provided by an embodiment of the present disclosure can be applied to electronic devices such as terminal devices and servers. Taking a terminal device as an example, the terminal device is, for example, a smart phone, which implements the video generation method provided by this embodiment by running a video processing application (Application, APP). Specifically, as shown in FIG1 , after loading a pre-generated video to be processed 11, the terminal device adds camera movement effects to the video to be processed by triggering a camera movement template control 12 provided by the video processing application, thereby generating an output video 13 with a camera movement effect on the lens screen. Among them, the camera movement effect includes, for example, movement and scaling of the lens screen.

[0034] In the related art, the camera movement effect provided by the camera movement template is achieved by manually or automatically cropping the image of the initial video to generate multiple moving lens images to simulate real camera movement. Therefore, the camera movement effect achieved by the camera movement template can match the image of the initial video. However, for the video to be processed that needs to apply the camera movement template to obtain the same camera movement effect, the corresponding video content is different from the initial video. Therefore, the camera movement effect provided by the camera movement template does not match the content of the video to be processed actually shot by the user. This causes the camera movement special effects of the output video generated by applying the camera movement template to have problems such as the center of the lens image cannot focus on the portrait, the lens image is zoomed too large or too small, and the visual perception is affected.

[0035] The embodiments of the present disclosure provide a video generation method to solve the above problems.

[0036] Referring to FIG2 , FIG2 is a flow chart of a video generation method provided by an embodiment of the present disclosure. The method of this embodiment can be applied in a terminal device or a server, and the video generation method includes:

[0037] Step S101: Obtain an initial camera movement sequence based on an initial camera movement template. The initial camera movement sequence includes at least one camera movement node corresponding to an initial video, and the camera movement node is used to represent the camera movement characteristics of the shot image during video playback.

[0038] First, let's briefly introduce the concept of camera movement. Camera movement refers to the process of moving the camera lens during video capture, through operations such as pushing, pulling, shaking, shifting, following, swinging, raising, and lowering. This ensures the subject (such as a dancer) remains in focus and maintains a coherent movement, thereby highlighting the subject and enhancing the visual experience. This is a common video shooting technique.

[0039] Camera movement effects, on the other hand, involve processing already-filmed videos and dynamically cropping the original video. This process involves scaling, rotating, and other image processing steps to create continuous, moving footage, simulating real-world camera movement. A camera movement template is the data used to achieve these camera movement effects. By applying this template to the video being processed, the resulting video (i.e., the output video) can be imbued with the corresponding camera movement effects.

[0040] Furthermore, the initial camera movement template in this embodiment is generated based on the initial video. The initial camera movement template includes an initial camera movement sequence, which is used to represent one or more independent camera movement effects. Specifically, the camera movement sequence is composed of at least one ordered camera movement node, which is used to characterize the camera movement characteristics of the camera lens during video playback, such as left-right shaking, up-and-down shaking, zooming, etc., thereby achieving an independent camera movement effect. The camera movement characteristics also include specific parameters of the camera lens motion, such as shaking amplitude and frequency. Each camera movement node in the camera movement sequence is associated with a timestamp, which may include a start timestamp and an end timestamp, respectively corresponding to the start time and end time of the camera movement effect represented by the camera movement node. The initial camera movement sequence corresponds to one or more camera movement effects in the initial video, and the camera movement nodes in the initial camera movement sequence characterize the camera movement characteristics of the camera lens during the initial video playback. Furthermore, by parsing the model data corresponding to the initial camera movement model, the initial camera movement sequence in the initial camera movement model can be obtained. The initial camera sequence in the initial camera model can be manually set by the user using keyframes (K frames) based on the content of the initial video. Alternatively, it can be automatically generated by the terminal device or server after performing content recognition based on the content of the initial video. The specific generation process of the initial camera model will not be described here.

[0041] The above steps can be executed based on a triggering operation within the application. For example, after a user clicks the control for camera movement template A (the initial camera movement template), the terminal device loads the template data corresponding to camera movement template A, thereby obtaining the initial camera movement sequence. The specific interactive operation process is not specifically limited here and will not be further described.

[0042] Step S102: Obtain video content features of the video to be processed.

[0043] Exemplarily, the terminal device can load the video to be processed by responding to user instructions. The video to be processed can be a video shot by the terminal device, or a video shot or generated by other devices. More specifically, the content of the video to be processed can be various, such as dance videos, landscape videos, cartoon videos, etc., which are not specifically limited here. After the terminal device loads the video to be processed, the corresponding video content features can be obtained by performing content recognition on the video to be processed. Among them, the video content features are information used to characterize the video content of the video to be processed. In one possible implementation, the video content features can be implemented in the form of an identifier. For example, the video content feature is "0", which means that no face appears in the video to be processed. The video content feature is "1", which means that a face appears in the video to be processed. In another possible implementation, the video content features can be implemented in the form of more complex feature vectors and feature matrices to characterize more complex content. For example, the outline of a portrait in the image to be processed, the size of a specific object, etc.

[0044] In a possible implementation, as shown in FIG3 , a specific implementation of step S102 includes:

[0045] Step S1021: Acquire a corresponding target video frame in the video to be processed according to the timestamp of at least one camera movement node in the initial camera movement sequence.

[0046] Step S1023: Perform content detection on the target video frame to obtain video content features.

[0047] Exemplarily, the initial camera movement sequence includes at least one camera movement node. The camera movement node corresponds to at least one timestamp, and the timestamp is used to characterize the marking time of the camera movement node in the initial camera movement sequence in the initial video. More specifically, the marking time can be the camera movement start time, the camera movement end time, etc. Afterwards, based on the timestamp of at least one camera movement node, the corresponding video frame, that is, the target video frame, is read in the video frame to be processed. Content detection is performed on the target video frame to obtain the video frame content feature corresponding to the target video frame. Afterwards, the video frame content feature corresponding to one of the target video frames can be used as the video content feature. Alternatively, a collection of video frame content features corresponding to multiple target video frames can be used as the video content feature.

[0048] FIG4 is a schematic diagram of a process for determining video content features provided by an embodiment of the present disclosure. The above process will be further described with reference to FIG4. As shown in FIG4, exemplarily, the initial camera movement sequence includes camera movement node #1, camera movement node #2, and camera movement node #3. Among them, the timestamp of camera movement node #1 is [00:02, 00:03], which indicates that the start time of the camera movement effect corresponding to camera movement node #1 is 00:02 (0 minutes and 2 seconds, the same below) and the end time is 00:03. The timestamp of camera movement node #2 is [00:04, 00:05], which indicates that the start time of the camera movement effect corresponding to camera movement node #2 is 00:04 and the end time is 00:05. The timestamp of camera movement node #3 is [00:06, 00:08], which indicates that the start time of the camera movement effect corresponding to camera movement node #3 is 00:06 and the end time is 00:08. Afterwards, based on the start times corresponding to camera movement nodes #1, #2, and #3, i.e., 00:02, 00:04, and 00:06, the target video frames at the corresponding playback timestamps are obtained from the video to be processed, namely, video frames P1, P2, and P3. Content detection is then performed on video frames P1, P2, and P3, respectively, to obtain the corresponding video frame content features F_1, F_2, and F_3. Finally, the set of the above video frame content features F_1, F_2, and F_3 (shown as [F_1, F_2, F_3] in the figure) is used as the video content feature.

[0049] Step S103: obtaining an optimized camera movement sequence according to the video content features and the initial camera movement sequence, wherein the optimized camera movement sequence includes at least one optimized camera movement node, and the camera movement features corresponding to the optimized camera movement node match the video content features.

[0050] Exemplarily, after obtaining video content features and an initial camera movement sequence, at least one initial camera movement node in the initial camera movement sequence is optimized based on the video content features to generate a corresponding optimized camera movement node, thereby obtaining an optimized camera movement sequence. Similar to the initial camera movement node in the initial camera movement sequence, the optimized camera movement node is also used to characterize the camera movement features of the camera shot during video playback. However, since the optimized camera movement node is generated based on the initial camera movement node in the initial camera movement sequence and configured based on the video content features, the generated optimized camera movement node can match the video content features. The optimized camera movement node can match the video content features. For example, this includes optimizing the motion amplitude corresponding to the camera movement node (an exemplary implementation of the camera movement feature) to match the scale of the scene in the video to be processed (an exemplary implementation of the video content feature). Another example includes optimizing the center point of the camera shot corresponding to the camera movement node (an exemplary implementation of the camera movement effect) to match the position of the portrait in the video to be processed (an exemplary implementation of the video content feature). Thus, when the content of the video to be processed and the initial video differ, adaptive optimization of the camera movement effect can be achieved by optimizing the camera movement node.

[0051] In one possible implementation, the video content feature includes a first content feature, which represents the location of a human figure in the video to be processed. As shown in FIG5 , a possible implementation of step S103 includes:

[0052] Step S1031: determining a lens focus area of ​​a lens frame corresponding to at least one lens movement node in an initial lens movement sequence during the movement process according to a first content feature.

[0053] Step S1032: configuring corresponding camera movement nodes according to the lens focus area during the movement of the lens image, and generating optimized camera movement nodes.

[0054] Step S1033: Generate an optimized camera movement sequence according to the optimized camera movement nodes.

[0055] Exemplarily, the camera movement node has a first parameter, and the first parameter is used to determine the center point of the lens picture corresponding to the camera movement node during the movement process, that is, the lens focus area. When the video content feature is the first content feature that characterizes the position of the portrait in the video to be processed, the terminal device can determine the position of the portrait in the video to be processed through the first content feature. Afterwards, the first parameter of the camera movement node is set based on the position of the portrait, and the optimized camera movement node is generated, so that in the camera movement special effect generated by the camera movement node, the center point of the lens picture during the movement process is the position of the portrait, thereby achieving the focus of the portrait in the video to be processed. Compared with the solution of directly using the initial camera movement sequence (initial camera movement template) to process the video to be processed, it can avoid the problem of the inconsistency between the center of the lens picture and the position of the character in the video to be processed when adding camera movement special effects such as "zoom camera movement" and "shaking camera movement", thereby improving the visual effect of the camera movement special effect.

[0056] Referring to the introduction of video content features in the previous embodiment, the video content features can be a set consisting of multiple video frame content features, and the video frame content features correspond to a video frame at a certain playback position in the video to be processed (for example, the starting position of the camera movement special effect). Therefore, the first content feature can include the coordinates of the position of the portrait in the video at multiple playback positions in the video to be processed. More specifically, for example, the coordinates of the position of the center point of the character, or the coordinates of the center of the character's face. Then, based on the position represented by the first content feature, the first parameter of at least one camera movement node in the initial camera movement sequence is set. Exemplarily, when the first content feature includes multiple video frame content features, each video frame content feature represents a portrait position, and the camera movement node corresponding to the video frame content feature is set according to each video frame content feature (the video frame content feature is determined based on the timestamp of the camera movement node, so the two have a corresponding relationship). This ensures that when the character in the video to be processed is in a moving state (the position of the portrait changes), the center point of the lens image is actually focused on the character position, further highlighting the portrait and improving the visual perception of the camera movement effect.

[0057] Figure 6 is a schematic diagram of an optimized camera movement node generated based on a first content feature, as provided in an embodiment of the present disclosure. As shown in Figure 6, optimized camera movement node #1, generated based on a first content feature representing the location of a human figure in the video being processed, achieves a "zooming camera movement" effect. During this zooming effect, the center of the camera image is aligned with reference point P1, where the human figure is located, in the video being processed, thereby achieving a focused image.

[0058] In another possible implementation, the video content feature includes a second content feature, and the second content feature represents the scale of the scene in the video to be processed. As shown in FIG7 , another possible implementation of step S103 includes:

[0059] Step S1034: determining the movement amplitude of the shot image corresponding to at least one camera movement node in the initial camera movement sequence during the movement process according to the second content feature.

[0060] Step S1035: configuring corresponding camera movement nodes according to the movement amplitude of the lens image during the movement process, and generating optimized camera movement nodes.

[0061] Step S1036: Generate an optimized camera movement sequence according to the optimized camera movement nodes.

[0062] Exemplarily, a camera movement node has a second parameter that is used to determine the amplitude of motion of the lens image corresponding to the camera movement node during motion. When the video content feature is a second content feature that characterizes the scale of shot angles in the video being processed, the terminal device can determine the scale of shot angles in the video being processed based on the second content feature. More specifically, this scale can be a close shot, a medium shot, a long shot, etc. Of course, in other implementations, the scale of shot angles represented by the second content feature can be a continuously varying value, representing a more refined scale of shot angles, which will not be exemplified here. Subsequently, the second parameter of the camera movement node is set based on the scale of shot angles to generate an optimized camera movement node. This ensures that the amplitude of the camera image shaking during camera movement effects generated by the camera movement node matches the shot angle. Specifically, the farther the shot angle, the greater the relative amplitude of motion; the farther the shot angle, the smaller the relative amplitude of motion. This avoids the feeling of dizziness caused by inappropriate camera movement amplitudes during camera movement effects such as "shaking camera movement," thereby improving the visual quality of camera movement effects.

[0063] Figure 8 is a schematic diagram of an optimized camera movement node generated based on a second content feature provided by an embodiment of the present disclosure. As shown in Figure 8, the camera movement effect achieved by the optimized camera movement node #2 generated based on the second content feature that characterizes the scale of the scene in the video to be processed is "left and right shaking camera movement". In the process of achieving this camera movement effect, when the scale of the scene in the video to be processed is 03 (characterizing the distant view), the shaking amplitude is a during the movement of the lens image P1 to the lens image P2. When the scale of the scene in the video to be processed is 01 (characterizing the close view), the shaking amplitude is b during the movement of the lens image P3 to the lens image P4. Wherein, for example, b = 0.2a. By setting the second parameter of the camera movement node through the first content feature, the effect of adaptive amplitude adjustment of the left and right shaking camera movement is achieved.

[0064] In another possible implementation, the video content feature includes a third content feature, and the third content feature represents a target object in the video to be processed. As shown in FIG9 , another possible implementation of step S103 includes:

[0065] Step S1037: determining a change frequency of a shot image corresponding to at least one camera movement node in the initial camera movement sequence during the camera movement process according to the third content feature, wherein the change frequency is less than a preset frequency threshold.

[0066] Step S1038: configuring corresponding camera movement nodes according to the frequency of changes in the lens image during the camera movement process, and generating optimized camera movement nodes.

[0067] Step S1039: Generate an optimized camera movement sequence according to the optimized camera movement nodes.

[0068] Exemplarily, a camera movement node has a third parameter that is used to determine the frequency of change of the lens image corresponding to the camera movement node during the camera movement process. When the video content feature is a third content feature that characterizes a target object in the video to be processed, the terminal device can determine whether the target object exists in the video to be processed based on the third content feature. For example, whether important parts such as the face and hands are present, as well as the location and size of the above parts. Subsequently, the second parameter of the camera movement node is set based on the target object to generate an optimized camera movement node, thereby achieving the control of the shaking frequency of the lens image in the camera movement special effects generated by the camera movement node. Specifically, for example, for camera movement special effects such as "shaking camera movement", in one possible implementation, when it is determined based on the third content feature that a portrait face or finger (target object) exists in the video to be processed, high-frequency shaking is reduced. For example, the shaking frequency (changing frequency) is reduced to below a preset frequency value using the third parameter, thereby improving the focusing effect of the target object and avoiding the problem that the user cannot clearly see the target object such as the portrait facial expression or finger movements due to excessive shaking.

[0069] In another possible implementation, as shown in FIG10 , the specific implementation of step S1037 includes:

[0070] Step S1037A: Obtain the outline size of the target object.

[0071] Step S1037B: Obtain the corresponding target motion frequency according to the outline size, wherein the target motion frequency is inversely proportional to the outline size.

[0072] Step S1037C: according to the target movement frequency, a third parameter is set, where the third parameter is used to determine the frequency of change of the lens image corresponding to the camera movement node during the camera movement process.

[0073] Exemplarily, after obtaining the outline size of the target object (e.g., the area of ​​the facial outline) based on the third content feature, the corresponding shaking frequency (target motion frequency) is set based on the outline size of the target object. Specifically, for example, when the outline size of the facial outline is larger, the shaking frequency is lower. Conversely, when the outline size is smaller, the shaking frequency is higher (but not greater than the initial value of the third parameter). This achieves shaking frequency control in the camera movement special effect, reduces the visual impact of high-frequency shaking on target objects such as faces and fingers in the video, and improves the visual effect of camera movement.

[0074] It should be noted that the specific implementation of the above-mentioned video content features can be one of the first, second, and third content features, or multiple of the first, second, and third content features. Specifically, the camera movement node can be configured using at least two of the first, second, and third parameters to generate an optimized camera movement node and a corresponding optimized camera movement sequence. The specific implementation method can be configured as needed and will not be further described here.

[0075] Step S104: Process the video to be processed based on the optimized camera movement sequence to generate an output video, wherein the shot images of the output video have camera movement features corresponding to the optimized camera movement nodes.

[0076] For example, after obtaining an optimized camera sequence, camera effects are generated at corresponding locations in the video to be processed based on each optimized camera node in the optimized camera sequence, thereby generating a special effects video with one or more camera effects, i.e., the output video. Because the camera effects in the output video are generated based on the optimized camera sequence, the shot images of the output video have the camera characteristics corresponding to the optimized camera nodes, thereby ensuring that the camera effects in the output video match the video content.

[0077] In this embodiment, an initial camera movement sequence is obtained based on an initial camera movement template. The initial camera movement sequence includes at least one camera movement node corresponding to the initial video, which is used to characterize the camera movement characteristics of the camera shots during video playback. Video content features of the video to be processed are obtained, and an optimized camera movement sequence is obtained based on the video content features and the initial camera movement sequence. The optimized camera movement sequence includes at least one optimized camera movement node, and the camera movement characteristics corresponding to the optimized camera movement node match the video content features. The video to be processed is processed based on the optimized camera movement sequence to generate an output video, wherein the camera shots of the output video have the camera movement characteristics corresponding to the optimized camera movement node. The initial camera movement template is configured using the video content features of the video to be processed to obtain an optimized camera movement sequence composed of optimized camera movement nodes. An output video with camera movement effects is generated based on the optimized camera movement sequence, because the camera movement characteristics corresponding to the optimized camera movement nodes in the optimized camera movement sequence match the video content features of the video to be processed. Therefore, generating an output video based on the optimized camera movement sequence and the video to be processed avoids the mismatch between the camera movement effects generated by the initial camera movement template and the content of the video to be processed, thereby improving the visual perception of the camera movement effects.

[0078] Referring to FIG11 , FIG11 is a second flow chart of the video generation method provided by the embodiment of the present disclosure. Based on the embodiment shown in FIG2 , this embodiment further refines step S103 , and the video generation method includes:

[0079] Step S201: obtaining an initial camera movement sequence according to an initial camera movement template, wherein the initial camera movement sequence includes at least one camera movement node corresponding to an initial video, and the camera movement node is used to represent camera movement features of a shot during video playback.

[0080] Step S202: Obtain a first video content feature of the video to be processed.

[0081] Step S203: Acquire the second video content feature of the initial video.

[0082] Step S204: obtaining feature deviation information based on the second video content feature and the second video content feature, where the feature deviation information represents the feature deviation amount in video content between the video to be processed and the initial video.

[0083] Exemplarily, the video used to generate the initial camera movement sequence when generating the initial video is, that is, the video whose video content matches the camera movement effects generated by the initial camera movement sequence. In one possible implementation, the template data corresponding to the initial camera movement template includes the initial video corresponding to the initial camera movement template. Afterwards, the initial video is subjected to content detection to obtain the video content features corresponding to the initial video, that is, the second video content features. Among them, the specific implementation method of obtaining the second video content features is the same as the implementation method of obtaining the video content features of the video to be processed (the first video content features). Please refer to the introduction of the specific implementation method of obtaining the video content features of the video to be processed in the embodiment shown in Figure 2, which will not be repeated here.

[0084] Furthermore, after obtaining the first video content feature and the second video content feature, a feature comparison can be performed based on the first video content feature and the second video content feature to obtain feature deviation information. This feature deviation represents the feature deviation between the video content of the initial video and the video to be processed. Specifically, for example, the first video content feature represents the first position where a portrait is located in the video to be processed, and the second video content feature represents the second position where a portrait is located in the initial video. Based on the first video content feature and the second video content feature, feature deviation information representing the deviation between the first position and the second position is obtained.

[0085] Step S205: setting the node parameters of at least one camera movement node in the initial camera movement sequence according to the feature deviation information to obtain an optimized camera movement sequence.

[0086] For example, after obtaining the characteristic deviation information, the node parameters of the camera movement nodes in the initial camera movement sequence are modified based on the characteristic deviation amount represented by the characteristic deviation information. This allows the camera movement nodes to match the video content characteristics (first video content characteristics) of the video to be processed, thereby achieving the purpose of camera movement node optimization and obtaining an optimized camera movement sequence.

[0087] In one possible implementation, the feature deviation information includes a position deviation, which represents the deviation between a first position of the portrait in the initial video and a second position of the portrait in the video to be processed. As shown in FIG12 , the specific implementation of step S205 includes:

[0088] Step S2051: Obtain the target camera movement node corresponding to the position deviation in the initial camera movement sequence.

[0089] Step S2052: according to the position deviation, the center point of the lens picture corresponding to the target camera movement node during the camera movement process is set to obtain an optimized camera movement sequence.

[0090] For example, since the feature deviation information is obtained based on a feature comparison between the first video content feature and the second video content feature, and the first video content feature and the second video content feature are obtained based on video frames corresponding to the same camera movement node, the feature deviation information corresponds to a camera movement node in the initial camera movement sequence. In this embodiment, the camera movement node corresponding to the feature deviation information (position deviation amount) is the target camera movement node.

[0091] Furthermore, when the characteristic deviation information is a positional deviation amount representing the deviation between a first position where a portrait is located in the initial video and a second position where the portrait is located in the video to be processed, the characteristic deviation information can be used to adjust the center point of the lens image corresponding to the camera movement node during the camera movement process. Specifically, for example, the positional deviation amount is a deviation vector, and the center point of the lens image corresponding to the target camera movement node during the camera movement process is set by using this deviation vector. This causes the center point of the image to be offset by the deviation vector, thereby optimizing the target camera movement node. Furthermore, with the optimization of at least one target camera movement node, the initial camera movement sequence is optimized simultaneously, thereby generating an optimized camera movement sequence corresponding to the initial camera movement sequence. The camera movement effect generated by the optimized camera movement sequence can be focused on the position of the portrait in the video to be processed, thereby improving the visual perception experience. In this embodiment, the step of setting the center point of the lens image corresponding to the target camera movement node during the camera movement process can be achieved by setting the first parameter of the target camera movement node. The specific implementation method can be referred to the relevant introduction in the embodiment shown in Figure 2, and will not be repeated here.

[0092] In another possible implementation, the feature deviation information includes a scale coefficient, which represents the ratio of the scene scale in the initial video to the scene scale in the video to be processed. The specific implementation of step S205 includes:

[0093] Step S2053: Obtain the target camera movement node corresponding to the scale coefficient in the initial camera movement sequence.

[0094] Step S2054: according to the scale coefficient, the movement amplitude of the lens frame corresponding to the target camera movement node during the movement process is set to obtain an optimized camera movement sequence.

[0095] For example, the process of obtaining an optimized camera movement sequence based on position deviation is similar to the above-mentioned process. When the characteristic deviation information is a scale coefficient representing the ratio of the shot scale in the initial video to the shot scale in the processed video, the target camera movement node corresponding to the scale coefficient is first obtained. The implementation method is similar to the step of obtaining the target camera movement node corresponding to the position deviation in the above-mentioned embodiment and will not be repeated here. Then, based on the scale coefficient, the motion amplitude of the lens image corresponding to the target camera movement node during the motion process is set, thereby achieving the setting of the camera movement amplitude in camera movement effects such as "shaking camera movement". This achieves the optimization of the target camera movement node and generates an optimized camera movement sequence corresponding to the initial camera movement sequence. The motion amplitude of the camera movement effect generated by the optimized camera movement sequence matches the shot scale. This avoids visual fatigue and dizziness caused by large-scale movement in close-up shots, and the problem of unclear camera movement effects caused by too small movement in long shots, thereby improving the visual experience. In this embodiment, the step of setting the motion amplitude of the lens image corresponding to the target camera movement node during the motion process can be achieved by setting the second parameter of the target camera movement node. The specific implementation method can be found in the relevant introduction of the embodiment shown in Figure 2, which will not be repeated here.

[0096] Optionally, in a possible implementation, the initial video has a first playback duration, and the video to be processed has a second playback duration. When the second playback duration is greater than the first playback duration, after step S205, the method further includes:

[0097] Step S206: generating an extended camera movement sequence based on at least one camera movement node in the optimized camera movement sequence, wherein the camera movement duration of the shot corresponding to the extended camera movement sequence is not less than the difference between the second playback duration and the first playback duration.

[0098] Step S207: Process the video to be processed based on the optimized camera movement sequence and the extended camera movement sequence in sequence to generate an output video.

[0099] Exemplarily, when the playback duration of the initial video is longer than the playback duration of the video to be processed, the playback duration of the initial camera movement template generated based on the initial video will be shorter than that of the video to be processed, which in turn causes the camera movement special effects generated by the initial camera movement sequence to be unable to cover the entire video to be processed. To address this problem, this embodiment generates a delayed camera movement sequence by obtaining at least one camera movement node from the generated optimized camera movement sequence. Thereafter, the video to be processed is processed based on the optimized camera movement sequence and the extended camera movement sequence in sequence, so that the camera movement special effects generated by the optimized camera movement sequence and the extended camera movement sequence can cover the entire video to be processed, thereby avoiding the existence of blank special effects segments without camera movement special effects in the generated output video. At the same time, since the extended camera movement sequence is generated based on the camera movement nodes in the optimized camera movement sequence, it can be ensured that the extended camera movement sequence matches the remaining video clips in the video to be processed that are not covered by the optimized camera movement sequence, thereby improving the camera movement special effects of the output video.

[0100] For example, as shown in FIG13 , the specific implementation of step S206 includes:

[0101] Step S2061: Obtain identification features of the first video segment in the video to be processed, wherein the first video segment is the video segment after the first playback duration in the video to be processed, and the identification features include video content features and / or music melody features of the first video segment.

[0102] Step S2062: Obtain a second video segment from the video to be processed based on the identification feature, wherein the second video segment is a video segment before the first playback duration in the video to be processed. The identification feature of the second video segment is similar to the identification feature of the first video segment.

[0103] Step S2063: Generate an extended camera movement sequence according to at least one continuous camera movement node corresponding to the second video clip.

[0104] For example, when the playback duration of the video to be processed is longer than that of the initial video, the video to be processed can be divided into two parts: a part after the first playback duration and a part before the first playback duration. The part after the first playback duration can be divided into one or more sub-segments, namely, first video segments, based on the duration of the camera movement nodes. Feature extraction is then performed on the first video segment to obtain identification features corresponding to the first video segment, wherein the identification features include, for example, video content features and / or musical melody features of the first video segment. Based on the identification features, a search is then performed on the part before the first playback duration to obtain a second video segment similar to the identification features of the first video segment. The similarity of the identification features is determined by determining that the feature distance between the identification features of the second video segment and the identification features of the first video segment is less than a feature threshold. At least one continuous multiplexed camera movement node corresponding to the timestamp of the second video segment is then obtained. The above process is then repeated until multiplexed camera movement nodes corresponding to each of the first video segments are obtained, and the multiplexed camera movement nodes are then spliced ​​together to generate an extended camera movement sequence. Because the second video clip's identification features are similar to those of the first, the reused camera nodes selected for the first video clip can also, to a certain extent, match the video content features of the first video clip. This allows the generated extended camera sequence to inherit the effects of the optimized camera sequence, adapting to the video segments in the processed video that are not covered by the optimized camera sequence, and improving the overall visual quality of the camera effects in the generated output video.

[0105] In this embodiment, the implementation of step S201-step S202 is the same as the implementation of step S101-step S102 in the embodiment shown in FIG. 2 of the present disclosure, and will not be described in detail here.

[0106] Corresponding to the video generation method of the above embodiment, FIG14 is a structural block diagram of a video generation device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Referring to FIG14, the video generation device 3 includes:

[0107] A loading module 31 is configured to obtain an initial camera movement sequence based on the initial camera movement template. The initial camera movement sequence includes at least one camera movement node corresponding to the initial video. The camera movement node is used to represent the camera movement characteristics of the shot during video playback.

[0108] A processing module 32 is configured to obtain video content features of a video to be processed, and to obtain an optimized camera movement sequence based on the video content features and the initial camera movement sequence, wherein the optimized camera movement sequence includes at least one optimized camera movement node, and the camera movement features corresponding to the optimized camera movement node match the video content features;

[0109] The output module 33 is used to process the video to be processed based on the optimized camera movement sequence to generate an output video, and the shot images of the output video have the camera movement features corresponding to the optimized camera movement nodes.

[0110] In one embodiment of the present disclosure, the video content feature includes a first content feature, which represents the position of the portrait in the video to be processed; when the processing module 32 obtains the optimized camera movement sequence based on the video content feature and the initial camera movement sequence, it is specifically used to: generate a first parameter of at least one camera movement node in the initial camera movement sequence based on the first content feature, wherein the first parameter is used to determine the center point of the lens picture corresponding to the camera movement node during the movement process; configure the corresponding camera movement node according to the first parameter to generate an optimized camera movement node; and generate an optimized camera movement sequence based on the optimized camera movement node.

[0111] In one embodiment of the present disclosure, the video content feature includes a second content feature, and the second content feature characterizes the scale of the scene in the video to be processed; when the processing module 32 obtains the optimized camera movement sequence based on the video content feature and the initial camera movement sequence, it is specifically used to: generate a second parameter of at least one camera movement node in the initial camera movement sequence based on the second content feature, wherein the second parameter is used to determine the movement amplitude of the lens picture corresponding to the camera movement node during the movement process; configure the corresponding camera movement node according to the second parameter to generate an optimized camera movement node; and generate an optimized camera movement sequence based on the optimized camera movement node.

[0112] In one embodiment of the present disclosure, the video content feature includes a third content feature, and the third content feature characterizes the facial contour in the video to be processed; when the processing module 32 obtains the optimized camera movement sequence based on the video content feature and the initial camera movement sequence, it is specifically used to: generate a third parameter of at least one camera movement node in the initial camera movement sequence based on the third content feature, wherein the third parameter is used to determine the frequency of change of the lens picture corresponding to the camera movement node during the camera movement process; configure the corresponding camera movement node according to the third parameter to generate an optimized camera movement node; and generate an optimized camera movement sequence based on the optimized camera movement node.

[0113] In one embodiment of the present disclosure, when the processing module 32 generates the third parameter of at least one camera movement node in the initial camera movement sequence based on the third content feature, it is specifically used to: obtain the contour size of the facial contour; obtain the corresponding target motion frequency based on the contour size, wherein the target motion frequency is inversely proportional to the contour size; and set the third parameter based on the target motion frequency.

[0114] In one embodiment of the present disclosure, when the processing module 32 obtains the optimized camera movement sequence based on the video content features and the initial camera movement sequence, it is specifically used to: obtain the video content features of the initial video; obtain feature deviation information based on the video content features of the initial video and the video content features of the video to be processed, the feature deviation information representing the feature deviation amount in video content between the video to be processed and the initial video; and set the node parameters of at least one camera movement node in the initial camera movement sequence based on the feature deviation information to obtain the optimized camera movement sequence.

[0115] In one embodiment of the present disclosure, the characteristic deviation information includes a position deviation amount, which represents the deviation between a first position of a portrait in an initial video and a second position of a portrait in a video to be processed; when the processing module 32 sets the node parameters of at least one camera movement node in the initial camera movement sequence according to the characteristic deviation information to obtain an optimized camera movement sequence, it is specifically used to: obtain a target camera movement node corresponding to the position deviation amount in the initial camera movement sequence; and set the center point of the lens picture corresponding to the target camera movement node during the camera movement process according to the position deviation amount to obtain an optimized camera movement sequence.

[0116] In one embodiment of the present disclosure, the characteristic deviation information includes a scale coefficient, which represents the ratio of the shot scale in the initial video to the shot scale in the video to be processed; when the processing module 32 sets the node parameters of at least one camera movement node in the initial camera movement sequence according to the characteristic deviation information to obtain the optimized camera movement sequence, it is specifically used to: obtain the target camera movement node corresponding to the scale coefficient in the initial camera movement sequence; set the movement amplitude of the lens picture corresponding to the target camera movement node during the movement process according to the scale coefficient to obtain the optimized camera movement sequence.

[0117] In one embodiment of the present disclosure, when obtaining the video content features of the video to be processed, the processing module 32 is specifically used to: obtain the corresponding target video frame in the video to be processed based on the timestamp of at least one camera movement node in the initial camera movement sequence; perform content detection on the target video frame to obtain the video content features.

[0118] In one embodiment of the present disclosure, the initial video has a first playback duration, and the video to be processed has a second playback duration. When the second playback duration is greater than the first playback duration, the processing module 32 is further used to: generate an extended camera movement sequence based on at least one camera movement node in the optimized camera movement sequence, and the camera movement duration of the lens picture corresponding to the extended camera movement sequence is not less than the difference between the second playback duration and the first playback duration; when the processing module 32 processes the video to be processed based on the optimized camera movement sequence to generate the output video, it is specifically used to: process the video to be processed based on the optimized camera movement sequence and the extended camera movement sequence in sequence to generate the output video.

[0119] In one embodiment of the present disclosure, when the processing module 32 generates an extended camera movement sequence based on at least one camera movement node in the optimized camera movement sequence, it is specifically used to: obtain identification features of a first video clip in the video to be processed, wherein the first video clip is a video segment after a first playback time in the video to be processed, and the identification features include video content features and / or music melody features of the first video clip; according to the identification features, obtain a second video clip from the video to be processed, wherein the second video clip is a video segment before the first playback time in the video to be processed, and the identification features of the second video clip are similar to the identification features of the first video clip; and generate an extended camera movement sequence according to at least one continuous camera movement node corresponding to the second video clip.

[0120] The loading module 31, processing module 32 and output module 33 are connected in sequence. The video generating device 3 provided in this embodiment can implement the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0121] FIG15 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. As shown in FIG15 , the electronic device 4 includes:

[0122] A processor 41, and a memory 42 communicatively connected to the processor 41;

[0123] Memory 42 stores computer-executable instructions;

[0124] The processor 41 executes the computer-executable instructions stored in the memory 42 to implement the video generation method in the embodiments shown in FIG. 2 to FIG. 13 .

[0125] Optionally, the processor 41 and the memory 42 are connected via a bus 43 .

[0126] The relevant explanations can be understood by referring to the relevant descriptions and effects corresponding to the steps in the embodiments corresponding to Figures 2 to 13, and no further details will be given here.

[0127] An embodiment of the present disclosure provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by a processor, they are used to implement the video generation method provided in any of the embodiments corresponding to Figures 2 to 13 of the present disclosure.

[0128] An embodiment of the present disclosure provides a computer program product, including a computer program. When the computer program is executed by a processor, the method in the embodiments shown in Figures 2 to 13 is implemented.

[0129] Referring to FIG16 , a schematic diagram of the structure of an electronic device 900 suitable for implementing an embodiment of the present disclosure is shown. The electronic device 900 may be a terminal device or a server. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG16 is merely an example and should not limit the functionality and scope of use of the embodiments of the present disclosure.

[0130] As shown in FIG16 , the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. Various programs and data required for the operation of the electronic device 900 are also stored in the RAM 903. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0131] Typically, the following devices can be connected to the I / O interface 905: an input device 906 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 908 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 909. The communication device 909 can allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although FIG16 shows an electronic device 900 with various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0132] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 909, or installed from the storage device 908, or installed from the ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0133] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0134] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0135] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0136] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0137] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0138] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0139] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0140] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0141] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0142] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0143] Although the subject matter has been described using language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. In a first aspect, the present disclosure provides a video generation method, comprising:

[0144] According to the initial camera movement template, an initial camera movement sequence is obtained, wherein the initial camera movement sequence includes at least one camera movement node corresponding to the initial video, and the camera movement node is used to characterize the camera movement characteristics of the camera lens during video playback; the video content characteristics of the video to be processed are obtained, and according to the video content characteristics and the initial camera movement sequence, an optimized camera movement sequence is obtained, wherein the optimized camera movement sequence includes at least one optimized camera movement node, and the camera movement characteristics corresponding to the optimized camera movement node match the video content characteristics; the video to be processed is processed based on the optimized camera movement sequence to generate an output video, and the camera lens of the output video has the camera movement characteristics corresponding to the optimized camera movement node.

[0145] In one or more possible implementations, the video content feature includes a first content feature, which represents the position of the portrait in the video to be processed; obtaining the optimized camera movement sequence based on the video content feature and the initial camera movement sequence includes: determining the lens focus area of ​​the lens picture corresponding to at least one camera movement node in the initial camera movement sequence during the movement process based on the first content feature; configuring the corresponding camera movement node according to the lens focus area of ​​the lens picture during the movement process to generate an optimized camera movement node; and generating the optimized camera movement sequence based on the optimized camera movement node.

[0146] In one or more possible implementations, the video content feature includes a second content feature, and the second content feature represents the scale of the scene in the video to be processed; obtaining the optimized camera movement sequence based on the video content feature and the initial camera movement sequence includes: determining the movement amplitude of the lens picture corresponding to at least one camera movement node in the initial camera movement sequence during the movement process based on the second content feature; configuring the corresponding camera movement node according to the movement amplitude of the lens picture during the movement process to generate an optimized camera movement node; and generating the optimized camera movement sequence based on the optimized camera movement node.

[0147] In one or more possible implementations, the video content feature includes a third content feature, and the third content feature represents the target object in the video to be processed; obtaining the optimized camera movement sequence based on the video content feature and the initial camera movement sequence includes: determining the change frequency of the lens picture corresponding to at least one camera movement node in the initial camera movement sequence during the camera movement process based on the third content feature, and the change frequency is less than a preset frequency threshold; configuring the corresponding camera movement node according to the change frequency of the lens picture during the camera movement process to generate an optimized camera movement node; generating the optimized camera movement sequence based on the optimized camera movement node.

[0148] In one or more possible implementations, generating a third parameter of at least one camera movement node in the initial camera movement sequence based on the third content feature includes: obtaining the outline size of the target object; obtaining a corresponding target motion frequency based on the outline size, wherein the target motion frequency is inversely proportional to the outline size; and setting the third parameter based on the target motion frequency.

[0149] In one or more possible implementations, obtaining an optimized camera movement sequence based on the video content features and the initial camera movement sequence includes: acquiring video content features of the initial video; obtaining feature deviation information based on the video content features of the initial video and the video content features of the video to be processed, wherein the feature deviation information represents the feature deviation amount in video content between the video to be processed and the initial video; and setting node parameters of at least one camera movement node in the initial camera movement sequence based on the feature deviation information to obtain the optimized camera movement sequence.

[0150] In one or more possible implementations, the characteristic deviation information includes a position deviation, and the position deviation represents the deviation between a first position of the portrait in the initial video and a second position of the portrait in the video to be processed; according to the characteristic deviation information, the node parameters of at least one camera movement node in the initial camera movement sequence are set to obtain the optimized camera movement sequence, including: obtaining a target camera movement node corresponding to the position deviation in the initial camera movement sequence; according to the position deviation, setting the center point of the lens picture corresponding to the target camera movement node during the camera movement process to obtain the optimized camera movement sequence.

[0151] In one or more possible implementations, the characteristic deviation information includes a scale coefficient, which represents the ratio of the shot scale in the initial video to the shot scale in the video to be processed; according to the characteristic deviation information, the node parameters of at least one camera movement node in the initial camera movement sequence are set to obtain the optimized camera movement sequence, including: obtaining the target camera movement node corresponding to the scale coefficient in the initial camera movement sequence; according to the scale coefficient, setting the movement amplitude of the lens picture corresponding to the target camera movement node during the movement process to obtain the optimized camera movement sequence.

[0152] In one or more possible implementations, obtaining the video content features of the video to be processed includes: obtaining the corresponding target video frame in the video to be processed based on the timestamp of at least one of the camera movement nodes in the initial camera movement sequence; performing content detection on the target video frame to obtain the video content features.

[0153] In one or more possible implementations, the initial video has a first playback duration, and the video to be processed has a second playback duration. When the second playback duration is greater than the first playback duration, the method further includes: generating an extended camera movement sequence based on at least one camera movement node in the optimized camera movement sequence, and the camera movement duration of the lens picture corresponding to the extended camera movement sequence is not less than the difference between the second playback duration and the first playback duration; processing the video to be processed based on the optimized camera movement sequence to generate an output video includes: processing the video to be processed based on the optimized camera movement sequence and the extended camera movement sequence in sequence to generate an output video.

[0154] In one or more possible implementations, generating an extended camera movement sequence based on at least one camera movement node in the optimized camera movement sequence includes: obtaining identification features of a first video clip in the video to be processed, wherein the first video clip is a video segment after a first playback duration in the video to be processed, and the identification features include video content features and / or music melody features of the first video clip; obtaining a second video clip from the video to be processed based on the identification features, wherein the second video clip is a video segment before the first playback duration in the video to be processed, and the identification features of the second video clip are similar to the identification features of the first video clip; generating the extended camera movement sequence based on at least one continuous camera movement node corresponding to the second video clip.

Claims

1. A video generation method, comprising: According to the initial camera movement template, an initial camera movement sequence is obtained, wherein the initial camera movement sequence includes at least one camera movement node corresponding to the initial video, and the camera movement node is used to represent the camera movement characteristics of the lens picture when the video is played; Acquire video content features of the video to be processed, and obtain an optimized camera movement sequence according to the video content features and the initial camera movement sequence, wherein the optimized camera movement sequence includes at least one optimized camera movement node, and the camera movement features corresponding to the optimized camera movement node match the video content features; The video to be processed is processed based on the optimized camera movement sequence to generate an output video, wherein the shot images of the output video have camera movement features corresponding to the optimized camera movement nodes.

2. The method according to claim 1, wherein the video content feature comprises a first content feature, wherein the first content feature represents the position of the human figure in the video to be processed; The step of obtaining an optimized camera movement sequence according to the video content feature and the initial camera movement sequence includes: Determining, according to the first content feature, a lens focus area of ​​a lens picture corresponding to at least one lens movement node in the initial lens movement sequence during the movement process; According to the lens focus area of ​​the lens picture during the movement, a corresponding lens movement node is configured to generate an optimized lens movement node; The optimized camera movement sequence is generated according to the optimized camera movement node.

3. The method according to claim 1, wherein the video content feature comprises a second content feature, and the second content feature represents the scale of the scene in the video to be processed; The step of obtaining an optimized camera movement sequence according to the video content feature and the initial camera movement sequence includes: Determining, according to the second content feature, a movement amplitude of a shot picture corresponding to at least one camera movement node in the initial camera movement sequence during the movement process; According to the movement amplitude of the lens picture during the movement process, the corresponding camera movement node is configured. point, generate optimized camera movement nodes; The optimized camera movement sequence is generated according to the optimized camera movement node.

4. The method according to claim 1, wherein the video content feature comprises a third content feature, and the third content feature represents a target object in the video to be processed; The step of obtaining an optimized camera movement sequence according to the video content feature and the initial camera movement sequence includes: Determining, according to the third content feature, a change frequency of a shot picture corresponding to at least one camera movement node in the initial camera movement sequence during the camera movement process, wherein the change frequency is less than a preset frequency threshold; According to the change frequency of the lens picture during the lens movement process, a corresponding lens movement node is configured to generate an optimized lens movement node; The optimized camera movement sequence is generated according to the optimized camera movement node.

5. The method according to claim 4, wherein generating a third parameter of at least one camera movement node in the initial camera movement sequence according to the third content feature comprises: Obtaining the outline size of the target object; According to the outline size, a corresponding target motion frequency is obtained, wherein the target motion frequency is inversely proportional to the outline size; The third parameter is set according to the target motion frequency.

6. The method according to claim 1, wherein obtaining an optimized camera sequence according to the video content features and the initial camera sequence comprises: Acquire video content features of the initial video; According to the video content features of the initial video and the video content features of the video to be processed, feature deviation information is obtained, wherein the feature deviation information represents the feature deviation amount in video content between the video to be processed and the initial video; According to the characteristic deviation information, node parameters of at least one camera movement node in the initial camera movement sequence are set to obtain the optimized camera movement sequence.

7. The method according to claim 6, wherein the feature deviation information includes a position deviation, wherein the position deviation represents a deviation between a first position where the portrait in the initial video is located and a second position where the portrait in the video to be processed is located; According to the characteristic deviation information, setting a node parameter of at least one camera movement node in the initial camera movement sequence to obtain the optimized camera movement sequence includes: Obtaining a target camera movement node corresponding to the position deviation in the initial camera movement sequence; According to the position deviation, the center point of the lens picture corresponding to the target camera movement node during the camera movement process is set to obtain the optimized camera movement sequence.

8. The method according to claim 6, wherein the feature deviation information comprises a scale coefficient, wherein the scale coefficient represents a ratio of a scene scale in the initial video to a scene scale in the video to be processed; According to the characteristic deviation information, setting a node parameter of at least one camera movement node in the initial camera movement sequence to obtain the optimized camera movement sequence includes: Obtaining a target camera movement node corresponding to the scale coefficient in the initial camera movement sequence; According to the scale coefficient, the movement amplitude of the lens picture corresponding to the target camera movement node during the movement process is set to obtain the optimized camera movement sequence.

9. The method according to claim 1, wherein obtaining video content features of the video to be processed comprises: Acquire a corresponding target video frame in the video to be processed according to a timestamp of at least one of the camera movement nodes in the initial camera movement sequence; Content detection is performed on the target video frame to obtain the video content feature.

10. The method according to claim 1, wherein the initial video has a first playback duration, and the to-be-processed video has a second playback duration, and when the second playback duration is greater than the first playback duration, the method further comprises: generating an extended camera movement sequence according to at least one camera movement node in the optimized camera movement sequence, wherein the camera movement duration of the shot picture corresponding to the extended camera movement sequence is not less than the difference between the second playback duration and the first playback duration; The step of processing the video to be processed based on the optimized camera movement sequence to generate an output video includes: The video to be processed is processed based on the optimized camera movement sequence and the extended camera movement sequence in sequence to generate an output video.

11. The method according to claim 10, wherein generating an extended camera sequence according to at least one camera node in the optimized camera sequence comprises: Acquire identification features of a first video segment in the video to be processed, wherein the first video segment is a video segment after a first playback duration in the video to be processed, and the identification features include video content features and / or music melody features of the first video segment; According to the identification feature, a second video segment is obtained from the video to be processed, where the second video segment is a video segment before the first playback duration in the video to be processed, and the identification feature of the second video segment is similar to the identification feature of the first video segment; The extended camera movement sequence is generated according to at least one continuous camera movement node corresponding to the second video clip.

12. A video generating device, comprising: A loading module, used to obtain an initial camera movement sequence according to an initial camera movement template, wherein the initial camera movement sequence includes at least one camera movement node corresponding to an initial video, and the camera movement node is used to represent the camera movement characteristics of a lens picture during video playback; A processing module, used to obtain video content features of a video to be processed, and obtain an optimized camera movement sequence according to the video content features and the initial camera movement sequence, wherein the optimized camera movement sequence includes at least one optimized camera movement node, and the camera movement features corresponding to the optimized camera movement node match the video content features; An output module is used to process the video to be processed based on the optimized camera movement sequence to generate an output video, wherein the shot images of the output video have the camera movement features corresponding to the optimized camera movement nodes.

13. An electronic device comprising: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the video generating method according to any one of claims 1 to 11.

14. A computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the video generation method according to any one of claims 1 to 11 is implemented.

15. A computer program product tangibly stored in a computer storage medium and comprising computer executable instructions which, when executed by a device, cause the device to perform the method according to any one of claims 1 to 11.

Images