Video generation method and device, electronic equipment and medium

Abstract

The embodiment of the invention provides a video generation method and device, electronic equipment and a medium, and the method specifically comprises the steps: carrying out the cutting processing of a target picture according to a shot control mode, so as to obtain a plurality of sub-pictures according with the shot control mode, and taking the sub-pictures as a key frame sequence; generating a video according with a shot control mode according to the key frame sequence; wherein the plurality of sub-pictures in the key frame sequence are arranged according to the shot motion mode corresponding to the shot mode, so that the shot motion mode of the video is matched with the shot mode. The embodiment of the invention can improve the visual quality of the video.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of multimedia technology, and in particular to a video generation method, device, electronic device and medium. Background Art

[0002] In the era of multimedia technology, video generation technology that generates videos from pictures is of great significance. It has been widely used in many scenarios such as personal life recording, education, and commercial advertising production. For example, in personal life recording, video generation technology can transform travel photos into wonderful travel videos, thus bringing users a new way to record and share their lives. Another example is that in education, video generation technology can transform beautiful picture books into vivid videos, allowing users to more intuitively experience the story world in the picture books and stimulate their imagination and interest in reading.

[0003] Taking the conversion of picture books to videos in educational scenarios as an example, related technologies usually use shooting or scanning to obtain picture book images, and then use video generation tools such as SD (Stable Diffusion) to generate videos corresponding to the picture book images.

[0004] However, the principle of video generation tools such as SD is to randomly generate videos based on the understanding of the content of picture books; therefore, videos generated by video generation tools such as SD usually do not contain camera movement. Videos without camera movement usually lack a sense of dynamics, and the video images appear stiff and dull, which greatly reduces the visual effect of the video. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a video generation method, device, electronic device and medium, which can improve the visual quality of the video.

[0006] The specific technical solutions are as follows:

[0007] In a first aspect of the present invention, a video generation method is provided, the method comprising:

[0008] According to the lens control mode, the target image is cropped to obtain a plurality of sub-images conforming to the lens control mode as a key frame sequence;

[0009] A video conforming to the lens control mode is generated according to the key frame sequence; wherein the multiple sub-pictures in the key frame sequence are arranged according to the lens movement mode corresponding to the lens mode, so that the lens movement mode of the video matches the lens mode.

[0010] In a second aspect of the present invention, a video generation device is provided, the device comprising:

[0011] A cropping processing module, used for cropping the target image according to the lens control mode to obtain a plurality of sub-images conforming to the lens control mode as a key frame sequence;

[0012] A video generation module is used to generate a video that conforms to the lens control mode based on the key frame sequence; wherein the multiple sub-pictures in the key frame sequence are arranged according to the lens movement mode corresponding to the lens mode, so that the lens movement mode of the video matches the lens mode.

[0013] Optionally, the lens control mode includes any one of the following modes: a lens translation mode, a lens zoom-in mode, and a lens zoom-out mode.

[0014] Optionally, the lens control mode includes: a lens translation mode, and the key frame sequence includes: n sub-pictures;

[0015] The cutting processing module comprises:

[0016] A first parameter determination module is used to determine the number of pictures n included in the key frame sequence, and determine the translation direction and unit translation amount s corresponding to the lens translation mode;

[0017] The first cropping module is used to crop n sub-pictures from the target picture according to the translation direction and the unit translation amount s.

[0018] Optionally, the first parameter determination module includes:

[0019] A total translation amount determination module, used to determine the total translation amount according to the size of the target image in the translation direction and the expansion ratio of the target image in the translation direction;

[0020] The unit translation amount determination module is used to determine the unit translation amount s according to the total translation amount and the number of pictures.

[0021] Optionally, the lens control mode includes: a lens zoom-in mode, and the key frame sequence includes: n sub-pictures;

[0022] The cutting processing module comprises:

[0023] A second parameter determination module is used to determine the number n of pictures included in the key frame sequence, and determine the first cropping ratios or first intermediate picture sizes corresponding to the n sub-pictures respectively;

[0024] A second cropping module is configured to crop n first intermediate images from the target image according to the first cropping ratio or the first intermediate image size and the center of the target image; the n first intermediate images correspond to different first intermediate image sizes;

[0025] The first size unification module is used to perform size unification processing on the first intermediate image to obtain a sub-image that meets the target size.

[0026] Optionally, the lens control mode includes: a lens zoom-out mode, and the key frame sequence includes: n sub-pictures;

[0027] The cutting processing module comprises:

[0028] A third parameter determination module is used to determine the number n of pictures included in the key frame sequence, and determine the second cropping ratios or the second intermediate picture sizes corresponding to the n sub-pictures respectively;

[0029] a third cropping module, configured to crop n second intermediate images from the target image according to the second cropping ratio or the second intermediate image size and the center of the target image; the n second intermediate images correspond to different second intermediate image sizes;

[0030] The second size unification module is used to perform size unification processing on the second intermediate image to obtain a sub-image that meets the target size.

[0031] Optionally, the device further comprises:

[0032] The content expansion module is used to input the original image and prompt text into the image expansion model to obtain the target image output by the image expansion model; the prompt text includes: a preset position and the size of the target image; the preset position is used to represent the position for adding new details or new elements in the original image.

[0033] In a third aspect of the present invention, there is also provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0034] Memory, used to store computer programs;

[0035] The processor is used to implement the aforementioned method steps when executing the program stored in the memory.

[0036] In a fourth aspect of the implementation of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute any of the above-described methods.

[0037] The video generation method, device, electronic device and medium provided by the embodiments of the present invention first perform cropping processing on the target image according to the lens control mode to obtain a key frame sequence that conforms to the lens control mode; then, a video is generated according to the key frame sequence; wherein the multiple sub-images in the key frame sequence are arranged according to the lens movement mode corresponding to the lens mode, so that the lens movement mode of the video matches the lens mode.

[0038] The lens control mode determined by the embodiment of the present invention is used to characterize the motion mode of the lens picture. Moreover, the key frame sequence obtained according to the lens control mode and the target image can be arranged in order according to the lens motion mode characterized by the lens control mode. On this basis, the video generated by the embodiment of the present invention according to the key frame sequence can present the motion effect of the lens picture under the guidance of the lens motion mode corresponding to the key frame sequence. Therefore, the embodiment of the present invention can not only improve the visual quality of the video, but also bring a better viewing experience to the audience, thereby meeting the user's demand for high-quality video. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0040] Figure 1 A flowchart of a video generation method according to an embodiment of the present invention;

[0041] Figure 2 This is an example of cropping a target image according to an embodiment of the present invention;

[0042] FIG3(a) is an example of an original picture of the present invention, and FIG3(b) is an example of a target picture corresponding to FIG3(a);

[0043] Figure 4(a) to Figure 4(h) is a schematic diagram of a key frame sequence corresponding to the target image shown in FIG3( b ) in the lens left shift mode;

[0044] Figure 5(a) to Figure 5(h) is a schematic diagram of a key frame sequence corresponding to the target image shown in FIG3( b ) in the downward lens shift mode;

[0045] Figure 6(a) to Figure 6(h) is a schematic diagram of a key frame sequence corresponding to the target image shown in FIG3( b ) in the zoom-in mode;

[0046] Figure 7 A schematic diagram of the structure of a video generating device according to an embodiment of the present invention;

[0047] Figure 8The structure block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0049] The video generation method according to the embodiment of the present invention can be used to generate a video based on a picture to improve the visual effect of the video.

[0050] The embodiment of the present invention does not limit the specific application scenarios corresponding to the video generation method. For example, the application scenarios of the video generation method may include: personal life recording scenarios, education scenarios, commercial advertisement production scenarios and many other scenarios.

[0051] Taking the conversion of picture books to videos in educational scenarios as an example, related technologies usually use methods such as shooting or scanning to obtain picture book images, and then use video generation tools to generate videos corresponding to the picture book images. However, the principle of video generation tools is: based on the understanding of the content of picture book images, videos are randomly generated; therefore, videos generated by video generation tools usually do not contain camera movement. Videos without camera movement usually lack a sense of dynamics, and the video screen will appear stiff and dull, which greatly reduces the visual effect of the video.

[0052] In response to the technical problem of low visual effect of videos in related technologies, an embodiment of the present invention provides a video generation method, which specifically includes: according to a lens control mode, cropping a target image to obtain multiple sub-images that conform to the lens control mode as a key frame sequence; generating a video that conforms to the lens control mode according to the key frame sequence; wherein the multiple sub-images in the key frame sequence are arranged according to the lens movement mode corresponding to the lens mode, so that the lens movement mode of the video matches the lens mode.

[0053] In the technical solution of the embodiment of the present invention, the target image is firstly cropped according to the lens control mode to obtain a key frame sequence that conforms to the lens control mode; then, a video is generated according to the key frame sequence; wherein the multiple sub-images in the key frame sequence are arranged according to the lens movement mode corresponding to the lens mode, so that the lens movement mode of the video matches the lens mode.

[0054] The lens control mode determined by the embodiment of the present invention is used to characterize the motion mode of the lens picture. Moreover, the key frame sequence obtained according to the lens control mode and the target image can be arranged in order according to the lens motion mode characterized by the lens control mode. On this basis, the video generated by the embodiment of the present invention according to the key frame sequence can present the motion effect of the lens picture under the guidance of the lens motion mode corresponding to the key frame sequence. Therefore, the embodiment of the present invention can not only improve the visual quality of the video, but also bring a better viewing experience to the audience, thereby meeting the user's demand for high-quality video.

[0055] The embodiments of the present invention are described below through specific examples.

[0056] Reference Figure 1 , shows a flowchart of the steps of a video generation method according to an embodiment of the present invention, the method may specifically include the following steps:

[0057] Step 101: According to the lens control mode, the target image is cropped to obtain a plurality of sub-images conforming to the lens control mode as a key frame sequence;

[0058] Step 102: Generate a video that conforms to the lens control mode based on the key frame sequence; wherein the multiple sub-pictures in the key frame sequence are arranged according to the lens movement mode corresponding to the lens mode, so that the lens movement mode of the video matches the lens mode.

[0059] Figure 1 The illustrated method embodiment can be used to generate a video based on a picture to enhance the visual effect of the video. Figure 1 The method embodiment shown can be executed by a client or a server. It can be understood that the embodiment of the present invention is for Figure 1 The specific execution subject of the illustrated method embodiment is not limited.

[0060] In step 101, the target image may be an original image uploaded by a user, or may be an image obtained by performing content expansion processing on the original image uploaded by a user. It is understood that the embodiment of the present invention does not limit the specific target image and the specific method of obtaining the target image.

[0061] In a specific implementation, the client or server may provide a picture upload interface. The picture upload interface is used to receive the original picture uploaded by the user. The original picture may be one or more.

[0062] The original picture of the embodiment of the present invention may be different pictures in different application scenarios. Taking the conversion of picture books to videos in an educational scenario as an example, the original picture may be a picture of a picture book. The user may obtain the original picture by taking a picture or scanning a text. It is understood that the embodiment of the present invention does not limit the specific original picture and the specific method for obtaining the original picture.

[0063] The lens control mode of the embodiment of the present invention is used to characterize the movement mode of the lens picture.

[0064] In a specific implementation, the lens control mode includes any of the following modes: lens shift mode, lens zoom-in mode, and lens zoom-out mode. Among them, the lens shift mode may include: lens shift mode in any direction. Specifically, the lens shift mode may include: lens left shift mode, lens right shift mode, lens down shift mode, or lens up shift mode, etc.

[0065] In practical applications, the lens control mode may be specified by the user. For example, a mode setting interface may be provided, and the mode setting interface may be used to set the lens control mode corresponding to the original image. For example, the mode setting interface may include: an input box, and the user may enter the lens setting mode corresponding to the original image in the input box.

[0066] The embodiment of the present invention can also automatically determine the lens control mode corresponding to the original image. Accordingly, the process of determining the lens control mode corresponding to the original image may specifically include:

[0067] Step A1: determining a lens control mode corresponding to the original image according to semantic information corresponding to the original image; and / or

[0068] Step A2: determining the lens control mode corresponding to the original pictures according to the context information between the multiple original pictures.

[0069] For step A1, the semantic information may represent the content and meaning expressed by the original image. The semantic information may include, but is not limited to: the subject content, emotional tone, and objects and scenes involved in the image.

[0070] In practical applications, image recognition technology can be used to determine the semantic information corresponding to the original image. Alternatively, the semantic information corresponding to the original image can be determined based on the text information contained in the original image. Taking the original image as a picture book image as an example, the text in the picture book image can be the text information contained in the picture book image. It can be understood that the embodiment of the present invention does not limit the specific method for determining the semantic information corresponding to the original image.

[0071] Different semantic information can correspond to different lens control modes to better convey the connotation of the original picture and enhance the visual effect.

[0072] According to one implementation, for semantic information showing a wide scene, the panning lens control mode can allow viewers to more fully experience the grandeur of the picture. If the original picture is a picture of a wide grassland landscape, and the semantic information expresses the vastness and tranquility of nature, then the panning lens control mode can be determined to show different parts of the grassland by panning the lens.

[0073] According to another implementation, for semantic information that highlights specific details, the zoom-in and zoom-out lens control mode can better focus on the key parts. For example, if the original image is a close-up of a delicate flower, the semantic information focuses on highlighting the beauty and delicacy of the flower. At this time, the zoom-in and zoom-out lens control mode can be determined, first zooming in to show the details of the flower, such as the texture of the petals, the shape of the stamens, etc., and then zooming out to show the position of the flower in the flowers and the surrounding environmental background, so as to better highlight the uniqueness of the flower.

[0074] For step A2, the lens control mode is determined according to the context information between the multiple original pictures, so that the video can achieve smoother transition and more coherent visual effects between the multiple pictures.

[0075] In one implementation, if a series of original images describe a continuous story or scene, the embodiment of the present invention can determine an appropriate lens control mode based on context information, such as using panning to switch between different scenes, or using a zooming mode to highlight a key plot. This can make the generated video more logical and integrated, and enhance the audience's viewing experience.

[0076] In another implementation, the embodiment of the present invention can determine the behavior information of the object in the original picture according to the context information between the multiple original pictures, and then determine the lens control mode according to the above behavior information. The above method of determining the lens control mode according to the behavior information of the object in the original picture can make the generated video more consistent with the content of the picture, enhance the expressiveness and logic of the video, and bring a better viewing experience to the audience. Among them, the object can include: people, animals, objects, etc.

[0077] In one example, it is assumed that the picture book scene corresponding to the original picture is "forest adventure". Further, it is assumed that the original picture includes the following picture:

[0078] Image 1: A child stands at the entrance of a forest, with a winding path in front of him and tall trees on both sides. The child is carrying a backpack and has an expression of excitement and curiosity on his face.

[0079] Picture 2: The child walked a little further along the path. The trees next to him became more dense, and a small bird was standing on a branch.

[0080] Picture 3: The child continued to move forward, and a bunch of bright wild flowers appeared on the roadside. The child stopped and looked at the wild flowers with an expression of surprise on his face.

[0081] From picture one to picture two, the camera can slowly pan from left to right. Starting from the child at the entrance of the forest, pan to show the extension of the path and the changes of the trees on both sides, highlighting the process of the child gradually going deeper into the forest. The appearance of the bird can serve as a guide, allowing the camera to naturally pan to the location of the bird, adding a vivid feeling to the picture.

[0082] From picture 2 to picture 3, the camera continues to pan from left to right. As the child moves forward, the camera first shows the child's back and the denser trees around him, then pans to where the wildflowers appear, focusing on the wildflowers, and then returns to the child's surprised expression, reflecting the child's surprise and excitement in discovering new things during the adventure.

[0083] Compared with the original picture, the target picture obtained by the content expansion processing in the embodiment of the present invention contains more information. The content expansion processing can add more details on the basis of the original picture, making the picture content richer and fuller.

[0084] In practical applications, the size of the target image can be larger than the size of the original image. For example, if the size of the target image is H2×W2 and the size of the original image is H1×W1, then H2 is greater than or equal to H1, or W2 is greater than or equal to W2. H2 represents the height size of the target image, which can represent the number of pixels in the height direction of the target image. W2 represents the width size of the target image, which can represent the number of pixels in the width direction of the target image.

[0085] The content expansion process of the embodiment of the present invention can be performed on a preset position of the original image. The preset position can be: an edge position of the original image, or an object position of the original image.

[0086] The edge position of the original image refers to the border area around the image. In an image, the edge position is usually the outer part of the picture, which defines the range of the image. To perform content expansion processing on the edge position is to add new content around the image. Performing content expansion processing on the edge position of the original image can provide more space for the lens panning mode and better show the relationship between the object and the environment.

[0087] The object position refers to the area in the original image where a specific object is located. The object here can be any element that is important in the image, such as a person, an animal, or an object. The object position is mainly to add relevant content around the specific object. For example, in a photo of a person, the object position is the person itself and a small area around it. You can add some elements related to the person's activities around the person, such as props, background decorations, etc. Content expansion processing for the object position of the original image can enrich the background of the object and provide more details for the zoom mode or zoom mode.

[0088] Of course, those skilled in the art can use other preset positions besides edge positions and object positions to perform content expansion processing according to actual application requirements. For example, other preset positions can be the positions of objects commonly included in two adjacent original pictures. It can be understood that the embodiments of the present invention do not limit the specific preset positions.

[0089] The embodiment of the present invention can use the picture expansion technology to expand the content of the original picture. The picture expansion technology can add new details or new elements while maintaining the basic elements of the original image, so that the expanded image is more visually rich and complete.

[0090] The process of performing content expansion processing on the original image in the embodiment of the present invention may specifically include: performing content expansion processing on the original image using an image expansion model. Specifically, the original image and the prompt text may be input into the image expansion model to obtain a target image output by the image expansion model.

[0091] The image expansion model can capture the characteristic patterns of different types of images through a large amount of image data, so that when expanding the image, it can generate content consistent with the style of the original image. For example, using a generation tool based on a diffusion model such as SD, new elements that merge with the original image can be generated based on the input prompt text.

[0092] Hint text plays a vital role in image expansion technology. It provides clear guidance for the image expansion model, helping it to add new details or new elements while maintaining the original elements of the original image, making the expanded target image richer and more complete.

[0093] The prompt text may include information such as a preset position, a size of a target image, etc. The preset position may be used to represent a position for adding new details or new elements to an original image.

[0094] The embodiment of the present invention can perform a cropping process on the target image according to the lens control mode to obtain a key frame sequence that conforms to the lens control mode.

[0095] The embodiment of the present invention can perform multiple cropping processes on the target image to obtain a series of sub-images that conform to the lens control mode. These sub-images can form a key frame sequence for displaying the motion effect of the lens picture. The multiple sub-images in the key frame sequence can be arranged according to the lens motion mode corresponding to the lens mode, so that the lens motion mode of the video matches the lens mode.

[0096] Specifically, if the lens control mode is the left shift in the lens shift mode, the position of the sub-image cropped from the target image in the target image gradually moves from close to the right edge to the left. In other words, the first sub-image is close to the right edge of the target image, and as the key frame sequence advances, the positions of subsequent sub-images gradually move to the left. At this time, the pixels in the target image move from right to left, and as the key frame sequence advances, the pixels gradually move from the right side of the screen to the left side. Just like when the lens pans to the left, the content in the screen is displayed from right to left, and each sub-image displays the content of the screen at different positions in turn.

[0097] If the lens control mode is the right shift in the lens pan mode, the position of the sub-images cut from the target image in the target image gradually moves from close to the left edge to the right. In other words, the first sub-image is close to the left edge of the target image, and as the key frame sequence advances, the positions of subsequent sub-images gradually move to the right. At this time, the pixels in the target image move from left to right, and as the key frame sequence advances, the pixels gradually move from the left side of the screen to the right side. Just like when the lens pans to the right, the content in the screen is displayed from left to right, and each sub-image displays the content of the screen at different positions in turn.

[0098] If the lens control mode is the zoom-in mode, the position of the sub-image cropped from the target image in the target image gradually shrinks inward with a specific focus as the center. In other words, the first sub-image covers a larger screen range, and as the key frame sequence progresses, the subsequent sub-images gradually focus on a specific focus, and the screen range continues to shrink, showing more details.

[0099] If the lens control mode is the zoom mode, the position of the sub-images cropped from the target image in the target image gradually expands outward from a specific focus point. In other words, the first sub-image is focused near the specific focus point, and the image range is small. As the key frame sequence progresses, the subsequent sub-images gradually expand the image range to show a wider scene.

[0100] The embodiment of the present invention can provide the following technical solution for cropping the target image:

[0101] Technical solution 1,

[0102] In technical solution 1, the lens control mode specifically includes: lens shift mode, and the key frame sequence specifically includes: n sub-pictures; n can be a positive integer greater than 1;

[0103] The process of cropping the target image according to the lens control mode specifically includes:

[0104] Step B1, determining the number of pictures n included in the key frame sequence, and determining the translation direction and unit translation amount s corresponding to the lens translation mode;

[0105] Step B2: cropping n sub-images from the target image according to the translation direction and the unit translation amount s.

[0106] Regarding step B1, the embodiment of the present invention can determine the number n of pictures included in the key frame sequence according to the audio duration t corresponding to the original picture.

[0107] Taking the original picture as a picture book picture as an example, a picture book picture usually contains picture book text. The embodiment of the present invention can use speech synthesis technology to convert the picture book text into picture audio, and the audio duration t can be the duration of the picture audio.

[0108] There may be a positive relationship between the number of pictures n and the audio duration t. In other words, the longer the audio duration t, the greater the number of pictures n. It is understandable that those skilled in the art can determine the number of pictures n corresponding to the audio duration t according to actual application requirements.

[0109] The translation direction corresponding to the lens translation mode can be determined according to the lens translation mode. For example, the translation direction corresponding to the lens left shift mode is left shift. For another example, the translation direction corresponding to the lens down shift mode is down shift.

[0110] The process of determining the unit translation amount s corresponding to the lens translation mode specifically includes: determining the total translation amount according to the size of the target image in the translation direction and the expansion ratio of the target image in the translation direction; and determining the unit translation amount s according to the total translation amount and the number of images.

[0111] Assuming that the translation direction is the left direction, the size of the target image in the translation direction may be the size of the target image in the horizontal direction or the width direction, that is, the width size W2. Then the expansion ratio k corresponding to the target image in the translation direction may be: (W2-W1) / W1.

[0112] The total translation amount may be the product of the expansion ratio and the size of the original image in the translation direction. Assuming that the translation direction is the left direction, the total translation amount may be W1*k.

[0113] Further, the unit translation amount s may be a quotient obtained by dividing the total translation amount by the number of pictures, specifically expressed as: s=W1*k / n.

[0114] In step B2, n sub-pictures may be cropped from the target picture according to the translation direction and the unit translation amount s.

[0115] Assuming that the translation direction is the left direction, taking the cropping of the i-th sub-image as an example, the (right edge - i*s) of the target image can be used as the right edge of the i-th sub-image, and the i-th sub-image of a preset size can be cropped from the target image. It should be noted that the preset size can be the same as the size of the original image. The i-th target size and the target image can contain the same axis of symmetry, and this axis of symmetry can be parallel to the translation direction. i can be a positive integer. In the order of i from small to large (from 1 to n), the multiple sub-images in the above key frame sequence are arranged according to the lens movement mode corresponding to the above lens mode.

[0116] Reference Figure 2 , showing an example of cropping a target image according to an embodiment of the present invention, Figure 2 A target image with a width of W2 and a height of H2 is shown in FIG.

[0117] Assuming that the size of the original image is 1000×2000, assuming that the size of the target image is 1200×2400, assuming that the translation direction is left, the expansion ratio can be: k=(2400-2000) / 2000=0.2. Further, assuming that the number of images n=20, the unit translation amount s=W1*k / n=1000*0.2 / 20=10. The units of the unit translation amount and the total translation amount can be pixels.

[0118] The process of cropping the target image specifically includes:

[0119] First, the right edge of the target image is used as the right edge of the first sub-image, and the first sub-image with a size of 1000×2000 is cropped from the target image according to the symmetry axis parallel to the width direction of the target image. Figure 2 The target image area covered by the dotted rectangular frame 201 in represents the first sub-image.

[0120] Then, the (right edge - s) of the target image is used as the right edge of the second sub-image, and the second sub-image with a size of 1000×2000 is cropped from the target image according to the symmetry axis parallel to the width direction of the target image. Figure 2 The target image area covered by the dotted rectangular frame 202 in represents the second sub-image.

[0121] Similarly, the (right edge - i*s) of the target image is used as the right edge of the i-th sub-image, and the i-th sub-image of a preset size is cropped from the target image.

[0122] 3( a ) shows an example of an original picture of the present invention, and FIG. 3( b ) shows an example of a target picture corresponding to FIG. 3( a ).

[0123] Assuming the translation direction is leftward, then Figure 4(a) to Figure 4(h) A schematic diagram of a key frame sequence corresponding to the target image shown in FIG3( b ) in the left-shift mode of the lens is shown. The key frame sequence includes 8 sub-images, which are obtained by cropping the target image in sequence from left to right.

[0124] Taking other translation directions except the left translation direction as an example, since the corresponding cropping processing process is similar to the cropping processing process corresponding to the left translation direction, it will not be described here in detail and can be referred to each other.

[0125] For example, assuming that the translation direction is the downward direction, taking the cropping of the i-th sub-image as an example, the (upper edge - i*s) of the target image can be used as the upper edge of the i-th sub-image, and the i-th sub-image of the preset size can be cropped from the target image. In this case, the i-th target size and the target image can contain the same symmetry axis, and the symmetry axis can be parallel to the translation direction.

[0126] Assuming the translation direction is downward, then Figure 5(a) to Figure 5(h) A schematic diagram of a key frame sequence corresponding to the target image shown in FIG3( b ) in the downward lens shift mode is shown, where the key frame sequence includes 8 sub-images, which are obtained by cropping the target image in sequence from top to bottom.

[0127] Technical solution 2,

[0128] In technical solution 2, the lens control mode specifically includes: a lens zoom-in mode, and the key frame sequence specifically includes: n sub-pictures;

[0129] According to the lens control mode, the process of cropping the target image specifically includes:

[0130] Step C1, determining the number n of pictures included in the key frame sequence, and determining the first cropping ratios or first intermediate picture sizes corresponding to the n sub-pictures respectively;

[0131] Step C2: cropping n first intermediate pictures from the target picture according to the first cropping ratio or the first intermediate picture size and the center of the target picture; the n first intermediate pictures correspond to different first intermediate picture sizes;

[0132] Step C3: performing size unification processing on the first intermediate image to obtain a sub-image that meets the target size.

[0133] The embodiment of the present invention may first determine a ratio step or a size step; then, according to the ratio step or the size step, determine the first cropping ratio or the first intermediate image size corresponding to the n sub-images. The first cropping ratio corresponding to the first sub-image may be equal to 1. The first cropping ratio may represent the ratio of the first intermediate image to the target image.

[0134] Optionally, the first cropping ratio of the i-th sub-image may be determined according to the ratio step. For example, the first cropping ratio of the i-th sub-image may be expressed as: 1-(i-1)*ratio step. As i increases, the first cropping ratio decreases.

[0135] In an example, assuming that the size of the original image is 1000×2000, and assuming that the size of the target image is 1200×2400, the expansion ratio can be 1.2, and assuming that the number of sub-images is 20, the proportional step can be (1.2-1) / 20=0.01, and the first cropping ratio of the i-th sub-image can be: 1-(i-1)*0.01.

[0136] Optionally, the size of the target image may be decreased according to the size step to obtain the first intermediate image size of the ith sub-image. For example, the first intermediate image size of the ith sub-image may be expressed as: the size of the target image - (i-1) * the size step.

[0137] In an example, assuming that the size of the original image is 1000×2000, assuming that the size of the target image is 1200×2400, assuming that the number of sub-images is 20, the size step in the width direction may be (2400-2000) / 20=20, and the size step in the height direction may be (1200-1000) / 20=10. Therefore, the size of the first intermediate image of the first sub-image may be: 1200×2400, the size of the first intermediate image of the second sub-image may be: 1190×2380... The size of the first intermediate image of the 20th sub-image may be: 1010×2020.

[0138] In step C2, the size of the first intermediate image may be determined according to the first cropping ratio, and then, according to the center of the target image, a first intermediate image corresponding to the size of the first intermediate image may be cropped from the target image, wherein the centers of the first intermediate image and the target image are the same.

[0139] Since different first cropping ratios correspond to different first intermediate image sizes, step C3 performs size unification processing on the first intermediate image to obtain sub-images with uniform sizes. In other words, the size-unified images can be used as sub-images. The sizes of the n sub-images can all be the target size.

[0140] The target size can be determined by those skilled in the art according to actual application requirements. For example, the target size can be equal to the size of the original image. Alternatively, the target size can be x times the size of the target image, where x can be a positive number less than 1, for example, x can be a value such as 0.7 or 0.8. The size unification process can be a size reduction process or a size enlargement process.

[0141] The method for performing size unification processing on the first intermediate image may include: a linear interpolation method or a deep learning method, etc. It can be understood that the embodiment of the present invention does not limit the specific size unification processing method.

[0142] Figure 6(a) to Figure 6(h) A schematic diagram of a key frame sequence corresponding to the target image of FIG3(b) in the zoom-in mode is shown, wherein the key frame sequence includes 8 sub-images, and 8 first intermediate images are obtained by cropping from the target image in order of the first cropping ratio from large to small, and the 8 first intermediate images are resized to obtain 8 sub-images.

[0143] In the embodiment of the present invention, when the lens zoom-in mode is adopted, since the first intermediate pictures are cropped from the same target picture, the object size of the same object in different first intermediate pictures is the same. Since the size of the pictures from the 1st first intermediate picture to the nth first intermediate picture gradually decreases, this means that the proportion of the same object in the first intermediate picture (such as the tiger in Figure 6 (a)) in different first intermediate pictures gradually increases. On this basis, after the n first intermediate pictures are unified into n sub-pictures with the same target size, the object size of the same object in the n sub-pictures is increased. Therefore, the key frame sequence of the embodiment of the present invention can achieve the motion effect of lens zooming in.

[0144] The ratio can represent the relative relationship between the size of an object and the size of the image. When the image size is reduced, the proportion of an object of the same size in the image will increase accordingly.

[0145] Technical solution 3,

[0146] In technical solution 3, the lens control mode specifically includes: a lens zoom-out mode, and the key frame sequence specifically includes: n sub-pictures;

[0147] The process of cropping the target image according to the lens control mode specifically includes:

[0148] Step D1, determining the number n of pictures included in the key frame sequence, and determining the second cropping ratios or the second intermediate picture sizes corresponding to the n sub-pictures respectively;

[0149] Step D2: cropping n second intermediate images from the target image according to the second cropping ratio or the second intermediate image size and the center of the target image; the n second intermediate images correspond to different second intermediate image sizes;

[0150] Step D3: resize the second intermediate image to obtain a sub-image that meets the target size.

[0151] The embodiment of the present invention may first determine the ratio step or the size step, and then determine the second cropping ratio or the second intermediate image size corresponding to the n sub-images respectively according to the ratio step or the size step. The second cropping ratio corresponding to the nth sub-image may be equal to 1. The second cropping ratio may represent the ratio of the second intermediate image to the target image.

[0152] Optionally, the second cropping ratio of the ith sub-image may be determined according to the ratio step. For example, the second cropping ratio of the ith sub-image may be expressed as: 1-(ni)*ratio step. As i increases, the second cropping ratio increases.

[0153] In an example, assuming that the size of the original image is 1000×2000, and assuming that the size of the target image is 1200×2400, the expansion ratio can be 1.2, and assuming that the number of sub-images is 20, the proportional step can be (1.2-1) / 20=0.01, and the second cropping ratio of the i-th sub-image can be: 1-(ni)*0.01.

[0154] Optionally, the size of the target image may be decreased according to the size step to obtain the second intermediate image size of the ith sub-image. For example, the second intermediate image size of the ith sub-image may be expressed as: the size of the target image - (ni)*size step.

[0155] In an example, assuming that the size of the original image is 1000×2000, assuming that the size of the target image is 1200×2400, assuming that the number of sub-images is 20, the size step in the width direction may be (2400-2000) / 20=20, and the size step in the height direction may be (1200-1000) / 20=10. Therefore, the size of the second intermediate image of the first sub-image may be: 1010×2020, the size of the second intermediate image of the second sub-image may be: 1020×2040... The size of the second intermediate image of the 20th sub-image may be: 1200×2400.

[0156] In step D2, the size of the second intermediate image may be determined first according to the second cropping ratio, and then, according to the center of the target image, a second intermediate image corresponding to the size of the second intermediate image may be cropped from the target image, wherein the center of the second intermediate image is the same as that of the target image.

[0157] Since different second cropping ratios correspond to different sizes of the second intermediate image, step D3 performs size unification processing on the second intermediate image to obtain sub-images of uniform size. In other words, the image after size unification processing can be used as a sub-image. The sizes of the n sub-images can all be target sizes. The target size can be determined by those skilled in the art according to actual application requirements. For example, the target size can be equal to the size of the original image.

[0158] The method for performing size unification processing on the second intermediate image may include: a linear interpolation method or a deep learning method, etc. It can be understood that the embodiment of the present invention does not limit the specific size unification processing method.

[0159] In the embodiment of the present invention, when the lens zoom-out mode is adopted, since the second intermediate pictures are cropped from the same target picture, the object size of the same object in different second intermediate pictures is the same. Since the size of the second intermediate pictures from the 1st to the nth increases gradually, this means that the proportion of the same object in the second intermediate pictures (such as the tiger in Figure 6 (a)) in different second intermediate pictures is gradually reduced. On this basis, after the n second intermediate pictures are unified into n sub-pictures with the same target size, the object size of the same object in the n sub-pictures is reduced. Therefore, the key frame sequence of the embodiment of the present invention can achieve the motion effect of zooming out.

[0160] In step 102, a generation model of a key frame sequence to a video may be used to generate a video corresponding to the key frame sequence.

[0161] Among them, the generation model of key frame sequence to video can be an artificial intelligence model that can convert static images into animated videos.

[0162] This generative model is based on the stable diffusion model, which makes full use of the stable diffusion model's powerful image generation ability and takes it as a solid foundation for animation generation.

[0163] In addition, the generative model also introduces a motion prediction module, which can perform in-depth analysis of static images and predict the motion path corresponding to the static images.

[0164] The keyframe sequence to video generation model can generate a series of video frames based on the input keyframe sequence, and these video frames together build the basic skeleton of the animation. Each video frame is slightly different from the previous video frame and incorporates the predicted motion changes.

[0165] In practical applications, the generated video can be output. For example, a video download link can be displayed on a user interface so that the user can download the video according to the download link. Of course, a video viewing entry can also be displayed on the user interface so that the user can view the video according to the viewing entry. If the user is not satisfied with the video, a video regeneration interface can be triggered. The regeneration interface can be used to regenerate the video corresponding to the key frame sequence using a generation model from a key frame sequence to a video.

[0166] Of course, the key frame sequence to video generation model can output multiple videos based on the input key frame sequence for users to select and download.

[0167] It should be noted that the video of the embodiment of the present invention is a video with a lens control effect. In an educational scenario, the embodiment of the present invention can convert picture book images into picture book videos with dynamic effects according to a specific lens control mode, thereby enhancing the visual appeal and expressiveness of the picture book.

[0168] In summary, the video generation method of the embodiment of the present invention first crops the target image according to the lens control mode to obtain a key frame sequence that conforms to the lens control mode; then, generates a video according to the key frame sequence; wherein the multiple sub-images in the key frame sequence are arranged according to the lens movement mode corresponding to the lens mode, so that the lens movement mode of the video matches the lens mode.

[0169] The lens control mode determined by the embodiment of the present invention is used to characterize the motion mode of the lens picture. Moreover, the key frame sequence obtained according to the lens control mode and the target image can be arranged in order according to the lens motion mode characterized by the lens control mode. On this basis, the video generated by the embodiment of the present invention according to the key frame sequence can present the motion effect of the lens picture under the guidance of the lens motion mode corresponding to the key frame sequence. Therefore, the embodiment of the present invention can not only improve the visual quality of the video, but also bring a better viewing experience to the audience, thereby meeting the user's demand for high-quality video.

[0170] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0171] Reference Figure 7 , shows a schematic structural diagram of a video generating device according to an embodiment of the present invention. The video generating device may specifically include the following modules: a cropping processing module 701 and a video generating module 702.

[0172] The cropping processing module 701 is used to crop the target image according to the lens control mode to obtain a plurality of sub-images that conform to the lens control mode as a key frame sequence;

[0173] The video generation module 702 is used to generate a video that conforms to the lens control mode based on the key frame sequence; wherein the multiple sub-pictures in the key frame sequence are arranged according to the lens movement mode corresponding to the lens mode, so that the lens movement mode of the video matches the lens mode.

[0174] Optionally, the lens control mode includes any one of the following modes: a lens translation mode, a lens zoom-in mode, and a lens zoom-out mode.

[0175] Optionally, the lens control mode includes: a lens translation mode, and the key frame sequence includes: n sub-pictures;

[0176] The cutting processing module comprises:

[0177] A first parameter determination module is used to determine the number of pictures n included in the key frame sequence, and determine the translation direction and unit translation amount s corresponding to the lens translation mode;

[0178] The first cropping module is used to crop n sub-pictures from the target picture according to the translation direction and the unit translation amount s.

[0179] Optionally, the first parameter determination module includes:

[0180] A total translation amount determination module, used to determine the total translation amount according to the size of the target image in the translation direction and the expansion ratio of the target image in the translation direction;

[0181] The unit translation amount determination module is used to determine the unit translation amount s according to the total translation amount and the number of pictures.

[0182] Optionally, the lens control mode includes: a lens zoom-in mode, and the key frame sequence includes: n sub-pictures;

[0183] The cutting processing module comprises:

[0184] A second parameter determination module is used to determine the number n of pictures included in the key frame sequence, and determine the first cropping ratios or first intermediate picture sizes corresponding to the n sub-pictures respectively;

[0185] A second cropping module, configured to crop n first intermediate pictures from the target picture according to the first cropping ratio or the size of the first intermediate picture and the center of the target picture;

[0186] The first size unification module is used to perform size unification processing on the first intermediate image to obtain a sub-image that meets the target size.

[0187] Optionally, the lens control mode includes: a lens zoom-out mode, and the key frame sequence includes: n sub-pictures;

[0188] The cutting processing module comprises:

[0189] A third parameter determination module is used to determine the number n of pictures included in the key frame sequence, and determine the second cropping ratios or the second intermediate picture sizes corresponding to the n sub-pictures respectively;

[0190] A third cropping module, configured to crop n second intermediate pictures from the target picture according to the second cropping ratio or the size of the second intermediate picture and the center of the target picture;

[0191] The second size unification module is used to perform size unification processing on the second intermediate image to obtain a sub-image that meets the target size.

[0192] Optionally, the device further comprises:

[0193] The content expansion module is used to input the original image and prompt text into the image expansion model to obtain the target image output by the image expansion model; the prompt text includes: a preset position and the size of the target image; the preset position is used to represent the position for adding new details or new elements in the original image.

[0194] An embodiment of the present invention further provides an electronic device that can implement the functions of the aforementioned storage node or scheduling node or ledger node or management node.

[0195] like Figure 8As shown, the electronic device may include a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other via the communication bus 1004.

[0196] Memory 1003, used for storing computer programs;

[0197] The processor 1001 is used to implement the following steps when executing the program stored in the memory 1003:

[0198] Storing the storage object;

[0199] Sending the storage object to a coupled second storage node, so that the second storage node stores and / or forwards the storage object;

[0200] After the storage object is successfully stored, the object information of the storage object is sent to the ledger node, so that the ledger node saves the ledger information of the first storage node; the ledger information includes: the object information stored in the corresponding storage node.

[0201] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0202] The communication interface is used for communication between the above electronic device and other devices.

[0203] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one video generating device located away from the aforementioned processor.

[0204] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0205] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the computer-readable storage medium is executed on a computer, the computer executes the video generation method described in any one of the above embodiments.

[0206] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the video generation method described in any one of the above embodiments.

[0207] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium, or a semiconductor medium (e.g., a solid-state hard disk SolidState Disk (SSD)), etc. Examples of optical media may include DVD (Digital Video Disc) and the like.

[0208] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0209] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0210] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A video generation method, characterized in that: The method comprises: According to the lens control mode, the target image is cropped to obtain a plurality of sub-images conforming to the lens control mode as a key frame sequence; A video conforming to the lens control mode is generated according to the key frame sequence; wherein the multiple sub-pictures in the key frame sequence are arranged according to the lens movement mode corresponding to the lens mode, so that the lens movement mode of the video matches the lens mode.

2. The method according to claim 1, characterized in that: The lens control mode includes any one of the following modes: a lens shift mode, a lens zoom-in mode, and a lens zoom-out mode.

3. The method according to claim 1, characterized in that The lens control mode includes: a lens shift mode, and the key frame sequence includes: n sub-pictures; The step of performing cropping processing on the target image according to the lens control mode includes: Determine the number of pictures n included in the key frame sequence, and determine the translation direction and unit translation amount s corresponding to the lens translation mode; According to the translation direction and the unit translation amount s, n sub-pictures are cropped from the target picture.

4. The method according to claim 3, characterized in that The step of determining the number of pictures n included in the key frame sequence and determining the translation direction and unit translation amount s corresponding to the lens translation mode includes: Determining a total translation amount according to a size of the target image in the translation direction and an expansion ratio of the target image in the translation direction; The unit translation amount s is determined according to the total translation amount and the number of pictures.

5. The method according to claim 1, characterized in that The lens control mode includes: a lens zoom-in mode, and the key frame sequence includes: n sub-pictures; The step of performing cropping processing on the target image according to the lens control mode includes: Determine the number n of pictures included in the key frame sequence, and determine first cropping ratios or first intermediate picture sizes corresponding to the n sub-pictures respectively; Cropping n first intermediate pictures from the target picture according to the first cropping ratio or the first intermediate picture size and the center of the target picture; the n first intermediate pictures correspond to different first intermediate picture sizes; The first intermediate image is processed to be uniform in size to obtain a sub-image that meets the target size.

6. The method according to claim 1, characterized in that The lens control mode includes: a lens zoom-out mode, and the key frame sequence includes: n sub-pictures; The step of performing cropping processing on the target image according to the lens control mode includes: Determine the number n of pictures included in the key frame sequence, and determine the second cropping ratios or second intermediate picture sizes corresponding to the n sub-pictures respectively; According to the second cropping ratio, or the second intermediate picture size, and the center of the target picture, cropping n second intermediate pictures from the target picture; the n second intermediate pictures correspond to different second intermediate picture sizes; The second intermediate image is processed to be uniform in size to obtain n sub-images that meet the target size.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: The original image and the prompt text are input into the image expansion model to obtain the target image output by the image expansion model; the prompt text includes: a preset position and the size of the target image; the preset position is used to represent the position for adding new details or new elements in the original image.

8. A video generating device, characterized in that: The device comprises: A cropping processing module, used for cropping the target image according to the lens control mode to obtain a plurality of sub-images conforming to the lens control mode as a key frame sequence; A video generation module is used to generate a video that conforms to the lens control mode based on the key frame sequence; wherein the multiple sub-pictures in the key frame sequence are arranged according to the lens movement mode corresponding to the lens mode, so that the lens movement mode of the video matches the lens mode.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images