Video generation method and device, equipment, storage medium and program product

Abstract

The invention discloses a video generation method and device, equipment, a storage medium and a program product. Obtaining input content; wherein the input content comprises line text and shot information; generating a plurality of candidate shot sequences based on the line text and the shot information; determining a target lens sequence from the plurality of candidate lens sequences according to the adjacent relation between the lenses in each candidate lens sequence; and generating a target video based on the target shot sequence. According to the video generation method provided by the embodiment of the invention, the multiple candidate shot sequences are automatically generated based on the line text and the shot information, so that diversification of the candidate shot sequences is realized, then the target shot sequence is determined from the multiple candidate shot sequences, and the target video is generated based on the target shot sequence. When the target video is played, the target video displays different video contents at different visual angles and distances along with the sequential switching of various shots in the target shot sequence, so that the diversity of video content display is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of video generation technology, and in particular to a video generation method, device, equipment, storage medium and program product. Background Art

[0002] The metaverse is a virtual digital world that creates an immersive and interactive virtual environment by combining technologies such as virtual reality, augmented reality and artificial intelligence.

[0003] The metaverse needs to use technical means to create a virtual environment outside the real physical environment, so it is highly dependent on virtual reality technology. The basic implementation method of virtual reality technology is to use computers and other equipment to create a virtual world with realistic three-dimensional vision, touch, smell and other sensory experiences. As we all know, vision is the main way for humans to perceive the external world, so three-dimensional vision (abbreviated as "3D") content generation technology plays a vital role in the field of virtual reality technology.

[0004] 3D content generation technology can be further divided into different branches such as 3D action generation technology, 3D face generation technology, and 3D lens generation technology. Currently, it is possible to automatically generate 3D actions and 3D faces, but it is not possible to automatically generate 3D lenses. Therefore, automatically generated 3D videos usually use a single lens to display content, which affects the diversity of video content display. Summary of the invention

[0005] The embodiments of the present invention provide a video generation method, device, equipment, storage medium and program product, which can generate a video with continuously switched shots and improve the diversity of video content display.

[0006] In a first aspect, an embodiment of the present invention provides a video generation method, including:

[0007] Obtaining input content; wherein the input content includes dialogue text and shot information;

[0008] Based on the dialogue text and the shot information, generating multiple candidate shot sequences;

[0009] Determining a target shot sequence from a plurality of candidate shot sequences according to the adjacent relationship between shots in each of the candidate shot sequences;

[0010] Based on the target shot sequence, a target video is generated.

[0011] In a second aspect, an embodiment of the present invention further provides a video generating device, including:

[0012] An input content acquisition module, used to acquire input content; wherein the input content includes dialogue text and shot information;

[0013] A candidate shot sequence generation module, used to generate multiple candidate shot sequences based on the dialogue text and the shot information;

[0014] a target shot sequence determination module, which determines a target shot sequence from a plurality of candidate shot sequences according to the adjacent relationship between shots in each of the candidate shot sequences;

[0015] The target video generation module is used to generate a target video based on the target shot sequence.

[0016] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can execute the video generating method described in the embodiment of the present invention.

[0020] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video generation method described in the embodiment of the present invention when executed.

[0021] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the video generating method as described in the embodiment of the present invention.

[0022] The embodiment of the present invention discloses a video generation method, device, equipment, storage medium and program product. Input content is obtained; wherein the input content includes dialogue text and lens information; based on the dialogue text and the lens information, a plurality of candidate lens sequences are generated; according to the adjacent relationship between the lenses in each candidate lens sequence, a target lens sequence is determined from the plurality of candidate lens sequences; based on the target lens sequence, a target video is generated. The video generation method provided by the embodiment of the present invention automatically generates a plurality of candidate lens sequences based on the dialogue text and the lens information, thereby realizing the diversification of the candidate lens sequences, and then determines the target lens sequence from the plurality of candidate lens sequences, and generates a target video based on the target lens sequence. When playing the target video, as the various lenses in the target lens sequence are switched in sequence, the target video displays different video content at different viewing angles and distances, so as to improve the diversity of the video content display. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flow chart of a video generation method in an embodiment of the present invention;

[0024] Figure 2 is an example diagram of input content obtained in an embodiment of the present invention;

[0025] Figure 3 is an example diagram for determining a candidate shot sequence in an embodiment of the present invention;

[0026] Figure 4a is an example diagram of a video frame displayed by a shot of a content display category in an embodiment of the present invention;

[0027] Figure 4b is an example diagram of a video frame displayed by a shot showing a virtual person in an embodiment of the present invention;

[0028] Figure 5 is a flow chart of a video generation method in an embodiment of the present invention;

[0029] Figure 6 is a structural schematic diagram of a video generating device in an embodiment of the present invention;

[0030] Figure 7 It is a structural schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0032] Figure 1 A flowchart of a video generation method provided in an embodiment of the present invention. This embodiment can be applied to automatically generate a video in which content is displayed by switching multiple lenses in sequence. The method can be executed by a video generation device, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, a PC or a server, etc.

[0033] Based on the description of metaverse related technologies in the background technology section, it can be seen that the metaverse is a virtual environment outside the real physical environment. In the virtual environment, there are virtual people (also called "virtual digital people"), which have functions such as content display, information interaction, and communication. They can be competent for positions such as virtual news anchors, virtual teachers, and virtual anchors. The 3D action generation technology and 3D face generation technology in the 3D content generation technology are mainly used to generate virtual people.

[0034] In different scenarios, users can receive information from virtual people by watching videos, interact with virtual people by inputting text, and communicate with virtual people through voice and video conversations.

[0035] A video generation method provided in an embodiment of the present invention is mainly used to generate a video containing a virtual person to display content to a user.

[0036] Through the scheme of this embodiment, the user inputs at least one paragraph of text, and after generating audio according to the text, a video of a virtual person explaining the text can be quickly generated, for example: it can be a news broadcast video, a product introduction video, and a knowledge popularization video. In addition, in order to enhance the display effect of the video, this embodiment also allows the user to input some video materials, such as images or videos, while inputting the aforementioned text. The images or videos can be used as the display content when the virtual person explains in the generated video. Further, in order to facilitate user operation, the aforementioned video materials can be input in the form of static pictures, dynamic pictures (such as gif animations), videos, PPT documents, and PDF documents. It should be understood that each PPT document contains at least one slide, and one slide is equivalent to a picture, and text, artistic words, pictures, videos and other contents can be inserted into the slide. Similarly, each PDF document contains at least one PDF page, and one PDF page is equivalent to a picture, and the PDF page contains elements such as text and pictures. For example, in an application scenario, a user inputs a PPT document and corresponding dialogue text and shot information. The PPT document contains text, images and videos. This embodiment can generate a video of a virtual person explaining the PPT document in a sequence of shot switching based on the input content. The video can show the image of the virtual person and video materials to the user at different perspectives and distances.

[0037] It should be noted that each slide in the PPT document has corresponding notes, so the dialogue text and shot information corresponding to the PPT document can be input in the form of independent text or as notes of the PPT document, and this embodiment does not limit this.

[0038] like Figure 1 As shown, the method specifically comprises the following steps:

[0039] S110, obtaining input content.

[0040] The input content includes dialogue text and shot information.

[0041] The dialogue text may be directly input by the user through the client, or obtained from the local storage of the client, or downloaded from the server, and the acquisition method of the dialogue text is not limited here. The dialogue text may contain only one text segment or multiple text segments, and there is no limitation here.

[0042] Based on the above description, it can be known that the video generated in this embodiment is a video of a virtual person explaining a dialogue text, and the dialogue text is a manuscript used for the virtual person to explain the content, which can be expressed in a certain document format, such as: TXT document, WORD document or PDF document, etc. The dialogue text is used to generate the dubbing of the video, that is, the audio of the virtual person. The dubbing will run through the entire video, and the duration of the dubbing is the same as the duration of the video to be generated, and the time axis of the two is also the same.

[0043] Optionally, the input content may also include an image, where the image may specifically be a picture or a video, and the picture or video is reflected in the generated video in the form of display content to assist the virtual person in explaining the content.

[0044] It is understandable that in some formal occasions, such as product launches, education and training classes, only verbal explanations are boring, and pictures or videos are needed to deepen the audience's understanding of the explanation content and leave a deep impression. Therefore, this embodiment also allows users to input pictures or videos as explanation materials, combined with the language and actions of the virtual person, to form a high-quality video.

[0045] In addition, this embodiment can also convert the dialogue text into the dubbing corresponding to the voice template selected by the user (such as baritone, bass, soprano, etc.). The voice template can be a pre-designed voice for the user to select or generated by the voice recorded by the user, and the voice template selected by the user matches the image of the virtual person. That is, a suitable timbre is selected for the dubbing, so that the image of the virtual person is consistent in vision and hearing, and the entire character image is richer and fuller.

[0046] Based on the above description, it can be known that this embodiment can generate the dubbing of the video based on the dialogue text, and then generate the corresponding video based on the dubbing. Therefore, one possible implementation method is to directly use the dubbing instead of the dialogue text as the input content, and use the time axis of the dubbing as the time axis of the video to be generated.

[0047] In addition to the dialogue text, the input content in this embodiment also includes shot information. The shot information includes at least one shot category, and the shot category may include at least one of a content display shot and a character display shot.

[0048] Based on the above description, it can be known that the video generated in this embodiment is a video of a virtual person explaining a line text, and the main elements contained in the video are the image of the virtual person, the virtual environment in which the virtual person is located, the material explained by the virtual person, etc. Correspondingly, the lens category in this embodiment is mainly classified according to the object focused by the lens. Among them, the content display lens refers to a lens whose focus object is the video display content (i.e., the material explained by the virtual person). Under this type of lens, the video display content is focused and displayed, that is, in the picture corresponding to this type of lens, the virtual person is the accompanying body, and may not even appear in the picture. Panoramic lens and full-screen lens are typical content display lenses. Character display lens refers to a lens whose focus object is a virtual person. Under this type of lens, the virtual person is focused and displayed, that is, in the picture corresponding to this type of lens, the virtual person is the main body. Medium shot, close-up shot, and close-up shot are typical character display shots.

[0049] In addition, in order to make the generated video more professional, the shot categories in this embodiment may also include start shots and end shots. Start shots and end shots refer to shots without a clear focus object, that is, the pictures corresponding to such shots continuously move from top to bottom, from left to right, from far to near, from near to far, etc., to display the virtual environment where the virtual person is located in an all-round and three-dimensional manner, so that the audience can follow the changes in the shots and naturally approach / leave the current virtual environment.

[0050] It is understood that the start shot is usually set at the beginning of the entire video to show the overall environment of the video, establish the tone, or highlight the key elements of the picture. Exemplarily, the start shot can be a panoramic shot, a close-up shot, or a large scene shot. The end shot is usually set at the end of the entire video to summarize the storyline, to leave a deep impression on the audience, or to strengthen the theme or message of the video. Exemplarily, the end shot can be a slow-pull shot.

[0051] The shot information in this embodiment is used to mark the text content in the dialogue text, so that the text segment of the dialogue text is further divided into multiple shot segments according to the mark of the shot information, and each shot segment corresponds to a shot category. It should be understood that if the dialogue text contains only one text segment, the mark of the shot information is used to divide the entire dialogue text into multiple shot segments, and if the dialogue text contains multiple text segments, the mark of the shot information is used to divide each text segment into multiple shot segments. In other words, the dialogue text, the text segment, and the shot segment constitute a three-layer structure for dividing the dialogue text.

[0052] Specifically, when the user inputs the dialogue text, he can use different shot information to mark the text content in the dialogue text. The marked text content is segmented with the adjacent text content before and after to form different shot segments.

[0053] It should be noted that, for a video, each frame must correspond to a virtual camera lens (referred to as a "lens" for short). The virtual camera lens renders the scene by simulating the shooting effect of a real camera lens to generate a picture. Therefore, the lens can be used to define the viewing angle, distance and duration of the content displayed on the picture. Different lenses can be defined by different technical parameters. The parameters of each lens include at least one of focal length, camera posture and duration. Among them, the focal length can be understood as the distance from the optical center of the lens to the imaging plane. The larger the focal length (such as a telephoto lens), the smaller the viewing angle, and the narrower the scene range in the picture. The smaller the focal length (such as a wide-angle lens), the larger the viewing angle, and the wider the scene range in the picture; the camera posture can include the camera position and the camera posture, which can characterize the viewing angle direction of the lens; the duration can be understood as the duration of the lens. It should be understood that during the duration of the lens, the parameters such as the focal length and camera posture of the lens can change over time, so that the picture corresponding to the lens also changes accordingly.

[0054] Since each frame in the video needs to correspond to a shot, if the user only marks a part of the text content in the dialogue text, this embodiment will automatically mark the shot information for the unmarked text content.

[0055] In addition, considering the particularity of the beginning and end of the video, this embodiment allows for mandatory setting of the first 5 seconds and the last 5 seconds of the video to prevent the user from forgetting to mark the start shot and the end shot, or mistakenly setting the beginning and end of the video to other types of shots due to operational errors.

[0056] For example, Figure 2 is an example diagram of the input content obtained in this embodiment, such as Figure 2As shown, the dialogue text includes three text segments, namely, text segment 1, text segment 2, and text segment 3. The user only uses the content display lens to mark part of the text content in the dialogue text. This embodiment automatically marks the start lens at the beginning of text segment 1, marks the end lens at the end of text segment 3, and marks the character display lens for the remaining text content, thereby further dividing the three text segments into 2 to 3 lens segments, that is, the entire dialogue text is divided into 8 lens segments, and each lens segment carries lens information.

[0057] S120, generating multiple candidate shot sequences based on the dialogue text and shot information.

[0058] Based on the above description of generating dubbing from dialogue text, it can be known that the timeline of the dubbing is the same as the timeline of the video to be generated, that is, based on the dialogue text, the dubbing of the target video is generated, and the timeline of the dubbing is used as the timeline of the target video. The target video here is the video to be generated.

[0059] The dialogue text is divided into multiple lens segments. After the dubbing is generated based on the dialogue text, the entire dialogue text corresponds to the entire timeline, and each lens segment corresponds to a time period on the dubbing timeline, that is, the lens segment and the time period on the dubbing timeline correspond one to one. Each lens segment carries lens information, which is used to identify different lens categories. Therefore, the time period on the timeline can be associated with the lens category, that is, the time period on the timeline is also associated with the lens category carried by the corresponding lens segment.

[0060] In short, this embodiment first obtains the shot segments corresponding to the shot categories in the dialogue text, and then obtains the time periods corresponding to each shot segment on the timeline to associate each time period with the shot category, thereby realizing the division of the timeline into different types of shot periods based on the shot category.

[0061] Based on the above description, it can be known that the dialogue text in this embodiment may include only one text segment or multiple text segments. When the dialogue text includes multiple text segments, different text segments are also pre-divided, and the entire dialogue text corresponds to the entire timeline. Accordingly, the timeline is composed of multiple time slices, and each text segment corresponds to a time slice. Each text segment is divided into multiple lens segments, and each time slice is divided into different types of lens time periods.

[0062] In order to avoid that the types of shot periods contained in a single time slice are too single, such as only shot periods containing character display shots, this embodiment will detect the types of shot periods contained in each time slice after dividing the time axis into different types of shot periods. If the time slice does not contain a shot period of a content display shot, the starting time period of the time slice (for example, the first 5 seconds) will be replaced with a shot period of a content display shot, thereby diversifying the types of shot periods contained in the time slice, ensuring that each time slice includes at least one shot period of a content display shot, and avoiding the generated target video from not displaying content for a long time.

[0063] In order to ensure the professionalism of the shots in the generated video, this embodiment pre-designs a plurality of different shot combinations for various shot categories. These shot combinations are pre-designed by professional directors based on shooting theory and years of shooting experience, and are stored as materials in databases corresponding to various shot categories, thereby forming shot libraries corresponding to various shot categories. Each shot library contains a plurality of shot combinations, and each shot combination consists of one or more shots. For example, a shot combination in the character display shot library is: a medium shot of the front view - a close-up shot of the front view - a close-up shot of the side view.

[0064] After the timeline is divided into different types of shot periods, each period in the timeline is associated with a shot category, and each shot category corresponds to a shot library, thereby associating each period in the timeline with the shot library.

[0065] It can be understood that to determine the shot sequence corresponding to the entire video, you can first select the corresponding shot combination for each shot period of the timeline, and then connect all the shot combinations in sequence to obtain a shot sequence corresponding to the entire video.

[0066] In order to ensure the quality of the shot sequence, when selecting the corresponding shot combination for the shot period, this embodiment randomly selects the shot combination from the corresponding shot library according to the association between the shot period and the shot library, and concatenates them in sequence to obtain a candidate shot sequence. Repeating the random selection multiple times and concatenating them in sequence can obtain multiple candidate shot sequences. For example, Figure 3 For example, there are 8 shot periods on the timeline. A shot combination is randomly selected from the shot library corresponding to each shot period and connected in sequence to obtain a candidate shot sequence containing 8 shot combinations.

[0067] That is, at least one shot combination is selected from the shot library corresponding to different shot categories, and is bound to the corresponding shot time period to generate multiple candidate shot sequences. These candidate shot sequences are screened and scored, and the target shot sequence is determined as the shot sequence corresponding to the target video.

[0068] It can be understood that when generating multiple candidate shot sequences, the number of candidate shot sequences to be generated can be preset, and then the target shot sequence can be determined from the preset number of candidate shot sequences. On the one hand, the diversity of the candidate shot sequences can be increased by randomly selecting shot combinations, and on the other hand, the number of candidate shot sequences to be generated can be limited to improve the generation efficiency of the target shot sequence.

[0069] It should be noted that when selecting at least one lens combination from the lens library corresponding to different lens categories, it is also necessary to consider whether the duration of the lens period matches the duration of the lens combination. It can be understood that the parameters of each lens include the duration, and the duration of the lens combination is the sum of the durations of all the lenses it contains.

[0070] Correspondingly, a method of selecting at least one lens combination from the lens library corresponding to different lens categories may be: based on the duration of the lens period, a matching lens combination is screened out from the lens library corresponding to the lens category.

[0071] Specifically, for each shot period, firstly, all shot combinations in the shot library corresponding to the corresponding shot category are screened based on the duration of the shot period, and shot combinations with exactly the same duration or a duration less than 20% of the shot period are taken as matching shot combinations, and then a shot combination is randomly selected from the matching shot combinations as the shot combination corresponding to the shot period.

[0072] Optionally, if the duration of the shot combination is equal to the duration of the shot period, the shot combination is directly determined as a shot combination that matches the shot period; if the duration of the shot combination is greater than the duration of the shot period, the duration of the shot combination is proportionally reduced, and the proportionally reduced shot combination is determined as a shot combination that matches the shot period.

[0073] Specifically, the method of proportionally reducing the shot combination can be: dividing the duration of the shot combination by the duration of the shot period to obtain the ratio of the shot combination to the shot period, and reducing the duration of all shots included in the shot combination according to the ratio, that is, dividing the duration of each shot by the ratio as the reduced duration. Exemplarily, the duration of the shot period is T1, if the duration of the shot combination b is T2, T2>T1, and the shot combination b includes shot 1 (duration t1), shot 2 (t2) and shot 3 (t3), then calculate the value of T2 / T1, record it as T, and then divide the duration t1, t2 and t3 of shot 1, shot 2 and shot 3 by T as the reduced duration, then the proportionally reduced shot combination b includes shot 1 (t1 / T), shot 2 (t2 / T) and shot 3 (t3 / T).

[0074] Furthermore, the present embodiment can pre-set lens switching rules and / or lens concatenation rules to improve the quality of the candidate lens sequence, for example, a medium shot can be followed by a close-up shot, but a close-up shot cannot be followed by a close-up shot. Since the selection of the lens combination corresponding to each of the aforementioned lens time periods is random, after these lens combinations are concatenated in sequence, it may be that the last lens of the previous lens combination and the first lens of the next lens combination cannot be switched / concatenated. In order to solve this problem, when selecting the corresponding lens combination for each lens time period, the present embodiment can adopt a method of selecting from front to back and in sequence, that is, first select a lens combination for the first lens time period, and then screen the lens combination in the lens library corresponding to the second lens time period according to the last lens in the first lens combination, eliminate the lens combination that does not meet the preset lens switching rules and / or lens concatenation rules, and select from the remaining lens combinations, thereby ensuring the generation quality of the candidate lens sequence.

[0075] Taking into account that the video generated by this embodiment is a video of a virtual person explaining a line text, the virtual person in the video will walk and move during the explanation. Accordingly, the character display shot specifically includes a character following shot. This type of shot is a special character display shot. In addition to focusing on the virtual person, it can also follow the virtual person when he walks or moves, to prevent the virtual person from leaving the screen due to walking or moving.

[0076] In order to bind the character following shot and the virtual person walking or moving behavior on the timeline, this embodiment generates the dubbing of the target video based on the dialogue text, and then generates an initial video based on the dubbing. The initial video contains the scene of the virtual person acting or moving, and the initial video is a video that displays the content with a single shot.

[0077] Specifically, dubbing can be used as input to the virtual human video generation software, and a virtual human image can be selected in the virtual human video generation software. The virtual human video generation software will automatically generate an initial video of the virtual human explaining the content, and the virtual human may explain the content while walking and stopping.

[0078] The initial video is detected, and if it is detected that the virtual person walks or moves in the video, that is, the initial video contains a video clip of the person moving, then the corresponding period is determined in the time axis of the initial video as the moving period. Since the time axis of the initial video and the target video are the same, the corresponding moving period is also determined in the time axis of the target video.

[0079] In this embodiment, if there is a moving period in the timeline, a character following shot in the character display shot or a panoramic shot in the content display shot is bound to the moving period, and a shot combination is bound to other periods in the timeline except the moving period according to the aforementioned steps, so as to determine the shot sequence corresponding to the entire target video to be generated. It can be understood that whether it is a character following shot or a panoramic shot, it can ensure that the virtual person is always in the picture when walking or moving, so as to prevent the virtual person from being out of the picture.

[0080] The above method for determining the moving period requires generating an initial video based on dubbing. In addition, there is another method for determining the moving period. Specifically, the input content in this embodiment also includes a video script, and the video script includes at least one of the following: a moving mark. The moving mark can mark the text content in the dialogue text, and is used to mark that the virtual person in the video clip corresponding to the shot clip needs to walk or move.

[0081] Based on the above description, it can be known that the shot segments of the dialogue text correspond to the shot periods on the timeline. Therefore, the corresponding moving period can be determined on the timeline based on the shot segments corresponding to the moving marks. After the moving period is determined on the timeline, the moving period is bound to the character following shot in the character display shot, so that in the generated target video, when the virtual person walks or moves, the shot always follows the virtual person, ensuring that the virtual person is always in the preset position of the picture, preventing the virtual person from leaving the picture.

[0082] S130, determining a target shot sequence from multiple candidate shot sequences according to the adjacent relationship between shots in each candidate shot sequence.

[0083] It should be understood that the shot sequence in the target video finally generated is unique, so after generating multiple candidate shot sequences, it is necessary to determine the best shot sequence as the target shot sequence for generating the target video.

[0084] The quality of a shot sequence mainly depends on the switching and concatenation between different shots. Therefore, this embodiment determines the target shot sequence based on the adjacent relationship between shots in each candidate shot sequence.

[0085] One possible implementation method is to evaluate each candidate shot sequence based on a preset shot switching rule and / or a preset shot concatenation rule to obtain a score for each candidate shot sequence; and determine the candidate shot sequence with the highest score as the target shot sequence.

[0086] The method for evaluating at least one candidate shot sequence based on the preset shot switching rule may be: firstly, scoring the switching methods of the two adjacent shot combinations in the candidate shot sequence to obtain a plurality of shot switching sub-scores, and then weighting and summing the plurality of shot switching sub-scores to obtain the shot switching score of the candidate shot sequence. Specifically, the method for scoring the switching methods of the two adjacent shot combinations may be: for adjacent shot combinations, obtaining the last shot of the previous shot combination and the first shot of the next shot combination, and scoring the switching methods of the last shot and the first shot according to the preset shot switching rule to obtain the shot switching sub-score of the adjacent shot combination.

[0087] The method of evaluating at least one candidate shot sequence based on the preset shot concatenation rule may be: for each candidate shot sequence, the candidate shot sequence is evaluated according to the preset concatenation rule to obtain a shot concatenation score.

[0088] In this embodiment, after obtaining the shot switching score and the shot concatenation score, the shot switching score and the shot concatenation score are accumulated to obtain the score of the candidate shot sequence. Finally, the candidate shot sequence with the highest score is determined as the target shot sequence. In this embodiment, the target shot sequence is determined from at least one candidate shot sequence based on the preset shot switching rule and / or the preset shot concatenation rule, so that the shot switching of the generated video is more in line with the visual effect, thereby improving the display effect of the generated video.

[0089] S140, generating a target video based on the target shot sequence.

[0090] Specifically, after obtaining the target shot sequence, a target video is generated based on the target shot sequence, so that the target video performs shot switching according to the target shot sequence during playback.

[0091] Alternatively, the initial video generated in the above embodiment is adjusted in terms of shots based on the target shot sequence, thereby obtaining the target video.

[0092] For example, Figure 4a is an example diagram of a video frame displayed by a content display lens in this embodiment, such as Figure 4aAs shown, it is a video frame in the video of a virtual person reporting the weather forecast. In this video frame, the content is the main content in the video frame. Figure 4b is an example of a video frame displayed by a character display lens in this embodiment. Figure 4b As shown, it is another video frame in the video of the virtual person reporting the weather forecast. In this video frame, the virtual person is the main content.

[0093] The technical solution of this embodiment is to obtain input content; wherein the input content includes dialogue text and lens information; based on the dialogue text and lens information, generate multiple candidate lens sequences; according to the adjacent relationship between the lenses in each candidate lens sequence, determine the target lens sequence from the multiple candidate lens sequences; based on the target lens sequence, generate a target video. The video generation method provided by the embodiment of the present invention automatically generates multiple candidate lens sequences based on dialogue text and lens information, thereby realizing the diversification of candidate lens sequences, and then determines the target lens sequence from the multiple candidate lens sequences, and generates a target video based on the target lens sequence. When playing the target video, as the various lenses in the target lens sequence are switched in sequence, the target video displays different video content at different viewing angles and distances, so as to improve the diversity of video content display.

[0094] Based on the above embodiments, Figure 5 1 is a flow chart of a video generation method provided by the present invention. As shown in the figure, the method comprises the following steps:

[0095] S501, obtaining input content.

[0096] S502, generating dubbing of the target video based on the dialogue text; and using the timeline of the dubbing as the timeline of the target video.

[0097] S503: Divide the time axis into different types of shot periods based on the shot categories.

[0098] S504, based on the duration of the shot period, select a matching shot combination from a shot library corresponding to the shot category, and bind it to the shot period of the corresponding type.

[0099] S505, detecting the shot period type contained in each time slice; in response to the time slice not containing the shot period of the content display shot, replacing the start period of the time slice with the shot period of the content display shot.

[0100] S506, generating an initial video based on dubbing; in response to a video clip containing character movement in the initial video, determining a corresponding movement period in the timeline; and binding the movement period with a character follow-up shot in the character display shot.

[0101] S507: Generate multiple candidate shot sequences.

[0102] S508, evaluating each candidate shot sequence based on a preset shot switching rule and / or a preset shot concatenation rule to obtain a score for each candidate shot sequence; and determining the candidate shot sequence with the highest score as the target shot sequence.

[0103] S509, generating a target video based on the target shot sequence.

[0104] Figure 6 is a schematic diagram of the structure of a video generating device provided by an embodiment of the present invention, such as Figure 6 As shown, the device comprises:

[0105] The input content acquisition module 610 is used to acquire input content; wherein the input content includes dialogue text and shot information;

[0106] A candidate shot sequence generation module 620 is used to generate multiple candidate shot sequences based on the dialogue text and shot information;

[0107] A target shot sequence determining module 630 determines a target shot sequence from a plurality of candidate shot sequences according to the adjacent relationship between shots in each candidate shot sequence;

[0108] The target video generation module 640 is used to generate a target video based on a target shot sequence.

[0109] Optionally, the shot information includes at least one shot category, and the candidate shot sequence generation module 620 is further configured to:

[0110] Generate the dubbing of the target video based on the dialogue text;

[0111] Use the timeline of the dubbing as the timeline of the target video;

[0112] Based on the shot categories, the timeline is divided into different types of shot periods; wherein the shot categories include at least one of content display shots and character display shots;

[0113] At least one shot combination is selected from a shot library corresponding to different shot categories, and is bound to a shot period of a corresponding type to generate a plurality of candidate shot sequences.

[0114] Optionally, the dialogue text includes multiple segments, and accordingly, the timeline is composed of multiple time slices. The candidate shot sequence generation module 620 is further used to:

[0115] Detect the shot period type contained in each time slice;

[0116] In response to the time slice not including the shot period of the content display shot, the start period of the time slice is replaced with the shot period of the content display shot.

[0117] Optionally, each shot library contains multiple shot combinations, each shot combination is composed of one or more shots, and the candidate shot sequence generation module 620 is further used to:

[0118] Based on the duration of the shot period, a matching shot combination is selected from the shot library corresponding to the shot category.

[0119] Optionally, it also includes: an initial video generation module, used to:

[0120] Generate an initial video based on the dubbing;

[0121] Optionally, the candidate shot sequence generating module 620 is further configured to:

[0122] In response to a video segment in the initial video containing a person moving, determining a corresponding movement period in the timeline;

[0123] Bind the moving period to the character follow shot in the character display shot.

[0124] Optionally, the target shot sequence determination module 630 is further configured to:

[0125] Evaluate each candidate shot sequence based on a preset shot switching rule and / or a preset shot concatenation rule to obtain a score for each candidate shot sequence;

[0126] The candidate shot sequence with the highest score is determined as the target shot sequence.

[0127] Optionally, the lens parameters include at least one of focal length, camera position, and duration.

[0128] The above device can execute the methods provided by all the above embodiments of the present invention, and has the corresponding functional modules and beneficial effects of executing the above methods. For technical details not described in detail in this embodiment, please refer to the methods provided by all the above embodiments of the present invention.

[0129] Figure 7A schematic diagram of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, the connections and relationships between the components, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0130] like Figure 7 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0131] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0132] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a video generation method.

[0133] In some embodiments, the video generation method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the video generation method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the video generation method in any other appropriate manner (e.g., by means of firmware).

[0134] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0136] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0137] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0138] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0139] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0140] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the video generation method provided in any embodiment of the present application.

[0141] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0142] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0143] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A video generation method, characterized in that: include: Obtaining input content; wherein the input content includes dialogue text and shot information; Based on the dialogue text and the shot information, generating multiple candidate shot sequences; Determining a target shot sequence from a plurality of candidate shot sequences according to the adjacent relationship between shots in each of the candidate shot sequences; Based on the target shot sequence, a target video is generated.

2. The method according to claim 1, characterized in that The shot information includes at least one shot category, and the generating of multiple candidate shot sequences based on the dialogue text and the shot information includes: Based on the dialogue text, generating the dubbing of the target video; Using the timeline of the dubbing as the timeline of the target video; Based on the shot categories, the timeline is divided into different types of shot periods; wherein the shot categories include at least one of content display shots and character display shots; At least one shot combination is selected from a shot library corresponding to different shot categories, and is bound to a shot period of a corresponding type to generate a plurality of candidate shot sequences.

3. The method according to claim 2, characterized in that The dialogue text includes a plurality of segments, and accordingly, the timeline is composed of a plurality of time slices. After the timeline is divided into different types of shot periods based on the shot categories, the following is further included: Detecting the shot period type contained in each of the time slices; In response to the time slice not including the shot period of the content display shot, the start time period of the time slice is replaced with the shot period of the content display shot.

4. The method according to claim 2, characterized in that: Each of the lens libraries contains a plurality of lens combinations, each of the lens combinations consists of one or more lenses, and selecting at least one lens combination from lens libraries corresponding to different lens categories includes: Based on the duration of the shot period, a matching shot combination is screened out from a shot library corresponding to the shot category.

5. The method according to claim 4, characterized in that After generating the dubbing of the target video based on the dialogue text, the method further includes: Based on the dubbing, generating an initial video; Accordingly, the selecting at least one lens combination from the lens library corresponding to different lens categories further includes: In response to a video segment in the initial video containing a moving person, determining a corresponding moving period in the timeline; The moving period is bound to the character following shot in the character display shot.

6. The method according to any one of claims 1 to 5, characterized in that The step of determining a target shot sequence from a plurality of candidate shot sequences according to the adjacent relationship between shots in each of the candidate shot sequences comprises: Evaluate each of the candidate shot sequences based on a preset shot switching rule and / or a preset shot concatenation rule to obtain a score for each of the candidate shot sequences; The candidate shot sequence with the highest score is determined as the target shot sequence.

7. The method according to claim 6, characterized in that The parameters of the lens include at least one of focal length, camera position, and duration.

8. A video generating device, characterized in that: include: An input content acquisition module, used to acquire input content; wherein the input content includes dialogue text and shot information; A candidate shot sequence generation module, used to generate multiple candidate shot sequences based on the dialogue text and the shot information; a target shot sequence determination module, which determines a target shot sequence from a plurality of candidate shot sequences according to the adjacent relationship between shots in each of the candidate shot sequences; The target video generation module is used to generate a target video based on the target shot sequence.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the video generating method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video generation method according to any one of claims 1 to 7 when executed.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the video generation method according to any one of claims 1 to 7.

Images