Video generation method and apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to a video generation method and device, an electronic device and a storage medium. The method comprises: obtaining a target single-shot video in a preset video library as a first single-shot video of a multi-shot video to be generated, and the preset shot library comprising a plurality of single-shot videos; determining at least one group of video sequences starting from the first single-shot video according to visual correlation features of the first single-shot video and visual correlation features of a first group of candidate videos, the visual correlation features representing contextual correlation of the single-shot videos, and the first group of candidate videos comprising other single-shot videos in the preset video library except the first single-shot video; and splicing the at least one group of video sequences to obtain at least one multi-shot video. The embodiments of the present disclosure can improve the video production efficiency.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a video generation method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the popularity of video shooting devices such as mobile phones, cameras, etc., users can shoot a lot of single-shot videos by using the video shooting devices. If a user wants to make a multi-shot video by using multiple single-shot videos, the user usually needs to manually edit the multiple single-shot videos by using a video editing tool, and the video making efficiency is low. SUMMARY

[0003] The present disclosure provides a video generation technical solution.

[0004] According to an aspect of the present disclosure, a video generation method is provided, which includes: obtaining a target single-shot video in a preset video library as a first single-shot video of a to-be-generated multi-shot video, the preset video library including multiple single-shot videos; determining at least one group of video sequences starting from the first single-shot video according to visual correlation features of the first single-shot video and visual correlation features of a first group of candidate videos, the visual correlation features representing contextual correlation of single-shot videos, and the first group of candidate videos including other single-shot videos in the preset video library except the first single-shot video; and splicing the at least one group of video sequences to obtain at least one multi-shot video. In this way, at least one reasonable and smooth multi-shot video can be automatically generated, which is beneficial to shorten the time length of making a multi-shot video and improve the video making efficiency.

[0005] In a possible implementation, the determining, according to the visual correlation feature of the first single-shot video and the visual correlation features of the first group of candidate videos, at least one group of video sequences starting from the first single-shot video, comprises: determining, according to the visual correlation feature of the first single-shot video and the visual correlation features of the first group of candidate videos, at least one second single-shot video arranged in time sequence after the first single-shot video from the first group of candidate videos; determining, according to the visual correlation feature of at least one nth single-shot video and the visual correlation features of an nth group of candidate videos, at least one (n+1)th single-shot video arranged in time sequence after the at least one nth single-shot video from the nth group of candidate videos, wherein the nth group of candidate videos comprises other single-shot videos in the preset video library except the first single-shot video to the at least one nth single-shot video, 2≤nN, and N is a preset total number of single-shot videos in the multi-shot video; and combining N single-shot videos arranged in time sequence from the first single-shot video to an Nth single-shot video as a video sequence. In this way, the visual correlation features of each single-shot video in the preset video library can be preset, and at least one group of video sequences starting from the first single-shot video can be effectively determined.

[0006] In a possible implementation, the determining, according to the visual correlation feature of the first single-shot video and the visual correlation features of the first group of candidate videos, at least one second single-shot video arranged in time sequence after the first single-shot video from the first group of candidate videos, comprises: determining a feature distance between the visual correlation feature of the first single-shot video and the visual correlation feature of each single-shot video in the first group of candidate videos; and determining, as the at least one second single-shot video arranged in time sequence after the first single-shot video, at least one single-shot video in the first group of candidate videos with a feature distance less than a preset threshold. In this way, the second single-shot video with higher context correlation with the first single-shot video can be effectively determined from the first group of candidate videos.

[0007] In a possible implementation, the determining, according to the visual correlation features of the at least one nth single-shot video and the visual correlation features of the nth group of candidate videos, of at least one (n+1)th single-shot video that is arranged in time sequence after the at least one nth single-shot video, includes: determining, for any nth single-shot video, a feature distance between the visual correlation features of the nth single-shot video and the visual correlation features of each single-shot video in the nth group of candidate videos; and determining, as the at least one (n+1)th single-shot video that is arranged in time sequence after the nth single-shot video, at least one single-shot video in the nth group of candidate videos that has a feature distance less than a preset threshold. In this way, the (n+1)th single-shot video that has a high contextual correlation with the nth single-shot video can be determined from the nth group of candidate videos in a "chain" manner based on the visual correlation features.

[0008] In a possible implementation, the visual correlation features are extracted by a feature extraction network, and the feature extraction network is obtained by training an initial feature extraction network based on sample videos. The sample videos include multiple-shot videos in a same video and multiple-shot videos obtained by splicing at least two different videos. Each sample video includes a mask video segment that has been replaced by a preset video segment and a mask video segment that has not been replaced. The preset video segment includes at least one of a video segment in a different video and a blank video segment. The mask video segment is used to indicate a specified single-shot video segment in the sample video. In this way, the feature extraction network trained based on the sample videos can extract visual correlation features that can represent the contextual correlation.

[0009] In a possible implementation, the training of the initial feature extraction network based on the sample videos includes: extracting, by the initial feature extraction network, sample visual correlation features of the sample videos; determining, based on the sample visual correlation features, a network loss of the initial feature extraction network. The network loss includes at least one of a global continuity loss and a local continuity loss. The global continuity loss represents a global continuity loss between multiple single-shot video segments in the sample videos. The local continuity loss represents a local continuity loss between adjacent single-shot video segments in the sample videos. The initial feature extraction network is trained based on the network loss to obtain a trained feature extraction network. In this way, the feature extraction network that can extract visual correlation features representing the contextual correlation can be effectively trained.

[0010] In a possible implementation, the determining the network loss of the initial feature extraction network according to the sample visual correlation feature comprises: performing classification on the sample video according to the sample visual correlation feature to obtain a video classification result of the sample video, the video classification result being used to indicate a probability that the sample video is a multi-shot video in a same video or a probability that the sample video is a multi-shot video spliced from at least two different videos; and determining the global continuity loss according to a difference between the video classification result and a video annotation result of the sample video. In this way, the initial feature extraction network can be trained to pay attention to the global continuity between the multiple single-shot video segments in the sample video based on the determined global continuity loss, so that the visual correlation feature extracted by the trained feature extraction network contains relevant information representing the global continuity.

[0011] In a possible implementation, the determining the network loss of the initial feature extraction network according to the sample visual correlation feature comprises: performing classification on the mask video segment in the sample video according to a sub-visual correlation feature corresponding to the mask video segment in the sample visual correlation feature to obtain a video segment classification result of the mask video segment in the sample video, the video segment classification result being used to indicate a probability that the mask video segment in the sample video is a video segment in a different video, or a probability that the mask video segment is a blank video segment, or a probability that the mask video segment is an unreplaced video segment; and determining the local continuity loss according to a difference between the video segment classification result and a video segment annotation result of the mask video segment in the sample video. In this way, the initial feature extraction network can be trained to pay attention to the local continuity between adjacent single-shot video segments in the sample video based on the determined local continuity loss, so that the visual correlation feature extracted by the trained feature extraction network contains relevant information representing the local continuity.

[0012] In a possible implementation, in a case where a plurality of groups of video sequences are determined, the splicing the at least one group of video sequences to obtain at least one multi-shot video comprises: splicing each group of video sequences in the plurality of groups of video sequences to obtain a plurality of multi-shot videos; and after the plurality of multi-shot videos are obtained, the method further comprises: in response to a selection operation on the plurality of multi-shot videos, determining a selected multi-shot video as a recommended target multi-shot video. In this way, a user can select a satisfactory target multi-shot video from the plurality of generated multi-shot videos.

[0013] According to an aspect of the present disclosure, a video generation apparatus is provided, comprising: an acquisition module configured to acquire a target single-shot video in a preset video library as a first single-shot video of a multi-shot video to be generated, the preset video library comprising a plurality of single-shot videos; a determination module configured to determine at least one video sequence starting from the first single-shot video according to visual correlation features of the first single-shot video and visual correlation features of a first group of candidate videos, the visual correlation features representing contextual correlation of single-shot videos, the first group of candidate videos comprising other single-shot videos in the preset video library except the first single-shot video; and a splicing module configured to splice the at least one video sequence to obtain at least one multi-shot video.

[0014] In a possible implementation, the determination module comprises: a first determination submodule configured to determine at least one second single-shot video arranged in time sequence after the first single-shot video from the first group of candidate videos according to the visual correlation features of the first single-shot video and the visual correlation features of the first group of candidate videos; a second determination submodule configured to determine at least one (n+1)th single-shot video arranged in time sequence after at least one nth single-shot video from an nth group of candidate videos according to the visual correlation features of the at least one nth single-shot video and the visual correlation features of the nth group of candidate videos, wherein the nth group of candidate videos comprises other single-shot videos in the preset video library except the first single-shot video to the at least one nth single-shot video, 2≤n

[0015] In a possible implementation, the determination of the at least one second single-shot video arranged in time sequence after the first single-shot video from the first group of candidate videos according to the visual correlation features of the first single-shot video and the visual correlation features of the first group of candidate videos comprises: determining feature distances between the visual correlation features of the first single-shot video and the visual correlation features of each single-shot video in the first group of candidate videos; and determining at least one single-shot video in the first group of candidate videos with a feature distance less than a preset threshold as the at least one second single-shot video arranged in time sequence after the first single-shot video.

[0016] In a possible implementation, the determining, according to the visual correlation features of the at least one nth single-shot video and the visual correlation features of the nth group of candidate videos, of at least one (n+1)th single-shot video that is arranged in time sequence after the at least one nth single-shot video, includes: determining, for any nth single-shot video, a feature distance between the visual correlation features of the nth single-shot video and the visual correlation features of each single-shot video in the nth group of candidate videos; and determining, as the at least one (n+1)th single-shot video that is arranged in time sequence after the nth single-shot video, at least one single-shot video in the nth group of candidate videos that has a feature distance less than a preset threshold.

[0017] In a possible implementation, the visual correlation features are extracted by a feature extraction network, and the feature extraction network is obtained by training an initial feature extraction network based on sample videos; the sample videos include multi-shot videos in the same video and multi-shot videos spliced from at least two different videos; and each sample video includes a mask video segment that has been replaced by a preset video segment and a mask video segment that has not been replaced, the preset video segment including at least one of a video segment in a different video and a blank video segment, and the mask video segment being used to indicate a specified single-shot video segment in the sample video.

[0018] In a possible implementation, the training of the initial feature extraction network based on the sample videos includes: extracting sample visual correlation features of the sample videos by the initial feature extraction network; determining a network loss of the initial feature extraction network according to the sample visual correlation features, the network loss including at least one of a global continuity loss and a local continuity loss, the global continuity loss representing a global continuity loss between multiple single-shot video segments in the sample video, and the local continuity loss representing a local continuity loss between adjacent single-shot video segments in the sample video; and training the initial feature extraction network based on the network loss to obtain a trained feature extraction network.

[0019] In a possible implementation, the determining of the network loss of the initial feature extraction network according to the sample visual correlation features includes: classifying the sample videos according to the sample visual correlation features to obtain video classification results of the sample videos, the video classification results being used to indicate a probability that the sample videos are multi-shot videos in the same video or a probability that the sample videos are multi-shot videos spliced from at least two different videos; and determining the global continuity loss according to a difference between the video classification results and video annotation results of the sample videos.

[0020] In a possible implementation, the determining the network loss of the initial feature extraction network according to the sample visual correlation feature comprises: performing classification on the mask video segment in the sample video according to a sub visual correlation feature corresponding to the mask video segment in the sample visual correlation feature, to obtain a video segment classification result of the mask video segment in the sample video, the video segment classification result being used to indicate a probability that the mask video segment in the sample video is a video segment in different videos, or a probability that the mask video segment is a blank video segment, or a probability that the mask video segment is a video segment that is not replaced; and determining the local continuity loss according to a difference between the video segment classification result and a video segment annotation result of the mask video segment in the sample video.

[0021] In a possible implementation, in a case where a plurality of groups of video sequences are determined, the splicing module comprises a splicing sub-module configured to splice each group of video sequences in the plurality of groups of video sequences to obtain a plurality of multi-shot videos; and after the plurality of multi-shot videos are obtained, the apparatus further comprises a selection module configured to determine a selected multi-shot video as a recommended target multi-shot video in response to a selection operation on the plurality of multi-shot videos.

[0022] According to an aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the above method.

[0023] According to an aspect of the present disclosure, a computer-readable storage medium is provided, having stored thereon computer program instructions, which, when executed by a processor, implement the above method.

[0024] In the embodiments of the present disclosure, by acquiring a target single-shot video in a preset video library comprising a plurality of single-shot videos as a first single-shot video segment of a multi-shot video to be generated, it is equivalent to specifying the first single-shot video segment of the multi-shot video to be generated from the plurality of single-shot videos, and then according to the visual correlation feature of the first single-shot video segment and the visual correlation features of other single-shot videos in the preset video library except the first single-shot video segment, and the visual correlation features characterizing the contextual correlation of the single-shot videos, at least one group of video sequences having strong contextual correlation starting from the first single-shot video segment can be determined, and then the at least one group of video sequences is spliced, so that at least one reasonable and smooth multi-shot video can be automatically generated, which is conducive to shortening the time length of producing the multi-shot video and improving the video production efficiency.

[0025] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, but not limiting the present disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0027] Figure 1 A flowchart illustrating a video generation method according to an embodiment of the present disclosure is shown.

[0028] Figure 2 A schematic diagram illustrating the generation of a video sequence according to an embodiment of the present disclosure is shown.

[0029] Figure 3 This diagram illustrates a type of sample video according to an embodiment of the present disclosure.

[0030] Figure 4 This diagram illustrates a type of sample video according to an embodiment of the present disclosure.

[0031] Figure 5 A flowchart illustrating a video generation method according to an embodiment of the present disclosure is shown.

[0032] Figure 6 A block diagram of a video generation apparatus according to an embodiment of the present disclosure is shown.

[0033] Figure 7 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. Detailed Implementation

[0034] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0035] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0036] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0037] In addition, in order to better illustrate the present disclosure, numerous specific details are set forth in the following detailed description. One skilled in the art will understand that the present disclosure can be practiced without certain specific details. In some instances, well-known methods, structures, elements, and circuits have not been described in detail in order to avoid obscuring the subject matter of the present disclosure.

[0038] It is known that the key to editing a high-quality multi-shot video is to select a plurality of relevant single-shot videos and combine them into a complete multi-shot video sequence for splicing, and to ensure the continuity of vision and semantics. In the related art, the multi-shot video is automatically generated by selecting single-shot videos according to some manual rules or following the order of a handwritten script, and little attention is paid to the contextual relevance between single-shot videos, which makes it easy to lose the visual fluency between shots, and without considering the contextual relevance information, the edited multi-shot video sequence may lack global continuity, and the generated multi-shot video sequence is inflexible and monotonous.

[0039] According to the video generation method of the embodiments of the present disclosure, the feature extraction can be performed on each single-shot video provided in the preset video library, the visual association features for representing the contextual relevance between any single-shot video and other single-shot videos are extracted, and the single-shot videos with similar contexts are connected according to the visual association features, and a multi-shot video with global continuity and visual fluency is generated.

[0040] Figure 1 A flowchart of a video generation method according to an embodiment of the present disclosure is shown, which can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor invoking computer-readable instructions stored in a memory, or the method can be executed by a server. As shown in Figure 1 The video generation method includes:

[0041] In step S11, a target single-shot video in a preset video library is acquired as a first single-shot video of a multi-shot video to be generated, and the preset shot library includes a plurality of single-shot videos.

[0042] The single-lens video can be understood as a video continuously captured by a video capturing device, that is, a video captured by one lens. It should be understood that one video capturing device can capture at least one single-lens video, and the plurality of single-lens videos in the preset video library can be a plurality of single-lens videos captured by one or more video capturing devices. The number of the plurality of single-lens videos in the preset video library and the content of the videos are not limited in the embodiments of the present disclosure.

[0043] The plurality of single-lens videos in the preset video library can include a plurality of single-lens videos uploaded by a user or a plurality of single-lens videos built-in by a related system, platform, application, etc. The embodiments of the present disclosure do not limit this. The user can specify a single-lens video or randomly select a single-lens video from the preset video library as the first single-lens video of the multi-lens video to be generated, that is, the target single-lens video is the single-lens video specified by the user from the preset video library or the single-lens video randomly selected from the preset video library.

[0044] In step S12, at least one video sequence starting from the first single-lens video is determined according to the visual association feature of the first single-lens video and the visual association feature of the first group of candidate videos.

[0045] The visual association feature characterizes the contextual association of the single-lens video, so that a single-lens video with high association or reasonable and smooth splicing with the single-lens video can be found by using the contextual association. The visual association feature can be extracted by a feature extraction network, that is, the visual association feature of each single-lens video in the preset video library can be extracted by the feature extraction network. In this way, after the first single-lens video is obtained, at least one video sequence starting from the first single-lens video can be found according to the visual association feature of the first single-lens video and the visual association feature of the first group of candidate videos.

[0046] It should be understood that the same single-lens video is usually not reused in the editing and production of the multi-lens video. The first group of candidate videos includes other single-lens videos in the preset video library except the first single-lens video, that is, the single-lens video after the first single-lens video in the video sequence is determined from other single-lens videos except the first single-lens video.

[0047] In a possible implementation, the at least one video sequence starting from the first single-shot video is determined according to the visual correlation feature of the first single-shot video and the visual correlation feature of the first group of candidate videos, for example, which can include: calculating the feature distance between the visual correlation feature of the first single-shot video and the visual correlation feature of other single-shot videos in the first group of candidate videos, determining the single-shot video in the first group of candidate videos whose feature distance is less than a preset threshold as the second single-shot video arranged after the first single-shot video; then calculating the feature distance between the visual correlation feature of the second single-shot video and the visual correlation feature of other single-shot videos in the second group of candidate videos, the other single-shot videos in the second group of candidate videos including other single-shot videos except the first single-shot video and the second single-shot video, and determining the other single-shot video in the second group of candidate videos whose feature distance is less than the preset threshold as the third single-shot video arranged after the second single-shot video, and so on, until the Nth single-shot video is obtained, and each group of video sequences can include N single-shot videos arranged in time sequence from the first single-shot video to the Nth single-shot video.

[0048] wherein N can be a preset total number of single-shot videos in the multi-shot video to be generated set by the user, that is, the user can set to splice N single-shot videos, or can set that the multi-shot video includes N single-shot videos in total, and the embodiments of the present disclosure do not limit the specific value of N.

[0049] It should be understood that, based on the above manner of determining the video sequence based on the feature distance less than the preset threshold, at least one second single-shot video can be determined after the first single-shot video, at least one third single-shot video can be determined after any second single-shot video, or no third single-shot video can be determined, that is, there can be at least one third single-shot video in the second group of candidate videos whose feature distance with a certain second single-shot video is less than the preset threshold, or there can be no third single-shot video in the second group of candidate videos whose feature distance with a certain second single-shot video is less than the preset threshold, therefore, at least one (n+1) th single-shot video can exist after at least one nth single-shot video, 2≤n

[0050] Figure 2 A schematic diagram of generating a video sequence according to an embodiment of the present disclosure is shown as follows: Figure 2 N single-shot videos can be set to 4, that is, N single-shot videos are spliced, Figure 2 one circle in the figure represents one single-shot video, and multiple circles represent multiple single-shot videos in a preset video library, v1 can represent the first single-shot video, a third single-shot video after v1, a third single-shot video after a third single-shot video after v1, a third single-shot video after two third single-shot videos after a fourth single-shot video after and a fourth single-shot video after v1, and other third single-shot videos and other fourth single-shot videos are sequentially similar, and will not be described again.

[0051] As can be seen from Figure 2 , there can be at least one (n+1) th single-shot video after an n th single-shot video, or there can be no (n+1) th single-shot video, so a video sequence containing N single-shot videos can be taken as the video sequence to be spliced in step S13; of course, a video sequence containing less than N single-shot videos can also be taken as the video sequence to be spliced, and the embodiments of the present disclosure do not limit this.

[0052] In step S13, at least one group of video sequences is spliced to obtain at least one multi-shot video.

[0053] As described above, each group of video sequences can include N single-shot videos arranged in time sequence from a first single-shot video to an N th single-shot video. Therefore, the N single-shot videos in each group of video sequences can be spliced according to the time sequence of the N single-shot videos in each group of video sequences to obtain at least one multi-shot video. It should be understood that those skilled in the art can use video splicing technology known in the art to splice the N single-shot videos in each group of video sequences to obtain at least one multi-shot video, and the embodiments of the present disclosure do not limit this.

[0054] In a possible implementation, after obtaining at least one multi-shot video, the obtained at least one multi-shot video can also be recommended to the user, so that the user can directly use the obtained multi-shot video for uploading and sharing on social media, or the obtained multi-shot video can also be optimized and edited, which can shorten the production time of the multi-shot video and improve the video production efficiency.

[0055] In a possible implementation, in the case that a plurality of groups of video sequences are determined through step S12, splicing at least one group of video sequences to obtain at least one multi-shot video includes: splicing each group of video sequences in the plurality of groups of video sequences to obtain a plurality of multi-shot videos; after the plurality of multi-shot videos are obtained, the method further includes: in response to a selection operation on the plurality of multi-shot videos, determining the selected multi-shot video as a recommended target multi-shot video. In this way, the user can select a satisfactory target multi-shot video from the generated plurality of multi-shot videos.

[0056] It should be understood that a group of video sequences can be spliced into one multi-shot video, a plurality of groups of video sequences can be spliced into a plurality of multi-shot videos respectively, the user can select at least one target multi-shot video from the plurality of multi-shot videos, and after the target multi-shot video is selected, the target multi-shot video can be directly used for uploading and sharing on a social media, or the target multi-shot video can be further optimized and edited, which can shorten the production time of the multi-shot video and improve the video production efficiency.

[0057] In the embodiments of the present disclosure, by obtaining a target single-shot video in a preset video library including a plurality of single-shot videos as a first single-shot video of a to-be-generated multi-shot video, it is equivalent to specifying the first single-shot video of the to-be-generated multi-shot video from the plurality of single-shot videos, and then according to the visual association features of the first single-shot video and the visual association features of other single-shot videos in the preset video library except the first single-shot video, and the visual association features representing the contextual association of the single-shot video, at least one group of video sequences having strong contextual association starting from the first single-shot video can be determined, and then splicing the at least one group of video sequences can automatically generate at least one reasonable and smooth multi-shot video, which is beneficial to shorten the time for producing the multi-shot video and improve the video production efficiency.

[0058] As described above, the video sequence including N single-shot videos starting from the first single-shot video can be determined based on the visual association features of each single-shot video in the preset video library. In a possible implementation, in step S12, at least one group of video sequences starting from the first single-shot video is determined according to the visual association features of the first single-shot video and the visual association features of the first group of candidate videos, including:

[0059] Step S121: determining at least one second single-shot video arranged in time sequence after the first single-shot video from the first group of candidate videos according to the visual association features of the first single-shot video and the visual association features of the first group of candidate videos.

[0060] As described above, the first group of candidate videos includes other single-shot videos in the preset video library except the first single-shot video, the feature distance between the first single-shot video and the other single-shot videos can be calculated, and the single-shot video in the first group of candidate videos with the feature distance less than the preset threshold is taken as the second single-shot video. In a possible implementation, the at least one second single-shot video arranged in time sequence after the first single-shot video is determined from the first group of candidate videos according to the visual association feature of the first single-shot video and the visual association features of the first group of candidate videos, including: determining the feature distance between the visual association feature of the first single-shot video and the visual association features of each single-shot video in the first group of candidate videos; and determining the at least one single-shot video in the first group of candidate videos with the feature distance less than the preset threshold as the at least one second single-shot video arranged in time sequence after the first single-shot video. In this way, the second single-shot video with high context association with the first single-shot video can be effectively determined from the first group of candidate videos.

[0061] The smaller the feature distance between the two visual association features is, the more similar the context association between the two single-shot videos corresponding to the two visual association features is, and the two single-shot videos can be connected together. The feature distance between the two visual association features can also represent the similarity between the two visual association features. The higher the similarity between the two visual association features is, the higher the visual content association between the two single-shot videos is, and it can be considered that one single-shot video can be explained by its semantic feature of context association, so that the context association can be combined to form a reasonable video sequence with semantic meaning. In addition, if the two single-shot videos have similar context association, the two single-shot videos are likely to provide similar semantics, that is, have similar visual content, so that the multi-shot video spliced has continuous visual content and conveys consistent video theme.

[0062] The feature distance can be, for example, Euler distance, Manhattan distance, and the like, that is, the distance between the visual association features can be calculated by using a distance calculation manner known in the art, and the embodiments of the present disclosure are not limited in this regard. As described above, the preset threshold can be set by the user, and the embodiments of the present disclosure are not limited in this regard.

[0063] Step S122: determining at least one (n+1)th single-shot video arranged in time sequence after the at least one nth single-shot video from the nth group of candidate videos according to the visual association feature of the at least one nth single-shot video and the visual association features of the nth group of candidate videos.

[0064] wherein 2≤n

[0065] In a possible implementation, the at least one (n+1)th single-shot video arranged in time sequence after the at least one nth single-shot video is determined from the nth group of candidate videos according to the visual correlation features of the at least one nth single-shot video and the visual correlation features of the nth group of candidate videos, including: for any nth single-shot video, determining the feature distance between the visual correlation features of the nth single-shot video and the visual correlation features of each single-shot video in the nth group of candidate videos; and determining at least one single-shot video in the nth group of candidate videos with a feature distance less than a preset threshold as the at least one (n+1)th single-shot video arranged in time sequence after the nth single-shot video. In this way, the (n+1)th single-shot video with high contextual correlation with the nth single-shot video can be determined from the nth group of candidate videos in a "chain" manner based on the visual correlation features.

[0066] As described above, the nth single-shot video can be at least one, and for any nth single-shot video, at least one (n+1)th single-shot video arranged in time sequence after the nth single-shot video can be determined. The feature distance between the visual correlation features can be calculated by using the distance calculation method known in the art, and the preset threshold can be set by the user, which is not limited in the embodiments of the present disclosure.

[0067] Step S123: combining the N single-shot videos arranged in time sequence from the first single-shot video to the Nth single-shot video as a video sequence.

[0068] It should be understood that at least one Nth single-shot video can be obtained through step S122, and therefore for any Nth single-shot video, the N single-shot videos arranged in time sequence from the first single-shot video to the Nth single-shot video can be arranged as a group of video sequences, and the at least one Nth single-shot video can be arranged into at least one group of video sequences, each group of video sequences including a video sequence arranged in time sequence from the first single-shot video to the Nth single-shot video.

[0069] In the embodiments of the present disclosure, the visual correlation features of each single-shot video in the preset video library can be preset, and at least one group of video sequences starting from the first single-shot video can be effectively determined.

[0070] As described above, the visual correlation feature is extracted by the feature extraction network, and in a possible implementation, the feature extraction network is obtained by training an initial feature extraction network based on sample videos; each sample video includes multiple shots in a same video and multiple shots obtained by splicing at least two different videos; and each sample video includes a mask video segment that has been replaced by a preset video segment and a mask video segment that has not been replaced, the preset video segment includes at least one of a video segment in a different video and a blank video segment, and the mask video segment is used to indicate a specified single-shot video segment in the sample video. The feature extraction network trained based on the sample videos in this way can extract visual correlation features that can represent context correlation.

[0071] The multiple shots in the same video can be a video segment cut from any real video, or can be any complete video, for example, a movie segment cut from a movie, or a complete short video, and the like, which are not limited by the embodiments of the present disclosure. The multiple shots obtained by splicing at least two different videos can include a video obtained by splicing video segments cut from at least two different videos, or can be a video obtained by directly splicing at least two different videos (for example, two short videos). Figure 3 A type diagram of a sample video according to an embodiment of the present disclosure is shown in FIG. 2, which shows that a sample video includes multiple shots in a same video and multiple shots obtained by splicing at least two different videos. Figure 3 As shown in FIG. 2, a video segment 1 and a video segment 2 constitute the multiple shots in the same video, and a video segment 1 and a video segment 3 constitute the multiple shots obtained by splicing at least two different videos.

[0072] The mask video segment in each sample video can be randomly specified or artificially specified, for example, 15% of the single-shot video segments in the sample video can be randomly specified as the mask video segment, and the specified mask video segment can be replaced by a video segment in a different video, or can be replaced by a blank video segment, or can not be replaced, that is, the original single-shot video segment in which the mask video segment is located is used. Figure 4 A type diagram of a sample video according to an embodiment of the present disclosure is shown in FIG. 3, which shows that a sample video includes multiple shots in a same video and multiple shots obtained by splicing at least two different videos. Figure 4 As shown in FIG. 3, a mask video segment can be framed by a thick line box, and the framed mask video segment can be replaced by a video segment in a different video, or can be replaced by a blank video segment, or can remain the original video segment (that is, not replaced).

[0073] It is known that a well-edited multi-shot video should generally adhere to two principles: one is that the time sequence position of each single-shot video in the entire multi-shot video is appropriate, that is, global continuity; the other is that the transition between two adjacent single-shot videos in the multi-shot video is seamless and smooth, that is, local continuity. Global continuity means that a certain single-shot video segment does not look out of place in the entire multi-shot video, while local continuity means that each single-shot video segment in the multi-shot video is coordinated and consistent in visual content and semantics with adjacent single-shot video segments.

[0074] Based on the above principles, the embodiments of the present disclosure use the sample video constructed as described above to train an initial feature extraction network, to obtain a feature extraction network for extracting the above-mentioned visual correlation features. Through the above-mentioned sample video, the initial feature extraction network can not only be trained to pay attention to the global continuity between multiple single-shot video segments in the sample video, but also be trained to pay attention to the local continuity between adjacent single-shot video segments in the sample video, so that the visual correlation features extracted by the trained feature extraction network can represent the contextual correlation of the single-shot video, which can represent the global continuity and the local continuity, so as to obtain a reasonable and smooth multi-shot video sequence by using the visual correlation features representing the contextual correlation.

[0075] In a possible implementation, the training of the initial feature extraction network through the sample video comprises:

[0076] extracting sample visual correlation features of the sample video through the initial feature extraction network; determining a network loss of the initial feature extraction network according to the sample visual correlation features, wherein the network loss comprises at least one of a global continuity loss and a local continuity loss, the global continuity loss represents a global continuity loss between multiple single-shot video segments in the sample video, and the local continuity loss represents a local continuity loss between adjacent single-shot video segments in the sample video; and training the initial feature extraction network according to the network loss to obtain a trained feature extraction network. In this way, the visual correlation features representing the contextual correlation can be effectively trained.

[0077] The extraction of the sample visual correlation features of the sample video through the initial feature extraction network can comprise: extracting sub-visual correlation features of each single-shot video segment in the sample video through the initial feature extraction network, that is, the sample visual correlation features of the sample video comprise multiple sub-visual correlation features of multiple single-shot video segments in the sample video, or the sample visual correlation features are composed of multiple sub-visual correlation features corresponding to multiple single-shot video segments.

[0078] According to the network loss, the initial feature extraction network is trained, which can be understood as adjusting the network parameters of the initial feature extraction network according to the network loss. It should be understood that network training is usually performed multiple times, and the initial feature extraction network can be trained multiple times according to the above training process by using multiple sample videos until a preset training end condition is met to obtain the trained feature extraction network; wherein the preset end condition may, for example, include loss convergence, training times reaching a preset number, training duration reaching a preset duration, and the like, and the embodiments of the present disclosure are not limited thereto.

[0079] As described above, the initial feature extraction network can be trained by sample videos to focus on the global continuity between multiple single-shot video segments in the sample videos. In one possible implementation, the network loss of the initial feature extraction network is determined according to the sample visual correlation feature, which includes:

[0080] According to the sample visual correlation feature, the sample video is classified to obtain a video classification result of the sample video, and the video classification result is used to indicate the probability that the sample video is a multi-shot video in the same video or the probability that the sample video is a multi-shot video spliced from at least two different videos; and the global continuity loss is determined according to the difference between the video classification result and the video annotation result of the sample video.

[0081] In this embodiment of the present disclosure, the network structure, network type, and the like of the binary classification network are not limited. It should be understood that the binary classification network can be trained synchronously with the initial feature extraction network, that is, the network parameters of the binary classification network can be adjusted synchronously according to the determined global continuity loss.

[0082] It can be understood that by classifying the sample video, it is predicted whether the sample video is a multi-shot video in the same video or a multi-shot video spliced from at least two different videos, that is, it is predicted whether the multi-shot video is continuous as a whole, and then based on the global continuity loss determined by the video classification result of the sample video, the initial feature extraction network is trained to focus on the global continuity between multiple single-shot video segments in the sample video, so that the visual correlation feature extracted by the trained feature extraction network contains relevant information representing the global continuity.

[0083] The video annotation result can represent a real category of the sample video, that is, can indicate whether the sample video is a multi-shot video in the same video or a multi-shot video spliced from at least two different videos. Considering that the video classification result is used to indicate a probability that the sample video is a multi-shot video in the same video or a probability that the sample video is a multi-shot video spliced from at least two different videos, in a possible implementation, for example, a label "1" can represent that the sample video is a multi-shot video in the same video, and a label "0" can represent that the sample video is a multi-shot video spliced from at least two different videos. The video classification result of any sample video can be a value between 0 and 1.

[0084] Based on this, for example, a cross-entropy loss function can be used to determine the global continuity loss according to the difference between the video classification result and the video annotation result of the sample video. Equation (1) shows a global continuity loss L1 determined based on a cross-entropy loss function according to an embodiment of the present disclosure.

[0085]

[0086] wherein b cls represents the video annotation result of the sample video, "1" represents that the sample video is a multi-shot video in the same video, and "0" represents that the sample video is a multi-shot video spliced from at least two different videos, f cls represents the sample visual correlation feature, H cls represents a binary classification network, H cls (f cls ) represents the video classification result of the sample video.

[0087] In the embodiment of the present disclosure, the global continuity loss determined can be used to train the initial feature extraction network to pay attention to the global continuity between the plurality of single-shot video segments in the sample video, so that the visual correlation feature extracted by the trained feature extraction network contains relevant information representing the global continuity.

[0088] As described above, through the above sample video, the initial feature extraction network can not only be trained to pay attention to the global continuity between the plurality of single-shot video segments in the sample video, but also be trained to pay attention to the local continuity between adjacent single-shot video segments in the sample video. In a possible implementation, the network loss of the initial feature extraction network is determined according to the sample visual correlation feature, including:

[0089] According to the sub-visual correlation feature corresponding to the mask video segment in the sample visual correlation feature, the mask video segment in the sample video is classified to obtain a video segment classification result of the mask video segment in the sample video, and the video segment classification result is used to indicate a probability that the mask video segment in the sample video is a video segment in different videos, or a probability that the mask video segment is a blank video segment, or a probability that the mask video segment is a video segment that is not replaced; and the local continuity loss is determined according to a difference between the video segment classification result and a video segment annotation result of the mask video segment in the sample video.

[0090] As described above, the mask video segment is used to indicate a part of the specified single-shot video segment in the sample video, and each sample video can include multiple mask video segments, or in other words, each sample video can include mask video segments that are replaced by video segments in different videos, mask video segments that are replaced by blank video segments, and mask video segments that are not replaced.

[0091] As described above, the sample video correlation feature of the sample video includes multiple sub-visual correlation features of multiple single-shot video segments in the sample video, that is, the initial feature extraction network extracts the sub-visual correlation features of each single-shot video segment in the sample video, and the sample video correlation feature is composed of the multiple sub-visual correlation features of the multiple single-shot video segments. It can be understood that the mask video segment is equivalent to the single-shot video segment in the sample video, so after the initial feature extraction network outputs the sample correlation feature, the sub-visual correlation feature corresponding to the mask video segment can also be obtained.

[0092] For example, a three-classification network known in the art can be used to classify the mask video segment in the sample video according to the sub-visual correlation feature corresponding to the mask video segment in the sample visual correlation feature to obtain the video segment classification result of the mask video segment in the sample video. The network structure, network type, and the like of the three-classification network are not limited in the embodiments of the present disclosure. It should be understood that the three-classification network can be trained synchronously with the initial feature extraction network, that is, the network parameters of the three-classification network can be adjusted synchronously according to the determined local continuity loss.

[0093] It can be understood that by classifying each mask video segment in the sample video to predict the probability that each mask video segment is a video segment in different videos, or the probability that each mask video segment is a blank video segment, or the probability that each mask video segment is a video segment that is not replaced, it is equivalent to predicting whether a mask video segment and an adjacent single-shot video are seamless and smooth. Based on the local continuity loss determined by the video segment classification result of each mask video segment, the initial feature extraction network focuses on the local continuity between adjacent single-shot video segments in the sample video, so that the visual correlation feature extracted by the trained feature extraction network contains relevant information representing local continuity.

[0094] The video segment label result can represent a real category of the mask video segment, that is, can indicate whether the mask video segment is a video segment in different videos, or a blank video segment, or a video segment that is not replaced. Considering that the video segment classification result is used to indicate the probability that each mask video segment in the sample video is a video segment in different videos, or the probability that it is a blank video segment, or the probability that it is a video segment that is not replaced, in a possible implementation, for example, the label "0" can represent that the mask video segment is replaced by a video segment in different videos, the label "1" can represent that the mask video segment is replaced by a blank video segment, and the label "2" can represent that the mask video segment is not replaced. The video segment classification result of any mask video segment can be a value between 0 and 2.

[0095] Based on this, for example, a cross-entropy loss function can be used to determine the local continuity loss according to the difference between the video segment classification result and the video segment label result of the mask video segment. Formula (2) shows a local continuity loss L2 determined based on a cross-entropy loss function according to an embodiment of the present disclosure.

[0096]

[0097] wherein n represents n mask video segments, b cls represents the video segment label result of the mask video segment, "1" represents that the mask video segment is replaced by a blank video segment, and "2" represents that the mask video segment is not replaced, f n represents the sub-visual correlation feature of the n mask video segments, H mask represents a three-classification network, H mask (f n ) represents the video segment classification result of the n mask video segments.

[0098] The above determination of the local continuity loss based on the video segment classification result of the mask video segment is based on the principle that the shot language and the natural language have similarities. The connection mode of the shots in the video sequence can be regarded as a kind of grammar. This kind of grammar allows the feature extraction network to simultaneously focus on the context between adjacent single-shot video segments and learn the single-shot video segments with context correlation. The key idea of determining the local continuity loss here is that it is not necessary to predict which single-shot video segment in the sample video is a mask video segment, but to predict the category of the mask video segment. It is relatively easy to predict the category of the mask video segment, and the above three replacement modes of the mask video segment (including replacement by a video segment in different videos, replacement by a blank video segment, and no replacement of the video segment) are sufficient to cover most types of mask video segments, and regardless of the specific video content, the trained feature extraction network will retain the ability to distinguish between related shots and unrelated shots.

[0099] In the embodiments of the present disclosure, the initial feature extraction network can be trained to pay attention to the local continuity between adjacent single-shot video segments in the sample video based on the determined local continuity loss, so that the visual correlation features extracted by the trained feature extraction network contain information representing local continuity.

[0100] Figure 5 A flowchart of a video generation method according to an embodiment of the present disclosure is shown as follows, Figure 5 The video generation method includes:

[0101] The user specifies a first single-shot video segment in the multi-shot video to be generated from a preset video library; the preset video library includes a plurality of single-shot videos;

[0102] The lens sampling module determines a plurality of second single-shot video segments after the first single-shot video segment, and then determines a next single-shot video segment after each second single-shot video segment, and so on, until at least one group of N single-shot video segments is determined.

[0103] The lens sampling module can determine a plurality of groups of N single-shot video segments according to steps S121 to S122 in the above embodiments, that is, the visual correlation features of each single-shot video in the preset video library are extracted by the feature extraction network, and then the second single-shot video segment arranged in time sequence after the first single-shot video segment is determined based on the visual correlation features of the first single-shot video segment. Then, the (n+1) th single-shot video segment is determined based on the visual correlation features of each nth single-shot video segment, and at least one group of N single-shot video segments is obtained. The feature extraction network can be obtained by training the initial feature extraction network with the sample video, and the specific training process can refer to the related description of the embodiments of the present disclosure above, which will not be repeated here.

[0104] At least one group of N single-shot video segments can be arranged in time sequence to form at least one group of video sequences; and the N single-shot video segments in each group of video sequences are spliced by the video splicing module to generate at least one multi-shot video.

[0105] After generating the multi-shot video, the generated multi-shot video can be recommended to the user; in this way, the user can optimize and edit the multi-shot video through manual editing, and output the edited video.

[0106] According to the embodiments of the present disclosure, the visual correlation features obtained by the feature extraction network have more context-related information, so that the downstream single-shot video splicing becomes more reasonable and smooth; and based on the user's specification of the preset video library and the first single-shot video, a plurality of reasonable and smooth multi-shot videos can be output.

[0107] According to the embodiments of the present disclosure, the user can be more efficiently recommended with multiple groups of multi-lens videos, the user can be facilitated to share to various social media, and further optimization and editing can be performed on the basis of the generated multi-lens video, so that the time for video creation is shortened and the video creation efficiency is improved.

[0108] It can be understood that the above-mentioned various method embodiments of the present disclosure can be combined with each other to form combined embodiments without departing from the principle logic. Limited by the length, the present disclosure will not be repeated. Those skilled in the art can understand that in the above-mentioned method of the specific embodiment, the specific execution order of each step should be determined according to its function and possible internal logic.

[0109] In addition, the present disclosure also provides a video generation apparatus, an electronic device, a computer readable storage medium, and a program, which can be used to implement any one of the video generation methods provided by the present disclosure. The corresponding technical solutions and descriptions are described in the method part and will not be repeated.

[0110] Figure 6 A block diagram of a video generation apparatus according to an embodiment of the present disclosure is shown, as shown in Figure 6 The apparatus includes:

[0111] The acquisition module 101 is configured to acquire a target single-lens video in a preset video library as a first single-lens video of a multi-lens video to be generated, and the preset lens library includes a plurality of single-lens videos.

[0112] The determination module 102 is configured to determine at least one group of video sequences starting from the first single-lens video according to visual correlation features of the first single-lens video and visual correlation features of a first group of candidate videos, the visual correlation features representing the contextual correlation of the single-lens video, and the first group of candidate videos including other single-lens videos in the preset video library except the first single-lens video.

[0113] The splicing module 103 is configured to splice the at least one group of video sequences to obtain at least one multi-lens video.

[0114] In a possible implementation, the determining module 102 comprises: a first determining sub-module, configured to determine at least one second single-shot video that is arranged in time sequence after the first single-shot video from the first group of candidate videos according to the visual correlation feature of the first single-shot video and the visual correlation feature of the first group of candidate videos; a second determining sub-module, configured to determine at least one (n+1) th single-shot video that is arranged in time sequence after the at least one n th single-shot video from the n th group of candidate videos according to the visual correlation feature of the at least one n th single-shot video and the visual correlation feature of the n th group of candidate videos, wherein the n th group of candidate videos comprises other single-shot videos in the preset video library except the first single-shot video to the at least one n th single-shot video, 2≤n

[0115] In a possible implementation, the determining at least one second single-shot video that is arranged in time sequence after the first single-shot video from the first group of candidate videos according to the visual correlation feature of the first single-shot video and the visual correlation feature of the first group of candidate videos comprises: determining the feature distance between the visual correlation feature of the first single-shot video and the visual correlation feature of each single-shot video in the first group of candidate videos; determining at least one single-shot video in the first group of candidate videos with a feature distance less than a preset threshold as the at least one second single-shot video arranged in time sequence after the first single-shot video.

[0116] In a possible implementation, the determining at least one (n+1) th single-shot video that is arranged in time sequence after the at least one n th single-shot video from the n th group of candidate videos according to the visual correlation feature of the at least one n th single-shot video and the visual correlation feature of the n th group of candidate videos comprises: for any n th single-shot video, determining the feature distance between the visual correlation feature of the n th single-shot video and the visual correlation feature of each single-shot video in the n th group of candidate videos; determining at least one single-shot video in the n th group of candidate videos with a feature distance less than a preset threshold as the at least one (n+1) th single-shot video arranged in time sequence after the n th single-shot video.

[0117] In a possible implementation, the visual correlation feature is extracted by a feature extraction network, and the feature extraction network is obtained by training an initial feature extraction network based on sample videos; the sample videos include multi-shot videos in a same video and multi-shot videos spliced from at least two different videos; and each sample video includes a mask video segment that has been replaced by a preset video segment and a mask video segment that has not been replaced, the preset video segment including at least one of a video segment in a different video and a blank video segment, and the mask video segment is used to indicate a specified single-shot video segment in the sample video.

[0118] In a possible implementation, the training of the initial feature extraction network based on the sample videos includes: extracting sample visual correlation features of the sample videos by the initial feature extraction network; determining a network loss of the initial feature extraction network based on the sample visual correlation features, where the network loss includes at least one of a global continuity loss and a local continuity loss, the global continuity loss representing a global continuity loss between multiple single-shot video segments in the sample video, and the local continuity loss representing a local continuity loss between adjacent single-shot video segments in the sample video; and training the initial feature extraction network based on the network loss to obtain a trained feature extraction network.

[0119] In a possible implementation, the determining of the network loss of the initial feature extraction network based on the sample visual correlation features includes: classifying the sample video based on the sample visual correlation features to obtain a video classification result of the sample video, the video classification result being used to indicate a probability that the sample video is a multi-shot video in a same video or a probability that the sample video is a multi-shot video spliced from at least two different videos; and determining the global continuity loss based on a difference between the video classification result and a video annotation result of the sample video.

[0120] In a possible implementation, the determining of the network loss of the initial feature extraction network based on the sample visual correlation features includes: classifying a mask video segment in the sample video based on a sub-visual correlation feature corresponding to the mask video segment in the sample visual correlation features to obtain a video segment classification result of the mask video segment in the sample video, the video segment classification result being used to indicate a probability that the mask video segment in the sample video is a video segment in a different video, a probability that the mask video segment is a blank video segment, or a probability that the mask video segment is a video segment that has not been replaced; and determining the local continuity loss based on a difference between the video segment classification result and a video segment annotation result of the mask video segment in the sample video.

[0121] In a possible implementation, in a case where a plurality of groups of video sequences are determined, the splicing module 103 comprises a splicing sub-module configured to splice each group of video sequences in the plurality of groups of video sequences to obtain a plurality of multi-shot videos; after the plurality of multi-shot videos are obtained, the apparatus further comprises a selection module configured to determine a selected multi-shot video as a recommended target multi-shot video in response to a selection operation on the plurality of multi-shot videos.

[0122] In the embodiments of the present disclosure, by obtaining a target single-shot video in a preset video library comprising a plurality of single-shot videos as a first segment single-shot video of the multi-shot video to be generated, it is equivalent to specifying the first segment single-shot video of the multi-shot video to be generated from the plurality of single-shot videos, and then according to the visual correlation features of the first segment single-shot video and the visual correlation features of other single-shot videos in the preset video library except the first segment single-shot video, and the visual correlation features representing the contextual correlation of the single-shot video, at least one group of video sequences having strong contextual correlation starting from the first segment single-shot video can be determined, and then the at least one group of video sequences is spliced, so that at least one reasonable and smooth multi-shot video can be automatically generated, which is beneficial to shorten the time length of producing the multi-shot video and improve the video production efficiency.

[0123] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules which can be used to execute the methods described in the above method embodiments, and the specific implementation can be referred to the description of the above method embodiments. For briefness, details are not described herein.

[0124] The embodiments of the present disclosure also provide a computer-readable storage medium having computer program instructions stored therein, and the computer program instructions are executed by a processor to implement the above method. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0125] The embodiments of the present disclosure also provide an electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above method.

[0126] The embodiments of the present disclosure also provide a computer program product, comprising computer readable code, or a non-volatile computer readable storage medium carrying computer readable code, when the computer readable code is run in the processor of the electronic device, the processor in the electronic device executes the above method.

[0127] The electronic device can be provided as a terminal, a server or other forms of devices.

[0128] Figure 7A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 can be provided as a server or a terminal device. Referring to FIG. 19, the electronic device 1900 includes a processing component 1922, a memory 1932, a communication interface 1940, and an input / output (I / O) interface 1958. Figure 7 The processing component 1922 includes one or more processors. The memory 1932 is configured to store instructions such as an application program, which can be executed by the processing component 1922. The application program stored in the memory 1932 can include one or more modules each corresponding to a set of instructions. The processing component 1922 is configured to execute the instructions to perform the above-described methods.

[0129] The electronic device 1900 can further include a power component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and the I / O interface 1958. The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as a Microsoft Windows Server operating system (Windows Server TM ), an Apple Inc. operating system based on a graphical user interface (Mac OSX TM ), a multi-user multi-processing computer operating system (Unix TM ), a free and open-source Unix-like operating system (Linux TM ), an open-source Unix-like operating system (FreeBSD TM ), or the like.

[0130] In an exemplary embodiment, a non-transitory computer-readable storage medium, such as the memory 1932 including computer program instructions, is also provided, which can be executed by the processing component 1922 of the electronic device 1900 to complete the above-described methods.

[0131] The present disclosure can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0132] Computer readable storage media can be tangible storage media which can retain and store instructions for use by an instruction execution device. Computer readable storage media can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer readable storage media include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0133] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0134] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0135] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0136] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0137] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0138] The flow diagrams and the block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logic functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0139] The computer program product can be embodied in a tangible medium of

[0140] The above description of the various embodiments is intended to be illustrative in all aspects, rather than being restrictive. Those skilled in the art can refer to the description of the various embodiments to make modifications and / or improvements.

[0141] Those skilled in the art can understand that, in the above-described method of the specific embodiments, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible inherent logic.

[0142] If the technical solutions of the present application involve personal information, the product applying the technical solutions of the present application has been explicitly informed of the personal information processing rules before processing the personal information, and has obtained the personal independent consent. If the technical solutions of the present application involve sensitive personal information, the product applying the technical solutions of the present application has obtained the personal independent consent before processing the sensitive personal information, and at the same time meets the requirement of "explicit consent". For example, at the personal information collection device such as camera, a clear and prominent mark is set to inform that it has entered the personal information collection range and will collect personal information. If the individual voluntarily enters the collection range, it is considered to agree to collect personal information. Or on the device for processing personal information, through the pop-up information or by asking the individual to upload his personal information, the individual's authorization is obtained under the condition that the device uses obvious mark / information to inform the individual of the personal information processing rules. The personal information processing rules can include personal information processor, personal information processing purpose, processing method and personal information type, etc.

[0143] The above has described various embodiments of the present disclosure, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles, practical application or improvement of technology in the market of the embodiments, or to enable other ordinary skilled in the art to understand the embodiments disclosed herein.

Claims

1. A video generation method, characterized in that, include: Obtain a target single-shot video from a preset video library as the first single-shot video to be generated in a multi-shot video. The preset video library includes multiple single-shot videos. Based on the visual association features of the first single-shot video segment and the visual association features of the first group of candidate videos, at least one video sequence with contextual association is determined, starting from the first single-shot video segment. The visual association features characterize the contextual association of the single-shot videos. The first group of candidate videos includes other single-shot videos in the preset video library besides the first single-shot video segment. The contextual association characterizes global continuity and local continuity. By splicing together the at least one set of video sequences, at least one multi-camera video can be obtained; The visual association features are extracted through a feature extraction network, which is obtained by training an initial feature extraction network using sample videos. The step of training the initial feature extraction network through sample videos includes: The initial feature extraction network extracts the visual association features of the sample videos. Based on the visual association features of the samples, the network loss of the initial feature extraction network is determined, wherein the network loss includes at least one of global continuity loss and local continuity loss, wherein the global continuity loss characterizes the global continuity loss between multiple single-shot video segments in the sample video, and the local continuity loss characterizes the local continuity loss between adjacent single-shot video segments in the sample video. The initial feature extraction network is trained based on the network loss to obtain the trained feature extraction network.

2. The method according to claim 1, characterized in that, The step of determining at least one set of video sequences with contextual relevance starting from the first single-shot video, based on the visual association features of the first single-shot video segment and the visual association features of the first set of candidate videos, includes: Based on the visual association features of the first single-shot video segment and the visual association features of the first group of candidate videos, at least one second single-shot video segment that is sequentially arranged after the first single-shot video segment is determined from the first group of candidate videos. Based on the visual association features of at least one nth single-shot video segment and the visual association features of the nth group of candidate videos, at least one (n+1)th single-shot video segment sequentially arranged after the at least one nth single-shot video segment is determined from the nth group of candidate videos. The nth group of candidate videos includes other single-shot videos in the preset video library other than the first single-shot video segment to the at least one nth single-shot video segment, where 2≤n<N, and N is the preset total number of single-shot videos in the multi-shot video. The N single-shot videos, arranged chronologically from the first single-shot video to the Nth single-shot video, are combined into a video sequence.

3. The method according to claim 2, characterized in that, The step of determining at least one second single-shot video segment that is sequentially arranged after the first single-shot video segment from the first group of candidate videos based on the visual association features of the first single-shot video segment and the visual association features of the first group of candidate videos includes: Determine the feature distance between the visual association features of the first single-shot video segment and the visual association features of each single-shot video in the first group of candidate videos; At least one single-shot video in the first group of candidate videos whose feature distance is less than a preset threshold is identified as at least one second single-shot video that is sequentially arranged after the first single-shot video.

4. The method according to claim 2, characterized in that, The step of determining at least one (n+1)th single-shot video segment sequentially following the at least one nth single-shot video segment from the nth group of candidate videos, based on the visual association features of at least one nth single-shot video segment and the visual association features of the nth group of candidate videos, includes: For any nth single-shot video segment, determine the feature distance between the visual association features of the nth single-shot video segment and the visual association features of each single-shot video segment in the nth group of candidate videos. At least one single-shot video in the nth group of candidate videos whose feature distance is less than a preset threshold is determined as at least one (n+1)th single-shot video that is sequentially arranged after the nth single-shot video.

5. The method according to any one of claims 1 to 4, characterized in that, The sample videos include: multi-shot videos within the same video and multi-shot videos composed of at least two different videos; and each sample video includes a masked video segment that has been replaced by a preset video segment and a masked video segment that has not been replaced, wherein the preset video segment includes at least one of video segments from different videos and blank video segments, and the masked video segment is used to indicate a partially specified single-shot video segment in the sample video.

6. The method according to claim 1, characterized in that, The step of determining the network loss of the initial feature extraction network based on the visual association features of the samples includes: Based on the visual association features of the samples, the sample videos are classified to obtain the video classification results of the sample videos. The video classification results are used to indicate the probability that the sample video is a multi-shot video in the same video or a multi-shot video spliced ​​together from at least two different videos. The global continuity loss is determined based on the difference between the video classification result and the video annotation result of the sample video.

7. The method according to claim 1 or 6, characterized in that, The step of determining the network loss of the initial feature extraction network based on the visual association features of the samples includes: Based on the sub-visual association features corresponding to the mask video segments in the sample visual association features, the mask video segments in the sample video are classified to obtain the video segment classification results of the mask video segments in the sample video. The video segment classification results are used to indicate the probability that the mask video segment in the sample video is a video segment in a different video, or a blank video segment, or a video segment that has not been replaced. The local continuity loss is determined based on the difference between the video segment classification result and the video segment annotation result of the masked video segment in the sample video.

8. The method according to claim 1, characterized in that, When multiple video sequences are determined, the step of splicing the at least one set of video sequences to obtain at least one multi-camera video includes: Each video sequence in the multiple video sequences is spliced ​​together to obtain multiple multi-camera videos; After obtaining the multiple multi-camera videos, the method further includes: In response to the selection operation for the plurality of multi-camera videos, the selected multi-camera video is determined as the recommended target multi-camera video.

9. A video generation apparatus, characterized in that, include: The acquisition module is used to acquire a target single-lens video from a preset video library as the first single-lens video to be generated in a multi-lens video. The preset video library includes multiple single-lens videos. The determining module is used to determine at least one set of video sequences with contextual relevance starting from the first single-shot video, based on the visual relevance features of the first single-shot video and the visual relevance features of the first set of candidate videos. The visual relevance features characterize the contextual relevance of the single-shot videos, and the first set of candidate videos includes other single-shot videos in the preset video library besides the first single-shot video. The contextual relevance characterizes global continuity and local continuity. A splicing module is used to splice the at least one set of video sequences to obtain at least one multi-camera video; The visual association features are extracted through a feature extraction network, which is obtained by training an initial feature extraction network using sample videos. The step of training the initial feature extraction network through sample videos includes: The initial feature extraction network extracts the visual association features of the sample videos. Based on the visual association features of the samples, the network loss of the initial feature extraction network is determined, wherein the network loss includes at least one of global continuity loss and local continuity loss, wherein the global continuity loss characterizes the global continuity loss between multiple single-shot video segments in the sample video, and the local continuity loss characterizes the local continuity loss between adjacent single-shot video segments in the sample video. The initial feature extraction network is trained based on the network loss to obtain the trained feature extraction network.

10. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 8.

Images