Video feature extraction method and device, video generation method and device, medium and equipment

Abstract

The invention relates to a video feature extraction method and device, a video generation method and device, a medium and equipment. The method comprises the following steps: receiving a to-be-processed target video; feature information of the target video under multiple target dimensions is determined, and the target dimensions at least comprise a subject content dimension and an object content dimension; according to the feature information under the multiple target dimensions, performing sub-lens detection on the target video to determine sub-lens segments in the target video; and for each sub-lens segment, performing feature extraction on the video frame in the sub-lens segment, and generating sub-lens script information corresponding to the sub-lens segment according to the extracted information. Therefore, the sub-mirror detection can be realized based on the change of the subject content and the object content in the target video, so that the video sub-mirror accuracy can be improved, the feature extraction can be performed from the level of the sub-mirror segment, the accuracy and comprehensiveness of the video feature can be improved, and the automatic conversion from the target video to the sub-mirror script can be realized.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a video feature extraction method, a video generation method, a device, a medium and a device. Background Art

[0002] With the development of computer technology, video applications have gradually facilitated people's daily lives. In related technologies, one way to generate videos is for users to manually upload pictures or videos and then edit them, and the other is to automatically generate videos based on video understanding. In the process of video understanding, since it involves time dimensions and dynamically changing information, it is more difficult to understand than natural language and picture content, resulting in low efficiency and accuracy of video feature extraction. Summary of the invention

[0003] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, the present disclosure provides a video feature extraction method, the method comprising:

[0005] Receive a target video to be processed;

[0006] Determining feature information of the target video in multiple target dimensions, wherein the target dimensions at least include a subject content dimension and an object content dimension;

[0007] Performing frame detection on the target video according to the feature information under the multiple target dimensions to determine frame segments in the target video;

[0008] For each of the storyboard segments, feature extraction is performed on the video frames in the storyboard segment, and storyboard script information corresponding to the storyboard segment is generated based on the extracted information.

[0009] In a second aspect, the present disclosure provides a video generation method, the method comprising:

[0010] Receiving a target text, wherein the target text includes at least one storyboard script;

[0011] According to the target text and the video generation model, a target video corresponding to the target text is generated, wherein the video generation model is trained based on training video samples and annotation scripts corresponding to the training video samples, and the annotation scripts corresponding to the training video samples are generated based on the video feature extraction method described in the first aspect.

[0012] In a third aspect, the present disclosure provides a video feature extraction device, the device comprising:

[0013] A first receiving module, used for receiving a target video to be processed;

[0014] A first determination module, configured to determine feature information of the target video in multiple target dimensions, wherein the target dimensions at least include a subject content dimension and an object content dimension;

[0015] A detection module, configured to perform frame detection on the target video according to feature information under the plurality of target dimensions, so as to determine frame segments in the target video;

[0016] The processing module is used to extract features of the video frames in each storyboard segment, and generate storyboard script information corresponding to the storyboard segment based on the extracted information.

[0017] In a fourth aspect, the present disclosure provides a video generation device, the device comprising:

[0018] A second receiving module is used to receive a target text, wherein the target text includes at least one storyboard script;

[0019] A generation module is used to generate a target video corresponding to the target text based on the target text and a video generation model, wherein the video generation model is trained based on training video samples and annotation scripts corresponding to the training video samples, and the annotation scripts corresponding to the training video samples are generated based on the video feature extraction method described in the first aspect.

[0020] In a fifth aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect or the second aspect.

[0021] In a sixth aspect, the present disclosure provides an electronic device, including:

[0022] a storage device having a computer program stored thereon;

[0023] A processing device is used to execute the computer program in the storage device to implement the steps of the method described in the first aspect or the second aspect.

[0024] In the above technical solution, by determining the characteristic information of the target video under multiple target dimensions, the target video can be subjected to storyboard detection according to the characteristic information under multiple target dimensions to determine the storyboard segments in the target video, and then for each storyboard segment, the video frame in the storyboard segment can be subjected to feature extraction, and the storyboard script information corresponding to the storyboard segment can be generated according to the extracted information. Through the above technical solution, the storyboard detection can be realized based on the changes of the subject content and the object content in the target video, so as to improve the accuracy of the video storyboard, and then the feature extraction can be performed on each storyboard segment from the level of the storyboard segment, on the one hand, the correlation between the extracted video features and the matching degree between the video features and the overall content of the target video can be guaranteed, and the accuracy and comprehensiveness of the video features can be improved; on the other hand, the storyboard script information corresponding to the storyboard segment can be generated by extracting the video features, so as to realize the automatic conversion of the target video to the storyboard script, without the need for manual recording and annotation, which can save the manual workload and provide a large amount of accurate training data for the subsequent video generation model based on the script to generate the video, and provide effective data support for the subsequent training of the video generation model. At the same time, the timeline association between multiple storyboard segments can be obtained, and the arrangement characteristics of the target video and the multiple storyboard segments in the time dimension can be obtained, providing a reference for ensuring the logic accuracy of the video generation process in the subsequent video generation model.

[0025] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings:

[0027] Figure 1 It is a flowchart of a video feature extraction method provided according to an embodiment of the present disclosure.

[0028] Figure 2 It is a block diagram of a video feature extraction device provided according to an embodiment of the present disclosure.

[0029] Figure 3 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0030] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0031] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0032] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0033] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0034] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0035] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0036] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0037] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0038] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0039] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0040] At the same time, it is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0041] Figure 1 As shown, it is a flow chart of a video feature extraction method provided according to an embodiment of the present disclosure, as shown Figure 1 As shown, the method may include:

[0042] In step 11, a target video to be processed is received.

[0043] The target video may be a video uploaded by a user or a video received from other service interfaces.

[0044] In step 12, feature information of the target video in multiple target dimensions is determined, wherein the target dimensions at least include a subject content dimension and an object content dimension.

[0045] The subject content dimension is used to represent the features of the subject object in the target video, and the object content dimension is used to represent the features of the object object in the target video. The applicant has found through research that the switching of the storyboards usually occurs when the subject object and the object object change. Therefore, in this embodiment, the features of the target video in the subject content dimension and the object content dimension can be extracted respectively to provide data support for subsequent accurate storyboard detection.

[0046] In step 13, based on the feature information under multiple target dimensions, the target video is subjected to a storyboard detection to determine the storyboard segments in the target video.

[0047] Storyboard, also known as storyboard, usually refers to various image media such as movies, animations, TV series, advertisements, music videos, etc., which illustrate the composition of the image in a diagram before the actual shooting or drawing. The continuous picture is decomposed into a unit of one shot, and the shot movement method, time length, dialogue, special effects, etc. are marked. Therefore, a shot can be used to represent continuous and related content in a short period of time in the video. In this embodiment, the target video can be split into multiple shot segments, and the multiple shot segments are connected in series to form the complete target video.

[0048] Therefore, in this embodiment, the target video can be split into storyboards to obtain multiple storyboard segments contained in the target video. As described above, the storyboard segments can contain continuous and interrelated content, which can facilitate the analysis and feature extraction of the target video based on the content of each storyboard segment. Feature extraction of the video at the storyboard segment level can ensure the correlation between the extracted features to a certain extent, thereby improving the matching degree between the extracted features and the content of the target video.

[0049] In step 14, for each storyboard segment, feature extraction is performed on the video frames in the storyboard segment, and storyboard script information corresponding to the storyboard segment is generated based on the extracted information.

[0050] Among them, feature extraction can be performed on the video frames in the storyboard segments in a multimodal feature extraction manner, such as visual feature extraction based on the image content in the video frames, sound feature extraction based on the sound content in the video frames, and scene change feature extraction based on continuous video frames in the storyboard segments, etc. It can be implemented based on the image feature extraction and sound feature extraction algorithms or models in the art, which will not be described in detail here.

[0051] After the corresponding information is extracted, the information can be added to the storyboard script information to perform standardized storage and display of the information of the storyboard segment.

[0052] Therefore, in the above technical solution, by determining the characteristic information of the target video under multiple target dimensions, the target video can be subjected to storyboard detection according to the characteristic information under multiple target dimensions to determine the storyboard segments in the target video, and then for each storyboard segment, the video frame in the storyboard segment can be subjected to feature extraction, and the storyboard script information corresponding to the storyboard segment can be generated according to the extracted information. Through the above technical solution, the storyboard detection can be realized based on the changes of the subject content and the object content in the target video, so as to improve the accuracy of the video storyboard, and then the feature extraction can be performed on each storyboard segment from the level of the storyboard segment, on the one hand, the correlation between the extracted video features and the matching degree between the video features and the overall content of the target video can be guaranteed, and the accuracy and comprehensiveness of the video features can be improved; on the other hand, the storyboard script information corresponding to the storyboard segment can be generated by extracting the video features, so as to realize the automatic conversion of the target video to the storyboard script, without the need for manual recording and annotation, which can save the manual workload and provide a large amount of accurate training data for the subsequent video generation model based on the script to generate the video, and provide effective data support for the subsequent training of the video generation model. At the same time, the timeline association between multiple storyboard segments can be obtained, and the arrangement characteristics of the target video and the multiple storyboard segments in the time dimension can be obtained, providing a reference for ensuring the logic accuracy of the video generation process in the subsequent video generation model.

[0053] In a possible embodiment, an exemplary implementation method of performing frame detection on the target video according to the feature information under the multiple target dimensions to determine frame segments in the target video may include:

[0054] A multimodal content matrix is ​​generated according to feature information of the target video in multiple target dimensions.

[0055] Among them, the feature information under multiple target dimensions can be spliced ​​to obtain the multimodal content matrix, so as to comprehensively characterize the features of the target video based on the multimodal content matrix.

[0056] Afterwards, storyboard detection is performed based on the storyboard detection model and the multimodal content matrix to obtain the plurality of storyboard segments, wherein the storyboard detection model is determined based on a training video and training annotated storyboards corresponding to the training video, and the training annotated storyboards are annotated based on the plurality of target dimensions.

[0057] The split-screen detection model may be a pre-trained model. As an example, the split-screen detection model may be determined in the following manner:

[0058] A training sample set is obtained, where the training sample set may include a training video and a training annotated storyboard corresponding to the training video.

[0059] As an example, when annotating the storyboards corresponding to the training video, standard division can be performed according to the multiple target dimensions, so that the annotators can perform corresponding annotations based on the characteristics of the training video under the multiple target dimensions and determine the multiple storyboards corresponding to the training video.

[0060] As another example, process files in the video generation process may also be obtained, such as a training video and an unedited video corresponding to the training video, so as to determine a plurality of storyboards in the training video based on the content of the unedited video and the training video.

[0061] In a possible embodiment, the target dimension also includes at least one of a camera dimension, a scene dimension, a music dimension, and an aesthetic dimension.

[0062] For example, the segmentation annotation can be performed based on the change of the feature in the target dimension. For example, in the subject content dimension, it can be determined whether to perform segmentation based on the change of the subject object, the change of the number of subject objects, and the change of the relationship between subject objects. For example, in the object content dimension, it can be determined whether to perform segmentation based on the change of the object object, the change of the number of object objects, and the change of the relationship between object objects.

[0063] The camera movement dimension can be used to represent the changes in the shooting method during the video shooting process. Usually, the continuous picture can be decomposed into a storyboard in units of one camera movement. The changes in the camera movement method can be determined to provide a reference for whether to split the storyboard, such as camera movement changes, scene changes, etc.

[0064] The scene dimension can be used to represent scenes in the video shooting process. Different scenes usually correspond to different storyboards. The changes in the scenes can be determined to provide a reference for whether to split the storyboards, such as changes in location and composition.

[0065] The music dimension can be used to represent the background music in the video. Different background music usually corresponds to different storyboards. The changes in the music can be determined to provide a reference for whether to split the storyboards, switch, pause or add background music, etc.

[0066] The aesthetic dimension can be used to represent the style and atmosphere of the video. Different styles or atmospheres usually correspond to different storyboards. The changes in the aesthetic dimension can be used to provide a reference for whether to split the storyboards, such as changes in picture style or picture atmosphere.

[0067] Among them, the standards and rules for determining whether storyboard splitting is required based on the features under multiple target dimensions can be pre-set based on the actual application scenario. The present disclosure does not limit this, and it is sufficient to ensure that the standards and rules for splitting are the same in the same model.

[0068] In this embodiment, the training annotated storyboards can be referenced and annotated from the above-mentioned feature dimensions when annotating, so that the segmentation standard of the training annotated storyboards conforms to the actual storyboard shooting process, thereby improving the accuracy of multiple storyboards obtained by the storyboard detection model to a certain extent, and making the predicted division of multiple storyboard segments match the actual video shooting process, thereby improving the accuracy and rationality of the storyboard segment division.

[0069] Afterwards, the training video can be used as the input of the model, and the training annotated storyboards can be used as the target output of the model for training, thereby obtaining a storyboard detection model.

[0070] Among them, the storyboard detection model can be implemented based on a neural network model with an attention mechanism, and then the corresponding multimodal content matrix can be obtained based on the input training video, and the attention mechanism is processed to obtain multiple predicted storyboards output by the model. Afterwards, it can be determined whether the predicted storyboard is reasonable based on the division standards and rules of multiple target dimensions, and the model can be adjusted based on the similarity between the predicted storyboard and the training annotated storyboard until a storyboard detection model that meets the training requirements is obtained. Meeting the training requirements can be that the accuracy of the predicted storyboard reaches a preset threshold, or that the number of iterations of the model reaches a threshold number, which is not limited in this disclosure.

[0071] Therefore, through the above technical solution, the target video can be split into storyboards through the storyboard detection model to obtain multiple storyboard segments, so that video features can be extracted and understood in units of storyboard segments. The training annotated storyboards of the training video corresponding to the storyboard detection model are annotated based on the multiple target dimensions, thereby ensuring the interpretability of the storyboard segments obtained based on the storyboard detection model, ensuring the accuracy and effectiveness of the storyboard splitting, and improving the effective guidance of the tuning direction of the video generation model when the obtained storyboard script is used for the subsequent video generation model.

[0072] In a possible embodiment, determining the feature information of the target video in multiple target dimensions may include:

[0073] A target video frame in the target video is obtained.

[0074] Among them, all video frames in the target video can be used as the target video frame, or frames can be extracted from the target video at intervals of a preset time length to obtain the target video frame. The preset time length can be set based on the actual application scenario, and the present disclosure does not limit this.

[0075] Object recognition is performed on the target video frame to determine candidate objects in the target video.

[0076] Among them, for each target video frame, object recognition can be performed on it. As an example, the object may include people, animals, static objects, etc., and the objects in the target video frame can be classified and recognized by pre-training an object classification and recognition model, such as can be implemented based on a classification model commonly used in the field.

[0077] As an example, each object identified from the target video may be used as a candidate object. As another example, the category corresponding to the candidate object may be preset, and the objects belonging to the preset category among the identified objects may be used as candidate objects.

[0078] Display information of each candidate object in the target video is determined, and subject objects and object objects in the candidate objects are determined according to the display information.

[0079] The display information is used to represent the characteristics of the candidate object when it is displayed in the target video, such as the display information can be at least one of the display duration, display frequency, and display matching degree of the candidate object with the theme of the target video in the target video. The display duration can be represented by the number of video frames containing the candidate object, the display frequency can be determined by the ratio of the number of video frames containing the candidate object to the target video frame, and the display matching degree can be determined based on the vector similarity calculation between the candidate object and the theme of the target video, which will not be elaborated here.

[0080] As an example, for each candidate object, the display components can be determined based on the above display information, and the display components can be weighted based on the weight of each display information to obtain a comprehensive display score for the candidate object. The subject object and the object object are then determined based on the comprehensive display score. For example, the object with the largest comprehensive display score can be used as the subject object, and the objects ranked 2-N in order from large to small in terms of comprehensive display scores can be used as object objects, where N can be set according to the actual application scenario. For another example, the number S of subject objects and the number T of object objects can be preset respectively, and then the first S objects can be selected as the subject objects in order from large to small according to the comprehensive display scores, and then T objects can be selected as object objects.

[0081] For another example, the display information may also include the ratio of the candidate objects to the entire screen, and the subject object and the object object may be determined according to the ratio of the candidate objects in the screen to the entire screen. For another example, the above-mentioned weighted method may be combined with multiple display information to perform a comprehensive weighted determination to identify the subject object and the object object, which will not be repeated here.

[0082] The above embodiments are merely exemplary and do not limit the present disclosure. In this embodiment, the subject object and the object object in the target video can be determined based on the display information of the object in the target video according to a unified standard.

[0083] Afterwards, the feature information corresponding to the subject object is determined as the feature information under the subject content dimension, and the feature information corresponding to the object object is determined as the feature information under the object content dimension.

[0084] Accordingly, the feature information of the subject object determined in the object recognition process can be used as the feature information under the subject content dimension, such as the feature information of the subject object can be the coding feature corresponding to the subject object extracted in the model processing process, such as the coding feature of the subject object determined in the classification recognition model. Similarly, the feature information under the object content dimension can be determined in a similar manner.

[0085] Therefore, through the above technical solution, the main content and object content in the target video can be extracted and represented, providing support for subsequent storyboard detection based on changes in the main content and object content, so that the process of storyboard splitting corresponds to the process of storyboard splicing in the video generation process, thereby improving the accuracy of storyboard splitting to a certain extent.

[0086] In a possible embodiment, an exemplary implementation method of extracting features from the video frames in the storyboard segments and generating storyboard script information corresponding to the storyboard segments according to the extracted information may include:

[0087] According to the video frames in the storyboard fragments, feature information of the storyboard fragments under the classification identification dimension corresponding to each script dimension is extracted, wherein the script dimension includes a subject content dimension and an object content dimension, and at least one classification identification dimension is preset under each script dimension.

[0088] The script dimension can be pre-set according to the actual application scenario. As an example, the script dimension can also include at least one of the lens dimension, scene dimension, sound dimension, text dimension and aesthetic dimension, so that the video frame can be feature extracted from multiple dimensions to ensure the comprehensiveness and accuracy of feature extraction. The text dimension can be used to represent the subtitle information in the video frame, such as by OCR (Optical Character Recognition) technology.

[0089] The classification and identification dimensions under each of the script dimensions can be preset according to the actual application scenario, wherein the classification and identification dimensions are finer-grained divisions under the script dimension. For example, the classification and identification dimensions under the subject content dimension and corresponding to the object content dimension include at least one of the trait dimension, the action behavior dimension, the trait change dimension, and the subject-object interaction dimension.

[0090] In the following, the script dimension is used as the main content dimension for explanation. For example, the first feature information identified is that the trait dimension can represent the attribute characteristics of the subject object itself, such as a boy wearing black clothes; the action behavior dimension can be used to represent the behavior and action events that cause the state of the subject object to change, such as a boy preparing to shoot; the trait change dimension can be used to represent the event that causes the trait of the subject object to change, such as a boy looking at the basket; the subject-object interaction dimension is used to represent the relationship between the subject object and the object object, such as attribute association, relative spatial position and behavioral interaction, such as cheerleaders standing in a row on the court.

[0091] As another example, the classification and identification dimension corresponding to the lens dimension may include at least one of the dimension of scene, the dimension of camera movement, the dimension of angle, and the dimension of subject-object relationship. The scene refers to the difference in the size of the range of the subject presented in the camera recorder due to the different distances between the camera and the subject when the focal length is constant. The value of the scene dimension can be determined based on the scene division method commonly used in the field, such as long view, panorama, etc. The value of the camera movement dimension can be set based on the general camera movement method, such as push, pull, shake, etc.; the value of the angle dimension can be set based on the general angle setting method in the field, such as pitch, tilt, etc.; the subject-object relationship dimension is used to represent the relationship and time sequence change of scene, camera movement, angle, etc. relative to the subject object. Therefore, the feature information such as scene, camera movement and angle corresponding to the storyboard segment can be obtained by determining the extraction of video features in the storyboard segment.

[0092] The classification and identification dimension corresponding to the scene dimension may include at least one of the location dimension, the environment composition dimension, the environment style dimension, and the subject-object relationship dimension. Among them, the location dimension is used to describe the attribute characteristics of the location, such as time and space attributes, place names, etc.; the environment composition dimension is used to represent the elements existing in the scene environment and their characteristics and relative spatial positions; the environment style dimension is used to represent the style corresponding to the scene environment, and its value can be set by classification; the subject-object relationship dimension can be used to represent the relationship and time sequence changes of the location, environment composition, etc. relative to the subject object. Therefore, the above-mentioned classification and identification dimensions can be used to make a more comprehensive and accurate feature description of the scene in the storyboard fragment.

[0093] The sound dimension may include the human voice dimension, the background sound dimension, and the sound effect dimension, which may be obtained by extracting features from the audio in the storyboard clips so as to represent the audio features in the storyboard clips; the text dimension may include the subtitle dimension and the artistic word dimension, so as to represent the text features in the storyboard clips; the aesthetic dimension may include the tone dimension and the atmosphere dimension.

[0094] It should be noted that, for each script dimension, a sub-model for feature extraction under the classification and recognition dimension can be trained under the script dimension. The sub-model can be implemented based on a classification model or encoder model in the art, and the present disclosure does not limit this.

[0095] Afterwards, the key video frames in the storyboard segments are determined, and storyboard description texts corresponding to the storyboard segments and object change information between the key video frames are generated. The storyboard script information includes the key video frames corresponding to the storyboard segments, the storyboard description texts and content change information between the key video frames.

[0096] For example, the script dimension can be used as a first-level directory, the classification identification dimension under the script dimension can be used as a second-level directory, and the feature information under the classification identification dimension can be written into the value of the classification identification dimension accordingly, so as to obtain the feature information under the classification identification dimension, and then the feature information can be input into the video understanding model, so as to identify the key video frames in the storyboard segment based on the multiple feature information, and generate the storyboard description text corresponding to the storyboard segment and the content change information between the key video frames. Among them, the subject and object objects contained in different video frames may be different or the state of the subject or object objects is different, so they can be represented based on the content change information between the key video frames to describe the change state of the objects between different video frames.

[0097] The video understanding model can be trained in advance, for example, the training data of the video understanding model can be obtained based on the static image frames corresponding to each storyboard produced during the shooting of the target video, and the change description information between the storyboard script and the multiple static image frames. Afterwards, the target video can be input into the video understanding model, and the static image frames of the multiple storyboard segments contained in the target video, and the change description information between the storyboard script and the multiple static image frames are used as model target values ​​for training to obtain the video understanding model.

[0098] Therefore, in the process of generating storyboard script information, feature representation of secondary dimensions can be performed. On the one hand, the accuracy of feature extraction can be effectively improved, so that more comprehensive and rich features can be extracted from the storyboard fragments, thereby ensuring the accuracy of the storyboard script information and making the storyboard script information more comprehensive and effective in describing the storyboard fragments. On the other hand, the storyboard fragments can be characterized as a whole, and features can be extracted at the storyboard fragment level, further improving the matching degree between the feature extraction process and the actual video generation process.

[0099] In a possible embodiment, the method further includes:

[0100] Determine the storyboard display position corresponding to each storyboard segment in the target video.

[0101] The target video may include multiple storyboard segments, and the display order of the multiple storyboard segments represents the time logic of the target video. For example, the storyboard display position may be used to indicate which storyboard the storyboard segment is displayed in the target video.

[0102] Afterwards, the storyboard display position corresponding to the storyboard segment is associated with the storyboard script information corresponding to the storyboard segment.

[0103] In this embodiment, while extracting the storyboard script information corresponding to each storyboard segment in the target video, the corresponding storyboard display position is associated, so that the combination logic of different storyboard segments in the target video can be characterized. Accordingly, when the video generation model is trained based on the storyboard segments and storyboard script information obtained in the process, the combination logic of the storyboard segments in the video can be learned synchronously during the video generation model training process, which, on the one hand, ensures the consideration of the overall video display logic during the video feature extraction process, thereby ensuring the accuracy of the video feature extraction; on the other hand, it can provide a more comprehensive feature reference for subsequent video generation, and improve the reasonable arrangement of the generated video in the time dimension.

[0104] The present disclosure also provides a video generation method, which may include:

[0105] A target text is received, wherein the target text includes at least one storyboard script.

[0106] The target text may be a detailed description of the video that the user wants to generate.

[0107] According to the target text and the video generation model, a target video corresponding to the target text is generated, wherein the video generation model is trained based on training video samples and annotation scripts corresponding to the training video samples, and the annotation scripts corresponding to the training video samples are generated based on the video feature extraction method described above.

[0108] In this embodiment, feature extraction can be performed on the training video sample based on the video feature extraction method described above, so that the storyboard script corresponding to the training video sample can be obtained as the annotation script, and in the process of video feature extraction, the corresponding storyboard segments are first detected, and then the storyboard script information corresponding to each storyboard segment is determined. When the video generation model is trained based on the training video sample and its annotation script, the video generation model can learn the corresponding features in the storyboard detection process and the timeline logic between the multiple storyboard segments contained in the training video sample. While quickly determining a large number of training samples to save the workload of manual annotation, the video generation process of the video generation model is improved. Matching degree with the real video editing process is improved, thereby improving the efficiency and accuracy of video generation and enhancing the user experience.

[0109] In a possible embodiment, the target video and the storyboard clips and storyboard scripts corresponding to the target video can be used as training data to train the script generation model, the picture analysis model and the video shooting guidance model, which can improve the processing accuracy of the above models to a certain extent, and can also effectively reduce the amount of manual annotation required to generate training data.

[0110] The present disclosure also provides a video feature extraction device, such as Figure 2 As shown, the device 10 comprises:

[0111] A first receiving module 100 is used to receive a target video to be processed;

[0112] A first determination module 200 is used to determine feature information of the target video in multiple target dimensions, wherein the target dimensions at least include a subject content dimension and an object content dimension;

[0113] A detection module 300, configured to perform frame detection on the target video according to feature information under the plurality of target dimensions, so as to determine frame segments in the target video;

[0114] The processing module 400 is used to extract features of the video frames in each storyboard segment, and generate storyboard script information corresponding to the storyboard segment based on the extracted information.

[0115] Optionally, the detection module includes:

[0116] A first generating submodule, configured to generate a multimodal content matrix according to feature information of the target video in multiple target dimensions;

[0117] A detection submodule is used to perform storyboard detection based on a storyboard detection model and the multimodal content matrix to obtain a plurality of storyboard segments, wherein the storyboard detection model is determined based on a training video and training annotated storyboards corresponding to the training video, and the training annotated storyboards are annotated based on the multiple target dimensions.

[0118] Optionally, the first determining module includes:

[0119] An acquisition submodule, used for acquiring a target video frame in the target video;

[0120] An identification submodule, configured to perform object identification on the target video frame and determine candidate objects in the target video;

[0121] A first determination submodule is used to determine display information of each candidate object in the target video, and determine the subject object and the object object in the candidate object according to the display information;

[0122] The second determining submodule is used to determine the feature information corresponding to the subject object as the feature information under the subject content dimension, and to determine the feature information corresponding to the object object as the feature information under the object content dimension.

[0123] Optionally, the target dimension also includes at least one of a camera dimension, a scene dimension, a music dimension and an aesthetic dimension.

[0124] Optionally, the processing module includes:

[0125] An extraction submodule, configured to extract feature information of the storyboard segment under a classification identification dimension corresponding to each script dimension according to the video frames in the storyboard segment, wherein the script dimension includes a subject content dimension and an object content dimension, and at least one classification identification dimension is preset under each of the script dimensions;

[0126] The second generation submodule is used to determine the key video frames in the storyboard segment according to the feature information under the classification identification dimension, and generate the storyboard description text corresponding to the storyboard segment and the object change information between the key video frames, and the storyboard script information includes the key video frames corresponding to the storyboard segment, the storyboard description text and the content change information between the key video frames.

[0127] Optionally, the script dimension also includes at least one of a shot dimension, a scene dimension, a sound dimension, a text dimension and an aesthetic dimension, and the classification identification dimension under the subject content dimension and corresponding to the object content dimension includes at least one of a trait dimension, an action behavior dimension, a trait change dimension, and a subject-object interaction dimension.

[0128] Optionally, the device further comprises:

[0129] A second determination module is used to determine the corresponding storyboard display position of each storyboard segment in the target video;

[0130] An associating module is used to associate the storyboard display position corresponding to the storyboard segment with the storyboard script information corresponding to the storyboard segment.

[0131] The present disclosure also provides a video generation device, the device comprising:

[0132] A second receiving module is used to receive a target text, wherein the target text includes at least one storyboard script;

[0133] A generation module is used to generate a target video corresponding to the target text based on the target text and a video generation model, wherein the video generation model is trained based on training video samples and annotation scripts corresponding to the training video samples, and the annotation scripts corresponding to the training video samples are generated based on the video feature extraction method described above.

[0134] Reference below Figure 3 , which shows a schematic diagram of the structure of an electronic device (such as a terminal device or a server) 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0135] like Figure 3 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0136] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0137] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0138] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0139] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0140] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0141] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: receives a target video to be processed; determines feature information of the target video under multiple target dimensions, wherein the target dimensions at least include a subject content dimension and an object content dimension; performs storyboard detection on the target video based on the feature information under the multiple target dimensions to determine the storyboard segments in the target video; performs feature extraction on the video frames in the storyboard segment for each storyboard segment, and generates storyboard script information corresponding to the storyboard segment based on the extracted information.

[0142] Alternatively, the computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device: receives a target text, wherein the target text contains at least one storyboard script; generates a target video corresponding to the target text according to the target text and a video generation model, wherein the video generation model is trained based on training video samples and annotation scripts corresponding to the training video samples, and the annotation scripts corresponding to the training video samples are generated based on the above-mentioned video feature extraction method.

[0143] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0144] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0145] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a module does not limit the module itself in some cases. For example, the first receiving module may also be described as a "module for receiving a target video to be processed".

[0146] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0147] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0148] According to one or more embodiments of the present disclosure, Example 1 provides a video feature extraction method, wherein the method includes:

[0149] Receive a target video to be processed;

[0150] Determining feature information of the target video in multiple target dimensions, wherein the target dimensions at least include a subject content dimension and an object content dimension;

[0151] Performing frame detection on the target video according to the feature information under the multiple target dimensions to determine frame segments in the target video;

[0152] For each of the storyboard segments, feature extraction is performed on the video frames in the storyboard segment, and storyboard script information corresponding to the storyboard segment is generated based on the extracted information.

[0153] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, wherein the step of performing frame detection on the target video according to the feature information under the multiple target dimensions to determine frame segments in the target video includes:

[0154] Generate a multimodal content matrix according to feature information of the target video in multiple target dimensions;

[0155] Storyboard detection is performed based on a storyboard detection model and the multimodal content matrix to obtain a plurality of storyboard segments, wherein the storyboard detection model is determined based on a training video and training annotated storyboards corresponding to the training video, and the training annotated storyboards are annotated based on the multiple target dimensions.

[0156] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 1, wherein determining feature information of the target video in multiple target dimensions includes:

[0157] Acquire a target video frame in the target video;

[0158] Performing object recognition on the target video frame to determine candidate objects in the target video;

[0159] Determine display information of each candidate object in the target video, and determine the subject object and the object object in the candidate objects according to the display information;

[0160] The characteristic information corresponding to the subject object is determined as the characteristic information under the subject content dimension, and the characteristic information corresponding to the object object is determined as the characteristic information under the object content dimension.

[0161] According to one or more embodiments of the present disclosure, Example 4 provides the method of Example 1, wherein the target dimension further includes at least one of a camera dimension, a scene dimension, a music dimension, and an aesthetic dimension.

[0162] According to one or more embodiments of the present disclosure, Example 5 provides the method of Example 1, wherein the extracting features of the video frames in the storyboard segments and generating storyboard script information corresponding to the storyboard segments according to the extracted information includes:

[0163] Extracting feature information of the storyboard fragments under a classification identification dimension corresponding to each script dimension according to the video frames in the storyboard fragments, wherein the script dimension includes a subject content dimension and an object content dimension, and at least one classification identification dimension is preset under each script dimension;

[0164] Based on the feature information under the classification identification dimension, the key video frames in the storyboard segment are determined, and the storyboard description text corresponding to the storyboard segment and the object change information between the key video frames are generated. The storyboard script information includes the key video frames corresponding to the storyboard segment, the storyboard description text and the content change information between the key video frames.

[0165] According to one or more embodiments of the present disclosure, Example 6 provides the method of Example 5, wherein the script dimension also includes at least one of a shot dimension, a scene dimension, a sound dimension, a text dimension, and an aesthetic dimension, and the classification and identification dimension under the subject content dimension and corresponding to the object content dimension includes at least one of a trait dimension, an action behavior dimension, a trait change dimension, and a subject-object interaction dimension.

[0166] According to one or more embodiments of the present disclosure, Example 7 provides the method of Example 1, wherein the method further includes:

[0167] Determine a storyboard display position corresponding to each storyboard segment in the target video;

[0168] The storyboard display position corresponding to the storyboard segment is associated with the storyboard script information corresponding to the storyboard segment.

[0169] According to one or more embodiments of the present disclosure, Example 8 provides a video generation method, the method comprising:

[0170] Receiving a target text, wherein the target text includes at least one storyboard script;

[0171] According to the target text and the video generation model, a target video corresponding to the target text is generated, wherein the video generation model is trained based on training video samples and annotation scripts corresponding to the training video samples, and the annotation scripts corresponding to the training video samples are generated based on the video feature extraction method described in any one of Examples 1-7.

[0172] According to one or more embodiments of the present disclosure, Example 9 provides a video feature extraction device, the device comprising:

[0173] A first receiving module, used for receiving a target video to be processed;

[0174] A first determination module, configured to determine feature information of the target video in multiple target dimensions, wherein the target dimensions at least include a subject content dimension and an object content dimension;

[0175] A detection module, configured to perform frame detection on the target video according to feature information under the plurality of target dimensions, so as to determine frame segments in the target video;

[0176] The processing module is used to extract features of the video frames in each storyboard segment, and generate storyboard script information corresponding to the storyboard segment based on the extracted information.

[0177] According to one or more embodiments of the present disclosure, Example 10 provides a video generating device, the device comprising:

[0178] A second receiving module is used to receive a target text, wherein the target text includes at least one storyboard script;

[0179] A generation module is used to generate a target video corresponding to the target text based on the target text and a video generation model, wherein the video generation model is trained based on training video samples and annotation scripts corresponding to the training video samples, and the annotation scripts corresponding to the training video samples are generated based on the video feature extraction method described in any one of Examples 1-7.

[0180] According to one or more embodiments of the present disclosure, Example 11 provides a computer-readable medium having a computer program stored thereon, which implements the steps of any of the methods described in Examples 1-8 when executed by a processing device.

[0181] According to one or more embodiments of the present disclosure, Example 12 provides an electronic device, including:

[0182] a storage device having a computer program stored thereon;

[0183] A processing device is used to execute the computer program in the storage device to implement the steps of any one of the methods described in Examples 1-8.

[0184] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0185] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0186] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A video feature extraction method, characterized in that: The method comprises: Receive a target video to be processed; Determining feature information of the target video in multiple target dimensions, wherein the target dimensions at least include a subject content dimension and an object content dimension; Performing frame detection on the target video according to the feature information under the multiple target dimensions to determine frame segments in the target video; For each of the storyboard segments, feature extraction is performed on the video frames in the storyboard segment, and storyboard script information corresponding to the storyboard segment is generated based on the extracted information.

2. The method according to claim 1, characterized in that The step of performing frame detection on the target video according to the feature information under the multiple target dimensions to determine frame segments in the target video includes: Generate a multimodal content matrix according to feature information of the target video in multiple target dimensions; Storyboard detection is performed based on a storyboard detection model and the multimodal content matrix to obtain a plurality of storyboard segments, wherein the storyboard detection model is determined based on a training video and training annotated storyboards corresponding to the training video, and the training annotated storyboards are annotated based on the multiple target dimensions.

3. The method according to claim 1, characterized in that Determining feature information of the target video in multiple target dimensions includes: Acquire a target video frame in the target video; Performing object recognition on the target video frame to determine candidate objects in the target video; Determine display information of each candidate object in the target video, and determine the subject object and the object object in the candidate objects according to the display information; The characteristic information corresponding to the subject object is determined as the characteristic information under the subject content dimension, and the characteristic information corresponding to the object object is determined as the characteristic information under the object content dimension.

4. The method according to claim 1, characterized in that: The target dimension also includes at least one of a camera dimension, a scene dimension, a music dimension, and an aesthetic dimension.

5. The method according to claim 1, characterized in that The extracting features of the video frames in the storyboard segments and generating storyboard script information corresponding to the storyboard segments according to the extracted information includes: Extracting feature information of the storyboard fragments under a classification identification dimension corresponding to each script dimension according to the video frames in the storyboard fragments, wherein the script dimension includes a subject content dimension and an object content dimension, and at least one classification identification dimension is preset under each script dimension; Based on the feature information under the classification identification dimension, the key video frames in the storyboard segment are determined, and the storyboard description text corresponding to the storyboard segment and the object change information between the key video frames are generated. The storyboard script information includes the key video frames corresponding to the storyboard segment, the storyboard description text and the content change information between the key video frames.

6. The method according to claim 5, characterized in that The script dimension also includes at least one of a shot dimension, a scene dimension, a sound dimension, a text dimension and an aesthetic dimension, and the classification and identification dimension under the subject content dimension and corresponding to the object content dimension includes at least one of a trait dimension, an action and behavior dimension, a trait change dimension and a subject-object interaction dimension.

7. The method according to claim 1, characterized in that The method further comprises: Determine a storyboard display position corresponding to each storyboard segment in the target video; The storyboard display position corresponding to the storyboard segment is associated with the storyboard script information corresponding to the storyboard segment.

8. A video generation method, characterized in that: The method comprises: Receiving a target text, wherein the target text includes at least one storyboard script; According to the target text and the video generation model, a target video corresponding to the target text is generated, wherein the video generation model is trained based on training video samples and annotation scripts corresponding to the training video samples, and the annotation scripts corresponding to the training video samples are generated based on the video feature extraction method described in any one of claims 1-7.

9. A video feature extraction device, characterized in that: The device comprises: A first receiving module, used for receiving a target video to be processed; A first determination module, configured to determine feature information of the target video in multiple target dimensions, wherein the target dimensions at least include a subject content dimension and an object content dimension; A detection module, configured to perform frame detection on the target video according to feature information under the plurality of target dimensions, so as to determine frame segments in the target video; The processing module is used to extract features of the video frames in each storyboard segment, and generate storyboard script information corresponding to the storyboard segment based on the extracted information.

10. A video generating device, characterized in that: The device comprises: A second receiving module is used to receive a target text, wherein the target text includes at least one storyboard script; A generation module is used to generate a target video corresponding to the target text based on the target text and a video generation model, wherein the video generation model is trained based on training video samples and annotation scripts corresponding to the training video samples, and the annotation scripts corresponding to the training video samples are generated based on the video feature extraction method described in any one of claims 1-7.

11. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processing device, the steps of the method described in any one of claims 1 to 8 are implemented.

12. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 8.

Images