Video processing method, video generating method and device

Abstract

The invention discloses a video processing method and device and a video generating method and device. The video processing method comprises the steps of obtaining input data; the input data represents a target intention of a user for generating a video; according to the input data, a resource set is obtained from a multimedia library, the resource set comprises at least one piece of multimedia data, and the multimedia data at least comprises one of an image type or a video type; generating an index set corresponding to the target intention based on the input data and the resource set; based on the index information in the index set, extracting at least one target image corresponding to each piece of index information from the multimedia data in the resource set; and generating a target video based on the target image.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a video processing method, a video generating method and a device. Background Art

[0002] Currently, on electronic devices, you can mix pictures and videos and add music and motion effects to get edited videos.

[0003] However, in this method, the user is required to first select specific pictures and cut out key clips in the video, so there is a high degree of operational complexity, which affects the user experience. Summary of the invention

[0004] In view of this, the present application provides a video processing method, a video generating method and a device, as follows:

[0005] A video processing method, comprising:

[0006] Obtaining input data; the input data represents the user's target intention for generating a video;

[0007] According to the input data, a resource set is obtained from a multimedia library, wherein the resource set includes at least one multimedia data, and the multimedia data includes at least one of the following: an image type or a video type;

[0008] Based on the input data and the resource set, generate an index set corresponding to the target intent;

[0009] Based on the index information in the index set, extract at least one target image corresponding to each of the index information from the multimedia data in the resource set;

[0010] A target video is generated based on the target image.

[0011] Preferably, in the above method, after generating the target video based on the target image, the method further comprises:

[0012] Outputting the target video;

[0013] Obtaining adjustment data for the target video, wherein the adjustment data represents a user's adjustment intention for the target video;

[0014] At least according to the adjustment data, update data is obtained, the update data is used as the input data, and the step of returning to execute the step of obtaining a resource set from a multimedia library according to the input data is performed.

[0015] The above method preferably generates an index set corresponding to the target intent based on the input data and the resource set, including:

[0016] Obtaining multimedia features corresponding to the multimedia data in the resource set, the multimedia features comprising: a coarse-grained feature and at least one of a fine-grained feature corresponding to the coarse-grained feature, the fine-grained feature comprising: a sub-feature of the corresponding coarse-grained feature in at least one feature dimension;

[0017] Inputting the input data and the multimedia features into a first processing model to obtain an index set corresponding to the target intent output by the first processing model, wherein the index set includes at least one index information;

[0018] The story logic represented by the index information in the index set matches the target intent.

[0019] In the above method, preferably, the coarse-grained features are obtained during the process of collecting the multimedia data;

[0020] The fine-grained features are obtained by extracting features using a second processing model after the multimedia data is collected; the second processing model extracts the fine-grained features from the multimedia data according to the coarse-grained features;

[0021] The feature dimension includes at least one of sound, picture and optical flow.

[0022] In the above method, preferably, the coarse-grained features are saved in a text format; and the fine-grained features are saved in a binary format.

[0023] In the above method, preferably, the index information at least includes: index extraction attributes; the index extraction attributes include image identification attributes corresponding to image types or extraction timestamp attributes corresponding to video types;

[0024] Wherein, based on the index information in the index set, extracting at least one target image corresponding to each index information from the multimedia data in the resource set includes at least one of the following:

[0025] Using a third processing model, according to the image identification attribute, extracting at least one target image corresponding to the image identification attribute from the multimedia data of the image type in the resource collection;

[0026] By utilizing the third processing model, at least one target image of a corresponding timestamp is extracted from the multimedia data of the video type in the resource collection according to the extraction timestamp attribute.

[0027] In the above method, preferably, the target image has: a splicing timestamp, the splicing timestamp is determined based on the story logic represented by the index information, the splicing timestamp represents the splicing relationship between the target images, and the target duration composed of the splicing timestamp matches the target intent;

[0028] Wherein, generating a target video based on the target image includes:

[0029] The target image is input into the third processing model so that the third processing model performs splicing according to the splicing timestamp to obtain the target video.

[0030] The above method, preferably, obtains a resource set from a multimedia library according to the input data, including:

[0031] Parsing the input data to obtain at least one associated feature; wherein the associated feature includes: at least one or any multiple of time, person, and location;

[0032] By using the fourth processing model, at least one multimedia data is screened out from the multimedia library according to the association feature to obtain a resource set, wherein the multimedia data in the resource set are associated with each other based on the association feature.

[0033] In the above method, preferably, the input data includes data of at least one data type;

[0034] The data type includes at least one of a voice type and a text type.

[0035] A video generation method, comprising:

[0036] Obtaining input data; the input data represents the user's target intention for generating a video;

[0037] According to the input data, a resource set is obtained from a multimedia library, wherein the resource set includes at least one multimedia data, and the multimedia data includes at least one of the following: an image type or a video type;

[0038] At least based on the target model, the input data and the resource set are processed to generate a target video, wherein the target video includes multiple frames of target images that meet the target intent, and the target images come from at least one of the following situations:

[0039] Multimedia data in the resource set that matches the target intent;

[0040] Part of the multimedia data in the resource collection that matches the target intent.

[0041] A video processing device, comprising:

[0042] An interactive engine, configured to obtain input data, wherein the input data represents a user's target intention for generating a video;

[0043] A video processing model is used to obtain a resource set from a multimedia library according to the input data, wherein the resource set includes at least one multimedia data, and the multimedia data includes at least one of the following: an image type or a video type; based on the input data and the resource set, generate an index set corresponding to the target intent;

[0044] The image processing model is used to extract at least one target image corresponding to each index information from the multimedia data in the resource set based on the index information in the index set; and generate a target video based on the target image.

[0045] A video generating device, comprising:

[0046] An interactive engine, configured to obtain input data, wherein the input data represents a user's target intention for generating a video;

[0047] A target model is used to obtain a resource set from a multimedia library according to the input data, wherein the resource set includes at least one multimedia data, and the multimedia data includes at least one of the following: an image type or a video type; and process the input data and the resource set to generate a target video, wherein the target video includes multiple frames of target images that meet the target intent;

[0048] The target image is from at least one of the following situations:

[0049] Multimedia data in the resource set that matches the target intent;

[0050] Part of the multimedia data in the resource collection that matches the target intent.

[0051] An electronic device comprises: a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement any one of the above-mentioned video processing methods.

[0052] A computer storage medium stores a computer program / instruction, which implements any of the above-mentioned video processing methods when executed by a processor.

[0053] A computer program product comprises a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the video processing method described in any one of the above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 A flowchart of a video processing method provided in an embodiment of the present application;

[0056] Figure 2 This is an example diagram of obtaining user input data in an embodiment of the present application;

[0057] Figure 3 This is an example diagram of obtaining a resource set in an embodiment of the present application;

[0058] Figure 4 This is an example diagram of obtaining an index set in an embodiment of the present application;

[0059] Figure 5 This is an example diagram of extracting coarse-grained features and fine-grained features in an embodiment of the present application;

[0060] Figure 6 This is an example diagram of extracting a target image according to index information in an embodiment of the present application;

[0061] Figure 7 This is an example diagram of stitching target images to generate a target video in an embodiment of the present application;

[0062] Figure 8 Another flow chart of a video processing method provided in an embodiment of the present application;

[0063] Fig. 9 A flowchart of a video generation method provided in an embodiment of the present application;

[0064] Fig.10 A schematic diagram of the structure of a video processing device provided in an embodiment of the present application;

[0065] Fig.11 A schematic diagram of the structure of a video generating device provided in an embodiment of the present application;

[0066] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0067] Fig.13 This is an example of a number of video clips of a celebratory occasion;

[0068] Fig.14 This is a schematic diagram of the application being applicable to interactive quick video editing based on user intent on a mobile phone;

[0069] Fig.15 This is an example diagram of the application for implementing segmented video feature extraction on a mobile phone;

[0070] Fig.16 A flowchart of the application for interactive video story generation based on coarse-grained features on a mobile phone;

[0071] Fig.17 This is a flowchart of the application for interactive video story generation based on coarse-grained features combined with some fine-grained features on a mobile phone;

[0072] Fig.18 This is a flowchart of the interactive video story generation based on coarse-grained features and complete fine-grained features applicable to this application on a mobile phone. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0074] refer to Figure 1 As shown, it is a flow chart of a video processing method provided in an embodiment of the present application, which can be applied to electronic devices capable of data processing, such as mobile phones, tablet devices, notebooks, servers, etc. The technical solution in this embodiment is mainly used to reduce the complexity of user operations when generating videos, so as to improve the user experience.

[0075] Specifically, the method in this embodiment may include the following process:

[0076] Step 101: Obtain input data.

[0077] The input data represents the user's target intention for generating videos.

[0078] Specifically, in this embodiment, an interactive engine such as a voice assistant in an electronic device can be used to provide an interactive interface for the user, such as Figure 2 As shown in , the user can input content in the input box in the interactive interface, so that the user's input data can be obtained in this embodiment. The obtained input data represents the user's target intention, and the target intention represents the intention of generating a video.

[0079] For example, if the input data is "I want a 10-second wedding video", the target intent represented is: generate a 10-second wedding video. For another example, if the input data is "I want a 20-second team-building activity", the target intent represented is: generate a 20-second team-building activity video.

[0080] It should be noted that the input data may include data of at least one data type. The data type may include at least one of a voice type and a text type. For example, the input data is a voice of "I want a 10-second wedding"; for another example, the input data is a text of "20-second team-building activity".

[0081] Step 102: Obtain a resource set from a multimedia library according to the input data.

[0082] The resource set includes at least one multimedia data, and the multimedia data includes at least one of the following: image type data or video type data.

[0083] For example, taking a mobile phone as an example, the multimedia library is a mobile phone album, which contains multiple images or videos. Based on this, in this embodiment, according to the user's input data "want a 10-second wedding", images or videos matching "wedding" are obtained from the mobile phone album to obtain a resource set, that is, a set consisting of all images or videos related to "wedding" in the mobile phone album or in the past week, such as Figure 3 as shown in .

[0084] In one implementation, in step 102, the input data may be first parsed to obtain at least one associated feature; then, using the fourth processing model, at least one multimedia data may be screened out from the multimedia library according to the associated feature to obtain a resource set.

[0085] The associated features include at least one or any multiple of time, person, and location. For example, by parsing the input data "I want a 10-second wedding video", we can get the two keywords "10 seconds" and "wedding". According to these keywords, we can get the corresponding associated features, such as "bride", "groom", "bridesmaid", "best man", "hall", "wedding candles", "wedding photos" and other associated features.

[0086] Specifically, the fourth processing model can be a multimodal visual language model, and the trained visual language model can process the input association features and the multimedia library to output multimedia data that matches the association features selected from the multimedia library, and these selected multimedia data constitute a resource set. Based on this, the multimedia data in the resource set are associated with each other in the association features.

[0087] For example, the visual language model filters out images or videos in the mobile phone album that match related features such as "bride", "groom", "bridesmaid", "best man", "auditorium", "wedding candles", and "wedding photos", thereby associating images or videos in the mobile phone album that match the input data "I want a 10-second wedding video" based on the related features corresponding to "wedding".

[0088] Step 103: Generate an index set corresponding to the target intent based on the input data and the resource set.

[0089] The index set contains at least one index information. The story logic represented by the index information in the index set matches the target intent. For example, the index set contains index information corresponding to the story plots such as "bridegroom enters", "bride enters", "witness", "worship in the hall" and "toast", such as Figure 4 As shown in , the story logic represented by these index information is: the hosting process of a wedding, which matches the target intent of "generating a 10-second wedding video".

[0090] Specifically, in step 103, each multimedia data in the resource set may be parsed, and then according to the parsing result obtained by the parsing and the input data, index information matching the target intent may be constructed, that is, an index set may be obtained.

[0091] In one implementation, in step 103, multimedia data corresponding to the multimedia data in the resource set may be first obtained, and then the input data and multimedia features may be input into the first processing model to obtain an index set corresponding to the target intent output by the first processing model.

[0092] The multimedia features may include: coarse-grained features and at least one of the fine-grained features corresponding to the coarse-grained features; the fine-grained features include: sub-features of the corresponding coarse-grained features in at least one feature dimension; the feature dimension may include: at least one of sound, picture, and optical flow. For example, the coarse-grained features are the features of multimedia data in dimensions such as scenes, characters, pets, focus, and transitions; the fine-grained features include sub-features of the corresponding coarse-grained features in dimensions such as pictures, sounds, and optical flows.

[0093] It should be noted that the coarse-grained features are obtained during the process of collecting multimedia data, while the fine-grained features are obtained by extracting features using the second processing model after the multimedia data is collected. The second processing model can extract corresponding fine-grained features from the multimedia data based on the coarse-grained features.

[0094] Specifically, the coarse-grained features may be multimedia features obtained when multimedia data is collected in any image processing scenario. Afterwards, after the multimedia data is collected and saved, the second processing model in the electronic device where the multimedia data is located may extract fine-grained features from the saved multimedia data. It should be noted that after the coarse-grained features are identified, they are associated with the corresponding multimedia data, and after the fine-grained features are extracted, they are also associated with the corresponding multimedia data.

[0095] For example, Figure 5 As shown in , in the face recognition scenario, in the process of collecting multiple frames of images or videos composed of multiple video frames through a mobile phone camera, the face (person) within the collection range is recognized through a face recognition algorithm or a machine learning model, and the facial features obtained through face recognition (there may also be scene, pet, focus, transition and other features) are coarse-grained features. The coarse-grained features are associated with the corresponding image or video (MP4). After that, the hybrid expert model is used to extract features in feature dimensions such as picture, sound and optical flow from the image or video associated with the coarse-grained features to obtain fine-grained features, and then the fine-grained features are associated with the corresponding image or video.

[0096] The second processing model may be a hybrid expert model, which processes the collected multimedia data and coarse-grained features, thereby extracting fine-grained features more quickly.

[0097] In addition, coarse-grained features can be saved in a text format such as JSON, while fine-grained features can be saved in a binary format such as NPY.

[0098] The first processing model may be a multimodal visual language model. The trained visual language model may process input data and multimedia features to obtain at least one index information corresponding to the target intent to form an index set.

[0099] It should be noted that, in this embodiment, before the fine-grained features are extracted by the hybrid expert model, the visual language model can be used to process only the input data and the coarse-grained features to obtain an index set, thereby reducing the time consumption caused by waiting for the extraction of fine-grained features, thereby speeding up the efficiency of obtaining the index set, and further improving the efficiency of subsequent generation of the target video;

[0100] Alternatively, in this embodiment, when the hybrid expert model only extracts part of the fine-grained features, the visual language model can be used to process the input data, the coarse-grained features and part of the fine-grained features to obtain an index set, thereby generating an index set based on the input data and the coarse-grained features in combination with the part of the fine-grained features, thereby improving the accuracy of the generated index set, thereby improving the accuracy of the target video generated subsequently, and at the same time, reducing part of the time consumption caused by waiting for the extraction of fine-grained features, thereby speeding up the efficiency of obtaining the index set to a certain extent, thereby improving the efficiency of generating the target video subsequently;

[0101] Alternatively, in this embodiment, when the hybrid expert model extracts all the fine-grained features, the visual language model can be used to process only the input data and coarse-grained features to obtain an index set. In this way, the index set can be generated based on the input data and coarse-grained features combined with the complete fine-grained features, thereby further improving the accuracy of the generated index set, thereby improving the accuracy of the subsequently generated target video.

[0102] Step 104: based on the index information in the index set, extract at least one target image corresponding to each index information from the multimedia data in the resource set.

[0103] The target image corresponding to each index information can express the corresponding storyline. Figure 6 As shown in , the target image corresponding to the index information corresponding to the storyline of "witnessing a wedding" is an image of the wedding scene. Therefore, the target image corresponding to the index information can express the storyline of "witnessing a wedding".

[0104] For example, in this embodiment, for each index information, partial video segments or video frames in the image corresponding to the index information or the video corresponding to the index information can be extracted from all images or videos related to "wedding" screened out from the mobile phone album to obtain at least one frame of target image corresponding to the index information.

[0105] Specifically, the index information at least includes: index extraction attributes. The index extraction attributes include: image identification attributes corresponding to the image type, or extraction timestamp attributes corresponding to the video type. The image identification attribute corresponding to the image type can be an image name or number, such as the image "camera_123456". The extraction timestamp attribute corresponding to the video type also corresponds to a video identification attribute. The video identification attribute can be a video name or encoding, such as the video "camera_987654". The corresponding extraction timestamp attribute can represent the timing corresponding to the video frame in the video to which the corresponding video identification attribute belongs, such as the 6th and 9th seconds in the video "camera_987654".

[0106] Based on this, when extracting the target image in step 104, it can be achieved by at least one of the following:

[0107] In one implementation, the third processing model may be used in step 104 to extract at least one target image corresponding to the image identification attribute from the multimedia data of the image type in the resource collection according to the image identification attribute.

[0108] In another implementation, the third processing model may be used in step 104 to extract at least one target image of a corresponding timestamp from the multimedia data of the video type in the resource collection according to the extraction timestamp attribute.

[0109] The third processing model may be a video editing engine. The video editing engine may extract a target image having the same image identification attribute from a resource collection according to the image identification attribute. The video editing engine may also extract a target image having a corresponding timestamp from multimedia data of a video type according to the extraction timestamp attribute.

[0110] For example, in this embodiment, a video editing engine is pre-deployed. The video editing engine can extract images corresponding to the image identification attributes in the index information from all images matching "wedding" in the mobile phone album, i.e., the target image; and the video editing engine can also capture video frames whose timestamps are consistent with the timestamp attributes extracted from the index information from all videos matching "wedding" in the mobile phone album, such as capturing video frames at the 6th and 9th seconds in a wedding video clip, i.e., the target image.

[0111] It should be noted that the target image extracted in step 104 has a splicing timestamp. The splicing timestamp is determined based on the story logic represented by the index information. For example, the splicing timestamp of the target image corresponding to the index information with an earlier storyline is earlier, while the splicing timestamp of the target image corresponding to the index information with a later storyline is later.

[0112] For example, Figure 7 As shown in , the splicing timestamp of the target image extracted according to the index information corresponding to “bride enters” is smaller than the splicing timestamp of the target image extracted according to the index information corresponding to “witness the wedding”.

[0113] The splicing timestamps of the target images extracted according to the same index information can be determined according to the time when the target images were captured. For example, if there are multiple target images extracted according to the index information corresponding to "bride entering the venue", the splicing timestamps of these target images can be determined according to the time when they were captured by the mobile phone camera.

[0114] Step 105: Generate a target video based on the target image.

[0115] Among them, the splicing timestamps of different target images are different, and there is a sequence between the splicing timestamps of each target image (i.e., a small-large relationship). Accordingly, the splicing timestamp can represent the splicing relationship between the target images, and the target duration composed of the splicing timestamps matches the target intention.

[0116] For example, if the target intent is to generate a 10-second wedding video, then the target duration composed of the spliced ​​timestamps of each target image is "10 seconds".

[0117] Specifically, in step 105, the target images may be spliced ​​in sequence according to the splicing timestamps of the target images to obtain the target video. Based on this, the target video contains video images of each storyline in the story logic represented by the index information to meet the user's target intention for generating the video.

[0118] In one implementation, in step 105, the target image may be input into the third processing model, so that the third processing model performs splicing according to the splicing timestamp to obtain the target video.

[0119] For example, Figure 7 As shown in , taking the third processing model as a video editing engine as an example, after extracting the target images, the video editing engine splices the target images according to the splicing timestamps of the target images to obtain a wedding video that matches the target intent of “generating a 10-second wedding video”.

[0120] It can be seen from the above technical solution that in a video processing method provided by an embodiment of the present application, by obtaining input data representing the user's target intention for generating a video, a resource set containing multimedia data can be obtained from a multimedia library based on the input data, and then an index set corresponding to the target intention is generated based on the input data and the resource set, so that based on the index information in the index set, at least one target image corresponding to each index information is extracted from the multimedia data in the resource set, and finally a target video is generated based on these target images. It can be seen that in the present application, only the user needs to input data representing the target intention, and the user does not need to perform other operations. According to the user's target intention for generating a video, the corresponding target image can be extracted from the existing multimedia data based on the index information corresponding to the target intention, and then the target video can be generated, thereby reducing the user's operation complexity and improving the user experience.

[0121] Based on the above implementation scheme, after step 105, the method in this embodiment can also be processed as follows: Figure 8 As shown in:

[0122] Step 106: Output the target video.

[0123] Among them, in this embodiment, the target video can be output to the user's interactive interface, so that the user can view the generated target video on the interactive interface, and can also propose corresponding adjustment operations on the target video. Based on this, in this embodiment, adjustment data for the target video can be generated in response to the adjustment operation.

[0124] For example, in this embodiment, the "wedding video" that has been generated can be output to the user through the video player. If the user is not satisfied with the "wedding video", the input data previously entered can be modified on the interactive interface. For example, if the user thinks the video is too long, the initially entered input data "generate a 10-second wedding video" can be modified to "generate a 5-second wedding video"; or, the user can enter new data on the interactive interface, such as "remove the witness part".

[0125] Step 107: Obtain adjustment data for the target video.

[0126] The adjustment data represents the user's intention to adjust the target video, and may specifically include: the adjustment data corresponding to the user's modification operation on the initial input data in the interactive interface, or the new data input by the user in the interactive interface. For example, the data corresponding to the modification operation of changing "10" to "5"; for another example, the new data: "remove the part about the wedding ceremony".

[0127] Step 108: At least based on the adjustment data, update data is obtained, and the updated data is used as input data, and the process returns to step 102 to obtain a resource set from the multimedia library based on the new input data, thereby obtaining a new target video. If the user continues to adjust the target video after outputting the target video, the above process can be continued to generate a new target video until the user believes that the generated target video meets the current needs and no adjustment is performed on the target video.

[0128] In this embodiment, when obtaining the updated data, the input data may be modified according to the adjustment data to obtain the updated data.

[0129] For example, in this embodiment, according to the data corresponding to the modification operation of modifying "10" to "5", the input data "generate a 10-second wedding video" is updated to obtain the updated data: "generate a 5-second wedding video", and then the operation is re-executed based on the updated data. Figure 1 The process shown is used to generate a new target video, i.e. a 5-second wedding video;

[0130] For another example, in this embodiment, according to the new data "remove the witness part", the initial input data "generate a 10-second wedding video" is adjusted to obtain the updated data: "generate a 10-second wedding video without the witness part of the wedding", and then, based on the updated data, re-execute Figure 1 The process shown is used to generate a new target video, that is, a 10-second wedding video without a witness.

[0131] In another implementation, in this embodiment, after obtaining the adjustment data, the adjustment data can be input into the first processing model, and the first processing model outputs a new index set based on the input data, the adjustment data and the multimedia features. Then, based on the index information in the new index set, the third processing model is used to re-extract at least one target image corresponding to each index information from the multimedia data in the resource collection, and then these target images are spliced ​​to generate a new target video.

[0132] For example, in this embodiment, the new data "remove the witness part" is input into the visual language model, and the visual language model outputs a new index set. The video editing engine is then used to extract new target images from images or videos related to "wedding" based on the new index set, and then these target images are spliced ​​to generate a new target video, that is, a 10-second wedding video without the witness segment.

[0133] In another implementation, in this embodiment, after obtaining the adjustment data, the adjustment data can be input into a third processing model, and the third processing model processes the target images previously extracted from the resource collection, such as eliminating part of the target images or extracting new target images from the resource collection, and then splicing these target images to generate a new target video.

[0134] For example, in this embodiment, the new data "remove the witness part" is input into the video editing engine, and the video editing engine edits the target image extracted from the image or video related to the "wedding", such as deleting the target image of "witness", and then splicing the remaining target images to generate a new target video, that is, a 10-second wedding video without the witness segment.

[0135] It can be seen that in this embodiment, interaction with the user can be carried out, and the visual language model and the video editing engine can be used to generate multiple target videos, so that the generated target videos can better meet the needs of the user.

[0136] refer to Fig. 9, is a flow chart of a method for generating a video provided in an embodiment of the present application, which can be applied to electronic devices capable of data processing, such as mobile phones, tablet devices, notebooks, and servers. The technical solution in this embodiment is mainly used to reduce the complexity of user operations when generating videos to improve the user experience.

[0137] Specifically, the method in this embodiment may include the following process:

[0138] Step 901: Obtain input data.

[0139] The input data represents the user's target intention for generating videos.

[0140] Specifically, in this embodiment, an interactive interface can be provided to the user, such as Figure 2 As shown in , the user can input content in the input box in the interactive interface, so that the user's input data can be obtained in this embodiment. The obtained input data represents the user's target intention, and the target intention represents the intention of generating a video.

[0141] For example, if the input data is "I want a 10-second wedding video", the target intent represented is: generate a 10-second wedding video. For another example, if the input data is "I want a 20-second team-building activity", the target intent represented is: generate a 20-second team-building activity video.

[0142] It should be noted that the input data may include data of at least one data type. The data type may include at least one of a voice type and a text type. For example, the input data is a voice of "I want a 10-second wedding"; for another example, the input data is a text of "20-second team-building activity".

[0143] Step 902: Obtain a resource set from a multimedia library according to input data.

[0144] The resource set includes at least one multimedia data, and the multimedia data includes at least one of the following: image type data or video type data.

[0145] For example, taking a mobile phone as an example, the multimedia library is a mobile phone album, which contains multiple images or videos. Based on this, in this embodiment, according to the user's input data "want a 10-second wedding", images or videos matching "wedding" are obtained from the mobile phone album to obtain a resource set, that is, a set consisting of all images or videos related to "wedding" in the mobile phone album or in the past week, such as Figure 3 as shown in .

[0146] In one implementation, in step 902, the input data may be first parsed to obtain at least one associated feature; then, using the fourth processing model, at least one multimedia data may be screened out from the multimedia library according to the associated feature to obtain a resource set.

[0147] The associated features include at least one or any multiple of time, person, and location. For example, by parsing the input data "I want a 10-second wedding video", we can get the two keywords "10 seconds" and "wedding". According to these keywords, we can get the corresponding associated features, such as "bride", "groom", "bridesmaid", "best man", "hall", "wedding candles", "wedding photos" and other associated features.

[0148] Specifically, the fourth processing model can be a multimodal visual language model, and the trained visual language model can process the input association features and the multimedia library to output multimedia data that matches the association features selected from the multimedia library, and these selected multimedia data constitute a resource set. Based on this, the multimedia data in the resource set are associated with each other in the association features.

[0149] For example, the visual language model filters out images or videos in the mobile phone album that match related features such as "bride", "groom", "bridesmaid", "best man", "auditorium", "wedding candles", and "wedding photos", thereby associating images or videos in the mobile phone album that match the input data "I want a 10-second wedding video" based on the related features corresponding to "wedding".

[0150] Step 903: Process the input data and the resource set at least based on the target model to generate a target video.

[0151] The target video contains multiple frames of target images that meet the target intent, and the target images come from at least one of the following situations:

[0152] Multimedia data in the resource collection that matches the target intent, such as image-type multimedia data;

[0153] Part of the multimedia data in the resource collection that matches the target intent, such as at least part of the video frames in the multimedia data of the video type.

[0154] Specifically, the target model may include a visual language model, a video editing engine, a hybrid expert model, etc. After being trained, the target model can process input data and resource collections to generate a target video.

[0155] For example, a visual language model is first used to process the multimedia features corresponding to the input data and the multimedia data in the resource collection to obtain an index set. Then, a video editing engine is used to extract attributes according to the index in the index information to extract at least one target image corresponding to each index information from the resource collection. The video editing engine then splices these target images according to the splicing timestamps of the target images to obtain the target video.

[0156] It should be noted that the specific implementation of step 903 in this embodiment can refer to the corresponding contents of steps 103 to 105 in the previous text, and will not be described in detail here.

[0157] It can be seen from the above technical solution that in a video generation method provided by an embodiment of the present application, by obtaining input data representing the user's target intention for generating a video, a resource set containing multimedia data can be obtained from a multimedia library based on the input data, and then the input data and the resource set are processed based on the target model to generate a target video. It can be seen that in the present application, only the user needs to input data representing the target intention, and the user does not need to perform other operations. According to the user's target intention for generating a video, the corresponding target image can be extracted from the existing multimedia data, and the target video can be generated, thereby reducing the user's operation complexity and improving the user experience.

[0158] refer to Fig.10 , is a schematic diagram of the structure of a video processing device provided in an embodiment of the present application, which can be configured in an electronic device capable of data processing, such as a mobile phone, a tablet device, a notebook, and a server. The technical solution in this embodiment is mainly used to reduce the complexity of user operations when generating videos to improve the user experience.

[0159] Specifically, the device in this embodiment may include the following units:

[0160] The interactive engine 1001, such as a voice assistant, is used to obtain input data; the input data represents the user's target intention for generating a video;

[0161] The video processing model 1002, such as a visual language model, is used to obtain a resource set from a multimedia library according to the input data, wherein the resource set includes at least one multimedia data, and the multimedia data includes at least one of the following: an image type or a video type; based on the input data and the resource set, generate an index set corresponding to the target intent;

[0162] The image processing model 1003, such as a video editing engine, is used to extract at least one target image corresponding to each index information from the multimedia data in the resource set based on the index information in the index set; and generate a target video based on the target image.

[0163] It can be seen from the above technical solution that in a video processing device provided by an embodiment of the present application, by obtaining input data representing the user's target intention for generating a video, a resource set containing multimedia data can be obtained from a multimedia library based on the input data, and then an index set corresponding to the target intention is generated based on the input data and the resource set, so that based on the index information in the index set, at least one target image corresponding to each index information is extracted from the multimedia data in the resource set, and finally a target video is generated based on these target images. It can be seen that in this embodiment, the user only needs to input data representing the target intention, and the user does not need to perform other operations. According to the user's target intention for generating a video, the corresponding target image can be extracted from the existing multimedia data based on the index information corresponding to the target intention, and then the target video can be generated, thereby reducing the user's operation complexity and improving the user experience.

[0164] refer to Fig.11 , is a schematic diagram of the structure of a video generation device provided in an embodiment of the present application, which can be configured in an electronic device capable of data processing, such as a mobile phone, a tablet device, a notebook, and a server. The technical solution in this embodiment is mainly used to reduce the complexity of user operations when generating videos to improve the user experience.

[0165] Specifically, the device in this embodiment may include the following units:

[0166] The interactive engine 1101, such as a voice assistant, is used to obtain input data; the input data represents the user's target intention for generating a video;

[0167] The target model 1102 is used to obtain a resource set from a multimedia library according to the input data, wherein the resource set includes at least one multimedia data, and the multimedia data includes at least one of the following: an image type or a video type; process the input data and the resource set to generate a target video, wherein the target video includes multiple frames of target images that meet the target intent;

[0168] The target image is from at least one of the following situations:

[0169] Multimedia data in the resource set that matches the target intent;

[0170] Part of the multimedia data in the resource collection that matches the target intent.

[0171] It can be seen from the above technical solution that in a video generation device provided by an embodiment of the present application, by obtaining input data representing the user's target intention for generating a video, a resource set containing multimedia data can be obtained from a multimedia library based on the input data, and then the input data and the resource set are processed based on the target model to generate a target video. It can be seen that in the present application, only the user needs to input data representing the target intention, and the user does not need to perform other operations. According to the user's target intention for generating a video, the corresponding target image can be extracted from the existing multimedia data, and the target video can be generated, thereby reducing the user's operation complexity and improving the user experience.

[0172] In one implementation, after the video editing engine generates a target video based on the target image, the interactive engine is further used to output the target video; obtain adjustment data for the target video, the adjustment data representing the user's adjustment intention for the target video;

[0173] At least according to the adjustment data, update data is obtained, and the update data is used as the input data to return to trigger the visual language model to obtain a resource set from a multimedia library according to the input data.

[0174] In one implementation, when the visual language model generates an index set corresponding to the target intent based on the input data and the resource set, it is specifically used to: obtain multimedia features corresponding to the multimedia data in the resource set, the multimedia features including: coarse-grained features and at least one of the fine-grained features corresponding to the coarse-grained features, the fine-grained features including: sub-features of the corresponding coarse-grained features on at least one feature dimension; input the input data and the multimedia features into a first processing model to obtain an index set corresponding to the target intent output by the first processing model, the index set containing at least one index information; wherein the story logic represented by the index information in the index set matches the target intent.

[0175] The coarse-grained features are obtained during the process of collecting the multimedia data; the fine-grained features are obtained by extracting features using a second processing model after the multimedia data is collected; the second processing model extracts the fine-grained features from the multimedia data based on the coarse-grained features; the feature dimensions include at least one of sound, picture, and optical flow. The coarse-grained features are saved in a text format; the fine-grained features are saved in a binary format.

[0176] In one implementation, the index information at least includes: an index extraction attribute; the index extraction attribute includes an image identification attribute corresponding to an image type or an extraction timestamp attribute corresponding to a video type;

[0177] Wherein, when the video editing engine extracts at least one target image corresponding to each index information from the multimedia data in the resource set based on the index information in the index set, at least one of the following processes is included:

[0178] Using a third processing model, according to the image identification attribute, extracting at least one target image corresponding to the image identification attribute from the multimedia data of the image type in the resource collection;

[0179] By utilizing the third processing model, at least one target image of a corresponding timestamp is extracted from the multimedia data of the video type in the resource collection according to the extraction timestamp attribute.

[0180] The target image has: a splicing timestamp, the splicing timestamp is determined based on the story logic represented by the index information, the splicing timestamp represents the splicing relationship between the target images, and the target duration composed of the splicing timestamp matches the target intent;

[0181] Among them, when the video editing engine generates the target video based on the target image, it is specifically used to: input the target image into the third processing model, so that the third processing model performs splicing according to the splicing timestamp to obtain the target video.

[0182] In one implementation, when the visual language model obtains a resource set from a multimedia library based on the input data, it is specifically used to: parse the input data to obtain at least one associated feature; wherein the associated feature includes: at least one or any multiple of time, person, and location; and use a fourth processing model to filter out at least one multimedia data from the multimedia library according to the associated feature to obtain a resource set, wherein the multimedia data in the resource set are associated with each other based on the associated feature.

[0183] The input data includes data of at least one data type; the data type includes at least one of a voice type and a text type.

[0184] refer to Fig.12 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, and the electronic device may include the following structure:

[0185] Memory, used to store computer programs and data generated by the operation of computer programs;

[0186] The processor is used to execute a computer program to implement the video processing method or video generating method described in any of the above embodiments.

[0187] Taking a mobile phone as an example, the technical solution of the present application is described below:

[0188] like Fig.13 As shown in the video, in celebrations, photographers often use different angles, positions, and video clips to record the happy moments of the day. As a result, many videos are accumulated in the album, and most of them are not suitable for direct sharing in terms of time length, volume, or the degree of summary of the event.

[0189] To this end, currently, the personalized "memories" function in the album is mostly used to "mix and edit" pictures and videos according to time, people, events, locations and other information, and play them in the form of stories with music and animation. However, this "memories" function requires a lot of manual operations from users, especially video export and sharing.

[0190] In view of this, this application can use modules such as AI voice assistant, multimodal visual language model and AI video editing engine to achieve interaction and quickly edit short videos based on user intentions.

[0191] like Fig.14 As shown in , users have recorded a lot of long videos, which are not suitable for direct sharing on social media, but manual editing is time-consuming and laborious. To this end, based on the technical solution of the present application, the user conveys the intention to share through a voice assistant, such as "make a 10-second follow-up shot of the dog", and then the AI's multimodal video understanding engine, i.e., the visual language model, searches for clues in the recorded video that meet the user's expectations. Finally, the AI's video editing engine edits a short video suitable for sharing based on the clues and displays it to the user. Afterwards, the user uses the voice assistant to adjust the intention or re-enter the intention according to the displayed short video, and accordingly re-finds clues, re-edits the short video, and re-displays the short video until the user is satisfied.

[0192] Among them, the Vision Language Model (VLM) or Vision Large Language Model has the ability to understand videos, generate summaries, generate story clues, and conduct interactive dialogues, and is an important module for implementing this proposal. Video Feature Retrieval is the basis for the Vision Language Model to achieve the above capabilities, which usually requires a lot of computing power.

[0193] In response to the defects of existing visual language models: Existing visual language models all extract video features from scratch during the video consumption stage. This application fully utilizes the hardware and algorithm capabilities of the video production stage, divides the video features into two stages, coarse and fine, and extracts them separately, greatly improving the efficiency of video feature extraction and laying a good foundation for subsequent visual language models to quickly understand videos.

[0194] Among them, during the video production stage, the camera itself can extract many video features (people / pets / scenes / focus / transitions, etc.). These video features are for recording services and are also discarded as the recording progresses. In this application, these features carried in the video are directly used, and there is no need to re-extract these video features, thereby improving the efficiency of video feature extraction.

[0195] like Fig.15 As shown in the figure, it is the process of segmented video feature extraction:

[0196] ① Real-time coarse-grained feature extraction during recording; ② Background fine-grained feature extraction after recording.

[0197] Among them, multiple video frames are collected by the camera according to time, and a video file (MP4) is generated through a video encoder. In this process, a coarse-grained feature extractor is used to extract coarse-grained features (JSON) in dimensions such as scenes, characters, pets, focus, and transitions in real time. Coarse-grained video features are generated and associated with video files synchronously. Fine-grained video features are analyzed by the background service calling the mixed expert model (Mixture of Experts), such as analyzing fine-grained features (NPY) in picture, sound, optical flow and other dimensions. With the input of coarse-grained video features, the mixed expert model can complete the extraction of fine-grained video features more quickly. The fine-grained features are extracted and then associated with the video file.

[0198] If the user's requirement for video editing is to start after the video is recorded, then the fine-grained features may not have been extracted yet, or have been only partially extracted. Therefore, in this embodiment, interactive video editing is performed in the following three situations according to the extraction status of fine-grained features.

[0199] like Fig.16 As shown in the figure, it is a flowchart of interactive video story generation based on coarse-grained features. If the videos or images in the resource collection have not yet started to extract fine-grained features, then the coarse-grained features in the videos or images can be used to achieve rapid response, thereby achieving rapid video generation, but the generated video may deviate from the user's intention. The specific process is as follows:

[0200] ① The user enters data on the screen to represent the user's intention;

[0201] ②The user interaction engine obtains the user's input data and sends it to the multimodal visual language model;

[0202] ③ The visual language model selects images or videos that match the intent from the album according to the input data, and then associates videos or images of the same subject that are close in time to form a resource collection;

[0203] ④ Provide the resource collection to the multimodal visual language model, which constructs story clues based on the coarse-grained features (JSON) of the images or videos in the resource collection;

[0204] ⑤ Provide the resource collection to the video editing engine, and at the same time, provide the story clues to the video editing engine, that is, the index information mentioned above;

[0205] ⑥The video editing engine extracts the target image according to the index information and generates the target video (mp4);

[0206] ⑦Send the target image to the video player;

[0207] ⑧The video player outputs the target video to the screen so that the user can make modification suggestions for the target video, and returns to ① to regenerate the target video, and repeats this cycle until the user is satisfied.

[0208] like Fig.17 As shown in the figure, it is a flowchart of interactive video story generation based on coarse-grained features combined with some fine-grained features. If the video or image in the resource collection has started to extract fine-grained features but has not yet completed the fine-grained feature extraction, then the coarse-grained features and some fine-grained features in the video or image can be used to achieve a quick response, thereby achieving fast video generation. The generated video is faster than Fig.15 The process shown generates a video that is closer to the user's intention. The specific process is as follows:

[0209] ① The user enters data on the screen to represent the user's intention;

[0210] ②The user interaction engine obtains the user's input data and sends it to the multimodal visual language model;

[0211] ③ The visual language model selects images or videos that match the intent from the album according to the input data, and then associates videos or images of the same subject that are close in time to form a resource collection;

[0212] ④ Provide the resource collection to the multimodal visual language model, which constructs story clues based on the coarse-grained features (JSON) and some fine-grained features (NPY) of the images or videos in the resource collection;

[0213] ⑤ Provide the resource collection to the video editing engine, and at the same time, provide the story clues to the video editing engine, that is, the index information mentioned above;

[0214] ⑥The video editing engine extracts the target image according to the index information and generates the target video (mp4);

[0215] ⑦Send the target image to the video player;

[0216] ⑧The video player outputs the target video to the screen so that the user can make modification suggestions for the target video, and returns to ① to regenerate the target video, and repeats this cycle until the user is satisfied.

[0217] like Fig.18 As shown in , it is a flowchart of interactive video story generation based on coarse-grained features and complete fine-grained features. If the video or image in the resource collection has started and completed the extraction of fine-grained features, then the coarse-grained features and complete fine-grained features in the video or image can be used to achieve accurate response, thereby achieving video generation while ensuring that the generated video is more accurate, that is, the generated video best matches the user's intention. The specific process is as follows:

[0218] ① The user enters data on the screen to represent the user's intention;

[0219] ②The user interaction engine obtains the user's input data and sends it to the multimodal visual language model;

[0220] ③ The visual language model selects images or videos that match the intent from the album according to the input data, and then associates videos or images of the same subject that are close in time to form a resource collection;

[0221] ④ Provide the resource collection to the multimodal visual language model, which constructs the story clues based on the complete fine-grained features (NPY) of the images or videos in the resource collection (which may also include coarse-grained features (JSON));

[0222] ⑤ Provide the resource collection to the video editing engine, and at the same time, provide the story clues to the video editing engine, that is, the index information mentioned above;

[0223] ⑥The video editing engine extracts the target image according to the index information and generates the target video (mp4);

[0224] ⑦Send the target image to the video player;

[0225] ⑧The video player outputs the target video to the screen so that the user can make modification suggestions for the target video, and returns to ① to regenerate the target video, and repeats this cycle until the user is satisfied.

[0226] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0227] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0228] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0229] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video processing method, comprising: Obtaining input data; the input data represents the user's target intention for generating a video; According to the input data, a resource set is obtained from a multimedia library, wherein the resource set includes at least one multimedia data, and the multimedia data includes at least one of the following: an image type or a video type; Based on the input data and the resource set, generate an index set corresponding to the target intent; Based on the index information in the index set, extract at least one target image corresponding to each of the index information from the multimedia data in the resource set; A target video is generated based on the target image.

2. The method according to claim 1, after generating a target video based on the target image, the method further comprises: Outputting the target video; Obtaining adjustment data for the target video, wherein the adjustment data represents a user's adjustment intention for the target video; At least according to the adjustment data, update data is obtained, the update data is used as the input data, and the step of returning to execute the step of obtaining a resource set from a multimedia library according to the input data is performed.

3. The method according to claim 1 or 2, generating an index set corresponding to the target intent based on the input data and the resource set, comprising: Obtaining multimedia features corresponding to the multimedia data in the resource set, the multimedia features comprising: a coarse-grained feature and at least one of a fine-grained feature corresponding to the coarse-grained feature, the fine-grained feature comprising: a sub-feature of the corresponding coarse-grained feature in at least one feature dimension; Inputting the input data and the multimedia features into a first processing model to obtain an index set corresponding to the target intent output by the first processing model, wherein the index set includes at least one index information; The story logic represented by the index information in the index set matches the target intent.

4. The method according to claim 3, wherein the coarse-grained features are obtained during the process of collecting the multimedia data; The fine-grained features are obtained by extracting features using a second processing model after the multimedia data is collected; the second processing model extracts the fine-grained features from the multimedia data according to the coarse-grained features; in, The feature dimension includes at least one of sound, picture and optical flow.

5. According to the method of claim 3, the coarse-grained features are saved in text format; and the fine-grained features are saved in binary format.

6. The method according to claim 1, wherein the index information at least comprises: Index extraction attributes; The index extraction attribute includes an image identification attribute corresponding to an image type or an extraction timestamp attribute corresponding to a video type; Wherein, based on the index information in the index set, extracting at least one target image corresponding to each index information from the multimedia data in the resource set includes at least one of the following: Using a third processing model, according to the image identification attribute, extracting at least one target image corresponding to the image identification attribute from the multimedia data of the image type in the resource collection; By utilizing the third processing model, at least one target image of a corresponding timestamp is extracted from the multimedia data of the video type in the resource collection according to the extraction timestamp attribute.

7. The method according to claim 6, wherein the target image has: a splicing timestamp, the splicing timestamp is determined based on the story logic represented by the index information, the splicing timestamp represents the splicing relationship between the target images, and the target duration composed of the splicing timestamp matches the target intent; in, Generating a target video based on the target image includes: The target image is input into the third processing model so that the third processing model performs splicing according to the splicing timestamp to obtain the target video.

8. A video generation method, comprising: Obtaining input data; the input data represents the user's target intention for generating a video; According to the input data, a resource set is obtained from a multimedia library, wherein the resource set includes at least one multimedia data, and the multimedia data includes at least one of the following: an image type or a video type; At least based on the target model, the input data and the resource set are processed to generate a target video, wherein the target video includes multiple frames of target images that meet the target intent, and the target images come from at least one of the following situations: Multimedia data in the resource set that matches the target intent; Part of the multimedia data in the resource collection that matches the target intent.

9. A video processing device, comprising: An interaction engine, used to obtain input data; The input data represents the user's target intention for generating a video; A video processing model is used to obtain a resource set from a multimedia library according to the input data, wherein the resource set includes at least one multimedia data, and the multimedia data includes at least one of the following: an image type or a video type; based on the input data and the resource set, generate an index set corresponding to the target intent; An image processing model, used for extracting at least one target image corresponding to each index information from multimedia data in the resource set based on the index information in the index set; A target video is generated based on the target image.

10. A video generating device, comprising: An interaction engine, used to obtain input data; The input data represents the user's target intention for generating a video; A target model is used to obtain a resource set from a multimedia library according to the input data, wherein the resource set includes at least one multimedia data, and the multimedia data includes at least one of the following: an image type or a video type; and process the input data and the resource set to generate a target video, wherein the target video includes multiple frames of target images that meet the target intent; The target image is from at least one of the following situations: Multimedia data in the resource set that matches the target intent; Part of the multimedia data in the resource collection that matches the target intent.

Images