Video generation method, electronic device, computer storage medium, and program product

By extracting and fusing multi-scale interactive features from images, poses, and text, multimodal fusion features are generated, solving the problem of content inconsistency in video generation technology, achieving accuracy and coherence of video content, and improving user experience.

CN123123658APending Publication Date: 2026-11-17MIGU CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610976136.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-01
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing video generation technologies struggle to accurately capture the details of images and text, resulting in inconsistencies between the generated content and the original input, which negatively impacts the user's interactive experience.

Method used

The system extracts image features from the input image, pose features of the target object, and text features from the descriptive text. It also determines the interaction features of these features at multiple scales, generates multimodal fusion features through fusion processing, and finally generates the target video.

Benefits of technology

It improves the accuracy and coherence of video content, ensuring that the generated video content not only conforms to the semantics of the descriptive text but also matches the pose of the target object in the original input image, thereby enhancing the user interaction experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123658A_ABST
    Figure CN123123658A_ABST
Patent Text Reader

Abstract

This application provides a video generation method, an electronic device, a computer storage medium, and a program product. The method is applied in the field of artificial intelligence technology. The method includes: acquiring an input image and corresponding descriptive text; the input image includes a target object; extracting image features of the input image, pose features of the target object, and text features of the descriptive text; determining the interaction features between any two pairs of features at multiple scales; fusing the interaction features between any two pairs of features at multiple scales to obtain a first fusion feature between image features and text features, a second fusion feature between pose features and image features, and a third fusion feature between text features and pose features; fusing the first fusion feature, the second fusion feature, and the third fusion feature to obtain a multimodal fusion feature; and generating a target video based on the multimodal fusion feature.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a video generation method, electronic device, computer storage medium, and program product. Background Technology

[0002] With the development of generative artificial intelligence, video generation technology has become a research focus. Current video generation technologies, especially those using images and text as input, struggle to accurately capture the detailed information of the images and text, resulting in inconsistencies between the generated content and the original input, thus affecting the user's interactive experience. Summary of the Invention

[0003] This application provides a video generation method, an electronic device, a computer storage medium, and a program product.

[0004] The technical solution of this application is implemented as follows: This application provides a video generation method, the method comprising: Obtain an input image and corresponding descriptive text; the input image includes a target object; Extract the image features of the input image, the pose features of the target object, and the text features of the descriptive text; The interaction features of each pair of features among the image features, pose features, and text features at multiple scales are determined. The interaction features of each pair of features at multiple scales are fused to obtain a first fusion feature between the image features and the text features, a second fusion feature between the pose features and the image features, and a third fusion feature between the text features and the pose features. The first fusion feature, the second fusion feature, and the third fusion feature are fused together to obtain a multimodal fusion feature; The target video is generated based on the multimodal fusion features.

[0005] This application provides an electronic device, the device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the video generation method provided by one or more of the foregoing technical solutions.

[0006] This application provides a computer storage medium storing a computer program; when the computer program is executed, it can implement the video generation method provided by one or more of the aforementioned technical solutions.

[0007] This application provides a computer program product, including a computer program that, when executed by a processor, implements the video generation method provided by one or more of the aforementioned technical solutions.

[0008] This application provides a video generation method, an electronic device, a computer storage medium, and a program product. The method includes: acquiring an input image and descriptive text corresponding to the input image; the input image includes a target object; extracting image features of the input image, pose features of the target object, and text features of the descriptive text; determining the interaction features of each pair of features among the image features, pose features, and text features at multiple scales; performing fusion processing on the interaction features of each pair of features at multiple scales to obtain a first fusion feature between the image features and the text features, a second fusion feature between the pose features and the image features, and a third fusion feature between the text features and the pose features; and performing fusion processing on the first fusion feature, the second fusion feature, and the third fusion feature to obtain a multimodal fusion feature.

[0009] As can be seen, in addition to extracting image features and text features, the embodiments of this application also extract the pose features of the target object in the input image. That is, video generation is performed based on the features of these three different modalities, which can improve the accuracy and coherence of the video content. In addition, by determining and fusing the interaction features of each pair of features at multiple scales, the correlation information between each pair of features in the image features, pose features and text features can be mined. Based on this correlation information, video generation can ensure that the generated video content not only conforms to the semantics of the descriptive text, but also is consistent with the pose of the target object in the original input image. That is, it can ensure the consistency of the original input of the video content and improve the user's interactive experience. Attached Figure Description

[0010] Figure 1 A flowchart illustrating a video generation method provided in this application embodiment; Figure 2 This is a schematic diagram illustrating a scenario of updating query vectors at multiple scales, provided as an embodiment of this application. Figure 3 This is a schematic diagram illustrating a scenario of updating key-value vectors at multiple scales, provided as an embodiment of this application. Figure 4 A flowchart for obtaining multimodal fusion features is provided in an embodiment of this application; Figure 5 A flowchart illustrating another video generation method provided in this application embodiment; Figure 6 This is a schematic diagram of the composition structure of the video generation apparatus according to an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0011] The technical solutions in this application will now be clearly and completely described with reference to the accompanying drawings.

[0012] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments provided herein are merely illustrative of the present application and are not intended to limit the present application. Furthermore, the embodiments provided below are some embodiments for implementing the present application, and not all embodiments for implementing the present application. Unless otherwise specified, the technical solutions described in the present application can be implemented in any combination.

[0013] It should be noted that, in this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method or apparatus that includes a list of elements includes not only the elements expressly described, but also other elements not expressly listed, or elements inherent to implementing the method or apparatus. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other related elements (e.g., steps in the method or units in the apparatus, such as portions of a processor, program, or software, etc.) in the method or apparatus that includes that element.

[0014] For example, the video generation method provided in this application includes a series of steps, but the video generation method provided in this application is not limited to the steps described therein. Similarly, the video generation apparatus provided in this application includes a series of modules, but the video generation apparatus provided in this application is not limited to the modules explicitly described, but may also include modules that need to be set up for obtaining relevant information or processing based on the information.

[0015] In some embodiments of this application, the video generation method can be implemented using a processor in a video generation device. The processor can be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a CPU, a controller, a microcontroller, and a microprocessor.

[0016] Figure 1 A flowchart of a video generation method provided in an embodiment of this application is shown below. Figure 1 As shown, the method includes the following steps: Step 100: Obtain the input image and the corresponding descriptive text.

[0017] In this embodiment of the application, the entity executing the video generation method can be a terminal device; the terminal device can include, but is not limited to, electronic devices with video playback functions such as mobile phones, tablets, personal computers (PCs), and vehicle terminals.

[0018] For example, the input image can be a user of the terminal device, that is, an image actively entered by the user; here, there is no specific limitation on the source of the input image; for example, the input image can be an image selected and entered by the user from the album, or an image taken directly by the user using the camera function.

[0019] It should be noted that the input image may include the target object; here, the category of the target object is not specifically limited; for example, the target object may be a person, or various types of animals (such as cats, dogs, etc.), or other types of objects (such as vehicles, buildings, etc.). The following will use a person as an example for illustration.

[0020] For example, the descriptive text corresponding to the input image refers to the text associated with the target object in the input image. It can be natural language text input by the user to describe the action, behavior, or scene of the target object. For example, if the target object in the input image is a person, the descriptive text can be "a person is jogging" or "a person is dancing".

[0021] Step 101: Extract image features from the input image, pose features of the target object, and text features from the descriptive text.

[0022] In this embodiment of the application, after obtaining the input image and the descriptive text corresponding to the input image, the image features of the input image, the pose features of the target object, and the text features of the descriptive text can be extracted.

[0023] For example, a pre-trained pose estimation model can be used to extract pose features of a target object in an input image; here, pose features are also called pose feature maps, which can be represented as two-dimensional heatmaps or structure vectors.

[0024] It should be noted that when the target object is a person, the pose estimation model can be a human pose estimation model. The pose features of the target object can include the position coordinates and connection relationships of multiple joint key points (such as shoulder, elbow, knee, ankle, etc.).

[0025] For example, an image encoder can be used to extract image features from the input image; here, the type of image encoder is not specifically limited, for example, it can be a ViT (Vision Transformer) network, or a ResNet (Residual Network), etc.

[0026] For example, a text encoder can be used to extract text features that describe the text; here, the type of text encoder is not specifically limited, for example, it can be a Contrastive Language-Image Pre-training (CLIP) text encoder, or a Text-to-Text Transfer Transformer (T5) model, etc.

[0027] Step 102: Determine the interaction features of each pair of features in the image features, pose features and text features at multiple scales, and perform fusion processing on the interaction features of each pair of features at multiple scales to obtain the first fusion feature between image features and text features, the second fusion feature between pose features and image features, and the third fusion feature between text features and pose features.

[0028] In this embodiment of the application, after obtaining the image features of the input image, the pose features of the target object, and the text features of the descriptive text according to the above steps, the interaction features of each pair of features at multiple scales can be determined; that is, the interaction features of image features and text features at multiple scales, the interaction features of pose features and image features at multiple scales, and the interaction features of text features and pose features at multiple scales can be determined.

[0029] It should be noted that the number of scales and the size of each scale can be set according to the actual situation, and no specific limit is made here.

[0030] For example, the number of scales can be 3, and the size of each scale is 1×1, 3×3 and 5×5 respectively. Here, the 1×1 scale is also called the W1 scale, the 3×3 scale is also called the W3 scale and the 5×5 scale is also called the W5 scale. The W1 scale includes a single feature point, the W3 scale includes 9 feature points and the W5 scale includes 25 feature points. This will be used as an example for further explanation.

[0031] In some embodiments, determining the interaction features of each pair of features among image features, pose features, and text features at multiple scales may include: using image features as a first query vector and text features as a first key vector to determine the interaction features of the first query vector and the first key vector at multiple scales; using pose features as a second query vector and image features as a second key vector to determine the interaction features of the second query vector and the second key vector at multiple scales; and using text features as a third query vector and pose features as a third key vector to determine the interaction features of the third query vector and the third key vector at multiple scales.

[0032] For example, the process of determining the interaction features of image features and text features at multiple scales can be as follows: using image features as a first query vector and text features as a first key vector, determine the interaction features of the first query vector and the first key vector at multiple scales.

[0033] In some embodiments, determining the interaction features of the first query vector and the first key-value vector at multiple scales may include: determining the updated query vector of the first query vector at multiple scales, and the updated key-value vector of the first key-value vector at multiple scales; for each scale, determining a first interaction value between the first query vector and the updated key-value vector at that scale, and a second interaction value between the first key-value vector and the updated query vector at that scale; and determining the interaction features based on the first interaction value and the second interaction value at multiple scales.

[0034] Correspondingly, the first interaction value at multiple scales and the second interaction value at multiple scales can be determined as the interaction features of the first query vector and the first key value vector at multiple scales, that is, the interaction features of image features and text features at multiple scales.

[0035] For example, the first query vector can be represented as QUERY, and the first key-value vector can be represented as KEY; see also Figure 2 When there are multiple scales, including W1, W3 and W5, for W1 scale, the first query vector does not need to be offset fusion and can be directly used as the update query vector at this scale. That is, the update query vector at W1 scale is the same as the first query vector. For W3 scale, the update query vector of the first query vector at W3 scale is obtained by adaptively weighting the offsets of the first query vector with the surrounding 8 neighboring feature points. Similarly, for W5 scale, the update query vector of the first query vector at W5 scale is obtained by adaptively weighting the offsets of the first query vector with the surrounding 24 neighboring feature points.

[0036] Similarly, see Figure 3For the W1 scale, the first key value vector does not need to undergo offset fusion; the original first key value vector can be directly used as the updated key value vector at this scale. In other words, the updated key value vector at the W1 scale is the same as the first key value vector. For the W3 scale, the updated key value vector of the first key value vector at the W3 scale is obtained by adaptively weighting the offsets of the first key value vector with the offsets of the eight surrounding neighboring feature points. Similarly, for the W5 scale, the updated key value vector of the first key value vector at the W5 scale is obtained by adaptively weighting the offsets of the first key value vector with the offsets of the 24 surrounding neighboring feature points.

[0037] It should be noted that for each scale, adaptive weighting can be performed using offset interactive weights; offset interactive weights can be learned through an additional network that can automatically adjust the contribution of different neighborhood feature points according to task requirements.

[0038] In this embodiment of the application, after determining the updated query vector of the first query vector at multiple scales, a first interaction value between the first query vector and the updated key value vector at each scale can be determined; similarly, after determining the updated key value vector of the first key value vector at multiple scales, a second interaction value between the first key value vector and the updated query vector at each scale can be determined; the process is illustrated below.

[0039] For example, as described above, the updated key-value vector at the W1 scale does not require offset fusion. Determining the first interaction value between the first query vector and the updated key-value vector at the W1 scale is equivalent to directly performing interactive calculations on each KEY and QUERY. This interactive calculation can capture local fine features. The updated key-value vector at the W3 scale is obtained by adaptively weighting the offsets of the first key-value vector and the surrounding 8 neighboring feature points. Determining the first interaction value between the first query vector and the updated key-value vector at the W3 scale is equivalent to completing key-value vector fusion within a 3×3 scale range and then performing interactive calculations on each item with the first query vector. This interactive calculation can capture feature associations within a medium range. The updated key-value vector at the W5 scale is obtained by adaptively weighting the offsets of the first key-value vector and the surrounding 24 neighboring feature points. Determining the first interaction value between the first query vector and the updated key-value vector at the W5 scale is equivalent to completing key-value vector fusion within a 5×5 scale range and then performing interactive calculations on each item with the first query vector. This interactive calculation can capture feature associations within a larger range.

[0040] Similarly, as described above, the updated query vector at the W1 scale does not require offset fusion. Determining the second interaction value between the first key-value vector and the updated query vector at the W1 scale is equivalent to directly performing interactive calculations on each KEY and QUERY. This interactive calculation can capture local fine features. The updated query vector at the W3 scale is obtained by adaptively weighting the offsets of the first query vector and the surrounding 8 neighboring feature points. Determining the second interaction value between the first key-value vector and the updated query vector at the W3 scale is equivalent to performing query vector fusion within a 3×3 scale range and then performing interactive calculations on each item with the first key-value vector. This interactive calculation can capture feature associations within a medium range. The updated query vector at the W5 scale is obtained by adaptively weighting the offsets of the first query vector and the surrounding 24 neighboring feature points. Determining the second interaction value between the first key-value vector and the updated query vector at the W5 scale is equivalent to performing query vector fusion within a 5×5 scale range and then performing interactive calculations on each item with the first key-value vector. This interactive calculation can capture feature associations within a larger range.

[0041] For example, through the above steps, a first interaction value and a second interaction value at multiple scales can be obtained; that is, the interaction features of image features and text features at multiple scales.

[0042] In this embodiment of the application, after obtaining the interaction features of pose features and image features at multiple scales, the interaction features of image features and text features at multiple scales can be fused to obtain the first fused features between image features and text features. For example, the process may include: determining the weights corresponding to each of the multiple scales; and using each weight to perform a weighted summation of the first interaction value at multiple scales and the second interaction value at multiple scales.

[0043] For example, when there are multiple scales including W1, W3 and W5, the first interaction value at the W1 scale can be represented as g11(KEY1,QUERY), the first interaction value at the W3 scale can be represented as g31(KEY3,QUERY), and the first interaction value at the W5 scale can be represented as g51(KEY5,QUERY); the second interaction value at the W1 scale can be represented as g12(KEY,QUERY1), the second interaction value at the W3 scale can be represented as g32(KEY,QUERY3), and the second interaction value at the W5 scale can be represented as g52(KEY,QUERY5); where KEY1, KEY3 and KEY5 correspond to the updated key-value vectors at the W1, W3 and W5 scales, respectively; QUERY1, QUERY3 and QUERY5 correspond to the updated query vectors at the W1, W3 and W5 scales, respectively; and g11, g31, g51, g12, g32 and g52 represent the offsettable interaction weights at the corresponding scales.

[0044] In practical applications, additional networks can be used to determine the weights corresponding to multiple scales. These networks can adjust the contribution of different scales based on the importance of different features. Thus, when fusing interaction values ​​from different scales, it's not a simple averaging operation, but rather a dynamic assignment of different weights to interaction values ​​at different scales based on the model's learning of input features during training. This highlights information from more important scales, making the model more flexible in handling multi-scale features.

[0045] Suppose that for the first interaction value at multiple scales, the weights for each scale determined by the additional network are w11, w31, and w51, respectively; and for the second interaction value at multiple scales, the weights for each scale determined by the additional network are w12, w32, and w52, respectively. Then, fusing the interaction features of image and text features at multiple scales is equivalent to using the weights to perform a weighted summation of the first and second interaction values ​​corresponding to multiple scales, obtaining the first fused feature F between image and text features. The corresponding formula can be expressed as: F=w11*(g11(KEY1,QUERY))+w12*(g12(KEY,QUERY))+w31*(g31(KEY3,QUERY))+w32*(g32(KEY,QUERY3)+w51*(g51(KEY5,QUERY))+w52*(g52(KEY,QUERY5) Here, the weights w11, w31, w51, w12, w32, and w52 are learnable hyperparameters.

[0046] For example, the above describes the process of determining the interaction features between image features and text features at multiple scales and fusing the interaction features at multiple scales.

[0047] In some embodiments, determining the interaction features of the second query vector and the second key-value vector at multiple scales may include: determining the updated query vector of the second query vector at multiple scales, and the updated key-value vector of the second key-value vector at multiple scales; for each scale, determining a third interaction value between the second query vector and the updated key-value vector at that scale, and a fourth interaction value between the second key-value vector and the updated query vector at that scale; and determining the interaction features based on the third interaction value and the fourth interaction value at multiple scales.

[0048] Correspondingly, the third interaction value and the fourth interaction value at multiple scales can be determined as the interaction features of the second query vector and the second key value vector at multiple scales, that is, the interaction features of pose features and image features at multiple scales.

[0049] It should be noted that the process of determining the interaction features of the second query vector and the second key value vector at multiple scales can refer to the process of determining the interaction features of the first query vector and the first key value vector at multiple scales described above. To avoid repetition, it will not be repeated here.

[0050] In this embodiment of the application, after obtaining the interaction features of pose features and image features at multiple scales, the interaction features of pose features and image features at multiple scales can be fused to obtain a second fused feature between pose features and image features. It should be noted that the process of obtaining the second fused feature can refer to the process of obtaining the first fused feature described above. To avoid repetition, it will not be repeated here.

[0051] In some embodiments, determining the interaction features of the third query vector and the third key-value vector at multiple scales may include: determining the updated query vector of the third query vector at multiple scales, and the updated key-value vector of the third key-value vector at multiple scales; for each scale, determining a fifth interaction value between the third query vector and the updated key-value vector at that scale, and a sixth interaction value between the third key-value vector and the updated query vector at that scale; and determining the interaction features based on the fifth interaction value and the sixth interaction value at multiple scales.

[0052] Correspondingly, the fifth interaction value and the sixth interaction value at multiple scales can be determined as the interaction features of the third query vector and the third key value vector at multiple scales, that is, the interaction features of text features and pose features at multiple scales.

[0053] It should be noted that the process of determining the interaction features of the third query vector and the third key value vector at multiple scales can refer to the process of determining the interaction features of the first query vector and the first key value vector at multiple scales described above. To avoid repetition, it will not be repeated here.

[0054] In this embodiment of the application, after obtaining the interaction features of text features and pose features at multiple scales, the interaction features of text features and pose features at multiple scales can be fused to obtain a third fused feature between text features and pose features. It should be noted that the process of obtaining the third fused feature can refer to the process of obtaining the first fused feature described above. To avoid repetition, it will not be repeated here.

[0055] As can be seen, in this embodiment, by introducing multiple scales, local fine feature association, medium-range feature association, and large-range feature association between different modal features can be achieved. Thus, feature information of input images and descriptive text can be captured from multiple dimensions, avoiding the information limitations caused by a single scale. In addition, since the model can automatically adjust the learnable hyperparameters corresponding to each scale according to the importance of different modal features, it can adapt to the video generation requirements of different scenarios.

[0056] Step 103: Perform fusion processing on the first fusion feature, the second fusion feature and the third fusion feature to obtain the multimodal fusion feature.

[0057] In this embodiment of the application, after obtaining the first fusion feature, the second fusion feature, and the third fusion feature according to the above steps, these three fusion features can be further fused to obtain a multimodal fusion feature. For example, the first fusion feature, the second fusion feature, and the third fusion feature can be concatenated along the feature dimension to obtain the multimodal fusion feature.

[0058] To facilitate understanding, the following will be combined with Figure 4 For illustrative examples, see [link to example]. Figure 4 For image features V, pose features P, and text features L, after obtaining the first fusion feature LV between image features V and text features L, the second fusion feature VP between pose features P and image features V, and the third fusion feature PL between text features L and pose features P through the above query vector and key-value vector methods, the first fusion feature LV, the second fusion feature VP, and the third fusion feature PL can be fused to obtain multimodal fusion features.

[0059] Step 104: Generate the target video based on multimodal fusion features.

[0060] In this embodiment of the application, after obtaining the multimodal fusion features according to the above steps, a target video can be generated based on the multimodal fusion features. The target video includes a target object, which can be a video of a set duration.

[0061] For example, the target video can be a video of a set duration; here, the value of the set duration can be set according to the actual situation, and there is no specific limitation. For example, the set duration can be 5 seconds.

[0062] As can be seen, in addition to extracting image features and text features, the embodiments of this application also extract the pose features of the target object in the input image. That is, video generation is performed based on the features of these three different modalities, which can improve the accuracy and coherence of the video content. In addition, by determining and fusing the interaction features of each pair of features at multiple scales, the correlation information between each pair of features in the image features, pose features and text features can be mined. Based on this correlation information, video generation can ensure that the generated video content not only conforms to the semantics of the descriptive text, but also is consistent with the pose of the target object in the original input image. That is, it can ensure the consistency of the original input of the video content and improve the user's interactive experience.

[0063] In some embodiments, generating a target video based on multimodal fusion features may include: decoding the multimodal fusion features using a temporal modeling network to obtain an initial video; and post-processing the initial video to obtain the target video.

[0064] Here, the time modeling network can be a time generator with video generation capabilities, such as a Transformer or Diffusion structure; no specific limitation is made here.

[0065] For example, see Figure 5 After processing the input data (including input images and descriptive text) to obtain multimodal fusion features, a temporal modeling network can be used to decode the multimodal fusion features to generate an initial video of a set duration. After obtaining the initial video, further optimization processing, i.e., post-processing, can be performed on the initial video to obtain the target video and output it.

[0066] Post-processing here can include inter-frame smoothing, color enhancement, frame interpolation, etc., without specific limitations. Inter-frame smoothing refers to making the transition between frames smoother, color enhancement refers to enhancing the colors in the image, and frame interpolation refers to supplementing missing frames. Understandably, post-processing the initial video can effectively improve the viewing experience.

[0067] Figure 6 This is a schematic diagram of the composition structure of the video generation apparatus according to an embodiment of this application, such as... Figure 6As shown, the device includes: an acquisition module 600, an extraction module 601, a determination module 602, a fusion module 603, and a generation module 604, wherein: The acquisition module 600 is used to acquire an input image and descriptive text corresponding to the input image; the input image includes a target object. Extraction module 601 is used to extract image features of the input image, pose features of the target object, and text features of the descriptive text; The determining module 602 is used to determine the interaction features of each pair of features in the image features, the pose features and the text features at multiple scales, and to perform fusion processing on the interaction features of each pair of features at multiple scales to obtain a first fusion feature between the image features and the text features, a second fusion feature between the pose features and the image features, and a third fusion feature between the text features and the pose features. The fusion module 603 is used to perform fusion processing on the first fusion feature, the second fusion feature and the third fusion feature to obtain a multimodal fusion feature; The generation module 604 is used to generate a target video based on the multimodal fusion features.

[0068] In some embodiments, the determining module 602 is further configured to: Using the image features as the first query vector and the text features as the first key vector, the interaction features of the first query vector and the first key vector at multiple scales are determined. Using the pose features as the second query vector and the image features as the second key vector, the interaction features of the second query vector and the second key vector at multiple scales are determined. Using the text features as the third query vector and the pose features as the third key vector, the interaction features of the third query vector and the third key vector at multiple scales are determined.

[0069] In some embodiments, the determining module 602 is further configured to: Determine the updated query vector of the first query vector at multiple scales, and the updated key-value vector of the first key-value vector at multiple scales; For each scale, determine a first interaction value between the first query vector and the updated key vector at the scale, and a second interaction value between the first key vector and the updated query vector at the scale; The interaction features are determined based on the first interaction value at multiple scales and the second interaction value at multiple scales.

[0070] In some embodiments, the determining module 602 is further configured to: Determine the weights corresponding to each of the multiple scales; The first interaction value and the second interaction value at the multiple scales are weighted and summed using various weights to obtain the first fusion feature between the image features and the text features.

[0071] In some embodiments, the determining module 602 is further configured to: Determine the updated query vector of the second query vector at multiple scales, and the updated key-value vector of the second key-value vector at multiple scales; For each scale, a third interaction value between the second query vector and the updated key vector at that scale, and a fourth interaction value between the second key vector and the updated query vector at that scale are determined. The interaction features are determined based on the third interaction value at multiple scales and the fourth interaction value at multiple scales.

[0072] In some embodiments, the determining module 602 is further configured to: Determine the updated query vector of the third query vector at multiple scales, and the updated key-value vector of the third key-value vector at multiple scales; For each scale, determine the fifth interaction value between the third query vector and the updated key vector at that scale, and the sixth interaction value between the third key vector and the updated query vector at that scale; The interaction features are determined based on the fifth interaction value at multiple scales and the sixth interaction value at multiple scales.

[0073] In some embodiments, the generation module 604 is further configured to: The multimodal fusion features are decoded using a temporal modeling network to obtain the initial video; The initial video is post-processed to obtain the target video.

[0074] In practical applications, the above-mentioned acquisition module 600, extraction module 601, determination module 602, fusion module 603 and generation module 604 can all be implemented by a processor located in an electronic device. The processor can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller and microprocessor.

[0075] Furthermore, in this embodiment, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0076] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0077] Specifically, the computer program instructions corresponding to a video generation method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the computer program instructions corresponding to a video generation method in the storage media are read or executed by an electronic device, any one of the video generation methods in the aforementioned embodiments is implemented.

[0078] Based on the same technical concept as the foregoing embodiments, see Figure 7 It illustrates an electronic device 700 provided in an embodiment of this application, which may include: a memory 701 and a processor 702; wherein, Memory 701 is used to store computer programs and data; The processor 702 is configured to execute a computer program stored in a memory to implement any of the video generation methods described in the foregoing embodiments.

[0079] In practical applications, the memory 701 mentioned above can be volatile memory, such as RAM; or non-volatile memory, such as ROM, flash memory, hard disk drive (HDD) or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 702.

[0080] The processor 702 described above can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor. It is understood that, for different model training devices, the electronic device used to implement the above processor function can also be other types, and this application embodiment does not specifically limit the specific types.

[0081] This application provides a computer program product, including a computer program that, when executed by a processor, implements any of the video generation methods described in the foregoing embodiments.

[0082] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0083] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0084] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined to obtain new method embodiments without conflict.

[0085] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0086] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0087] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0088] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0089] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0090] The above are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application.

Claims

1. A video generation method, characterized in that, The method includes: Obtain an input image and corresponding descriptive text; the input image includes a target object; Extract the image features of the input image, the pose features of the target object, and the text features of the descriptive text; The interaction features of each pair of features among the image features, pose features, and text features at multiple scales are determined. The interaction features of each pair of features at multiple scales are fused to obtain a first fusion feature between the image features and the text features, a second fusion feature between the pose features and the image features, and a third fusion feature between the text features and the pose features. The first fusion feature, the second fusion feature, and the third fusion feature are fused together to obtain a multimodal fusion feature; The target video is generated based on the multimodal fusion features.

2. The method according to claim 1, characterized in that, Determining the interaction features of each pair of features among the image features, pose features, and text features at multiple scales includes: Using the image features as the first query vector and the text features as the first key vector, the interaction features of the first query vector and the first key vector at multiple scales are determined. Using the pose features as the second query vector and the image features as the second key vector, the interaction features of the second query vector and the second key vector at multiple scales are determined. Using the text features as the third query vector and the pose features as the third key vector, the interaction features of the third query vector and the third key vector at multiple scales are determined.

3. The method according to claim 2, characterized in that, Determining the interaction features of the first query vector and the first key-value vector at multiple scales includes: Determine the updated query vector of the first query vector at multiple scales, and the updated key-value vector of the first key-value vector at multiple scales; For each scale, determine a first interaction value between the first query vector and the updated key vector at the scale, and a second interaction value between the first key vector and the updated query vector at the scale; The interaction features are determined based on the first interaction value at multiple scales and the second interaction value at multiple scales.

4. The method according to claim 3, characterized in that, The interaction features of the image features and the text features at multiple scales are fused to obtain a first fused feature between the image features and the text features, including: Determine the weights corresponding to each of the multiple scales; The first interaction value and the second interaction value at the multiple scales are weighted and summed using various weights to obtain the first fusion feature between the image features and the text features.

5. The method according to claim 2, characterized in that, Determining the interaction features of the second query vector and the second key-value vector at multiple scales includes: Determine the updated query vector of the second query vector at multiple scales, and the updated key-value vector of the second key-value vector at multiple scales; For each scale, a third interaction value between the second query vector and the updated key vector at that scale, and a fourth interaction value between the second key vector and the updated query vector at that scale are determined. The interaction features are determined based on the third interaction value at multiple scales and the fourth interaction value at multiple scales.

6. The method according to claim 2, characterized in that, Determining the interaction features of the third query vector and the third key-value vector at multiple scales includes: Determine the updated query vector of the third query vector at multiple scales, and the updated key-value vector of the third key-value vector at multiple scales; For each scale, determine the fifth interaction value between the third query vector and the updated key vector at that scale, and the sixth interaction value between the third key vector and the updated query vector at that scale; The interaction features are determined based on the fifth interaction value at multiple scales and the sixth interaction value at multiple scales.

7. The method according to any one of claims 1 to 6, characterized in that, The generation of the target video based on the multimodal fusion features includes: The multimodal fusion features are decoded using a temporal modeling network to obtain the initial video; The initial video is post-processed to obtain the target video.

8. An electronic device, characterized in that, The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method according to any one of claims 1 to 7.

9. A computer storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.