Video generation methods, devices, storage media, and computer program products
By using audio track separation and shape masking techniques to accurately locate the image material of the audio source in the video, the problem of low matching between dynamic video and audio in existing technologies is solved, and the deep alignment of audio and vision and rich visualization are achieved.
Patent Information
- Application Number
- CN202610975954.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-01
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies rely on external text or preset images to generate dynamic videos, resulting in low consistency between the generated dynamic video images and audio attributes, as well as low visual expressiveness.
By separating audio tracks to identify the source of the audio, combining it with a preset video style to determine the corresponding image material, and using shape masking to accurately locate the display area in the initial video, the image material is finally rendered into the video, achieving multi-dimensional alignment between audio and visuals.
It significantly improves the fit between the target video and audio, and enhances the richness and visual expressiveness of the audio visualization.
Smart Images

Figure CN123123657A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a video generation method, device, storage medium, and computer program product. Background Technology
[0002] The integration of Artificial Intelligence (AI) and multimedia video processing technology is an important research direction in the field of computer science. With the evolution of deep learning architecture and large model technology, the cross-fusion of multimodal features and cross-modal joint modeling have become the core driving force for intelligent multimedia processing. By performing deep feature extraction, semantic representation, temporal correlation and content generation on multi-dimensional modal data such as audio, vision and text, computers can simulate human composite perception and logical reasoning of multi-sensory information, thereby possessing the ability to automatically create, intelligently edit and collaboratively generate multimedia content. Among them, dynamic video generation technology, as a typical application in the field of multimodal understanding and generation, aims to use automated algorithms, temporal alignment mechanisms or cross-modal mapping networks to synchronously fuse and collaboratively render multi-dimensional audio signals and dynamic visual images at the time axis and visual semantic level. However, related technologies mainly rely on external text or preset images to generate dynamic videos, resulting in low consistency between the generated dynamic video images and audio attributes and low visual expressiveness. Summary of the Invention
[0003] This application provides a video generation method, device, storage medium, and computer program product that can improve the matching degree between target video and target audio.
[0004] The technical solution of this application embodiment is implemented as follows: This application provides a video generation method, the method comprising: The target audio is acquired, and the target audio is subjected to track separation processing to obtain multiple audio tracks of the target audio; For each audio track of the target audio, the sound source is identified to obtain the sound source of the audio track. Based on the sound source and a preset video style, the first image material corresponding to the sound source is determined. Acquire an initial video, and determine the display area of each first image element in the initial video based on the shape mask of each first image element; Based on the display area of each first image material in the initial video, multiple first image materials are rendered into the initial video to obtain the rendered initial video; The rendered initial video and the target audio are combined to obtain the target video.
[0005] This application provides a video generation apparatus, the apparatus comprising: The data acquisition module is used to acquire the target audio, perform audio track separation processing on the target audio, and obtain multiple audio tracks of the target audio; The object recognition module is used to identify the sound source of each audio track of the target audio, obtain the sound source of the audio track, and determine the first image material corresponding to the sound source based on the sound source and a preset video style. The region segmentation module is used to acquire the initial video and determine the display area of each first image material in the initial video based on the shape mask of each first image material; The material rendering module is used to render multiple first image materials to the initial video based on the display area of each first image material in the initial video, so as to obtain the rendered initial video; The video compositing module is used to perform video compositing on the rendered initial video and the target audio to obtain the target video.
[0006] This application provides an electronic device, the electronic device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the video generation method provided in the embodiments of this application.
[0007] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the video generation method provided in this application.
[0008] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implements the video generation method provided in this application.
[0009] The embodiments of this application have the following beneficial effects: By separating the target audio into multiple tracks and identifying the sound source for each track, the system can meticulously deconstruct and extract the essential sound source components from the internal structure of the audio. Based on this, the system combines a preset video style to determine the first image material corresponding to each sound source, achieving a precise cross-modal association between the audio's acoustic properties and the semantics of specific visual images. Furthermore, the system uses shape masks of each first image material to accurately locate their respective display areas in the initial video and renders multiple first image materials into the initial video, constructing an orderly fusion and dynamic collaborative representation of multi-source visual elements in the spatial layout of the screen. Finally, the system synthesizes the rendered initial video with the target audio to obtain the target video, achieving a deep alignment of the visual content with the audio's arrangement hierarchy and the temporal characteristics of the sound source in multiple dimensions of time and space. This significantly improves the fit between the final generated target video and the internal attributes of the audio, achieving a synergistic enhancement of the richness of audio visualization and visual expressiveness. Attached Figure Description
[0010] Figure 1 This is a schematic diagram of the video generation system architecture provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of the video generation device provided in the embodiments of this application; Figure 3 This is a first flowchart illustrating the video generation method provided in this application embodiment; Figure 4 This is a second flowchart illustrating the video generation method provided in this application embodiment; Figure 5 This is a first schematic diagram of the video generation method provided in the embodiments of this application; Figure 6 This is a second schematic diagram of the video generation method provided in the embodiments of this application; Figure 7 This is a third schematic diagram of the video generation method provided in the embodiments of this application; Figure 8 This is the fourth schematic diagram of the video generation method provided in the embodiments of this application.
[0011] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation
[0012] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0013] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0014] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.
[0015] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.
[0016] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.
[0017] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.
[0018] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.
[0019] 1) Audio track: An audio track is a structured digital representation of an audio signal on an independent channel. The data processing flow separates the composite audio signal into independent audio streams containing signals in different frequency domains by performing signal separation processing such as short-time Fourier transform on the audio source file. Each audio track corresponds to a specific sound source element, such as human voice, a specific instrument, or natural background sound effects, and contains the acoustic properties of the sound source, such as frequency and amplitude, which change over time.
[0020] 2) Shape Mask: A shape mask is a digital matrix representation used to define the geometric outline and coverage area of an image or video element in a two-dimensional screen space. After receiving the generation instruction, the multimedia system determines the sound object material based on the sound object and the preset style, and records the corresponding shape mask. It usually exists in the form of a binary pixel matrix, channel weight value or vector path. Through the selective activation or transparency control of pixels, the effective visual boundary of the target object material is accurately isolated and non-target areas are masked.
[0021] 3) Video frame: A video frame is the smallest independent static image unit that constitutes a dynamic video sequence. It is a sample slice of digital video on a specific time series. It contains all the pixel color, brightness and spatial structure information of the two-dimensional screen space at the current time node. On the time axis, a large number of consecutive video frames are arranged in order and refreshed at high speed according to a preset frame rate. They are combined to present a continuous dynamic visual effect by utilizing the persistence of vision of the human eye.
[0022] This application provides a video generation method, apparatus, electronic device, computer-readable storage medium, and computer program product that can improve the fit between target video and target audio.
[0023] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the video generation system provided in this application embodiment. In the video generation system 10 provided in this application embodiment, in order to support a video generation application, the terminal 400 connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.
[0024] Terminal 400 can be used to obtain video generation requests carrying target audio.
[0025] In some embodiments, a video generation plugin may be embedded in the client running in the terminal 400 to implement the video generation method locally on the client. For example, the terminal 400 calls the video generation plugin to implement the video generation method, which involves: acquiring target audio; performing audio track separation processing on the target audio to obtain multiple audio tracks; identifying the sound source for each audio track to obtain the sound source of the audio track; determining the first image material corresponding to the sound source based on the sound source and a preset video style; acquiring an initial video; determining the display area of each first image material in the initial video based on the shape mask of each first image material; rendering multiple first image materials onto the initial video based on the display area of each first image material in the initial video to obtain the rendered initial video; and performing video synthesis between the rendered initial video and the target audio to obtain the target video.
[0026] It is worth noting that the target video can be stored locally on the terminal 400 and retrieved when needed.
[0027] In some embodiments, after the terminal 400 obtains a video generation request carrying the target audio, it calls the video generation interface of the server 200 (which can be provided as a cloud service, i.e., a video generation service). The server 200 implements a video generation method through a video generation plugin, performs audio track separation processing on the target audio to obtain multiple audio tracks of the target audio; for each audio track of the target audio, it identifies the sound source of the audio track to obtain the sound source of the audio track; based on the sound source of the sound source and a preset video style, it determines the first image material corresponding to the sound source of the sound source; it obtains an initial video, and based on the shape mask of each first image material, it determines the display area of each first image material in the initial video; based on the display area of each first image material in the initial video, it renders multiple first image materials onto the initial video to obtain a rendered initial video; and it performs video synthesis between the rendered initial video and the target audio to obtain the target video.
[0028] It is worth noting that the target video can be stored locally on server 200 and retrieved when needed, or it can be returned to terminal 400.
[0029] In some embodiments, server 200 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.
[0030] Terminal 400 can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, vehicle terminal, aircraft, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.
[0031] See Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device for video generation provided in an embodiment of this application. Figure 2 The electronic device 500 shown can be Figure 1 The terminal 400 or server 200, and the electronic device 500 include: at least one processor 510, memory 550, and at least one network interface 520. The various components in server 200 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 2 The general labeled all buses as Bus System 540.
[0032] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.
[0033] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls; In some embodiments, when the video generation task is completed independently by the terminal 400, the server 200 provided in this application embodiment does not include the user interface 530.
[0034] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 510.
[0035] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.
[0036] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.
[0037] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc. Presentation module 553 is used to enable the presentation of information (e.g., user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with user interface 530. In some embodiments, when the video generation task is completed independently by the terminal 400, the server 200 provided in this application embodiment may not include the presentation module 553.
[0038] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532; in some embodiments, when the video generation task is completed independently by the terminal 400, the server 200 provided in this application embodiment may not include the presentation module 553.
[0039] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2A video generation apparatus 555 stored in memory 550 is shown. This apparatus can be software in the form of programs and plugins, and includes the following software modules: a data acquisition module 5551, an object recognition module 5552, a region division module 5553, a material rendering module 5554, and a video compositing module 5555. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.
[0040] As mentioned above, the electronic device implementing the video generation method of this application embodiment can be a terminal, a server, or a combination of both. Therefore, the executing entity of each step will not be described again below. See [link to relevant documentation]. Figure 3 , Figure 3 This is a first flowchart illustrating the video generation method provided in this application embodiment, which will be combined with... Figure 3 The steps shown are explained.
[0041] In step 101, the target audio is acquired, and the target audio is subjected to track separation processing to obtain multiple audio tracks of the target audio.
[0042] As an example of step 101, the target audio is acquired and input into the audio separation module. In the audio separation module, a short-time Fourier transform is performed on the time-domain signal of the target audio to obtain multiple audio frequency-domain signals. An inverse Fourier transform is performed on the multiple audio frequency-domain signals to obtain multiple audio tracks of the target audio. The target audio can be pre-recorded audio file data read from a local storage medium, or it can be audio stream data received in real time through a network communication interface. No limitation is made here.
[0043] For example, in the application scenario of dynamic song video generation, the acquired target audio can be a mixed song audio file containing vocals and instrumental accompaniment. The acquired target audio is input to the audio separation module. In the audio separation module, a short-time Fourier transform is performed on the time-domain signal of the target audio using a set sliding frame window, mapping the one-dimensional time-domain signal to the time-frequency domain, thereby separating multiple audio frequency domain signals, such as the vocal frequency domain signal reflecting the frequency characteristics of the vocals and the instrument frequency domain signal reflecting the frequency characteristics of specific instruments. Subsequently, an inverse Fourier transform is performed on each of the separated audio frequency domain signals to reconstruct the signal, thereby obtaining multiple audio tracks of the target audio.
[0044] In step 102, for each audio track of the target audio, the sound source is identified to obtain the sound source of the audio track. Based on the sound source and the preset video style, the first image material corresponding to the sound source is determined.
[0045] The sound source object refers to the sound source entity that produces specific acoustic features in the target audio, and the first image material refers to the image used to represent its corresponding sound source object.
[0046] As an example of step 102, for each audio track of the target audio, the following processing is performed: The audio track is input into a pre-trained audio recognition model. In the audio recognition model, features are extracted from the audio track to obtain its audio spectrum features. The audio spectrum features are then convolutionally encoded using a convolutional neural network in the audio recognition model to extract the acoustic texture of its local frequency bands, resulting in audio encoded features. A self-attention mechanism network in the audio recognition model is used to perform global temporal modeling on the audio encoded features to extract the correlation between features at different time steps, resulting in attention-processed audio encoded features. Finally, a fully connected network layer in the audio recognition model is used to classify and predict the attention-processed audio encoded features, outputting the sound source of the audio track. A preset video style is obtained. Using the category label of the sound source and the preset video style as joint retrieval conditions, a search is performed in a preset material database to obtain visual multimedia materials that match the joint retrieval conditions. The found visual multimedia materials are then identified as the first image material corresponding to the sound source.
[0047] For example, in the application scenario of dynamic song video generation, a certain audio track can be independent audio data containing continuous instrument playing sounds. The audio track is input into a pre-trained audio recognition model, and the output of the audio recognition model is the category label of the sound-producing object, "piano". With the preset video style being "cartoon style", the category label of the sound-producing object, "piano", and the preset video style, "cartoon style", are used as joint search conditions to perform label matching search in a preset structured material database, and a black and white piano key dynamic image with cartoon outline effect is obtained (i.e., visual multimedia material that matches the joint search conditions). The retrieved dynamic image is determined as the first image material corresponding to the sound-producing object.
[0048] In some embodiments, the identification of the vocal object in an audio track is achieved through an audio recognition model. The training of the audio recognition model can be achieved through the following processes: acquiring multiple audio track samples; identifying the vocal object for each audio track sample using an initial audio recognition model to obtain the predicted vocal object corresponding to each audio track sample; constructing a loss function based on the label information of multiple audio track samples and the predicted vocal objects corresponding to multiple audio track samples; and training the initial audio recognition model based on the value of the loss function to obtain the audio recognition model.
[0049] In this context, label information refers to the standard benchmark data that is manually verified and assigned to training samples during the model training phase of machine learning, used to characterize the fact that the sample belongs to the true category.
[0050] As an example of "training an audio recognition model", multiple audio track samples and the label information of each audio track sample are obtained. For each audio track sample, the initial audio recognition model is used to identify the sound source of the audio track sample, and the predicted sound source of the audio track sample is obtained.
[0051] Based on the difference between the label information of each audio track sample and its corresponding predicted vocal object, a loss function is constructed. Based on the value of the loss function, the initial audio recognition model is backpropagated to iteratively update the network parameters of the initial audio recognition model, resulting in an audio recognition model with updated parameters. The audio recognition model with updated parameters is evaluated and verified based on a preset test dataset. If the evaluation and verification pass, it is determined as the trained audio recognition model, i.e., the pre-trained audio recognition model.
[0052] The loss function can be the connection-time classification loss function (CTC loss function), the cross-entropy loss function, or a composite loss function composed of the connection-time classification loss function and the cross-entropy loss function, etc. There are no restrictions here. During the iterative training process, the audio recognition model in the iterative training can also be validated and evaluated based on a preset validation dataset to dynamically adjust its learning strategy.
[0053] In step 103, an initial video is acquired, and the display area of each first image material in the initial video is determined based on the shape mask of each first image material.
[0054] Here, shape mask refers to the mask matrix data used to identify the outline range of the target entity in the first image material, and the target entity refers to the visual entity corresponding to the sound-emitting object.
[0055] As an example of step 103, based on the preset output resolution and the total duration of the target audio, a blank frame sequence matching the total duration of the target audio or a background image frame sequence matching the video style is created. The background image frame sequence refers to a set of background images that are continuously arranged on the timeline and match the video style. The background images that match the video style can be obtained by searching and matching from a preset material database, or they can be generated in real time by calling a generative image large model based on the preset video style. There is no limitation here.
[0056] The created blank frame sequence or background image frame sequence is determined as the initial video. The display area division method is determined. Based on the shape mask of each first image material, the display area of each first image material in the initial video is determined according to the display area division method.
[0057] For example, in the application scenario of dynamic song video generation, the preset output resolution can be 1920×1080, and the total duration of the target audio can be 200 seconds. Based on the preset output resolution and the total duration of the target audio, a 200-second blank frame sequence is created through the video rendering engine. The created blank frame sequence is determined as the initial video. Alternatively, if the preset video style is "retro", a background image with old film texture effects that matches the video style is obtained from the preset material database. The background image is then arranged continuously on the timeline to obtain a background image frame sequence, which is then determined as the initial video.
[0058] In some embodiments, see Figure 4 , Figure 4 This is a schematic diagram of the second process of the video generation method provided in the embodiments of this application. Figure 3 Step 103 shown can be implemented through steps 1031 to 1033, which will be explained in detail below.
[0059] In step 1031, for each first image material, the geometric complexity of the first image material is determined based on the shape mask of the first image material.
[0060] Geometric complexity refers to the index used to measure the degree of irregularity of the outline range of the target entity in the first image material, where the target entity refers to the visual entity corresponding to the sound-emitting object.
[0061] As an example of step 1031, for each first image material, the following processing is performed: the shape mask of the first image material is subjected to boundary extraction to obtain the boundary of the target entity in the first image material, and the pixel area of the target entity is determined. The ratio between the square of the perimeter of the boundary and the pixel area of the target entity is determined as the geometric complexity of the first image material. Alternatively, the minimum bounding polygon of the target entity can be determined based on the boundary of the target entity, and the ratio between the pixel area of the minimum bounding polygon and the pixel area of the target entity is determined as the geometric complexity of the first image material. No limitation is imposed here.
[0062] For example, in the application scenario of dynamic music video generation, for the first image material A corresponding to the sound-producing object being a guitar, the boundary of the shape mask of the first image material A is extracted to obtain the boundary of the target entity corresponding to the sound-producing object "guitar". If the perimeter of the boundary is determined to be 400 pixels, and the pixel area of the target entity corresponding to the sound-producing object "guitar" is 10,000 square pixels, then the ratio of the square of the perimeter of the boundary (i.e., 160,000) to the pixel area of the target entity corresponding to the sound-producing object "guitar" (i.e., 10,000), which is 16, is determined as the geometric complexity of the first image material A; in another calculation method... In the application scenario of this method, for the first image material B corresponding to the sound-emitting object "drum kit", the minimum bounding polygon of the target entity corresponding to the sound-emitting object "drum kit" is determined based on the boundary of the target entity. If the pixel area of the minimum bounding polygon is determined to be 15,000 square pixels, and the pixel area of the target entity corresponding to the sound-emitting object "drum kit" is 12,000 square pixels, then the ratio of the pixel area of the minimum bounding polygon (i.e., 15,000) to the pixel area of the target entity corresponding to the sound-emitting object "drum kit" (i.e., 12,000), which is 1.25, is determined as the geometric complexity of the first image material B.
[0063] In step 1032, the display area division method is determined based on multiple geometric complexities.
[0064] The display area can be divided into two types: dynamic adaptation and uniform division.
[0065] As an example of step 1032, the geometric complexity of all first image materials is calculated. The average of multiple geometric complexities can be used as the overall complexity reference value, or the maximum value of multiple geometric complexities can be used; no limitation is made here. If the overall complexity reference value is less than a preset complexity threshold, the dynamic adaptation partitioning method is determined as the display area partitioning method. If the overall complexity reference value is greater than or equal to the preset complexity threshold, the uniform partitioning method is determined as the display area partitioning method.
[0066] In another example of "determining the display area division method", the display area division method corresponding to the preset video style can also be directly determined based on the mapping relationship between different video styles and different display area division methods. This is not limited here. For example, if the preset video style is "cartoon style", the display area division method corresponding to "cartoon style" can be directly determined to be the dynamic adaptation division method based on the pre-configured style mapping relationship table (i.e., the mapping relationship between different video styles and different display area division methods). If the preset video style is "multi-camera realistic style", the display area division method corresponding to "multi-camera realistic style" can be directly determined to be the uniform division method based on the above style mapping relationship table.
[0067] In step 1033, the display area of each first image material in the initial video is determined based on the display area division method.
[0068] As an example of step 1033, when the display area division method is dynamic adaptation division method, the multiple first image materials are sorted in descending order based on the sound duration of the audio track corresponding to each first image material to obtain an image material sequence. Based on the shape mask of each first image material, the display area of each first image material in the image material sequence in the initial video is determined in turn. When the display area division method is uniform division method, the video screen of the initial video is uniformly divided into multiple sub-regions, and the multiple sub-regions obtained are determined one by one as the display area of the multiple first image materials in the initial video.
[0069] In this embodiment, the geometric complexity of each first image material is determined based on its shape mask, and the display area division method is determined based on multiple geometric complexities. This allows the division logic to adapt to the shape characteristics of different materials. Based on this, a display area is assigned to each first image material in the initial video, realizing a direct association between the display area and the geometric attributes of the material itself. Since the geometric complexity reflects the richness of detail and boundary change characteristics of the shape mask, the display area determined based on this can naturally match the shape mask of the material, thereby forming a layout structure in the initial video that is coordinated with the shape of each first image material.
[0070] In some embodiments, Figure 4 Step 1033 shown can be implemented through the following processing: When the display area division method is dynamic adaptation division method, based on the sound duration of the audio track corresponding to each first image material, the multiple first image materials are sorted in descending order to obtain the image material sequence; for the first first image material in the image material sequence, based on the shape mask of the first first image material, the display area of the first first image material is determined from the video frame of the initial video; for the nth first image material in the image material sequence, based on the display areas of the first n-1 first image materials, the displayable area of the nth first image material in the video frame is determined; based on the shape mask of the nth first image material, the display area of the nth first image material is determined from the displayable area.
[0071] Where n is a positive integer greater than 1, n≤N, and N is the number of the first image materials.
[0072] As an example of "sorting multiple first image materials in descending order," when the display area is divided using a dynamic adaptation method, for each audio track corresponding to a first image material, acoustic indicators (e.g., sound intensity, audio frequency, depth, BPM, etc.) are extracted from each audio frame. The time periods where the acoustic indicators exceed a preset energy threshold are counted to obtain at least one sound duration. The total duration of this at least one sound duration is determined as the sound duration of the audio track corresponding to the first image material. Based on the sound duration of the audio track corresponding to each first image material, the multiple first image materials are arranged in descending order of sound duration to obtain an image material sequence.
[0073] For example, in the application scenario of multi-track dynamic song video generation, for the first image material A, the sound intensity (i.e., acoustic index) of each audio frame in the audio track corresponding to the first image material A is extracted. The time periods when the sound intensity is greater than the preset sound intensity threshold (i.e., energy threshold) are counted to obtain two sound time periods, namely from the 10th second to the 30th second and from the 50th second to the 80th second. The duration of the above two sound time periods is added together to obtain the sound duration of the audio track corresponding to the first image material A, which is 50 seconds. Similarly, the sound duration of the audio track corresponding to the first image material B is determined to be 60 seconds and the sound duration of the audio track corresponding to the first image material C is determined to be 70 seconds. The above three first image materials are arranged in descending order of sound duration to obtain the image material sequence {first image material C, first image material B, first image material A}.
[0074] As an example of "determining the display area of each first image material", for the first first image material in the image material sequence, the display area of the first first image material is determined from the video frame of the initial video based on the shape mask of the first first image material. For the second first image material in the image material sequence, the displayable area of the second first image material in the video frame is determined based on the display area of the first first image material (i.e., the display area of the previous first image material). Based on the shape mask of the second first image material, the display area of the second first image material is determined from the displayable area, and so on. For the nth first image material in the image material sequence, the displayable area of the nth first image material in the video frame is determined based on the display areas of the previous n-1 first image materials. Based on the shape mask of the nth first image material, the display area of the nth first image material is determined from the displayable area.
[0075] In this embodiment, the audio tracks corresponding to each first image material are sorted in descending order, so that the first image material with a longer audio duration has a higher priority in the image material sequence. The first first image material in the sequence directly determines the display area from the video screen based on its shape mask. The subsequent nth first image material first defines the displayable area based on the previous n-1 defined display areas, and then determines the display area from the displayable area based on its shape mask. This step-by-step division method makes the display area of each first image material match its shape mask. At the same time, the display area of subsequent materials automatically avoids the allocated area, thereby achieving orderly arrangement and spatial coordination of the display area in the video screen.
[0076] In some embodiments, "determining the displayable area of the nth first image material in the video frame based on the display areas of the first n-1 first image materials" can be achieved through the following process: when at least one second image material is determined from the first n-1 first image materials, the area in the video frame other than the display area of the second image material is determined as the displayable area; when no second image material is determined from the first n-1 first image materials, the video frame is determined as the displayable area.
[0077] Among them, the audio track corresponding to the second image material overlaps with the audio track corresponding to the nth first image material in terms of sound output time.
[0078] As an example, iterate through each of the first image materials in the first n-1 first image materials. If the audio track corresponding to the first image material is overlapping with the audio track corresponding to the nth first image material in terms of sound time, the first image material is determined as the second image material.
[0079] After the traversal is completed, if at least one second image material is determined from the first n-1 first image materials, the area in the video frame other than the display area of the second image material can be determined as the displayable area. Alternatively, the area in the video frame other than the display area of the second image material, as well as the area in the display area of the second image material that can overlap with other areas, can all be determined as the displayable area. The area that can overlap with other areas is preset by the engineers. If no second image material is determined from the first n-1 first image materials, the video frame is determined as the displayable area.
[0080] For example, in the application scenario of multi-track dynamic song video generation, the image material sequence can be {first image material "piano", first image material "flute", first image material "drum set"}. If the audio track corresponding to the first image material "drum set" overlaps with the audio track corresponding to the first image material "piano" in terms of sound time, but does not overlap with the audio track corresponding to the first image material "flute", then the first image material "piano" is determined as the second image material. Furthermore, when determining the display area of the first image material "drum set", the display area of the video frame excluding the first image material "piano" can be used. The area outside the designated area is defined as the displayable area of the first image material "drum set". Alternatively, the area in the video frame other than the display area of the first image material "piano", as well as the pre-defined edge transition area in the display area of the first image material "piano" that can overlap with other areas, are all defined as the displayable area of the first image material "drum set". If the audio track corresponding to the first image material "drum set" does not overlap with the audio tracks corresponding to the first image material "piano" and the first image material "flute" in terms of sound time, then the complete video frame is defined as the displayable area of the first image material "drum set".
[0081] In this embodiment, the determination of the displayable area is carried out using two different processes based on whether there is a second image material whose audio track playback time overlaps with that of the nth first image material among the first n-1 first image materials. When there is at least one second image material, the displayable area is limited to the area in the video frame other than the display area of the second image material, so that the display area of the nth first image material will not encroach on the area occupied by other materials whose audio track playback time overlaps. When there is no second image material, the displayable area is the entire video frame, providing a complete spatial selection range for the nth first image material. This dynamic spatial constraint mechanism based on the audio track playback time overlap relationship keeps the allocation of the display area consistent with the temporal correlation between each image material.
[0082] In some embodiments, "determining the display area of the nth first image material from the displayable area based on the shape mask of the nth first image material" can be achieved by the following process: determining multiple candidate areas from the displayable area based on the shape mask of the nth first image material; determining the distance between the geometric center point of each candidate area and the center point of the video frame; and determining the candidate area corresponding to the minimum distance as the display area of the nth first image material.
[0083] The geometric center point refers to the arithmetic mean of the coordinates of all pixels within the candidate region. The geometric center point is used to characterize the spatial centroid of the candidate region in visual distribution.
[0084] As an example of "determining the display area of the nth first image material", based on the shape mask of the nth first image material, a spatial matching search is performed in the displayable area of the nth first image material to obtain multiple candidate areas. The two-dimensional coordinates of the center point of the video frame are determined. The distance between the two-dimensional coordinates of the geometric center point of each candidate area and the two-dimensional coordinates of the center point of the video frame is calculated. The calculated distances are compared, the minimum distance is selected, and the candidate area corresponding to the minimum distance is determined as the final display area of the nth first image material.
[0085] For example, in the application scenario of multi-track dynamic song video generation, if the resolution of the initial video frame is 1920×1080, then the coordinates of the center point of the video frame are (960, 540). Based on the shape mask of the first image material "bass", spatial matching search is performed in its displayable area to obtain candidate region A and candidate region B. The geometric center point coordinates of candidate region A are calculated to be (800, 500) and the geometric center point coordinates of candidate region B are (1500, 800). The straight-line distance between the geometric center point of each candidate region and the center point of the video frame is determined. If the distance corresponding to candidate region A is less than the distance corresponding to candidate region B, then candidate region A corresponding to the smallest distance is determined as the display area of the first image material "bass".
[0086] In this embodiment, after determining multiple candidate regions from the displayable area based on the shape mask of the nth first image material, the distance between the geometric center point of each candidate region and the center point of the video frame is calculated, and the candidate region corresponding to the smallest distance is determined as the display area of the first image material. This processing method makes the center point position of each candidate region the core basis for region selection. By comparing the distances from the center points of each candidate region to the center point of the video frame, the candidate region with the smaller distance is selected first, so that the final determined display area tends to be in the center of the video frame in terms of spatial distribution. This center-oriented region selection mechanism allows the display areas of each first image material to move towards the center of the frame in sequence, forming an overall clustered layout.
[0087] In some embodiments, "determining the display area of the nth first image material from the displayable area based on the shape mask of the nth first image material" can be achieved through the following processing: when n equals 1, if the sound duration of the audio track corresponding to the first first image material is greater than a preset duration threshold, the center area of the video frame is determined as the display area of the first first image material; if the sound duration of the audio track corresponding to the first first image material is less than or equal to the duration threshold, based on the shape mask of the first first image material, multiple first candidate areas are determined from the video frame of the initial video, and any first candidate area is determined as the display area of the first first image material; when n is greater than 1, based on the shape mask of the nth first image material, multiple second candidate areas are determined from the displayable area, and any second candidate area is determined as the display area of the nth first image material.
[0088] The coordinates of the geometric center point of the central region of the video frame are the same as the coordinates of the center point of the video frame.
[0089] As an example, when n equals 1, the duration of the audio track corresponding to the first first image material is obtained, and compared with a preset duration threshold. If the duration of the audio track corresponding to the first first image material is greater than the preset duration threshold, it indicates that the audio track corresponding to the first first image material dominates the target audio, and the central area of the video frame is determined as the display area of the first first image material. If the duration of the audio track corresponding to the first first image material is less than or equal to the duration threshold, spatial matching is performed on the complete initial video frame based on the shape mask of the first first image material to filter out multiple first candidate areas that meet the mask size and do not exceed the frame boundary, and any one of these first candidate areas is determined as the display area of the first first image material. For subsequent materials in the image material sequence (i.e., when n is greater than 1), matching is performed on the corresponding displayable area based on the shape mask of the nth first image material to filter out multiple second candidate areas that do not cross the boundary, and any one of these second candidate areas is determined as the display area of the nth first image material.
[0090] In some embodiments, Figure 4 The step 1033 shown can be implemented by the following process: when the display area is divided in a uniform manner, the video frame of the initial video is evenly divided into multiple sub-regions; the multiple sub-regions are determined as the display areas of multiple first image materials, wherein the sub-regions and the display areas correspond one-to-one.
[0091] As an example, when the display area is divided in a uniform manner, the first number of first image materials is obtained. Through a spatial cutting algorithm, the video frame of the initial video is evenly divided into a first number of rectangular sub-regions of the same size that do not overlap. The multiple first image materials are mapped one-to-one with the multiple sub-regions obtained by spatial positioning. Each sub-region is determined as the display area of the corresponding first image material in the initial video.
[0092] For example, in the application scenario of multi-track dynamic music video generation, the number of first image materials is 4. Given that the determined display area is divided in a uniform manner and the initial video resolution is 1920×1080, a 2-row, 2-column grid layout is adopted to evenly divide the complete video screen into 4 rectangular sub-regions with equal area and resolution of 960×540. Subsequently, the four first image materials are respectively assigned to the four rectangular sub-regions of upper left, upper right, lower left, and lower right, and these four independent sub-regions are determined as the display areas of the four first image materials.
[0093] In this embodiment of the application, when a uniform division method is adopted, the video frame of the initial video is uniformly divided into multiple sub-regions, and these sub-regions are directly determined as the display areas of multiple first image materials, so that each sub-region and each display area form a one-to-one correspondence. Since the division process does not depend on the specific shape mask or audio track features of each image material, but is based on the geometric equal division principle of the video frame, each display area has the same positional distribution pattern and similar spatial size in the video frame. This regularized area allocation method makes multiple display areas present a neat and consistent arrangement in the video frame, thereby ensuring that the display areas of each first image material have a balanced spatial proportion and coordinated positional relationship.
[0094] In step 104, based on the display area of each first image material in the initial video, multiple first image materials are rendered into the initial video to obtain the rendered initial video.
[0095] In some embodiments, Figure 3 Step 104 shown can be achieved by performing the following processing on the m-th first image material: based on the audio track corresponding to the m-th first image material, determine at least one display time period of the m-th first image material; for each display time period, determine at least one video frame corresponding to the display time period from the initial video after the (m-1)-th drawing; based on the display area of the m-th first image material, draw the m-th first image material in each video frame to obtain the initial video after the m-th drawing.
[0096] Where m is an integer, 0 < m ≤ M, M is the number of the first image materials, the initial video after the Mth drawing is the initial video after rendering, and the initial video after the zeroth drawing is the initial video.
[0097] As an example of step 104, at least one sound time period of the audio track corresponding to the m-th first image material is obtained, and the at least one sound time period is determined as at least one display time period of the m-th first image material. For each display time period, the following processing is performed: at least one video frame corresponding to the display time period is determined from the initial video after the (m-1)-th drawing. The m-th first image material can be drawn directly in the target area of each video frame based on Alpha channel mixing technology to obtain the initial video after the m-th drawing, where the target area is the display area of the m-th first image material.
[0098] In another example, during the process of rendering the first image material to the target area of the video frame, the rendering effect of the m-th first image material can be dynamically adjusted based on the changes in the acoustic index of the audio track corresponding to the m-th first image material during the display time period. For example, when the acoustic index starts to rise from the preset index threshold, the m-th first image material is applied with entrance effects such as fading in, becoming clearer, or being magnified. When the acoustic index drops to the preset index threshold, the m-th first image material is applied with exit effects such as fading out, becoming blurred, or shrinking.
[0099] In this embodiment, after determining at least one display time period based on the audio track corresponding to the m-th first image material, at least one video frame corresponding to the initial video after the (m-1)-th drawing is obtained for each display time period. The material is then drawn in each video frame according to the display area of the m-th first image material, thereby generating the initial video after the m-th drawing. This frame-by-frame drawing method based on display time periods ensures that each first image material is drawn into the video frame only within a specific time interval corresponding to its audio track. As m increases from 1 to M, each first image material is sequentially superimposed on the previous drawing result according to its own time interval, ultimately ensuring that all first image materials appear precisely in the corresponding video frames within their respective display time periods, achieving the alignment of the image material with the audio track timeline.
[0100] In step 105, the rendered initial video and the target audio are combined to obtain the target video.
[0101] As an example of step 105, the display timestamps of each video frame in the rendered initial video and the display timestamps of each audio frame in the target audio are extracted respectively. Using the display timestamps of each video frame and the display timestamps of each audio frame as alignment references, the rendered initial video and the target audio are synthesized by an audio-video multiplexer to obtain the target video.
[0102] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.
[0103] With the synergistic development of artificial intelligence (AI) and video technology, many innovative application models have emerged in the music playback field. Dynamic song videos are an automated video technology that uses AI to integrate lyrics with dynamic special effects and display them synchronously with the music rhythm, thereby enhancing the user's audiovisual experience. The generation of dynamic song videos in related technologies mainly involves overlaying lyrics onto static images and combining this with font color filling (such as karaoke coloring), font motion scaling, particle effects, and shaking animation effects. These videos are primarily generated around the lyrics themselves, or synthesized by matching a material library based on user preferences. Videos generated in these ways rarely reflect the music itself, especially for songs in less common languages, chants (without specific lyrics), and purely instrumental music, where the expressiveness of these dynamic song videos is poor. Furthermore, while dynamic song videos in related technologies can achieve precise synchronization between lyrics and music, they struggle to visually showcase the diversity of the musical arrangement itself, such as the gradual coordination of different instruments, the strength of drum beats, chanting, and special sounds (e.g., whistling, percussion, flowing water, wind, insect chirping).
[0104] This application addresses the problem that videos generated in related technologies often fail to reflect the characteristics of the music itself. It proposes a dynamic song video generation method based on the composition of the music. Specifically, this application uses signal separation technology to separate the song's audio file into multiple tracks, determining the start and end times and relative intensity of each track's sound. Then, an audio model identifies the source of sound for each track, such as the singer's voice, a musical instrument (piano, guitar, drums, wind chimes, etc.), or a natural sound (cicada chirping, birdsong, wind sounds, flowing water, etc.). Based on the source of sound at each time point and its intensity changes, a large video model is used to generate materials (images or videos) containing each sound source. Finally, the materials from multiple tracks are superimposed to generate a dynamic song video, thus more intuitively reflecting the song's musicality and arrangement diversity.
[0105] In its specific implementation, this application embodiment adds large model recognition to the intelligent audio track separation, which can identify the sound source of each audio track, helping to enhance the user's understanding of the music itself (not limited to lyrics); for video synthesis of a single audio track, based on the source of sound at each time point and the changes in their respective sound intensity, the video large model is called to generate materials (images or videos) containing each sound source, so as to accurately and intuitively reflect the composition of the music and the diversity of the arrangement; at the same time, this application embodiment can control the entrance and exit video animation effects of the corresponding target object material according to the sound intensity of a single audio track; and the video materials corresponding to multiple audio tracks are fused and superimposed through shape matching, so that the final generated music video can dynamically visualize the arrangement of the song; in addition, this application embodiment processes the features of the song into the format of audio track dimension vectors and saves them to a structured database, effectively saving storage resources while preserving the dimensional features of the music.
[0106] See Figure 5 , Figure 5 This is a first schematic diagram of the video generation method provided in the embodiments of this application. The following will be combined with... Figure 5 The specific implementation of the video generation method provided in this application embodiment will be described (in order to more clearly illustrate the embodiments of this application, the embodiments of this application will be described in the form of steps, and it is worth noting that the following steps are not shown in the figures).
[0107] Step 1: Obtain the audio source file of the song (i.e., the target audio mentioned above) and input the audio source file into the audio separation module 501.
[0108] Step 2: The audio separation module 501 separates multiple audio tracks from the audio source file of the song. Specifically, the audio source file of the song is separated by short-time Fourier transform to obtain the frequency domain signals of different audio tracks. For the frequency domain signal of each separated audio track, the audio signal of the track is restored by inverse Fourier transform.
[0109] Step 3: Using the audio track sound source identification module 502, the audio signal of the audio track generated in Step 2 is input into the audio recognition model to identify and output its sound source (i.e., the sound object mentioned above). Examples include singer's vocals, piano, drums, water sounds, insect chirps, and some special sound effects. The sound source can be characterized as... .
[0110] The training process of the recognition model is as follows: In terms of dataset preparation, multiple datasets are prepared according to actual needs, including spectral features of various audio signals (e.g., MFCC spectrum / FBANK features) and labeled audio sources. Specifically, the datasets include: basic human voice datasets (e.g., LibriSpeech), musical instrument sound datasets (e.g., Nsynth), animal sound datasets, natural sound datasets, and special scene sound datasets, etc. In terms of modeling, a CNN+Transformer hybrid architecture is adopted, where CNN is used to process spectral features, and Transformer is used to capture long-term temporal dependencies. Its configured loss function is used for... The algorithm optimizes both CTC loss (to solve sequence alignment problems) and cross-entropy loss (to improve classification accuracy). During training, the dataset is first divided into training, validation, and test sets. A pre-trained audio model (such as PANNs) is used for transfer learning. Then, based on the input spectral features, the predicted probability and loss function are calculated. Parameters are updated via backpropagation, and the learning strategy is dynamically adjusted through validation evaluation using the validation dataset. This iterative training process is repeated. Finally, the audio recognition model is evaluated and validated using the test dataset. For application deployment, FastAPI provides a REST interface for outputting the corresponding audio source name when an audio file is input.
[0111] Step 4: Using the audio track sound intensity recognition module 503, for the audio signals of the audio track generated in Step 2, record the change in sound intensity (sound intensity is just a common indicator; it can also be other indicators such as audio frequency, depth, BPM, etc.) of each audio signal over time. This time-varying relationship can be characterized as... , , , These represent the start and end times of the song's audio source file, respectively, combined with the sound source generated in step 3. This yields the audio track dimension vector. ,in, =( , ), The track dimension vector corresponding to the i-th track is stored in the song track dimension storage module 504.
[0112] Step 5: When the instruction to generate a dynamic video of a song is received, the corresponding song's audio track dimension vector is read from the song audio track dimension storage module 504. .
[0113] Step 6: Using the single-track target material generation module 505, generate vectors for each track dimension. To obtain the source of their voice Based on the user's selected lyric video style (e.g., realistic, minimalist, comic, cyberpunk, etc., i.e., the video style mentioned above), the corresponding audio source material is determined (e.g., for human voices, lyrics can be displayed; for pianos, piano keys can be displayed, i.e., the first image material mentioned above), and the shape mask of the audio source material is recorded. .
[0114] See Figure 6 , Figure 6 This is a second schematic diagram of the video generation method provided in the embodiments of this application. Figure 6 This visually demonstrates the complete workflow architecture from audio processing to video rendering. The process begins with acquiring the audio source file and then... (The sentence is incomplete and ends abruptly.) to The timing is defined first, then the audio source file is separated into multiple tracks. Next, for the i-th track, the target material "piano" is determined, and the screen area is divided to define independent display spaces for drums, piano, guitar, and lyrics within the video canvas. Then, for the i-th track, its sound intensity is determined... exist to The dynamic curves on the axis control the corresponding visual elements to go through the animation stages of material entry, displaying target element material, and material exit in the time dimension. Finally, the materials corresponding to all audio tracks are merged and superimposed in the spatial and temporal dimensions to output a target video composed of multiple frames of static images arranged continuously. Each frame dynamically combines elements such as piano, drums, and guitar, as well as content with lyrics.
[0115] Step 7: Mask the shape of the sound-producing object material obtained in Step 6. ~ Within the entire video frame (i.e., the initial video frame mentioned above), a display area (i.e., the display area mentioned above) is selected sequentially for each audio track dimension vector, resulting in multiple display areas. ,in, This is the display area corresponding to the i-th audio track.
[0116] Depending on the visual style of the chosen lyrics video, the method for selecting the display area can vary; see [link / reference]. Figure 7 , Figure 7 This is a third schematic diagram of the video generation method provided in the embodiments of this application, showing multiple display areas. It can be like Figure 7 The multiple regularly shaped regions shown in part (a) can also be as follows: Figure 7 Part (b) shows multiple irregularly shaped regions, and there may be overlap between different regions.
[0117] Step 7.1: For each audio track dimension vector The relationship between the corresponding sound intensity and time is calculated to satisfy... The time set is , Calculate the total duration of the sound source. (i.e., the duration of the speech mentioned above). ,in, This represents the critical value for sound production; the sound intensity value is below a certain threshold. It is considered as if no voice was spoken. express There are a total The segment is greater than or equal to The sub-function (i.e., the time period shown above). Indicates the first The segment is greater than or equal to The start time of the sub-function. Indicates the first The segment is greater than or equal to The end time of the sub-function.
[0118] Step 7.2: Total vocalization time based on the vocalization source of each audio track dimension vector Sort the multiple audio track dimension vectors in descending order to obtain the sorted audio track dimension vectors, denoted as . ,in, The sequence number is used for sorting. .
[0119] Step 7.3: For the audio track dimension vector with index 0 after sorting The corresponding relationship between the sound intensity and time is as follows: Determine the source of the sound and the total duration of the sound production. If the proportion of the sound source to the total song duration is above 90%, then the sound source is considered the main framework of the entire piece and can occupy the central position, allowing for the acquisition of its corresponding shape mask. And in the video footage China selected a display area for it. .
[0120] Step 7.4: Obtain the dimension vector of the audio track with the sorted index 1. Corresponding shape mask Determine its corresponding time set and (Audio track dimension vector) The relationship between the corresponding time sets: If This indicates that the audio tracks corresponding to the two audio track dimension vectors have a temporal overlap, then according to... In the video footage (The area shown above) contains the audio track dimension vector. Select display area ,in, , Indicates video frame Not belonging to Part of express Edges are allowed to overlap with other areas. If This indicates that the two audio tracks do not overlap in time, therefore according to... In the video footage Selected .
[0121] Step 7.5: Starting from sort number M=2, obtain the shape mask of the sound source corresponding to the audio track dimension vector of that sort number. Determine the corresponding time set for each one. and Relationship: If This indicates that the two audio tracks overlap in time. ,in, Indicates video frame Not belonging to Part of express Edges are allowed to overlap with other areas. If This indicates that the two sound sources do not overlap in time. ,go through Round iteration ( From 0 to ) to obtain the final ,according to Shape in the video frame Selected display area .
[0122] Step 7.6: Increment M one by one and repeat the operation of Step 7.5 until M=N (inclusive), to complete the selection of the display area corresponding to all audio tracks.
[0123] See Figure 8 , Figure 8 This is the fourth schematic diagram of the video generation method provided in the embodiments of this application. Figure 8 From left to right, the entire evolution process of audio track feature calculation, sorting, and multi-round iterative overlay of screen layout is shown in detail. First, the top left corner shows the relationship between the sound intensity of each audio track and time. , , , , On the timeline to The process involves "calculation" within the system; subsequently, it is "sorted" by descending order, and the corresponding sorted relationship curve is obtained in the lower left corner. , , , Arranged from top to bottom Next, the iterative steps of layer-by-layer cropping and material fusion of the canvas area are clearly shown within the dotted box on the right: the first round is done in the "blank area" In the initial state, determine the display area corresponding to the first audio track. After the second update, in the blank space... The selected display area is then overlaid into the image. and The third round, based on this, will fill in the blank spaces. The image incorporates a bottom strip display area. In the fourth iteration, the blank space was... The image incorporates a display area shaped like a musical instrument. Finally, in the fifth round, in the "blank space" Animal-shaped display areas are superimposed on the image. It meticulously implemented the cascading selection of all audio tracks' corresponding display areas.
[0124] Step 8: In this embodiment of the application, for each audio track dimension vector Corresponding relationship between sound intensity and time When time is satisfied At that time, based on the change in its sound intensity value, it will appear in its corresponding display area. The video frame of the sound source material is generated internally. Specifically, if the sound intensity value is from If the sound intensity begins to rise, the corresponding material will be controlled to display an entrance effect (such as fade-in, becoming clearer, or becoming larger). If the sound intensity value drops to... Then control the corresponding material to display on-screen effects (such as fade-out, blurring, shrinking, etc.), and repeat step 8 until all audio track dimension vectors are superimposed. The corresponding materials are used to create the superimposed video (i.e., the initial video rendered above).
[0125] Step 9: In the video overlay and fusion generation module 506, the overlay video generated in step 8 is overlaid and synthesized with the song audio to obtain a complete dynamic song video (i.e., the target video mentioned above).
[0126] The following description continues to illustrate the exemplary structure of the video generation apparatus 555 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software module stored in the video generation device 555 of the memory 550 may include: The data acquisition module 5551 is used to acquire the target audio, perform track separation processing on the target audio, and obtain multiple tracks of the target audio.
[0127] The object recognition module 5552 is used to identify the sound source of each audio track of the target audio, obtain the sound source of the audio track, and determine the first image material corresponding to the sound source based on the sound source and a preset video style.
[0128] The region segmentation module 5553 is used to acquire the initial video and determine the display area of each first image material in the initial video based on the shape mask of each first image material.
[0129] The material rendering module 5554 is used to render multiple first image materials to the initial video based on the display area of each first image material in the initial video, so as to obtain the rendered initial video.
[0130] The video compositing module 5555 is used to perform video compositing on the rendered initial video and the target audio to obtain the target video.
[0131] In some embodiments, the region segmentation module 5553 is further configured to, for each first image material, determine the geometric complexity of the first image material based on the shape mask of the first image material; determine a display region segmentation method based on multiple geometric complexities; and determine the display region of each first image material in the initial video based on the display region segmentation method.
[0132] In some embodiments, the region division module 5553 is further configured to, when the display region division method is a dynamic adaptation division method, sort multiple first image materials in descending order based on the sound duration of the audio track corresponding to each first image material to obtain an image material sequence; for the first first image material in the image material sequence, determine the display area of the first first image material from the video frame of the initial video based on the shape mask of the first first image material; for the nth first image material in the image material sequence, determine the displayable area of the nth first image material in the video frame based on the display areas of the first n-1 first image materials, where n is a positive integer greater than 1, n≤N, and N is the number of first image materials; and determine the display area of the nth first image material from the displayable area based on the shape mask of the nth first image material.
[0133] In some embodiments, the region division module 5553 is further configured to, when at least one second image material is determined from the first n-1 first image materials, define the region in the video frame other than the display area of the second image material as the displayable region, wherein the audio track corresponding to the second image material overlaps with the audio track corresponding to the nth first image material in terms of sound emission time; when no second image material is determined from the first n-1 first image materials, define the video frame as the displayable region.
[0134] In some embodiments, the region segmentation module 5553 is further configured to determine a plurality of candidate regions from the displayable region based on the shape mask of the nth first image material; determine the distance between the geometric center point of each candidate region and the center point of the video frame; and determine the candidate region corresponding to the smallest distance as the display region of the nth first image material.
[0135] In some embodiments, the region division module 5553 is further configured to, when the display region division method is a uniform division method, uniformly divide the video frame of the initial video into multiple sub-regions; and determine the multiple sub-regions as display regions of multiple first image materials, wherein the sub-regions and the display regions correspond one-to-one.
[0136] In some embodiments, the material rendering module 5554 is further configured to perform the following processing for the m-th first image material: based on the audio track corresponding to the m-th first image material, determine at least one display time period for the m-th first image material, where m is an integer, 0 < m ≤ M, and M is the number of first image materials; for each display time period, determine at least one video frame corresponding to the display time period from the initial video after the (m-1)-th drawing, and based on the display area of the m-th first image material, draw the m-th first image material in each video frame to obtain the initial video after the m-th drawing, where the initial video after the M-th drawing is the rendered initial video, and the initial video after the zeroth drawing is the initial video.
[0137] This application provides a computer program product, which includes computer-executable instructions or a computer program. The processor of an electronic device reads the computer-executable instructions or computer program from a computer-readable storage medium, and executes the computer-executable instructions or computer program, causing the electronic device to perform the video generation method described in this application.
[0138] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the video generation method provided in this application.
[0139] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.
[0140] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.
[0141] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).
[0142] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.
[0143] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.
Claims
1. A video generation method, characterized in that, The method includes: The target audio is acquired, and the target audio is subjected to track separation processing to obtain multiple audio tracks of the target audio; For each audio track of the target audio, the sound source is identified to obtain the sound source of the audio track. Based on the sound source and a preset video style, the first image material corresponding to the sound source is determined. Acquire an initial video, and determine the display area of each first image element in the initial video based on the shape mask of each first image element; Based on the display area of each first image material in the initial video, multiple first image materials are rendered into the initial video to obtain the rendered initial video; The rendered initial video and the target audio are combined to obtain the target video.
2. The method according to claim 1, characterized in that, Determining the display area of each first image element in the initial video based on the shape mask of each first image element includes: For each of the first image materials, the geometric complexity of the first image material is determined based on the shape mask of the first image material; Based on the aforementioned geometric complexities, the method for dividing the display area is determined; Based on the aforementioned display area division method, the display area for each of the first image materials in the initial video is determined.
3. The method according to claim 2, characterized in that, Determining the display area of each first image material in the initial video based on the display area division method includes: When the display area is divided in a dynamic adaptation manner, the multiple first image materials are sorted in descending order based on the sound duration of the audio track corresponding to each first image material to obtain an image material sequence. For the first first image in the image material sequence, based on the shape mask of the first first image, the display area of the first first image is determined from the video frame of the initial video; For the nth first image material in the image material sequence, based on the display areas of the previous n-1 first image materials, the displayable area of the nth first image material in the video frame is determined, where n is a positive integer greater than 1, n≤N, and N is the number of first image materials; Based on the shape mask of the nth first image material, the display area of the nth first image material is determined from the displayable area.
4. The method according to claim 3, characterized in that, The step of determining the displayable area of the nth first image material in the video frame based on the display areas of the first n-1 first image materials includes: When at least one second image material is determined from the first n-1 first image materials, the area in the video frame other than the display area of the second image material is determined as the displayable area, wherein the audio track corresponding to the second image material overlaps with the audio track corresponding to the nth first image material in terms of sound output time; When no second image material is determined from the first n-1 first image materials, the video frame is determined as the displayable area.
5. The method according to claim 3, characterized in that, The step of determining the display area of the nth first image material from the displayable area based on the shape mask of the nth first image material includes: Based on the shape mask of the nth first image material, multiple candidate regions are determined from the displayable area; Determine the distance between the geometric center point of each candidate region and the center point of the video frame; The candidate region corresponding to the minimum distance is determined as the nth display region of the first image material.
6. The method according to claim 2, characterized in that, Determining the display area of each first image material in the initial video based on the display area division method includes: When the display area is divided in a uniform manner, the video frame of the initial video is evenly divided into multiple sub-areas. The plurality of sub-regions are defined as display areas for the plurality of the first image materials, wherein the sub-regions and the display areas correspond one-to-one.
7. The method according to claim 1, characterized in that, The step of rendering multiple first image materials onto the initial video based on the display area of each first image material in the initial video to obtain the rendered initial video includes: For the m-th first image material, perform the following processing: Based on the audio track corresponding to the m-th first image material, at least one display time period for the m-th first image material is determined, where m is an integer, 0 < m ≤ M, and M is the number of first image materials; For each display time period, at least one video frame corresponding to the display time period is determined from the initial video after the (m-1)th drawing. Based on the display area of the m-th first image material, the m-th first image material is drawn in each video frame to obtain the initial video after the m-th drawing. The initial video after the M-th drawing is the rendered initial video, and the initial video after the zeroth drawing is the initial video.
8. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, configured to execute computer-executable instructions or computer programs stored in the memory, implements the video generation method according to any one of claims 1 to 7.
9. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the video generation method according to any one of claims 1 to 7 is implemented.
10. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the video generation method according to any one of claims 1 to 7 is implemented.
