Music generation methods and devices, electronic devices, computer media

Abstract

This disclosure provides a music generation method and apparatus, electronic device, and computer medium, relating to the field of data processing technology, particularly to the fields of artificial intelligence, video data processing, and music generation. The specific implementation scheme is as follows: The video to be processed is analyzed to obtain multi-layered analysis results, which include at least global semantic information and shot structure information; wherein, the global semantic information is used to characterize the overall content semantic information of the video to be processed; the shot structure information is used to characterize multiple shot segments in the video to be processed, divided chronologically, and the content semantic information of each shot segment; based on the multi-layered analysis results, multimodal control features are constructed; based on the multimodal control features, a pre-constructed music generation model is driven to generate background music matching the duration of the video to be processed.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and more particularly to the fields of artificial intelligence, video data processing, and music generation. Specifically, this disclosure relates to a method and apparatus for music generation based on video analysis, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the explosive growth of multimedia content such as short videos, film and television editing, and advertising production, automatically matching suitable background music to videos has become a widespread and urgent need. Traditional music selection methods rely on manual selection and editing, which is inefficient and highly dependent on professional experience. To address this, the industry has proposed various AI-based video music selection or music generation solutions, aiming to automatically generate or recommend matching music based on video content. Summary of the Invention

[0003] This disclosure provides a method and apparatus for generating music based on video parsing, an electronic device, and a computer-readable storage medium.

[0004] According to a first aspect of this disclosure, a method for generating music based on video parsing is provided, the method comprising: The video to be processed is parsed to obtain multi-layered parsing results, which include at least global semantic information and shot structure information. The global semantic information is used to characterize the overall content semantic information of the video to be processed. The shot structure information is used to characterize multiple shot segments in the video to be processed in chronological order and the content semantic information of each shot segment. Based on the multi-layer analysis results, multimodal control features are constructed; Based on the multimodal control features, a pre-built music generation model is driven to generate background music that matches the duration of the video to be processed; The music generation model is divided into multiple generation stages corresponding to each shot segment based on the shot structure information. In each generation stage, the music generation model is constrained to generate background music only based on the multimodal control features of the shot segment corresponding to the current generation stage.

[0005] According to a second aspect of this disclosure, a music generation apparatus based on video analysis is provided, the apparatus comprising: The video parsing module is used to parse the video to be processed and obtain multi-layer parsing results. The multi-layer parsing results include at least global semantic information and shot structure information. The global semantic information is used to characterize the overall content semantic information of the video to be processed. The shot structure information is used to characterize multiple shot segments in the video to be processed in chronological order and the content semantic information of each shot segment. The feature generation module is used to construct multimodal control features based on the multi-layer parsing results; The music generation module is used to drive a pre-built music generation model to generate background music that matches the duration of the video to be processed, based on the multimodal control features. The music generation model is divided into multiple generation stages corresponding to each shot segment based on the shot structure information. In each generation stage, the music generation model is constrained to generate background music only based on the multimodal control features of the shot segment corresponding to the current generation stage.

[0006] According to a third aspect of this disclosure, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to at least one of the aforementioned processors; wherein, The memory stores instructions that can be executed by at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the video-based music generation method.

[0007] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the above-described video-based music generation method.

[0008] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described video-based music generation method.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart illustrating a video parsing-based music generation method provided in an embodiment of this disclosure; Figure 2 This is a flowchart illustrating some steps of another video-based music generation method provided in this embodiment of the present disclosure; Figure 3 This is a flowchart illustrating some steps of another video-based music generation method provided in this embodiment of the present disclosure; Figure 4This is a flowchart illustrating some steps of another video-based music generation method provided in this embodiment of the present disclosure; Figure 5 This is a flowchart illustrating some steps of another video-based music generation method provided in this embodiment of the present disclosure; Figure 6 This is a schematic diagram of the structure of a music generation device based on video parsing provided in an embodiment of this disclosure; Figure 7 This is a block diagram of an electronic device used to implement the video parsing-based music generation method of the present disclosure embodiments. Detailed Implementation

[0011] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0012] Some related technologies focus on the global matching of video content with musical mood. These technologies typically utilize deep learning models, such as convolutional neural networks or visual Transformers, to extract global semantic or emotional features from the video. These features are then used as conditions to drive music generation models to generate music whose emotional tone matches the overall atmosphere of the video.

[0013] However, the essence of this type of technology is to understand the video as a whole. While the music it generates may roughly match the "thematic mood" of the video, it completely ignores the dynamic and structured narrative changes within the video. For a video that includes introduction, development, climax, and transitions, as well as multiple shots and scene changes, the music generated is often a single emotion or a smooth transition, unable to respond to specific plot twists, shot changes, or action climaxes in the video. This results in the music and visuals being disconnected on the timeline, lacking the narrative drive and rhythm that film scores should have.

[0014] The video-based music generation method, apparatus, electronic device, and computer-readable storage medium disclosed herein are intended to solve at least one of the above-mentioned technical problems of the prior art.

[0015] The video-based music generation method provided in this disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device can be an in-vehicle device, user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The method can be implemented by a processor calling computer-readable program instructions stored in memory. Alternatively, the method can be executed by a server.

[0016] Figure 1 A flowchart illustrating the video parsing-based music generation method provided in this disclosure is shown. Figure 1 As shown in the embodiments of this disclosure, the music generation method based on video parsing may include the following steps: S110. Analyze the video to be processed to obtain multi-layered analysis results. The multi-layered analysis results include at least global semantic information and shot structure information. Among them, global semantic information is used to represent the overall content semantic information of the video to be processed; shot structure information is used to represent multiple shot segments in the video to be processed in chronological order and the content semantic information of each shot segment.

[0017] The video to be processed refers to any video file or video stream that needs to have background music generated.

[0018] The process of parsing the video to be processed and obtaining multi-layered parsing results can be a collection of information obtained from analyzing the video from different dimensions, which includes at least two types of key information: Global semantic information: Abstract information that represents the theme, tone, mood and atmosphere of the entire video, such as "outdoor travel", "healing", "light atmosphere", etc., used to define the overall tone of music generation; Shot structure information: describes the timeline structure of the video, representing the shot segments divided into time sequences and the attributes of each segment, including the start time, end time, boundary position, and duration of each shot segment, as well as the semantic tags of each segment (such as "main character entrance", "action enhancement", "climax scene"), which are used to provide temporal structural constraints for music generation.

[0019] In some possible implementations, a pre-trained visual encoder can be used to encode the video frames of the video to be processed, capture information such as objects, scenes, and colors within the video frames, obtain high-dimensional visual feature vectors, and aggregate the visual feature vectors of all video frames to obtain global visual features. By calling text generation models, such as the GPT (Generative Pre-trained Transformer) series and the BERT model (a pre-trained language model based on the Transformer architecture), global visual features are mapped to natural language descriptive text, such as "relaxing outdoor travel scene, overall mood healing, and atmosphere gentle". This text is the global semantic information, which can be further refined into keyword tags, such as "outdoor travel", "healing", and "gentle", for subsequent feature construction.

[0020] In some possible implementations, a visual encoder can be used to encode the video frames of the video to be processed, obtain the visual semantic features of the video frames, and input these feature sequences into a sequence aggregation model. The semantic features of all video frames are fused through an attention mechanism, and the input is a global semantic vector. Then, a lightweight text decoder generates a text description summarizing the overall content of the video based on the global semantic vector, which serves as the global semantic information.

[0021] In some possible implementations, a shot boundary detection algorithm can be used to identify all shot transition points in the video to be processed, and the video segments between adjacent transition points are defined as a shot segment. The shot boundary detection algorithm may include calculating inter-frame difference data (pixel grayscale value difference), color distribution similarity (HSV histogram Bach distance), and the degree of change of objects and actions (sparse optical flow detection results) between adjacent frames.

[0022] For each shot segment, the start timestamp, end timestamp, and duration of the shot segment are recorded. Then, using the method described above, i.e., using a visual encoder to encode the video frames of the shot segment, the feature vectors of the video frames are obtained. These feature vectors are then fused, and a text generation model is used to generate semantic tags for the shot segment based on the fused features. For example, "Shot 1: The subject slowly enters, with gentle movements" and "Shot 2: The character begins to run, with increased intensity of movements". Finally, a shot structure information table containing "segment number - start time - end time - semantic tag" is formed.

[0023] In some possible implementations, an image description generation model can be used to generate a text description for each video frame of a shot segment, and then these text descriptions can be summarized or clustered to obtain semantic information representing the core content of the segment.

[0024] S120. Based on the multi-layer analysis results, construct multi-modal control features.

[0025] Constructing multimodal control features based on multi-level analysis results is the process of transforming multi-level analysis results into structured feature signals that can be understood by the music generation model. It is the core bridge connecting video analysis and music generation.

[0026] The constructed multimodal control features include at least multi-dimensional fusion features such as text features, visual features, and temporal structure features.

[0027] The text features can include global semantic information and content semantic information for each shot segment; Visual features can include visual feature vectors obtained through a visual encoder.

[0028] Temporal structure features can include structured tags such as timestamps (start and end times) and relative durations for each shot segment.

[0029] In some possible implementations, the text features, visual features, and temporal structure features corresponding to the same shot segment can be concatenated or fused through a feature fusion network to form the multimodal control features corresponding to that shot segment.

[0030] S130. Based on multimodal control features, drive a pre-built music generation model to generate background music that matches the duration of the video to be processed; The music generation model's generation process is divided into multiple generation stages corresponding to each shot segment based on the shot structure information. In each generation stage, the music generation model is constrained to generate background music only based on the multimodal control features of the shot segment corresponding to the current generation stage.

[0031] Music generation models refer to pre-trained machine learning models that can generate music audio or musical symbol sequences (such as MIDI) based on conditional input, including but not limited to generative models based on the Transformer architecture, diffusion models, or recurrent neural networks.

[0032] The generation stage refers to the division of the working space of the music generation model in the time dimension. Each stage strictly corresponds to the time range of a shot segment, and only generates suitable music segments based on the features of a single shot segment to ensure accurate matching between music and shot content.

[0033] Constraints refer to the limiting conditions imposed during the music generation process to ensure that the music generation model's decisions, within a specific generation stage, rely solely on the multimodal control of the shot segment associated with the current stage, thus avoiding interference from information from other segments.

[0034] In some possible implementations, the timeline of the music generation model is strictly divided into generation stages that correspond one-to-one with the shot segments, and the duration of the music generated in each generation stage is exactly the same as the duration of the corresponding shot segment.

[0035] When generating music for the i-th shot segment, the music generation model is constrained to only "see" and utilize the multimodal control features corresponding to the i-th shot segment. The music generation model generates note sequences or audio spectra progressively within each constrained generation stage, using either an autoregressive or non-autoregressive approach. The style and rhythm of the generated note sequences or audio spectra are adapted to the semantic and visual features of the current shot segment.

[0036] After generating music segments in all stages, adjacent music segments can be spliced ​​together using audio smoothing algorithms to eliminate abrupt transitions between segments, ultimately outputting a complete background music track with the same duration as the video to be processed.

[0037] In the video parsing-based music generation method provided in this disclosure, multi-layer parsing and structured feature construction provide clear control signals for the music generation model. Specifically, by extracting global semantic information, the overall tone of music generation is clarified, avoiding the problem of "choosing a style based solely on feeling" in related technologies. By forcibly dividing the music generation process into stages according to the video's shot segments, the semantic information of each shot segment provides precise constraints on the corresponding music segment, ensuring that the music style is highly consistent with the shot content, and that the generated music accurately follows the changes in video content in terms of emotion and style. By constraining the attention range of the music generation model, it is ensured that the music generation at each stage is only affected by the features of the current shot segment, and the music styles corresponding to different shot segments do not interfere with each other, improving the coherence and adaptability of the music.

[0038] Meanwhile, the music generation process is forcibly divided into stages according to the video's shot segments, breaking down long music generation into multiple short music generation segments. This reduces the complexity of music generation, increases the speed of music generation, and facilitates subsequent fine-tuning and optimization of music segments for individual shots.

[0039] In some possible implementations, the understanding of the video can be presented in a human-readable way by outputting global semantic information and the content semantic information of each shot segment. This allows users to intuitively understand the basis for music generation and intervene or adjust the description of specific segments when necessary, thereby achieving more refined and interpretable control over the generated music.

[0040] The music generation method based on video parsing provided in this disclosure does not limit the type or duration of the video. Whether it is a short video, an e-commerce advertisement, or a long video, it can generate suitable music through the multi-layer information obtained from the parsing. Compared with the music generation methods in related technologies that can only process short videos, it has a wider range of applications and fewer limitations.

[0041] The following is a detailed description of each step in the video parsing-based music generation method provided in this embodiment.

[0042] As mentioned above, in some possible implementations, a visual encoder and a text decoder can be used to generate global semantic information of the video to be processed.

[0043] Figure 2 A flowchart illustrating one method for obtaining global semantic information of the video to be processed is shown, such as... Figure 2 As shown, step S110 may include the following steps: S210. Extract multiple keyframes from the video to be processed.

[0044] Among them, keyframes refer to representative video frames extracted from the video to be processed that can represent the core content and scene features of the video. The information they contain is sufficient to reflect the main content, objects, scenes and emotional tendencies of the video within the corresponding time period. They are the core carriers for extracting local semantics and aggregating global semantics.

[0045] In some possible implementations, the video to be processed is first decoded and analyzed. Specifically, a basic keyframe set is obtained at a base sampling rate, such as 1 frame per second. Simultaneously, the content variation between consecutive frames is calculated in real time, such as calculating the average value of optical flow amplitude or the Bach distance of the color space histogram. When the variation exceeds a preset threshold, supplementary keyframes are obtained by sampling at a higher density, such as 5-10 frames per second, within that time period to ensure that the core dynamic scene is captured. For segments with smooth motion, the base sampling interval is maintained to avoid redundant frames.

[0046] The base keyframe and supplementary keyframe together constitute multiple keyframes for semantic analysis, ensuring that the keyframes can evenly cover the video timeline and capture semantically rich moments such as action changes and scene transitions.

[0047] S220. Perform visual semantic understanding on each keyframe and generate the corresponding local semantic description.

[0048] Local semantic description refers to the natural language text generated after understanding the content of a single keyframe, which can accurately describe the core content of the frame, such as "a cat playing on the grass" or "traffic on the streets of the city at night". It contains key information such as the subject, action, and scene within the frame.

[0049] In some possible implementations, each extracted keyframe is input into a visual semantic understanding model. This model can be a large-scale vision-language pre-trained model, such as CLIP (Contrastive Language-Image Pre-training) with a lightweight text decoder, or image description generation models such as BLIP (Bootstrapping Language-Image Pre-training) or GIT (Generative Image Transformer), which have been trained on massive amounts of image and text data and have accurate image semantic mapping capabilities.

[0050] The model encodes and understands each keyframe. First, it identifies objects, actions, scenes, and emotional atmosphere within the frame. Then, it calls the text generation interface to generate a natural language sentence that describes the core content of the frame, i.e., a local semantic description. For example, for a travel video, it may generate descriptions such as "plane taking off", "mountain and sea of ​​clouds", and "hotel lobby".

[0051] The generated local semantic descriptions are then validated for compliance (meaningless text is filtered out and semantic deviations are corrected) to ensure that the semantic descriptions of each keyframe accurately reflect the content within the frame.

[0052] S230. Based on the local semantic descriptions corresponding to all keyframes, generate global semantic information to characterize the overall semantic content of the video to be processed.

[0053] Global semantic information refers to the abstract information that represents the overall semantic content of the entire video to be processed, obtained by aggregating and summarizing the local semantic descriptions based on all keyframes, such as "outdoor pet daily life" and "urban sports scene". It is the core basis for the overall tone of music generation.

[0054] In some possible implementations, after obtaining the local semantic descriptions of all keyframes, these descriptive texts are treated as a "document" describing the video content. This document is then fed into a semantic aggregation model, which could be a text sequence model such as the Transformer encoder. This model uses a self-attention mechanism to analyze the relationships between all local descriptions, capturing recurring topics, objects, and actions. Finally, through a pooling layer or a dedicated summarization head, a fixed-dimensional semantic vector incorporating all local information is output, or a short summary text is generated directly.

[0055] For example, combining the above descriptions such as "airplane takeoff", "mountain clouds", and "hotel lobby", the aggregation model may output "a record about a mountain trip" as global semantic information.

[0056] Based on this, global semantic tags, such as "outdoor tourism," "healing," and "gentle," can be further refined to form the final global semantic information. At the same time, local semantic descriptions can be stored to support the subsequent construction of multimodal features.

[0057] By understanding discrete but representative keyframes one by one, the problem of information dilution or averaging in long videos is avoided. This allows for the capture of multiple important visual themes scattered across different time points, resulting in richer global semantic information that more closely reflects the actual video content. Accurate global semantic information can directly guide the music generation model in determining the overall style, avoiding the problem of the tone being disconnected from the video content in related technologies.

[0058] By employing a hierarchical logic of "local semantics - global induction," the ambiguity of directly generating global semantics is avoided. For example, generating only "outdoor scene" without specifying the subject and emotion may not be possible. The generated global semantic information includes both natural language summaries and structured labels, providing a clear basis for subsequent music style mapping.

[0059] Meanwhile, the extracted keyframes and local semantic descriptions can be directly reused in multimodal feature construction (see the detailed description of the multimodal feature construction steps), without having to repeatedly process the original video frames, thus improving computational efficiency.

[0060] In some possible implementations, a visual encoder can be used to encode video frames of a specific shot segment to generate content semantic tags for that shot segment.

[0061] Figure 3 The diagram illustrates a flowchart of one method for obtaining the semantic information of a shot segment, such as... Figure 3 As shown, step S110 may include the following steps: S310. For each shot segment, extract and aggregate features from the video content within that shot segment to obtain a unified feature representation that characterizes the visual semantics of that shot segment.

[0062] Among them, a shot segment refers to a continuous and semantically coherent video segment divided based on video shot boundary detection. It has a clear start time, end time and duration, and is the basic unit for music segmentation generation. The specific method of dividing the video to be processed into multiple shot segments is as described above, and will not be repeated here.

[0063] Feature extraction and aggregation of video content within a shot segment to obtain a unified feature representation characterizing the visual semantics of that shot segment can include: Visual semantic features are extracted from multiple video frames (which can be equally spaced samples or keyframes) within the shot segment. These visual semantic features are typically high-dimensional feature vectors extracted by deep convolutional neural networks. These features are then aggregated, such as by calculating the temporal average or maximum value, or by using a temporal model or attention mechanism to assign different weights to the features of different frames and then performing weighted fusion. This makes the aggregated features more focused on the salient or key content within the segment, resulting in a unified feature representation.

[0064] The extracted high-dimensional feature vectors can contain feature vectors corresponding to information such as the outline of objects within the frame, color distribution, and motion intensity. Therefore, the final unified feature representation can comprehensively characterize the visual semantics of the shot segment (including objects, actions, colors, emotions, etc.).

[0065] S320. Input the unified feature representation into the text generation model to obtain the semantic summary text of the paragraph content, which serves as the semantic information of the scene paragraph.

[0066] Among them, semantic summary text refers to natural language text that accurately describes the core content and semantic tendency of a single shot segment by mapping its unified feature representation through a text generation model. For example, "The character begins to run quickly, and the intensity of the action increases" is a direct constraint on the musical style and rhythm of the shot segment.

[0067] In some possible implementations, the unified feature representation obtained in the previous step is used as conditional input to a text generation model. This model is typically a pre-trained language model whose input layer or cross-attention layer is designed to accept visual feature vectors as additional conditional context. Based on this visual conditional vector, the model autoregressively generates a fluent and accurate sentence, i.e., a semantic summary text. Alternatively, structured prompts can be set for the model, and the unified feature representation along with the prompts can be input into the text generation model, with the model outputting a semantic summary text.

[0068] In some possible implementations, the generated semantic summary text is quality-checked: texts with ambiguous semantics or missing information are filtered out, and if they do not meet the requirements, they are re-entered into the model to ensure the accuracy and practicality of the semantic information. Finally, the semantic summary text that passes the check is combined with the extracted keyword tags to form the semantic information of the scene segment.

[0069] By compressing visual information into a unified feature representation and leveraging LLM (a text generation model pre-trained on large-scale text) to "interpret" and "represent" that feature, this essentially combines visual understanding tasks with powerful natural language generation capabilities. This produces more natural, diverse, and human-friendly descriptions than directly training a classifier from pixels to specific labels, significantly improving the quality of semantic information.

[0070] Meanwhile, the process generates semantic summaries in text form, which are naturally compatible with text-based conditional control methods commonly used in music generation (such as cue-based generation). This allows the semantic information parsed from the video to be more directly and effectively transformed into control signals to guide music generation; direct analysis of a shot segment with clear start and end boundaries also ensures that the generated semantic summary naturally corresponds to a narrative unit of the video, giving the final shot structure information a clear temporal and semantic structure, providing a precise "script" for subsequent segmented music generation.

[0071] In some possible implementations, the multi-layer parsing results also include visual rhythm point information; whereby visual rhythm point information is used to characterize the time sequence of key visual events in the video to be processed that are associated with the rhythm of the music.

[0072] Among them, visual rhythm point information refers to the time points (usually represented by a time stamp sequence) of a series of key visual events that are strongly correlated with the potential musical rhythm and are automatically detected and extracted from the video to be processed.

[0073] Key visual events refer to those events in a video that are discrete in time, visually prominent, and often perceptually synchronized with musical beats or rhythm changes. Typical examples include: the moment of a shot change, the start or end point of a subject's large-scale movement in the frame (such as jumping and landing, or hitting), the key flashing white point in a product display, the climax of a transition effect (such as a flash or wipe), and the instant that an important subtitle or graphic pops up.

[0074] Key visual events are detected using a video content change detection algorithm. During the parsing of the video to be processed, the video is analyzed frame by frame to calculate signals reflecting dynamic changes in the scene, such as: Inter-frame difference intensity: The degree of drastic pixel changes between consecutive frames.

[0075] Optical flow energy: the speed and scale of motion of objects in an image.

[0076] Lens switching detection confidence: Lens boundary probability output by a dedicated algorithm.

[0077] These signals are smoothed and peaked. The timestamps corresponding to peaks exceeding a threshold are extracted to form an ordered sequence of time points, i.e., visual rhythm point information. For example, a video containing a jumping action may detect three time points [1.23s, 1.87s, 2.45s], corresponding to the take-off, the highest point in the air, and the landing, respectively.

[0078] In some specific implementations, camera transitions are typically identified by sudden changes in the image's color histogram or optical flow; large-scale jumps / landings are detected by optical flow peaks or key point speed changes; flashing white points are identified by sudden increases in brightness values; transition lighting effects are detected by edge changes and brightness patterns; and pop-up subtitles are identified by a combination of text region detection and time changes. All these algorithms are based on the premise that changes in inter-frame information exceeding a certain threshold determine that an event has occurred.

[0079] An ordered set, such as [t1, t2, ..., tn], is formed by arranging all key visual events in the video according to their chronological order of occurrence. This set represents the visual rhythm points and is used to clarify the specific time position for beat alignment. In some possible implementations, a type label can be added to each event, such as "camera transition" or "action landing," to determine the priority of subsequent beat adjustments (e.g., camera transition points have higher priority than ordinary action points).

[0080] In some possible implementations, after obtaining the visual rhythm point information, the rhythm of the generated background music can be adjusted according to the visual rhythm point information so that the beat points in the music are aligned with the time points of key visual events.

[0081] Beat adjustment refers to the process of processing generated or being generated music waveforms or symbol sequences to change the absolute time position of their internal beat points (such as drum beats and strong beats) and precisely match them with the time points of key visual events. Its goal is to achieve a "visual event - music beat" timing effect.

[0082] In some possible implementations, a sequence of visual rhythm points can be used as an additional conditional input during the music generation model's generation process. During training, the model learns the association between "generating beats (such as drum sounds) at specified time points." During inference, the music generation model tends to generate significant rhythmic elements near the time corresponding to the visual rhythm points, thus achieving soft alignment in the music generation process and aligning the beat points in the music with the time points of key visual events.

[0083] In some possible implementations, a pre-built music generation model can be driven to generate initial background music (without adjusted beats), record the temporal sequence of the beat points of the initial music, where the beat points include strong beats and weak beats, with a focus on the strong beats and drum beats, and adjust the beat points of the initial music according to the temporal sequence corresponding to key visual events.

[0084] Specifically, the time sequence corresponding to the key visual event is compared with the time position sequence of the beat points in the background music. Based on the comparison results, the time position of the music beat is adjusted by time-domain audio processing, or the music beat points near the time point corresponding to the key visual event are replaced with rests, so that the adjusted music beat points fall into the preset neighborhood of the corresponding key visual event time point on the time axis.

[0085] Among them, temporal audio processing refers to the technical means of adjusting the musical beat in the time dimension of audio, including beat stretching (extending the beat duration) and beat compression (shortening the beat duration), without changing the frequency characteristics of the audio (such as pitch and timbre), only adjusting the time position.

[0086] Rest replacement refers to replacing the original beat point with a short rest when the deviation between the beat point and the key visual event time point is small, and then adding a beat point at the key visual event time point to achieve alignment without disrupting the musical rhythm.

[0087] The preset neighborhood refers to an acceptable range of time error (e.g., ±50 milliseconds) defined around a key visual event time point. Alignment is considered successful as long as the music beat falls within this range. This threshold can be adjusted according to the accuracy requirements of the application scenario.

[0088] In some possible implementations, the generated background music can be analyzed using a beat tracking algorithm to obtain the temporal sequence of its beat points. This sequence is then compared with the temporal sequences corresponding to key visual events.

[0089] The optimal correspondence between two sequences is found using sequence alignment algorithms such as dynamic time warping. Misaligned beat points are identified as "target positions where the musical beats need to be adjusted."

[0090] For beats that require fine-tuning of their timing, a time-domain audio processing algorithm (such as a speed-changing algorithm based on a phase vocoder) is used to locally stretch or compress the music waveform at the millisecond level, so that the beat point "falls into the preset neighborhood of the corresponding key visual event time point".

[0091] For certain visual events (such as a rapid flash of light), if the nearby musical beats cannot be satisfactorily aligned through fine-tuning, the audio of that beat can be selectively replaced with a rest or a very short decaying note to create an auditory "pause" or "emphasis," thereby achieving rhythmic emphasis and alignment.

[0092] In some possible implementations, the alignment of high-priority events (camera switching points, flashing white points) is adjusted first, and then ordinary motion points are processed to ensure that the core visual events are accurately matched with the strong beats / drum beats of the music.

[0093] Related technologies only pursue statistical matching of the overall rhythm of the video and the overall rhythm of the music, and cannot guarantee the synchronization of specific event points. By detecting discrete key visual events and actively adjusting the music beat to align with them, professional-grade and high-impact synchronization effects can be achieved, such as "drum beats falling precisely at the moment of camera transition" and "musical accents accompanying the character's punching action", which greatly enhances the audiovisual performance and professionalism.

[0094] Music is no longer a static background element, but a dynamic element that responds to the video's changes. When the video action is intense and events are frequent, the music rhythm automatically becomes denser and more precisely aligned; when the video is calm, the music rhythm also becomes more relaxed. This dynamic correlation makes the generated music more intelligent and customized, as if it were "tailor-made" for the video. In the past, achieving high-quality "beat-matching" effects required composers or editors to spend a lot of time manually aligning the music. However, this disclosed embodiment fully automates this highly experience- and time-dependent process. Users only need to provide the video, and the system can automatically analyze and generate synchronized music, greatly reducing the threshold and time cost of professional video production, allowing creators to focus more on the content itself.

[0095] This disclosure provides two optional implementation paths: post-processing alignment and process control alignment, adapting to different system architectures and accuracy requirements. The method combining time-domain audio processing with rest replacement ensures alignment accuracy while also maintaining the naturalness of the processed music's sound, avoiding sound distortion or a mechanical rhythm that might result from simple speed changes.

[0096] In some possible implementations, constructing multimodal control features based on multi-level analysis results is essentially a unified conditional signal used to control the music generation model based on multi-level analysis results.

[0097] Figure 4 The diagram illustrates a flowchart of an implementation method for constructing multimodal control features based on multi-level analysis results, as shown below. Figure 4 As shown, step S120 may include the following steps: S410. Based on the multi-layer parsing results, construct text features, visual features, and temporal structure features; Among them, text features are generated based on global semantic information and the content semantic information of each shot segment; visual features are extracted based on the video content of each shot segment; and temporal structure features are generated based on shot structure information, which are used to indicate the duration of each shot segment and its structural role in music generation.

[0098] Text features are high-dimensional vector features obtained by converting the global semantic information of a video and the semantic information of each shot segment through text encoding. They can quantitatively represent the semantic attributes and emotional tendencies of a video and are the core semantic constraints that guide music style.

[0099] Visual features are a set of features extracted from the video content of each shot segment that can characterize the attributes of the image, including information such as the outline of objects in the image, color distribution, intensity of motion, and emotional atmosphere. They serve as a bridge connecting the visual presentation of the video with the timbre / rhythm of the music.

[0100] Temporal structure features are feature vectors generated based on shot structure information, representing the temporal attributes and structural effects of shot segments. They include parameters such as segment length, time percentage, and structural weight (e.g., opening segment, climax segment), which are used to constrain the segment layout and intensity changes of music.

[0101] In some possible implementations, multimodal control features are constructed by generating three types of features according to the three dimensions of "semantic-visual-temporal" and clarifying their relationships. This ensures that the multimodal control features are logically clear and interpretable, and avoids feature confusion and redundancy.

[0102] In some specific implementations, global semantic information and the semantic information of each shot segment are processed separately by a pre-trained text encoder to obtain corresponding semantic feature vectors. For example, semantic text is converted into word embedding vectors, and then a multi-head self-attention mechanism is used to capture semantic relationships, outputting global semantic feature vectors and semantic feature vectors of each shot segment. The global semantic features and the semantic features of each shot segment are then concatenated and fused to obtain the final text feature vector, which preserves the overall tone while highlighting the semantic differences between shots.

[0103] For each shot segment, its representative frames (such as intermediate frames or keyframes) are extracted, and its high-dimensional image features are extracted using a deep convolutional neural network. The image features of all single frames within the shot segment are then averaged and pooled to obtain a unified image feature vector for that shot segment. This vector serves as the basic component of the visual features, ensuring that the features can represent the overall content of the segment.

[0104] In some possible implementations, visual features include at least one of image features, motion energy features, color style features, and emotional trend features; Among them, motion energy features are calculated based on the optical flow information between video frames of each shot segment; color style features are characterized by extracting the dominant hue and its distribution ratio of the video content of each shot segment in a specific color space; and emotion trend features are characterized by performing emotion calculations on the video frame sequence within the shot segment to obtain an emotion dimension time sequence based on the valence-arousal model.

[0105] In other words, visual features, in addition to the unified image feature vector extracted above, can also include motion energy features, color style features, and emotional trend features.

[0106] Among them, motion energy features refer to the features that characterize the dynamic intensity of a shot segment, calculated based on the motion changes between video frames. They are used to quantify the intensity of the motion of objects in the picture. For example, "walking slowly" has low motion energy, while "running fast" has high motion energy.

[0107] Color style features refer to the features obtained by analyzing the color distribution of video frames, including the dominant color tone, such as "warm yellow" and "cool blue", brightness distribution, color saturation, etc., which are used to characterize the visual atmosphere of the picture (such as warm or cool).

[0108] Emotional trend features refer to the temporal features obtained by performing emotion calculations on video frame sequences within a shot segment based on the valence-arousal model. These features are used to quantify the dynamic changes in the emotions of the scene (such as "from calm to excitement").

[0109] In some specific implementations, a sparse optical flow algorithm can be used to calculate the motion vectors of adjacent frames within a shot segment: extract corner features for each frame, track the displacement changes of corner points between adjacent frames to obtain the motion vector of each corner point; calculate the average amplitude of all corner point motion vectors as the motion energy of a single frame; perform temporal smoothing on the motion energy of all single frames within the segment to obtain a motion energy temporal sequence; map the temporal sequence to a motion energy feature vector through a fully connected layer as the motion energy feature, where a higher vector value indicates a higher dynamic intensity of the shot segment.

[0110] Representative frames within a shot segment are converted to the HSV color space, and the hue (H), saturation (S), and value (V) channels are separated. Histogram statistics are performed on the H channel, and at least one hue with the highest proportion is selected as the dominant color tone. Histogram statistics are performed on the S and V channels respectively to obtain the saturation and value distributions. The dominant color tone encoding, saturation distribution, and value distribution are concatenated and dimension-reduced to obtain the color style feature vector as the color style feature.

[0111] A pre-trained visual sentiment analysis model is used to perform sentiment calculation on sampled frames within a segment, outputting the valence (V) and arousal (A) coordinates for each frame. Linear interpolation is performed on the VA coordinate sequence to fill in the sentiment values ​​of missing frames, resulting in a continuous temporal sequence of emotions. Features of the temporal sequence are extracted: mean (overall emotional level), variance (degree of emotional fluctuation), and trend slope (direction of emotional change), which are then concatenated into an emotional trend feature vector as the emotional trend feature.

[0112] In some specific implementations, image features, motion energy features, color style features, and emotional trend features can be input into the cross-attention fusion module to capture the correlation between features of each dimension; the fused features are mapped to preset dimensions through linear transformation, and after optimization by activation function, the final visual feature vector of the shot segment is obtained as the visual feature of the shot segment.

[0113] Compared to related technologies that extract only single visual features, this disclosure extracts features from four dimensions: content, dynamics, color, and emotion. This allows for a complete characterization of the visual attributes of a shot, enabling the music generation model to more accurately understand visual information. Multi-dimensional features support more refined style mapping (e.g., "high motion energy + warm color tone" corresponds to "high rhythm density + strings," and "low motion energy + cool color tone" corresponds to "low rhythm density + piano"), improving the matching degree between music and visual atmosphere, dynamic intensity, and emotional changes. Among these, the emotion trend feature quantifies the emotional changes in a shot, guiding the music generation model to dynamically adjust emotional expression, making the music more narrative and avoiding the problem of "flat emotional delivery." The fusion of multi-dimensional features effectively distinguishes shot segments in similar scenes, preventing the music generation model from confusing styles and improving the accuracy of generation.

[0114] In some possible implementations, visual feature vectors are arranged in the order of shot segments to form a visual feature sequence, which is consistent with the time dimension of text features.

[0115] Based on shot structure information, a structured label is generated for each shot segment. This can be a multi-dimensional vector containing: a) Absolute duration (number of seconds the paragraph lasts); b) Relative position (the proportion of the segment's start time to the total video length); c) Paragraph type labels (such as “introduction paragraph”, “development paragraph”, “climax paragraph”, “transition paragraph”, etc., obtained through rules or learning).

[0116] In some possible implementations, weighting coefficients can be assigned to different structure types to indicate the intensity changes in music generation.

[0117] The above information is mapped into a time structure feature vector, with each shot segment corresponding to a time structure feature vector, forming a time structure feature sequence.

[0118] Following the order of "text features - visual features - temporal structure features", the three types of feature sequences are aligned in the time dimension, and the final output is a structured multimodal control feature set, which includes text feature sequences, visual feature sequences, and temporal structure feature sequences, providing constraints for subsequent music generation.

[0119] In some possible implementations, the feature similarity between text features and visual features within the same shot segment can be calculated. If it is lower than a threshold, the semantic label or visual feature extraction results can be corrected (such as regenerating the shot semantic description) to avoid feature conflicts.

[0120] Compared to related technologies that rely on only a single feature, the three types of features in this disclosure constrain music generation from the perspectives of "style tone, visual presentation, and structural layout," making the generation process more targeted and improving the compatibility between the generated music and the video.

[0121] Structured features can also provide clear control signals for music generation models (such as text features controlling style and time structure features controlling paragraph intensity), avoiding randomness in generation, making music structure switching more precise, and solving the pain point of "style abrupt change" in related technologies.

[0122] In some possible implementations, it is necessary to convert text features, visual features, and temporal structure features into feature signals that the music generation model can understand. Therefore, step S120 also includes: S420. Text features, visual features, and temporal structure features are encoded and fused to generate a unified conditional signal for controlling the music generation model.

[0123] By employing cross-modal fusion technology, textual features, visual features, and temporal structure features are mapped to the same feature space, eliminating modal differences and feature conflicts, and forming a unified feature that is logically consistent and informationally complete, namely a unified conditional signal.

[0124] In some specific implementations, the textual features, visual features, and temporal structure features of the current shot segment are input into a feature fusion encoder.

[0125] This encoder typically consists of fully connected layers and an attention mechanism. It first maps each feature to the same dimension using independent linear projection layers. Then, a multi-head cross-attention module is used, for example, with temporal structure features as the query and textual and visual features as the key and value, allowing the temporal structure information to actively query and aggregate relevant semantic and visual content. Finally, the processed features from each modality are summed or concatenated.

[0126] The output is a unified conditional signal, a vector that encapsulates all the information about "what the current shot is about (semantics)," "how it looks (visuals)," and "how long it lasts and where it is located (temporal structure)." This conditional signal serves as the final conditional input, directly fed into the music generation model to control the generation of that piece of music. Specifically, each shot segment corresponds to a unified conditional signal, arranged chronologically to form a conditional signal sequence, which is then directly input into the music generation model.

[0127] By using encoding fusion, heterogeneous multi-source information is integrated into a unified, dense conditional signal. This solves the problem of information redundancy or conflict between different features, and generates a synergistic effect where 1+1>2, producing a more powerful conditional representation that no single feature possesses. This ensures that the conditional signal input to the music generation model is highly integrated and information-rich, thereby directly driving the generation of higher-quality music segments that better meet all constraints.

[0128] Meanwhile, the unified condition signal is a standardized high-dimensional vector that can be adapted to various mainstream music generation models without requiring modifications to the model architecture for different modal features, thus improving the compatibility and scalability of the solution.

[0129] In some possible implementations, textual and visual features can be mapped to music style control signals, and these music style control signals, along with temporal structure features, can be used as unified conditional signals to control the music generation model.

[0130] This involves mapping textual and visual features into music style control signals. Among them, the music style control signal is used to control at least the rhythm density and timbre tendency of the background music generated by the music generation model; the mapping is achieved through a pre-trained style mapping model, which is trained based on paired samples of video clips and corresponding background music.

[0131] Among them, music style control signals refer to a series of quantized parameters or conditional vectors that directly and specifically correspond to a certain aspect of the style attributes of a musical work, including at least rhythm density and timbre tendency. They are used to precisely control the auditory characteristics of the generated music and are the direct control input of the music generation model.

[0132] Style mapping models are pre-trained deep learning models specifically designed to map multimodal features of videos to music style control signals. They are trained based on paired samples of video clips and corresponding background music and have accurate cross-modal mapping capabilities.

[0133] The paired samples are training data consisting of “video clips - background music - style tags”. The video clips contain semantic and visual information, the background music is labeled with style parameters such as rhythm density and timbre type, and the style tags are manually annotated natural language descriptions.

[0134] In some specific implementations, a large number of video clips covering different scenarios (outdoor, indoor, e-commerce, sports) and emotions (healing, excitement, sadness, lighthearted) are collected. At least one suitable background music is matched to each video clip, and the rhythm density, timbre tendency, and style tag of the background music are manually labeled. The style mapping model adopts an "encoder-decoder" architecture, which concatenates the video text features and visual features into a fusion vector. The encoder is input to extract cross-modal features, and the decoder predicts the rhythm density and timbre tendency based on these features. The style mapping model is trained by jointly training the rhythm density prediction error and the timbre tendency classification error using a loss function.

[0135] The text feature vector and visual feature vector constructed in step S410 are combined through the feature concatenation module to obtain a cross-modal fusion vector. The fusion vector is then input into the trained style mapping model, where the encoder extracts feature association information and the decoder outputs a quantized music style control signal.

[0136] In some possible implementations, after generating corresponding style control signals for the characteristics of each shot segment, fine-tuning is performed by combining the weight coefficients of the temporal structure features: for example, the rhythm density of the climax shot is increased by 10% based on the model output, and decreased by 10% for the fallback shot; the style control signals of adjacent shot segments are smoothed to avoid abrupt changes in musical style. The final style control signals are then used as conditions input into the music generation model to guide the music generation process.

[0137] In some possible implementations, the textual features, visual features, temporal structure features, and music style control signals of the current shot segment can be input into a feature fusion encoder. Each feature is mapped to the same dimension using independent linear projection layers. A multi-head cross-attention module is then used to sum or concatenate the features processed by each modality, outputting a unified conditional signal. This signal encapsulates all the information about "what the current shot is about (semantics)," "how it looks (visuals)," "how long it lasts and where it is located (temporal structure)," and "what style of music should be used (style control)." This unified conditional signal serves as the final conditional input, directly fed into the music generation model to control the generation of the music segment.

[0138] By introducing a style mapping model, the complex relationship between "what kind of scene corresponds to what kind of music style" is automatically learned from massive film and television music data. Compared with relying on manual rules, this greatly improves the accuracy and diversity of style control, making the generated music style more professional and more in line with industry practices, and realizing the leap from "understanding the content" to "understanding the music".

[0139] Compared to related technologies that rely on "fuzzy matching of styles based on semantic tags," transforming style into quantifiable control signals (rhythm density, timbre tendency) can improve the controllability of music style. The style mapping model completes the transformation from "video features to style parameters" in advance, and the music generation model only needs to generate music based on the parameters, without directly processing high-dimensional video features, which greatly improves the inference speed and enhances generation stability.

[0140] As mentioned above, in some possible implementations, after obtaining the unified condition signal, it is input into the music generation model to guide the music generation model to generate background music.

[0141] Music generation models can generate background music using the following paradigms: Paradigm 1: For each shot segment, background music is generated independently based on its corresponding multimodal control features.

[0142] Independent generation means that the music generation model processes each video clip in a state-isolated manner. The generation of music for each clip does not depend on any music content generated in the previous clip; it uses only the multimodal control features of the current clip as the sole starting point and basis for creation.

[0143] In this paradigm, the music generation model initializes its generation state independently for each shot segment. For example, for a Transformer-based decoder, its initial context buffer or memory state is reset to empty at the beginning of each segment. The music generation model generates all note sequences or audio tags belonging to the i-th shot segment autoregressively, starting from the start marker of the music sequence, based solely on the multimodal control features or unified conditional signals corresponding to the i-th shot segment of the input.

[0144] The generated duration is exactly the same as the duration of the shot segment (accurate to milliseconds); the music style, rhythm, and timbre are determined separately by the multimodal control features of the current segment; the audio quality assessment model ensures no distortion and no noise, and the segment integrity meets the requirements (no abrupt beginnings or incomplete endings).

[0145] Once the generation process is complete, the state of the segment is discarded or archived, and the process for the (i+1)th shot segment begins in the same manner.

[0146] The core advantage of this paradigm lies in its extremely high semantic fidelity and stylistic consistency. Since the generation of each piece of music is entirely determined by the characteristics of its corresponding video segment, and is unaffected by other segments, it can most precisely achieve the effect of "music changes with every shot." It is particularly suitable for scenes with significant differences in content and emotion between video segments (such as a sudden switch from a tranquil landscape to intense battle), ensuring that the abrupt change in music style is clear, accurate, and perfectly matches the scene transition.

[0147] In some possible implementations, under this paradigm, after generating the corresponding music segments sequentially according to the time order of the shot segments, the rhythmic similarity and timbre similarity of adjacent segments can be calculated. If the similarity does not meet the standard, a second fine-tuning is performed at the connection point (such as adjusting the starting rhythm and timbre of the next segment to match the previous segment) to ensure the continuity of the complete background music.

[0148] Paradigm 2: For each shot segment, based on the feature sequence of the background music generated in the previous shot segment and the multimodal control features corresponding to the shot segment, the background music for that shot segment is generated again.

[0149] The feature sequence refers to the internal representation of the generated musical content. It can be the hidden state sequence of the music generation model when generating the preceding musical passage (such as the intermediate layer output of the Transformer decoder), the symbolic representation of the preceding musical passage (such as the pitch, duration, and velocity vectors of a note sequence), or the acoustic feature sequence extracted from the generated audio clip. This sequence encodes the structure, melody, harmony, and rhythmic information of the generated music.

[0150] Continuous generation refers to the fact that when the music generation model processes subsequent shot segments, its generation process is continuous in state. The music generation model uses the feature sequences generated in the previous segment as context, and on the basis of this context, combined with the multimodal control features of the current segment, it naturally evolves and develops the music of the current segment, thereby achieving a smooth connection and style transition of the music.

[0151] In this paradigm, music generation is viewed as a continuous creative process spanning the entire video timeline. When generating music for the i-th shot segment, the model uses the feature sequence generated in the previous segment (i-1-th segment) as additional conditional input. Specifically, the final hidden state vector of the previous segment can be used as the initial state of the current generation process; or, in an architecture similar to a music autoregressive Transformer, the complete note sequence (or its feature representation) generated in the previous segment can be directly appended to the input sequence of the current segment as "historical context." Then, the model continues to generate music for the current segment while simultaneously "remembering" the previous music and "seeing" the multimodal control features of the current segment, allowing the newly generated melody, harmony, and rhythm to naturally develop from the previous music.

[0152] The core advantage of this paradigm lies in the overall coherence and fluidity of the generated music. It mimics the approach of a composer scoring a complete film, where the music changes with the plot but maintains a consistent theme, motive, harmonic progression, or rhythmic pattern. Through "continuous generation," the system can create complete musical movements with strong internal logic and seamless sonic flow, avoiding the loose structure or abrupt transitions that can result from independently generated segments. It is particularly suitable for videos with coherent narratives and natural emotional transitions (such as a complete short story or documentary), generating more artistic and professional scores.

[0153] These two paradigms can be implemented independently or as configurable options during system design. Within a complete system, both paradigms can be flexibly applied. For example, the system can default to a continuous generation mode to ensure overall smoothness; however, when a drastic change in the semantic features (such as emotion tags) of adjacent shot segments is detected, it can automatically switch to an independent generation mode to handle the boundary and ensure the accuracy of style changes. This dynamic strategy combines the advantages of both paradigms.

[0154] Regardless of the paradigm, when a music generation model generates background music for a certain shot segment, the multimodal control features it references are only the multimodal control features of that shot segment.

[0155] In some possible implementations, constraining the music generation model to generate background music only based on the multimodal control features of the shot segment corresponding to the current generation stage can be achieved through attention masks.

[0156] Figure 5 The diagram illustrates a flowchart of one approach to constraining a music generation model using attention masks, as shown below. Figure 5 As shown, the following steps may be included: S510: Construct a temporal attention mask for each shot segment; Temporal attention masks are used to prevent the music generation model from focusing on multimodal control features of other shot segments outside of that specific shot segment when generating background music belonging to that particular shot segment during the current generation phase.

[0157] Temporal attention masks are Boolean matrices or weight matrices with specific structures used in neural network attention mechanisms. Their core function is to impose hard restrictions on information access based on the time dimension.

[0158] In this embodiment of the disclosure, the time attention mask is designed such that when generating music belonging to a specific time period (segment), the music generation model is only allowed to "pay attention" (i.e. calculate the relevance and extract information) to the multimodal control features corresponding to that time period, while forcibly "ignoring" the features of other time periods.

[0159] Based on the shot structure information extracted in step S110, the start and end times [start_k, end_k] of N shot segments are defined, k = 1, 2, ..., N.

[0160] For the k-th shot segment, construct a corresponding temporal attention mask M_k. M_k is a two-dimensional matrix (or an additive mask vector applied to the attention score) whose dimension matches the dimension of the attention score matrix between the query sequence and the key sequence when the music generation model generates the k-th music segment.

[0161] Each element M_k(i, j) in the M_k matrix defines whether it is allowed to “notice” the j-th conditional feature (a multimodal control feature sequence spliced ​​from all shot segments) at the i-th time step (or i-th music mark) when generating the k-th music segment.

[0162] The assignment rule is as follows: if the original time point corresponding to the j-th conditional feature falls within the interval [start_k, end_k] (i.e., belongs to the k-th paragraph), then M_k(i, j) is set to a value that allows passage (such as 0 or 1); otherwise, it is set to a value that indicates "access is prohibited" (usually a very large negative number in additive attention, such as -10^9, or 0 in multiplicative attention). In this way, M_k "draws a boundary" for the k-th paragraph in the time dimension.

[0163] Music generation models are typically autoregressive models based on Transformer decoders, or conditional generation models that include cross-attention layers (such as the conditional injection layer in a diffusion model). At each time step when the music generation model generates the music for the k-th shot segment, the temporal attention mask M_k corresponding to that segment is applied to the model's cross-attention calculation.

[0164] For the autoregressive Transformer decoder: Each layer's cross-attention module uses the currently generated music tag sequence as the Query and the sequence concatenated from the multimodal control features of all shot segments as the Key and Value. After calculating the dot product score of the Query and Key, M_k is added to the score matrix. Since M_k has extremely negative values ​​at non-corresponding segment positions, after the Softmax function, the attention weights at these positions are almost zero. Therefore, when the model generates the music for that segment, the weighted sum of its Values ​​almost entirely comes from the features of the k-th segment itself, achieving hard feature isolation.

[0165] For diffusion models, when injecting conditional features into the intermediate layers of networks such as U-Net through cross-attention, the M_k mask is also applied to ensure that the denoising process is only guided by the corresponding paragraph conditions at each time step.

[0166] As the generation process progresses, the system dynamically switches and applies the corresponding mask M_k based on the shot segment to which the current generation time point belongs.

[0167] The temporal attention masking mechanism is the core algorithmic guarantee for realizing the "segmented forced control" concept in the embodiments of this disclosure, and it solves a problem that related technologies cannot overcome: Unlike related technologies that can only perform overall or soft conditional control, this mechanism physically severs the flow path between feature information of different segments within the neural network through hard, algorithmic masking. This ensures that the style, mood, and rhythm of the music generated for shot A are 100% determined by the content of shot A, and are completely unaffected by any "contamination" or "dilution" from the content of shot B or shot C. This is the fundamental prerequisite for achieving high-fidelity, highly consistent soundtracks.

[0168] Because the generation process is strictly constrained, the entire process becomes highly deterministic and predictable. For the same video input and model parameters, the output is stable. More importantly, when a user modifies the video content or semantic tags of a particular shot segment, it can be precisely anticipated and implemented that only the music corresponding to that segment will change, while the music in other parts will remain unchanged. This modular, locally controllable characteristic is of immense value for professional video editing and music composition workflows.

[0169] The following is a detailed description of the music generation method based on video parsing provided in this disclosure, using a specific embodiment.

[0170] This specific embodiment will use a 12-second short video of a "pet cat playing" as an example to fully demonstrate how the music generation method based on video parsing provided in this embodiment works. The video content is as follows: the cat first walks slowly (0-3 seconds), then suddenly jumps up to catch a toy (3-8 seconds), and finally is picked up and petted by its owner (8-12 seconds).

[0171] Step 1: Multi-layer video analysis Global semantic information parsing: The system samples at a rate of 2 frames per second and, combined with image change detection, extracts approximately 15 keyframes from the video.

[0172] Each keyframe is analyzed by a visual semantic model to generate local descriptions, such as: "The cat is lying on the carpet", "The toy feathers are swaying", "The cat is leaping into the air", "The paws are touching the toy", and "The owner's hands are in the picture".

[0173] These descriptions are fed into a semantic aggregation model (such as an encoder), which analyzes them and outputs global semantic information: "A heartwarming everyday scene showing a domestic cat going from relaxation to play, and finally interacting with people." This sets the overall tone for the music as "warm, lively, and healing."

[0174] Lens structure information analysis: The system automatically divides the video into three shot segments using shot boundary detection: Paragraph 1 (0-3 seconds): The meaning of the content is "the cat walks slowly and observes its environment".

[0175] Paragraph 2 (3-8 seconds): The content means "The cat jumps and catches the toy, and the action is intense".

[0176] Paragraph 3 (8-12 seconds): The semantic meaning is "The cat was picked up and its movements returned to calm".

[0177] For each paragraph, the system extracts and aggregates the visual features of its keyframes, then inputs them into a text generation model to generate the human-readable semantic summary text described above.

[0178] Visual rhythm point information analysis: The system analyzes motion energy and scene changes frame by frame to detect key visual events and their timestamps: 3.15s: Shot transition point (from paragraph 1 to paragraph 2).

[0179] 4.02s: The cat pushes off the ground with its hind legs and jumps.

[0180] 4.75s: The cat reaches its highest point in the air.

[0181] 5.20s: The cat successfully grabbed the toy with its front paws.

[0182] 8.00s: Shot change point (from paragraph 2 to paragraph 3).

[0183] These time points constitute a visual rhythmic sequence, serving as anchor points for subsequent musical "timing".

[0184] Step 2: Constructing multimodal control features The system constructs a set of fine-grained control signals for each shot segment: Text features: Global semantics and the semantics of each paragraph are encoded into vectors respectively. For example, the text features of paragraph 2 will include information such as "catching toys" and "intense action".

[0185] Visual features: Image features: Describe the objects and scenes in the image.

[0186] Motion energy characteristics: Calculate the average optical flow amplitude within the paragraph. Paragraph 1 has a low energy value, paragraph 2 has an extremely high energy value, and paragraph 3 has a low energy value.

[0187] Sentiment trend characteristics: based on valence-arousal model analysis. Paragraph 1 is (high valence, low arousal), paragraph 2 is (high valence, high arousal), and paragraph 3 is (high valence, low arousal).

[0188] Time structure characteristics: Includes the absolute duration of each paragraph (3 seconds, 5 seconds, 4 seconds) and its structural function ("introduction", "climax", "conclusion").

[0189] Generate music style control signals (core mapping): The textual features of paragraph 2 (“intense play”) and the extremely high motion energy features are input into the style mapping model.

[0190] Based on patterns learned from massive amounts of data, the model outputs a set of music style control signals: {rhythm density: high, drum intensity: strong, timbre tendency: bright and impactful}.

[0191] Similarly, paragraphs 1 and 3 will map signals such as "rhythmic density: low, timbre tendency: soft".

[0192] Coding Fusion: All features of each segment (text, visual, temporal structure, style control signals) are processed through a feature fusion encoder to generate a final, unified conditional signal. This signal encapsulates all the information on "what to shoot" and "how to compose the music".

[0193] Step 3: Segmented Constraint Music Generation This is the most innovative aspect of this embodiment. The music generation model begins to work, but its process is strictly constrained.

[0194] Generation phase division: The total generation time of the model (12 seconds) is strictly divided into three phases, which correspond one-to-one with the three shot segments.

[0195] Applying a temporal attention mask (core control): When generating the music for paragraph 2 (3-8 seconds), the system uses a time attention mask specifically designed for paragraph 2.

[0196] This mask ensures that the model's attention mechanism can only access the conditional signals from paragraph 2 itself, while completely "not seeing" the conditional signals from paragraphs 1 and 3. This is like providing the composer with only the script and atmosphere requirements for paragraph 2 when composing the music for paragraph 2, while hiding the content of the other paragraphs.

[0197] The generation process of paragraphs 1 and 3 is similar, each confined within its own "information boundary".

[0198] Result generated: Paragraph 1 (0-3 seconds): Under the constraint of "slow walking", a soft, sparse piano monotone melody is generated with a soothing rhythm.

[0199] Section 2 (3-8 seconds): Under the constraints of "intense play" and "high energy," a brisk, rhythmic passage with clear drumbeats is generated. The musical mood shifts to excitement.

[0200] Section 3 (8-12 seconds): Under the constraint of the characteristic of "restoring calm", the music naturally comes to an end, the drumbeats disappear, and it returns to the warm and lingering string pad tone.

[0201] Step 4: Visual-Audio Rhythm Alignment (Automatic Timing) After generating the initial music, the system performs final refinement: Comparison: The system analyzes the generated music beat sequence and compares it with the previously detected visual rhythm sequence.

[0202] Fine-tune alignment: A strong beat in the music was found to be at 4.90s, but it was slightly off from the moment the cat "grabbed the toy" (5.20s).

[0203] The system initiates time-domain audio processing, performing extremely subtle time stretching on the waveform within the 4.8s to 5.3s interval, precisely shifting the strong beat point to 5.20s.

[0204] Similarly, align the camera transition points at 3.15s and 8.00s with the rhythmic start points or chord change points of the music.

[0205] Final output: A 12-second, fully customized background music track. Its emotional structure (soothing-intense-soothing) perfectly matches the video narrative, and key action moments are emphasized with precise musical accents.

[0206] Through this simple embodiment, it can be seen that the music generation based on video parsing provided in this disclosure deconstructs the video into semantic "segments" rather than a bunch of frames; it maps visual semantics, dynamics, and emotional data into specific music style parameters; it ensures that the music is generated strictly according to segments through time attention masks, eliminating style contamination; and it achieves automated and high-precision alignment between visual events and music beats.

[0207] Based on and Figure 1 The method shown follows the same principle. Figure 6 A schematic diagram of a video-based music generation device provided in an embodiment of this disclosure is shown, such as... Figure 6 As shown, the video-based music generation device 60 may include: The video parsing module 610 is used to parse the video to be processed and obtain multi-layer parsing results. The multi-layer parsing results include at least global semantic information and shot structure information. Among them, the global semantic information is used to represent the overall content semantic information of the video to be processed; the shot structure information is used to represent multiple shot segments in the video to be processed in chronological order and the content semantic information of each shot segment. The feature generation module 620 is used to construct multimodal control features based on multi-layer parsing results; The music generation module 630 is used to drive a pre-built music generation model to generate background music that matches the duration of the video to be processed, based on multimodal control features. The music generation model's generation process is divided into multiple generation stages corresponding to each shot segment based on the shot structure information. In each generation stage, the music generation model is constrained to generate background music only based on the multimodal control features of the shot segment corresponding to the current generation stage.

[0208] In the video-based music generation apparatus provided in this embodiment, multi-layered analysis and structured feature construction provide clear control signals for the music generation model. Specifically, by extracting global semantic information, the overall tone of music generation is clarified, avoiding the problem of "choosing a style based solely on feeling" in related technologies. By forcibly dividing the music generation process into stages according to the video's shot segments, the semantic information of each shot segment provides precise constraints on the corresponding music segment, ensuring that the music style is highly consistent with the shot content, and that the generated music accurately follows the changes in video content in terms of emotion and style. By constraining the attention range of the music generation model, it is ensured that the music generation at each stage is only affected by the features of the current shot segment, and the music styles corresponding to different shot segments do not interfere with each other, improving the coherence and adaptability of the music.

[0209] In some possible implementations, the music generation module is used to: construct a temporal attention mask for each shot segment; the temporal attention mask is used to prevent the music generation model from focusing on the multimodal control features of other shot segments outside of that specific shot segment when generating background music belonging to that specific shot segment during the current generation stage.

[0210] In some possible implementations, the music generation module is used to: independently generate background music for each shot segment based on its corresponding multimodal control features; or, for each shot segment, continue to generate background music based on the feature sequence of background music generated in the previous shot segment and the multimodal control features corresponding to the shot segment.

[0211] In some possible implementations, the feature generation module is used to: construct text features, visual features, and temporal structure features based on the multi-layer parsing results; wherein, the text features are generated based on global semantic information and the content semantic information of each shot segment; the visual features are based on the video content extraction of each shot segment; and the temporal structure features are generated based on shot structure information to indicate the duration of each shot segment and its structural role in music generation.

[0212] In some possible implementations, the feature generation module is used to: map text features and visual features to music style control signals; wherein the music style control signals are used at least to control the rhythm density and timbre tendency of the background music generated by the music generation model; the mapping is achieved through a pre-trained style mapping model, which is trained based on paired samples of video clips and corresponding background music.

[0213] In some possible implementations, visual features include at least one of image features, motion energy features, color style features, and emotional trend features; wherein, motion energy features are calculated based on the optical flow information between video frames of each shot segment; color style features are characterized by extracting the dominant hue and its distribution ratio of the video content of each shot segment in a specific color space; and emotional trend features are characterized by performing sentiment calculations on the video frame sequence within the shot segment to obtain a temporal sequence of the emotional dimension based on the valence-arousal model.

[0214] In some possible implementations, the feature generation module is used to encode and fuse text features, visual features, and temporal structure features to generate a unified conditional signal for controlling the music generation model.

[0215] In some possible implementations, the multi-layer parsing results also include visual rhythm point information; the visual rhythm point information is used to characterize the time sequence of key visual events in the video to be processed that are associated with the music rhythm; the music generation module is used to: adjust the tempo of the generated background music according to the visual rhythm point information so that the tempo points in the music are aligned with the time points of the key visual events.

[0216] In some possible implementations, the music generation module is used to: compare the time sequence corresponding to the key visual event with the time position sequence of the beat points in the background music; based on the comparison result, adjust the time position of the music beats through time-domain audio processing, or replace the music beats near the time point corresponding to the key visual event with rests, so that the adjusted music beats fall into the preset neighborhood of the corresponding key visual event time point on the time axis.

[0217] In some possible implementations, the video parsing module is used to: extract multiple keyframes from the video to be processed; perform visual semantic understanding on each keyframe to generate a corresponding local semantic description; and generate global semantic information to characterize the overall content semantics of the video to be processed based on the local semantic descriptions corresponding to all keyframes.

[0218] In some possible implementations, the video parsing module is used to: extract and aggregate features of the video content within each shot segment to obtain a unified feature representation that characterizes the visual semantics of the shot segment; input the unified feature representation into the text generation model to obtain a semantic summary text of the content of the segment, which serves as the semantic information of the shot segment.

[0219] It is understood that the above-described modules of the video-based music generation device in this embodiment of the present disclosure have the ability to implement... Figure 1 The embodiments shown illustrate the functionality of corresponding steps in the video-based music generation method. This functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described functions. These modules can be software and / or hardware, and each module can be implemented individually or integrated from multiple modules. For a detailed description of the functions of each module in the above-described video-based music generation device, please refer to [link to relevant documentation]. Figure 1 The corresponding description of the video parsing-based music generation method in the illustrated embodiments will not be repeated here.

[0220] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of users' personal information comply with the provisions of relevant laws and regulations, necessary measures have been taken, and there is no violation of public order and good morals.

[0221] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0222] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0223] The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the video parsing-based music generation method provided in the embodiments of this disclosure.

[0224] Compared with existing technologies, this electronic device provides clear control signals for the music generation model through multi-layer analysis and structured feature construction. Specifically, by extracting global semantic information, it clarifies the overall tone of music generation, avoiding the problem of "choosing a style based solely on feeling" in related technologies. By forcibly dividing the music generation process into stages according to video shot segments, the semantic information of each shot segment provides precise constraints on the corresponding music segment, ensuring that the music style is highly consistent with the shot content. The generated music can accurately follow the changes in video content in terms of emotion and style. By constraining the attention range of the music generation model, it ensures that the music generation at each stage is only affected by the features of the current shot segment, and the music styles corresponding to different shot segments will not interfere with each other, improving the coherence and adaptability of the music.

[0225] The readable storage medium is a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the video parsing-based music generation method provided in the embodiments of this disclosure.

[0226] Compared with existing technologies, this readable storage medium provides clear control signals for the music generation model through multi-layer parsing and structured feature construction. Specifically, by extracting global semantic information, it clarifies the overall tone of music generation, avoiding the problem of "choosing a style based solely on feeling" in related technologies. By forcibly dividing the music generation process into stages according to video shot segments, the semantic information of each shot segment provides precise constraints on the corresponding music segment, ensuring that the music style is highly consistent with the shot content, and the generated music can accurately follow the changes in video content in terms of emotion and style. By constraining the attention range of the music generation model, it ensures that the music generation at each stage is only affected by the features of the current shot segment, and the music styles corresponding to different shot segments will not interfere with each other, improving the coherence and adaptability of the music.

[0227] The computer program product includes a computer program that, when executed by a processor, implements the video-based music generation method provided in the embodiments of this disclosure.

[0228] Compared with existing technologies, this computer program provides clear control signals for the music generation model through multi-layered analysis and structured feature construction. Specifically, by extracting global semantic information, it clarifies the overall tone of music generation, avoiding the problem of "choosing a style based solely on feeling" in related technologies. By forcibly dividing the music generation process into stages according to video shot segments, the semantic information of each shot segment provides precise constraints on the corresponding music segment, ensuring that the music style is highly consistent with the shot content. The generated music can accurately follow the changes in video content in terms of emotion and style. By constraining the attention range of the music generation model, it ensures that the music generation at each stage is only affected by the features of the current shot segment, and the music styles corresponding to different shot segments will not interfere with each other, improving the coherence and adaptability of the music.

[0229] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0230] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded into random access memory (RAM) 703 from storage unit 708. The RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0231] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0232] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as a video-based music generation method. For example, in some embodiments, the video-based music generation method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the video-based music generation method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the video-based music generation method by any other suitable means (e.g., by means of firmware).

[0233] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0234] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0235] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0236] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0237] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0238] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0239] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0240] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A music generation method based on video parsing, comprising: The video to be processed is parsed to obtain multi-layered parsing results, which include at least global semantic information and shot structure information. The global semantic information is used to characterize the overall content semantic information of the video to be processed. The shot structure information is used to characterize multiple shot segments in the video to be processed in chronological order and the content semantic information of each shot segment. Based on the multi-layer analysis results, multimodal control features are constructed; Based on the multimodal control features, a pre-built music generation model is driven to generate background music that matches the duration of the video to be processed; The music generation model is divided into multiple generation stages corresponding to each shot segment based on the shot structure information. In each generation stage, the music generation model is constrained to generate background music only based on the multimodal control features of the shot segment corresponding to the current generation stage.

2. The method according to claim 1, wherein, The constraint that the music generation model can only generate background music based on the multimodal control features of the shot segment corresponding to the current generation stage is achieved in the following way: A temporal attention mask is constructed for each shot segment; the temporal attention mask is used to prevent the music generation model from paying attention to the multimodal control features of other shot segments outside of that specific shot segment when generating background music belonging to that specific shot segment in the current generation stage.

3. The method according to claim 1, wherein, The music generation model generates background music in at least one of the following ways: For each shot segment, background music is generated independently based on its corresponding multimodal control features; or, For each shot segment, based on the feature sequence of the background music generated in the previous shot segment and the multimodal control features corresponding to the shot segment, the background music for that shot segment is generated again.

4. The method according to claim 1, wherein, The construction of multimodal control features based on the multi-layer analysis results includes: Based on the multi-layer analysis results, text features, visual features, and temporal structure features are constructed. The text features are generated based on the global semantic information and the content semantic information of each shot segment; the visual features are extracted based on the video content of each shot segment; and the temporal structure features are generated based on the shot structure information, used to indicate the duration of each shot segment and its structural role in music generation.

5. The method according to claim 4, wherein, The step of driving a pre-built music generation model to generate background music matching the duration of the video to be processed, based on the multimodal control features, includes: The text features and the visual features are mapped into music style control signals; The music style control signal is used at least to control the rhythm density and timbre tendency of the background music generated by the music generation model; the mapping is achieved through a pre-trained style mapping model, which is trained based on paired samples of video clips and corresponding background music.

6. The method according to claim 4, wherein, The visual features include at least one of image features, motion energy features, color style features, and emotional trend features; The motion energy feature is calculated based on the optical flow information between video frames of each shot segment; the color style feature is characterized by extracting the dominant hue and its distribution ratio of the video content of each shot segment in a specific color space; and the emotion trend feature is characterized by performing emotion calculation on the video frame sequence within the shot segment to obtain an emotion dimension time sequence based on the valence-arousal model.

7. The method according to claim 4, wherein, The construction of multimodal control features based on the multi-layer analysis results includes: The text features, visual features, and temporal structure features are encoded and fused to generate a unified conditional signal for controlling the music generation model.

8. The method according to claim 1, wherein, The multi-layer analysis results also include visual rhythm point information; the visual rhythm point information is used to characterize the time point sequence corresponding to the key visual events in the video to be processed that are associated with the music rhythm. The step of driving a pre-built music generation model to generate background music matching the duration of the video to be processed, based on the multimodal control features, includes: Based on the visual rhythm point information, the generated background music is rhythmically adjusted so that the beat points in the music are aligned with the timing points of the key visual events.

9. The method according to claim 8, wherein, The step of adjusting the tempo of the generated background music based on the visual rhythm point information, so that the beat points in the music are aligned with the time points of the key visual events, includes: The time sequence corresponding to the key visual events is compared with the time position sequence of the beat points in the background music; Based on the comparison results, the time position of the music beat is adjusted by time-domain audio processing, or the music beat points near the time point corresponding to the key visual event are replaced with rests, so that the adjusted music beat points fall into the preset neighborhood of the corresponding key visual event time point on the time axis.

10. The method according to claim 1, wherein, The process involves parsing the video to be processed to obtain multi-layered parsing results, including: Extract multiple keyframes from the video to be processed; Perform visual semantic understanding on each keyframe to generate a corresponding local semantic description; Based on the local semantic descriptions corresponding to all the keyframes, global semantic information is generated to characterize the overall semantic content of the video to be processed.

11. The method according to claim 1, wherein, The process involves parsing the video to be processed to obtain multi-layered parsing results, including: For each shot segment, features are extracted and aggregated from the video content within that shot segment to obtain a unified feature representation that characterizes the visual semantics of that shot segment; The unified feature representation is input into the text generation model to obtain a semantic summary text of the paragraph content, which serves as the semantic information of the scene paragraph.

12. A music generation device based on video parsing, comprising: The video parsing module is used to parse the video to be processed and obtain multi-layer parsing results. The multi-layer parsing results include at least global semantic information and shot structure information. The global semantic information is used to characterize the overall content semantic information of the video to be processed. The shot structure information is used to characterize multiple shot segments in the video to be processed in chronological order and the content semantic information of each shot segment. The feature generation module is used to construct multimodal control features based on the multi-layer parsing results; The music generation module is used to drive a pre-built music generation model to generate background music that matches the duration of the video to be processed, based on the multimodal control features. The music generation model is divided into multiple generation stages corresponding to each shot segment based on the shot structure information. In each generation stage, the music generation model is constrained to generate background music only based on the multimodal control features of the shot segment corresponding to the current generation stage.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-11.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-11.

Images