Music generation model training method, music generation method and device

Abstract

The invention provides a music generation model training method and device, equipment and a storage medium, and relates to the technical field of computers, in particular to the technical fields of artificial intelligence, large models, deep learning, music generation and the like. According to the specific implementation scheme, a video sample is input into a to-be-trained first music generation model, and a first video vector output by the first music generation model and a corresponding music marking sequence are obtained; generating a music file based on the music mark sequence; calculating a loss function according to the video sample, the music file, the first video vector and the music mark sequence; and training the first music generation model based on the loss function to obtain a trained second music generation model. According to the invention, the model can be guided to generate music matched with the video content, and the diversity and quality of music generation are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to technical fields such as artificial intelligence, large models, deep learning, and music generation. Background Art

[0002] In recent years, more and more people have made personal videos through video platforms. In the process of video creation, matching the video with appropriate background music is a time-consuming and technically demanding task. With the development of artificial intelligence technology, the technology of automatically generating corresponding music from videos has solved most of people's needs. However, most of the current music generation methods rely on symbolic music annotations, which makes it difficult to generate real and diverse music. Therefore, how to automatically generate high-quality and diverse music based on videos has become a problem that needs to be solved urgently. Summary of the invention

[0003] The present invention provides a music generation model training method, a music generation method and a device.

[0004] According to one aspect of the present disclosure, a music generation model training method is provided, comprising:

[0005] Inputting the video sample into a first music generation model to be trained, obtaining a first video vector output by the first music generation model and a corresponding music tag sequence;

[0006] generating a music file based on the music tag sequence;

[0007] Calculating a loss function based on the video sample, the music file, the first video vector, and the music tag sequence;

[0008] Based on the loss function, the first music generation model is trained to obtain a trained second music generation model.

[0009] According to another aspect of the present disclosure, there is provided a music generation method, comprising:

[0010] Get the input video;

[0011] By inputting the video into a second music generation model, music related to the video output by the second music generation model is obtained; wherein the second music generation model is obtained by training the first music generation model using the training method provided by the present disclosure.

[0012] According to one aspect of the present disclosure, there is provided a music generation model training device, comprising:

[0013] A first input module, used for inputting a video sample into a first music generation model to be trained, and obtaining a first video vector and a corresponding music tag sequence output by the first music generation model;

[0014] A music generation module, used for generating a music file based on the music tag sequence;

[0015] A loss function calculation module, used for calculating a loss function according to the video sample, the music file, the first video vector and the music tag sequence;

[0016] A training module is used to train the first music generation model based on the loss function to obtain a trained second music generation model.

[0017] According to another aspect of the present disclosure, there is provided a music generating device, comprising:

[0018] An acquisition module is used to acquire input video;

[0019] The second input module is used to obtain music related to the video output by the second music generation model by inputting the video into the second music generation model; wherein the second music generation model is obtained by training the first music generation model using the training method provided by the present disclosure.

[0020] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0021] at least one processor; and

[0022] a memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0024] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0025] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0026] The present disclosure proposes a music generation model training method, which can train a music generation model, and proposes a method for using the music generation model to generate music matching a video. When using the model to generate music, only the video needs to be provided to the model, and music matching the video can be generated without relying on symbolic music annotations, thereby avoiding the generation of templated music, thereby improving the diversity and authenticity of music generation. At the same time, the music generation model can generate coherent music by capturing the dependency between the generated music tags, thereby improving the quality of music generation.

[0027] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0029] Figure 1 is a schematic diagram of an application scenario according to an embodiment of the present disclosure;

[0030] Figure 2 is a flow chart of an implementation of a music generation model training method according to an embodiment of the present disclosure;

[0031] Figure 3 is a schematic diagram of the architecture and training method of a music generation model according to an embodiment of the present disclosure;

[0032] Figure 4 is a schematic diagram of a music generation method according to an embodiment of the present disclosure;

[0033] Figure 5 is a structural schematic diagram of a music generation model training device 500 according to an embodiment of the present disclosure;

[0034] Figure 6 is a structural schematic diagram of a music generation model training device 600 according to an embodiment of the present disclosure;

[0035] Figure 7 is a structural schematic diagram of a music generating device 700 according to an embodiment of the present disclosure;

[0036] Figure 8 A schematic block diagram of an example electronic device 800 that may be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0037] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0038] The “and / or” in the embodiments of the present disclosure indicates that three relationships may exist. For example, A and / or B may indicate the following three situations: A exists alone, A and B exist at the same time, and B exists alone. The term “at least one” herein indicates any combination of at least two of any one or more of a plurality of types. For example, at least one of A, B, and C may indicate any one or more elements selected from the set consisting of A, B, and C. The terms “first” and “second” herein refer to and distinguish between multiple similar technical terms, and do not mean to limit the order or to limit the meaning to only two. For example, the first feature and the second feature refer to two types / two features. The first feature may be one or more, and the second feature may also be one or more.

[0039] In recent years, with the rapid development of Internet technology and the increasing popularity of video platforms, more and more people have begun to use these platforms to create and share personal videos. In this process, selecting and matching appropriate background music for the video has become a crucial link. It can not only enhance the appeal of the video, but also guide the audience's emotions and enhance the overall viewing experience. However, this work is a time-consuming and technically demanding challenge. Creators need to spend a lot of time browsing and screening massive music libraries, and at the same time have to have a certain musical aesthetic ability and video editing skills to ensure that the selected music is highly consistent with the video content in terms of rhythm, emotion, atmosphere, etc.

[0040] With the continuous breakthroughs and progress of artificial intelligence technology, a new technology that automatically generates corresponding background music based on video content has emerged, which has largely alleviated the pressure on creators in this regard and met most of people's needs. However, current music generation methods still mostly rely on symbolic music annotations, which represent the structure and elements of music in a simplified form, such as notes, chords, etc., which to some extent limits the diversity and authenticity of music generation.

[0041] Specifically, due to the limited number of symbolic music annotations, automated music generation systems can only select and combine from these limited resources when creating, making it difficult to produce truly novel and unique music works, and unable to meet people's needs for music diversity and personalization. At the same time, existing symbolic music annotations can only abstract and represent the basic structure and content of music, and it is difficult to fully capture and express the complexity of music. This results in the lack of realism in music generated by relying on these annotations.

[0042] In order to solve the above problems, the embodiments of the present disclosure propose a music generation model training method, which can be used to train a music generation model. The music generation model does not rely on symbolic music annotations. Users only need to input a video into the trained music generation model to generate background music that matches the video content. Since the music generation model is no longer restricted by symbolic music annotations, the diversity of music generation is improved. In addition, the music generation model can capture the dependencies between the generated music tags, generate more coherent music, and improve the quality of music generation.

[0043] Figure 1 is a schematic diagram of an application scenario according to an embodiment of the present disclosure, such as Figure 1 As shown, the application scenario diagram of the embodiment of the present disclosure may include but is not limited to a model training device 110 and a music generation device 120, and the model training device 110 and the music generation device 120 may communicate with each other through any type of wired or wireless network. Specifically, the model training device 110 can be used to train a music generation model, and send the trained music generation model to the music generation device 120; the music generation device 120 can load and install the music generation model, and after receiving a video input by a user, input the video into the music generation model, and generate music matching the video by the music generation model. Among them, the behavior sequence acquisition device 110 proposed in the embodiment of the present disclosure includes but is not limited to electronic devices such as mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, game consoles, e-book readers, multimedia playback devices, wearable devices, etc.; the music generation device 120 may include an electronic device or server for providing background management for the model training device 110. In addition, the embodiment of the present disclosure does not specifically limit the number of model training devices 110. For example, the application scenario diagram of the embodiment of the present disclosure may include one or more model training devices 110.

[0044] Figure 2 is a flowchart of a music generation model training method according to an embodiment of the present disclosure, including:

[0045] S210, inputting the video sample into a first music generation model to be trained, and obtaining a first video vector and a corresponding music tag sequence output by the first music generation model;

[0046] S220, generating a music file based on the music tag sequence;

[0047] S230, calculating a loss function according to the video sample, the music file, the first video vector and the music tag sequence;

[0048] S240. Based on the loss function, train the first music generation model to obtain a trained second music generation model.

[0049] It can be seen from the above training method that when training the first music generation model, a video sample is used to enable the model to generate a first video vector and a corresponding music tag sequence. The music tag sequence is processed to the corresponding music file, and based on the video sample, the music file, the first video vector and the music tag sequence, the loss function is calculated, so that the parameters of the model are adjusted according to the loss function. During the training process, since the first music generation model training method does not rely on symbolic music annotations, when using the trained second music generation model to generate music, it does not rely on symbolic music annotations, avoids generating templated music, and thus improves the diversity and authenticity of music generation. In addition, the second music generation model can capture the dependencies between the generated music tags, thereby generating more coherent music and improving the quality of music generation.

[0050] Figure 3 It is a schematic diagram of the architecture and training method of a music generation model according to an embodiment of the present disclosure.

[0051] like Figure 3 As shown, the main part of the first music generation model to be trained in the embodiment of the present disclosure is divided into a video encoding module 310 and a music decoding module 320, wherein the video encoding module 310 is used to convert a video sample into a first video vector, and the first video vector is used to characterize the video sample; the music decoding module 320 is used to generate a music marker sequence based on the first video vector, and the music marker sequence is used to characterize the music features corresponding to the video sample.

[0052] Through this architecture, the model realizes cross-modal conversion between video and music, a process that does not require human intervention and improves the level of automation and intelligence of music generation.

[0053] like Figure 3As shown, in order to process the video sample, the video encoding module 310 of the first music generation model involved in the embodiment of the present disclosure includes a convolution submodule 311 and a downsampling submodule 312. Among them, the convolution submodule 311 is used to extract the feature information of the video frame in the video sample; the downsampling submodule 312 is used to downsample the feature information to determine the first video vector.

[0054] In the convolution submodule 311, by designing appropriate convolution kernels, feature information of video frames in video samples can be effectively extracted. These convolution kernels not only have specific sizes and shapes, but also carry specific weights that can respond to specific structures in the input data. When the convolution kernel slides on the video frame, a feature map is generated by calculating the dot product between the input data and its own weights, which highlights the part of the input data that matches the convolution kernel.

[0055] For video samples, three-dimensional convolution can be used for processing, capturing both spatial and temporal features in the video. Unlike two-dimensional convolution, which only focuses on spatial information within a single frame, three-dimensional convolution can analyze the correlation between consecutive frames and understand dynamic changes in the video by introducing convolution kernels in the time dimension. When processing video samples, the three-dimensional convolution kernel slides on the video cube (i.e., a three-dimensional structure formed by stacking multiple consecutive video frames together), while calculating the dot product in space and time, which enables the model to capture complex spatial features in the video such as object motion, scene transitions, and character interactions.

[0056] In the downsampling submodule 312, a hierarchical structure is adopted to process the input data through multiple stages of downsampling operations. Figure 3 As shown, the 2nd, 5th, 21st and 24th layers of the downsampling submodule respectively perform downsampling operations on the feature information of the video frame, and after one downsampling operation, the video feature will enter the next stage of the downsampling process. The module divides the entire downsampling process into four stages to be carried out in sequence, wherein the 1st to 2nd layers are the first stage, the 3rd to 5th layers are the second stage, the 6th to 21st layers are the third stage, and the 22nd to 24th layers are the fourth stage, until the first video vector is output after the downsampling operation of the last layer. Through this design, different feature information of the video frame in the video sample can be captured at different stages, and efficient processing of the feature information of the video frame is achieved.

[0057] The above method can effectively extract key feature information from the video frames of video samples and reduce the dimension of the feature information, thereby reducing the amount of data and computational complexity, which helps the model to efficiently process large-scale video data.

[0058] In some implementations, the music decoding module 320 includes a plurality of sequentially connected layers, each layer is used to generate a music tag, and the music tag generated by each layer is used to synthesize a music tag sequence.

[0059] like Figure 3 As shown, each layer of the music decoding module 320 includes a causal attention block, a multi-head self-attention block and a multi-layer perceptron connected in sequence; wherein,

[0060] The input port of each layer of the multi-head self-attention block is the input port of the music decoding module 320, and the input information of each layer of the multi-head self-attention block includes the first video vector, the output information of the multi-head self-attention block in the adjacent high layer, and the output information of the causal attention block in the current layer;

[0061] The output port of each layer of the multi-layer perceptron is the output port of the music decoding module 320, and the output information of each layer of the multi-layer perceptron includes the music tag generated by the layer;

[0062] The input information of the causal attention block at each layer includes the music tags generated by the adjacent high layers.

[0063] The first video vector output by the video encoding module 310 is used as initial data to provide the necessary video feature information for the multi-head self-attention block of the music decoding module 320. At the highest level of the music decoding module 320, the multi-head self-attention block uses the first video vector to generate an abstract music representation. Subsequently, this music representation is used as guidance information and input into the multi-head self-attention block at the adjacent lower level.

[0064] In these adjacent lower layers, the multi-head self-attention block generates a corresponding, more specific and detailed music representation based on the music representation received from the higher layer, combined with the first video vector and the output of the causal attention block in the same layer. Among them, the multi-head self-attention block at each level receives the first video vector and incorporates it into the processing process as additional context information.

[0065] The music representation generated by the multi-head self-attention block of each layer will be input into the multi-layer perceptron of the same layer, and the multi-layer perceptron of each layer will output the corresponding music tag, that is, Figure 3 Y1, Y2, Y3…Y N-1, YN in the video. These music markers not only represent the status of the music at different time points or different sections, but also reflect the model's deep understanding of the synchronization relationship between video and music. For example, Y1 may correspond to the beginning of the music at the beginning of the video, and Y3 may reflect the music corresponding to a certain transition in the video. These music markers capture the structure and melodic features of the music and incorporate video context information, so that the final generated music content not only conforms to the inherent laws of the audio data, but is also closely related to the video content.

[0066] The music tags generated by each high-level layer will be input into the causal attention block of the adjacent low-level layer, which will determine the temporal dependency vector based on the music tags generated by the adjacent high-level layer. This vector can reflect the dependency between the music tags generated by the adjacent high-level layer and the pre-generated music tags of this layer. This dependency can guide the multi-head self-attention block of this layer to avoid generating incoherent or abrupt music representations, thereby ensuring the quality of the generated music tags. The music tags generated by each layer can be combined to form a music tag sequence corresponding to the content of the video sample, which is convenient for the subsequent generation of the corresponding music.

[0067] Since each layer contains a causal attention block, it can capture the temporal dependencies between music tags, ensuring that the generated music remains coherent and consistent in time. At the same time, the multi-head self-attention block can handle the global dependencies in the music, so that the generated music not only conforms to the local features, but also reflects the overall structure and style. This combination enables the model to generate more complex, diverse and creative music tag sequences.

[0068] In some implementations, generating a music file based on a music tag sequence includes:

[0069] Parsing the music tag sequence to obtain music generation instructions corresponding to the music tag sequence;

[0070] Based on the music generation instruction, a corresponding music file is generated.

[0071] The step of parsing the music tag sequence to obtain a music generation instruction corresponding to the music tag sequence includes:

[0072] According to the music mark sequence, identifying a timbre feature and at least one note corresponding to the music mark sequence;

[0073] extracting note information of at least one note;

[0074] Use timbre features and note information to construct music generation instructions;

[0075] The note information includes at least one of pitch, duration and loudness.

[0076] The above process mainly includes two stages: parsing the music tag sequence to obtain the music generation instruction, and generating the corresponding music file based on the music generation instruction.

[0077] The music generation device receives and reads the music tag sequence, which is composed of the music tags generated layer by layer by the music decoding module. After the initial analysis, the device needs to interpret the music tag sequence more deeply to identify the timbre characteristics and each note corresponding to the sequence. The timbre characteristics may include the type of instrument, the brightness of the timbre, etc. The notes represent the symbols of the sound.

[0078] In the process of processing the music tag sequence to generate a music file, a key step is to accurately match the vector representation of the timbre in the music tag sequence with the specific timbre characteristics according to the preset timbre mapping table. The realization of this step depends on the device's in-depth analysis of the music tag sequence and accurate query of the timbre mapping table.

[0079] The timbre vector representation in the musical notation sequence is usually described by a series of values ​​and parameters, which can reflect various characteristics of the timbre, such as the brightness, warmth, roughness, etc. The timbre vector representation provides an abstract, digital timbre description for the device, allowing the device to plan and adjust the timbre of the musical work without directly relying on specific audio samples.

[0080] During the note recognition process, a unique note name and attribute set are defined for each possible note vector according to the encoding rules of the note. The note name is usually a short text label, such as "C4" (for middle C, pitch 4) or "G#5" (for G sharp, pitch 5). The attribute set includes more detailed information, such as the duration and loudness of the note. The pitch, duration, and loudness of the note together constitute the note information.

[0081] In the music generation instruction construction stage, a suitable instruction format is designed to encode the timbre characteristics and note information into a code that the device can understand. This format may include a series of parameters, each of which corresponds to a specific aspect of the timbre or note. Then, the timbre characteristics and note information are mapped to the corresponding parameters of the instruction format, and specific music generation instructions are generated according to the mapped parameter values. These instructions exist in the form of binary code.

[0082] During the music file generation stage, the music generation device receives the music generation instruction and will parse the instruction. Since the instruction contains all the key information that constitutes the music work, the device needs to accurately interpret this information to ensure that the corresponding music file can be generated. After parsing the instruction, the device will synthesize the corresponding music file through a series of complex calculations and processing based on the note information and timbre characteristics in the instruction. It is worth noting that when generating music files, the music generation device usually takes into account multiple factors to ensure the sound quality and expressiveness of the final work. For example, the device may adjust the amplitude and frequency of the audio signal according to the volume and pitch changes of the notes; at the same time, it will also filter, modulate and process the audio signal according to the characteristics of the timbre to simulate the unique sound quality of different instruments or sound sources.

[0083] By using the above method, different timbre features and notes can be identified by parsing the music tag sequence, and then a variety of music generation instructions can be constructed. In addition, since the music generation instructions are constructed based on the specific information in the music tag sequence, the generated music file can accurately reflect the intention and style contained in the music tag sequence, avoiding the deviation and misunderstanding that may occur in traditional creation methods.

[0084] In some implementations, a loss function is calculated based on the video sample, the music file, the first video vector, and the music tag sequence.

[0085] In some examples, the loss function is calculated as follows:

[0086] Determine multiple single loss functions, and use the multiple single loss functions to calculate the loss function.

[0087] For example, a weighted sum of multiple single loss functions may be calculated to obtain a loss function, wherein the weight of each single loss function is set according to a preset rule. The single loss function may include: determining a first loss function using a first video vector and a music tag sequence; and determining a second loss function using a video sample and a music file.

[0088] Different loss functions can evaluate the performance of the model from different perspectives. In the embodiment of the present disclosure, the first loss function focuses on evaluating the matching degree between the generated music and video samples in terms of high-level content (such as genre, style, etc.), while the second loss function focuses on the matching degree between the generated music and video samples in terms of rhythm. By combining these loss functions, a more comprehensive and accurate performance evaluation can be obtained. The first loss function and the second loss function are introduced as follows:

[0089] In some examples, the first loss function is determined using the first video vector and the music tag sequence. In this process, it is necessary to extract the style features of the video sample based on the first video vector to generate the second video vector; and, based on the music tag sequence, extract the style features of the music file to generate the second music vector.

[0090] Style features refer to the uniqueness of video samples and music files, such as classical style, modern style, etc. Based on the first video vector obtained by the video encoding module, style features are extracted from the video sample. In this step, a series of image processing and analysis techniques, such as color space transformation, texture analysis, shape detection, etc., are used to capture the key style elements in the video sample, such as color matching, texture pattern, composition method, etc. By integrating these style features, a new video feature vector is generated, namely the second video vector. This vector not only contains the basic visual information of the first video vector, but also incorporates the unique style features of the video sample.

[0091] At the same time, it is necessary to extract style features from the music file, which is performed based on the music tag sequence. Specifically, a mapping algorithm is applied to map the extracted music style features into the feature space, wherein the mapping algorithm may include linear mapping, dimensionality reduction algorithm, etc. In the feature space, the music style features are represented as a specific data structure, namely, the second music vector, for subsequent analysis and comparison.

[0092] The first loss function is determined by calculating the cosine similarity between the second video vector and the second music vector, wherein the first loss function can be determined using the following calculation formula:

[0093] in,

[0094] represents the first loss function;

[0095] B, represents the number of second video vectors or second music vectors, wherein the number of second video vectors and second music vectors should be consistent;

[0096] represents the i-th second video vector, where the value of i ranges from 1 to B;

[0097] M i , represents the i-th second music vector, and the value of i ranges from 1 to B;

[0098] M j , represents the jth second music vector, and the value of j ranges from 1 to B;

[0099] cos(), which means calculating cosine similarity.

[0100] Among them, cosine similarity evaluates the similarity between two vectors by measuring the cosine value of the angle between them. In two-dimensional space, this vector can be regarded as an arrow starting from the origin, and cosine similarity is the cosine value of the angle between the two arrows. This value is between -1 and 1. The larger the value, the more similar the two vectors are, and the smaller the value, the less similar the two vectors are.

[0101] The first loss function aims to quantitatively evaluate the consistency and coordination of the generated music and video samples in terms of style. When the styles of the generated music and video samples are more consistent, the value of this function is closer to 0, and the loss value is smaller. Through this loss function, the degree of integration of the generated content at the style level is effectively measured, thereby guiding the model to adjust the style consistency.

[0102] In some examples, the second loss function is determined using the video sample and the music file. In this process, it is necessary to identify the video rhythm of the video sample and the music beat of the music file, and determine the second loss function by calculating the overlap between the video rhythm and the music beat.

[0103] The video rhythm of the video sample can be used to identify dynamic moments in the video (such as scene changes, dynamic human movement, etc.) by calculating the optical flow between consecutive frames in the video and taking the spatial average of the optical flow amplitude. Specifically, the algorithm analyzes the video frame by frame and calculates the optical flow by comparing the pixel changes between adjacent frames. Optical flow is a physical quantity that describes the direction and speed of movement of pixels in an image, which can reflect the dynamic changes between video frames. When calculating optical flow, the Lucas-Kanade (LK) method is usually used to ensure the accuracy and efficiency of the calculation.

[0104] The algorithm calculates the optical flow amplitude of each pixel, that is, the vector length of the optical flow. Then, the optical flow amplitude of the entire video frame is spatially averaged to obtain a global optical flow amplitude value, which reflects the overall dynamic change of the current frame relative to the previous frame. By continuously calculating the optical flow amplitude of each frame and taking the average value, the algorithm can generate an optical flow amplitude sequence. This sequence describes the continuity and intensity of dynamic changes in the video and is an important basis for identifying dynamic moments.

[0105] Identifying "video rhythm" in the optical flow amplitude sequence is a key step. Video rhythm can be understood as the points where dynamic changes occur in the video, usually corresponding to visual events such as scene transitions and dynamic human movements. In order to identify these beats, a series of signal processing techniques, such as peak detection and autocorrelation analysis, are used to find significant change points in the optical flow amplitude sequence. These change points often correspond to dynamic moments in the video and can be regarded as video rhythm. By vectorizing the video rhythm, the video rhythm vector Pv can be obtained.

[0106] The music beat of the music file can be analyzed in depth by using the beat detection algorithm to extract a stable music beat. Specifically, the algorithm pre-processes the input music signal, including removing noise and normalizing the volume, to ensure the accuracy of subsequent analysis. Through signal processing techniques such as Fourier transform, the music signal is converted from the time domain to the frequency domain, so that different frequency components in the music can be more clearly identified.

[0107] Next, this frequency information is used to detect periodic changes in the music, especially those related to the beat. This usually involves performing autocorrelation analysis or peak detection on the signal to find recurring, high-intensity signal points, which often represent the beat positions in the music.

[0108] After identifying potential beat points, further screening and optimization will be performed to exclude those false detections caused by noise or other non-rhythmic factors. This step can be done by setting some music-related parameters, such as beat speed range, beat intensity distribution, etc., to further verify and correct the position of the beat points, and finally obtain the music beat in the music file. By vectorizing the music beat, the music rhythm vector Pa can be obtained.

[0109] In order to determine the second loss function, it is necessary to first align the timestamps of the video sample and the music file to ensure that the video rhythm and the music beat are synchronized in time. On this basis, by calculating the overlap between the video rhythm vector and the music beat vector, the second loss function can be finally determined, which can be determined using the following calculation formula:

[0110]

[0111] represents the second loss function;

[0112] n, represents the number of video rhythm or music beats, where the number of video rhythm and music beats should be consistent;

[0113] Pv k , represents the kth video rhythm vector, and the value of k ranges from 1 to n;

[0114] Pa k , represents the kth music beat vector, and the value of k ranges from 1 to n;

[0115] (Pv k -Pa k ), which represents the overlap between the k-th video rhythm vector and the k-th music beat vector.

[0116] The second loss function aims to quantitatively evaluate the consistency and coordination between the video rhythm and the music beat. The more the video rhythm and the music beat overlap, the smaller the second loss function will be. Through this loss function, the matching degree between the video rhythm and the music beat is effectively measured, which helps to determine whether the two are coordinated, thereby guiding the model to generate music files synchronized with the video rhythm.

[0117] Using the above-mentioned single loss function, the loss function for training the model can be calculated. In some examples, the weighted sum of the first loss function and the second loss function is calculated to obtain the loss function; wherein the weight of each single loss function is set according to a preset rule.

[0118] For example, the loss function can be determined using the following calculation formula:

[0119] in,

[0120] represents the loss function;

[0121] Calculate the weighted sum of the above-mentioned first loss function and the second loss function to obtain the loss function. Among them, the weight (λ) used when calculating the weighted sum can be set according to the preset rules. The embodiment of the present disclosure can set the weights of the first loss function and the second loss function, so as to achieve focused training of different capabilities of the model. For example, if the focus is on urging the model to learn to generate music with the same style as the video, increase the weight of the first loss function; if the focus is on training the model to learn to generate music with the same beat as the video, increase the weight of the second loss function.

[0122] Using the above method, the weights of different loss functions can be adjusted according to specific needs to optimize the performance of the model in specific aspects.

[0123] The above introduces the training process of the first music generation model proposed in the embodiment of the present disclosure.

[0124] After training the first music generation model, a second music generation model is obtained, and corresponding music can be generated using the video input by the user.

[0125] The disclosed embodiment also proposes a music generation method, such as Figure 4 As shown, the method includes: obtaining an input video 410; obtaining video-related music 430 output by the second music generation model 420 by inputting the video 410 into a second music generation model 420; wherein the second music generation model 420 is obtained by training the first music generation model using the above-mentioned training method.

[0126] In some embodiments, the second music generation model 420 includes a video encoding module and a music decoding module. The video encoding module is used to receive a video input by a user, and generate a first video vector corresponding to the video by performing three-dimensional convolution and multiple downsampling on the video, wherein the first video vector carries context information contained in the video; and the music decoding module is used to generate a music tag sequence using the first video vector.

[0127] The music decoding module includes a plurality of layers connected in sequence, each layer is used to generate a music tag, and the music tag generated by each layer is used to synthesize a music tag sequence. In each layer, there are causal attention blocks, multi-head self-attention blocks and multi-layer perceptrons connected in sequence. The multi-head self-attention block is used to receive the first video vector output by the video encoder, the first music representation output by the multi-head self-attention block in the adjacent high-level layer, and the output content of the causal attention block in the current layer to generate the first music representation of the current layer. The multi-layer perceptron of the current layer generates the music tag of the current layer based on the first music representation generated by the current layer.

[0128] The music tags generated by each high-level layer will be input into the causal attention block of the adjacent low-level layer. Based on the music tags generated by the high-level layer, the causal attention block determines the temporal dependency vector between the pre-generated music tags of the current layer and the music tags generated by the adjacent high-level layer. The causal attention block inputs the temporal dependency vector into the multi-head self-attention block of the same layer to subsequently generate the music tags of the current layer.

[0129] When each layer generates corresponding music tags, the music tags are combined into a music tag sequence so that the second music generation model can generate corresponding music using the music tag sequence.

[0130] Using this method, users only need to input a video into the second music generation model, and the model will automatically generate music that matches the video content. At the same time, in the process of music generation, there is no need to rely on symbolic music annotations. This approach effectively avoids the template tendency of generated music, thereby significantly enhancing the diversity and authenticity of music generation. In addition, the second music generation model can capture the dependencies between the generated music markers, generate more coherent music, and further improve the overall quality of the music.

[0131] The disclosed embodiment also provides a music generation model training device, Figure 5 is a structural diagram of a music generation model training device 500 according to an embodiment of the present disclosure, comprising:

[0132] A first input module 510, used to input a video sample into a first music generation model to be trained, and obtain a first video vector and a corresponding music tag sequence output by the first music generation model;

[0133] A music generation module 520, for generating a music file based on the music tag sequence;

[0134] A loss function calculation module 530, configured to calculate a loss function based on the video sample, the music file, the first video vector and the music tag sequence;

[0135] The training module 540 is used to train the first music generation model based on the loss function to obtain a trained second music generation model.

[0136] In some implementations, the first music generation model includes a video encoding module and a music decoding module; wherein,

[0137] A video encoding module, used for converting a video sample into a first video vector, where the first video vector is used for representing the video sample;

[0138] The music decoding module is used to generate a music tag sequence based on the first video vector, and the music tag sequence is used to characterize the music features corresponding to the video sample.

[0139] In some embodiments, the video encoding module includes a convolution submodule and a downsampling submodule;

[0140] in,

[0141] A convolution submodule, used to extract feature information of video frames in video samples;

[0142] The down-sampling submodule is used to down-sample the feature information to determine the first video vector.

[0143] In some implementations, the music decoding module includes a plurality of sequentially connected layers, each layer is used to generate a music tag, and the music tag generated by each layer is used to synthesize a music tag sequence.

[0144] In some embodiments, each layer includes a causal attention block, a multi-head self-attention block, and a multi-layer perceptron connected in sequence; wherein,

[0145] The input port of each layer of multi-head self-attention block is the input port of the music decoding module. The input information of each layer of multi-head self-attention block includes the first video vector, the output information of the multi-head self-attention block in the adjacent high layer, and the output information of the causal attention block in the current layer;

[0146] The output port of each layer of multi-layer perceptron is the output port of the music decoding module, and the output information of each layer of multi-layer perceptron includes the music tag generated by the layer;

[0147] The input information of the causal attention block at each layer includes the music tags generated by the adjacent high layers.

[0148] In some implementations, the music generation module 520 is used to:

[0149] Parsing the music tag sequence to obtain music generation instructions corresponding to the music tag sequence;

[0150] Based on the music generation instruction, a corresponding music file is generated.

[0151] In some implementations, the music generation module 520 is used to:

[0152] According to the music mark sequence, identifying the timbre characteristics and at least one note corresponding to the music mark sequence;

[0153] extracting note information of at least one note;

[0154] Use timbre features and note information to construct music generation instructions;

[0155] The note information includes at least one of pitch, duration and loudness.

[0156] Figure 6 is a structural diagram of a music generation model training device 600 according to an embodiment of the present disclosure. In some implementations, the loss function calculation module 530 includes a single loss function determination module 650 and a loss function determination module 660;

[0157] The single loss function determination module 650 is used to determine multiple single loss functions, and the determination of multiple single loss functions includes: determining a first loss function using a first video vector and a music tag sequence; and determining a second loss function using a video sample and a music file;

[0158] The loss function determination module 660 is used to calculate the loss function using multiple single loss functions.

[0159] In some implementations, the single loss function determination module 650 is used to:

[0160] Based on the first video vector, extracting style features of the video sample to generate a second video vector; and,

[0161] Extracting style features of the music file based on the music tag sequence to generate a second music vector;

[0162] Calculating a cosine similarity between the second video vector and the second music vector;

[0163] Based on the cosine similarity, a first loss function is determined.

[0164] In some implementations, the single loss function determination module 650 is used to:

[0165] Identify the video rhythm of video samples and the music beat of music files;

[0166] Calculate the overlap between the video rhythm and the music beat;

[0167] Based on the overlap, a second loss function is determined.

[0168] In some embodiments, the loss function determination module 660 is used to calculate a weighted sum of the first loss function and the second loss function to obtain a loss function;

[0169] Among them, the weight of each single loss function is set according to the preset rules.

[0170] The disclosed embodiment also provides a music generating device, Figure 7 is a structural diagram of a music generating device 700 according to an embodiment of the present disclosure, comprising:

[0171] An acquisition module 710 is used to acquire an input video;

[0172] The second input module 720 is used to obtain music related to the video output by the second music generation model by inputting the video into the second music generation model; wherein the second music generation model is obtained by training the first music generation model using the training device provided by the present disclosure.

[0173] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0174] In the technical solution disclosed in the present invention, the acquisition, storage and application of personal information of users involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0175] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0176] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0177] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0178] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0179] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as detection methods. For example, in some embodiments, the detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the detection method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the detection method in any other appropriate manner (e.g., by means of firmware).

[0180] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0181] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0182] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0183] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0184] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0185] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0186] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0187] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A music generation model training method, the method comprising: Inputting the video sample into a first music generation model to be trained, and obtaining a first video vector and a corresponding music tag sequence output by the first music generation model; generating a music file based on the music tag sequence; Calculating a loss function based on the video sample, the music file, the first video vector and the music tag sequence; Based on the loss function, the first music generation model is trained to obtain a trained second music generation model.

2. The method according to claim 1, wherein: The first music generation model includes a video encoding module and a music decoding module; wherein, The video encoding module is used to convert the video sample into the first video vector, where the first video vector is used to represent the video sample; The music decoding module is used to generate the music tag sequence based on the first video vector, and the music tag sequence is used to characterize the music features corresponding to the video sample.

3. The method according to claim 2, wherein: The video encoding module includes a convolution submodule and a downsampling submodule; wherein, The convolution submodule is used to extract feature information of video frames in the video sample; The down-sampling submodule is used to down-sample the feature information to determine the first video vector.

4. The method according to claim 2 or 3, wherein: The music decoding module includes a plurality of layers connected in sequence, each of which is used to generate a music tag, and the music tag generated by each layer is used to synthesize the music tag sequence.

5. The method according to claim 4, wherein: Each of the layers includes a causal attention block, a multi-head self-attention block and a multi-layer perceptron connected in sequence; wherein, The input port of the multi-head self-attention block of each layer is the input port of the music decoding module, and the input information of the multi-head self-attention block of each layer includes the first video vector, the output information of the multi-head self-attention block in the adjacent high layer, and the output information of the causal attention block in the current layer; The output port of the multilayer perceptron at each layer is the output port of the music decoding module, and the output information of the multilayer perceptron at each layer includes the music tag generated by the layer; The input information of the causal attention block at each layer includes the music tags generated by the adjacent high layer.

6. The method according to claim 1 or 5, wherein: The step of generating a music file based on the music tag sequence comprises: Parsing the music tag sequence to obtain a music generation instruction corresponding to the music tag sequence; Based on the music generation instruction, the corresponding music file is generated.

7. The method according to claim 6, wherein: The parsing of the music tag sequence to obtain a music generation instruction corresponding to the music tag sequence includes: According to the music mark sequence, identifying a timbre feature and at least one note corresponding to the music mark sequence; extracting note information of at least one of the notes; Using the timbre characteristics and the note information, construct the music generation instruction; The note information includes at least one of pitch, duration and loudness.

8. The method according to at least one of claims 1 to 7, wherein: The calculating the loss function according to the video sample, the music file, the first video vector and the music tag sequence comprises: Determine a plurality of individual loss functions, and calculate the loss function using the plurality of individual loss functions; Wherein, determining multiple single loss functions includes: determining a first loss function using the first video vector and the music tag sequence; and, A second loss function is determined using the video sample and the music file.

9. The method according to claim 8, wherein: The determining a first loss function using the first video vector and the music tag sequence comprises: Based on the first video vector, extracting style features of the video sample to generate a second video vector; and Extracting style features of the music file based on the music tag sequence to generate a second music vector; Calculating the cosine similarity between the second video vector and the second music vector; Based on the cosine similarity, the first loss function is determined.

10. The method according to claim 8, wherein: The determining a second loss function using the video sample and the music file comprises: identifying a video rhythm of the video sample and a music beat of the music file; Calculating the overlap between the video rhythm and the music beat; Based on the overlap, the second loss function is determined.

11. The method according to any one of claims 8 to 10, wherein: The calculating the total loss function by using the multiple individual loss functions comprises: Calculating a weighted sum of the first loss function and the second loss function to obtain the loss function; Among them, the weight of each of the single loss functions is set according to preset rules.

12. A music generation method, the method comprising: Get the input video; By inputting the video into a second music generation model, music related to the video output by the second music generation model is obtained; wherein the second music generation model is obtained by training the first music generation model using the training method described in any one of claims 1-11.

13. A music generation model training device, the device comprising: A first input module, used for inputting a video sample into a first music generation model to be trained, and obtaining a first video vector and a corresponding music tag sequence output by the first music generation model; A music generation module, used for generating a music file based on the music tag sequence; A loss function calculation module, used to calculate a loss function according to the video sample, the music file, the first video vector and the music tag sequence; A training module is used to train the first music generation model based on the loss function to obtain a trained second music generation model.

14. The device according to claim 13, wherein: The first music generation model includes a video encoding module and a music decoding module; wherein, The video encoding module is used to convert the video sample into the first video vector, where the first video vector is used to represent the video sample; The music decoding module is used to generate the music tag sequence based on the first video vector, and the music tag sequence is used to characterize the music features corresponding to the video sample.

15. The device according to claim 14, wherein: The video encoding module includes a convolution submodule and a downsampling submodule; wherein, The convolution submodule is used to extract feature information of video frames in the video sample; The down-sampling submodule is used to down-sample the feature information to determine the first video vector.

16. The device according to claim 14 or 15, wherein: The music decoding module includes a plurality of layers connected in sequence, each of which is used to generate a music tag, and the music tag generated by each layer is used to synthesize the music tag sequence.

17. The device according to claim 16, wherein: Each of the layers includes a causal attention block, a multi-head self-attention block and a multi-layer perceptron connected in sequence; wherein, The input port of the multi-head self-attention block of each layer is the input port of the music decoding module, and the input information of the multi-head self-attention block of each layer includes the first video vector, the output information of the multi-head self-attention block in the adjacent high layer, and the output information of the causal attention block in the current layer; The output port of the multilayer perceptron at each layer is the output port of the music decoding module, and the output information of the multilayer perceptron at each layer includes the music tag generated by the layer; The input information of the causal attention block at each layer includes the music tags generated by the adjacent high layer.

18. The device according to claim 13 or 17, wherein: The music generation module is used for: Parsing the music tag sequence to obtain a music generation instruction corresponding to the music tag sequence; Based on the music generation instruction, the corresponding music file is generated.

19. The device according to claim 18, wherein: The music generation module is used for: According to the music mark sequence, identifying a timbre feature and at least one note corresponding to the music mark sequence; extracting note information of at least one of the notes; Using the timbre characteristics and the note information, construct the music generation instruction; The note information includes at least one of pitch, duration and loudness.

20. The device according to at least one of claims 13 to 19, wherein: The loss function calculation module includes: A single loss function determination module, used to determine multiple single loss functions, wherein the determining multiple single loss functions comprises: determining a first loss function using the first video vector and the music tag sequence; and determining a second loss function using the video sample and the music file; A loss function determination module is used to calculate the loss function using the multiple individual loss functions.

21. The device according to claim 20, wherein: The single loss function determination module is used for: Based on the first video vector, extracting style features of the video sample to generate a second video vector; as well as, Extracting style features of the music file based on the music tag sequence to generate a second music vector; Calculating the cosine similarity between the second video vector and the second music vector; Based on the cosine similarity, the first loss function is determined.

22. The device according to claim 20, wherein: The single loss function determination module is used for: identifying a video rhythm of the video sample and a music beat of the music file; Calculating the overlap between the video rhythm and the music beat; Based on the overlap, the second loss function is determined.

23. The device according to any one of claims 20 to 22, wherein the loss function determination module is used to calculate a weighted sum of the first loss function and the second loss function to obtain the loss function; in, The weights of the individual loss functions are set according to preset rules.

24. A music generating device, comprising: An acquisition module is used to acquire input video; The second input module is used to obtain music related to the video output by the second music generation model by inputting the video into the second music generation model; wherein the second music generation model is obtained by training the first music generation model using the training device described in any one of claims 13-23.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

27. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images