A short video automatic music matching method and system based on reinforcement learning

Abstract

The application discloses a short video automatic music matching method and system based on reinforcement learning, which comprises the following steps: constructing an adaptive video encoder and extracting video deep features; obtaining a MIDI file of short video music matching, constructing an audio encoder, and extracting a plurality of sets of audio feature vector sets from the MIDI format file; taking the video features as input and the audio feature vector sets as output, and constructing a short video music matching model based on reinforcement learning; according to the reinforcement learning music matching model, a sampling method is designed to obtain a music feature vector set; and the music feature vector set is decoded into a MIDI format file to complete the automatic music matching of the short video. The application solves the problems of difficulty in selecting artificial short video music matching, the need for music background knowledge and common sense, music copyright infringement and the like, greatly improves the speed and quality of short video music matching, and realizes automatic music matching.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence technology and music scoring, and in particular to a method and system for automatic music scoring of short videos based on reinforcement learning. Background Technology

[0002] Currently, short videos have become a very important form of online social interaction and entertainment. Among these, background music is a crucial aspect of short video production. Typically, people spend a significant amount of time selecting suitable music for their short videos, while also considering issues such as music copyright.

[0003] Automatic background music for short videos refers to methods that use artificial intelligence algorithms to automatically generate background music for short video content. By using reinforcement learning methods, configuring suitable reward functions and action parameters, and designing reasonable encoding methods for video and audio, it can learn relevant background music knowledge from real background music datasets. The generated music data is more consistent with the video content, effectively saving time and labor costs.

[0004] Some short video websites have launched automatic background music (IMM) features for short videos, but these features only extract music manually selected by the creator based on the video length. This method cannot automatically select background music based on the video content, requiring short video creators to spend a significant amount of time selecting music that matches the video content from massive amounts of data. To address these shortcomings, this paper proposes an automatic background music method and system for short videos based on reinforcement learning. This system can automatically generate suitable background music based on the short video content without manual intervention. It employs a professional encoding method, using multi-hot and one-hot encoding for note parameters, instrument parameters, velocity parameters, tempo parameters, and controller parameters, ensuring the background music conforms to music theory rules and that the background music information is more complete and accurate. Furthermore, a reasonable action value function is designed, enabling the reinforcement learning algorithm to quickly generate background music, significantly saving video creators' time. Summary of the Invention

[0005] In view of this, one of the objectives of this invention is to provide a method for automatically adding background music to short videos based on reinforcement learning, which is mainly used to solve the drawback of traditional methods that cannot quickly select background music for short videos.

[0006] One of the objectives of this invention is achieved through the following technical solution:

[0007] This reinforcement learning-based method for automatically adding background music to short videos includes the following steps:

[0008] Step S1: Construct an adaptive video encoder to extract video depth features;

[0009] Step S2: Obtain the MIDI file of the background music for the short video, build an audio encoder, and extract multiple sets of audio feature vectors from the MIDI format file;

[0010] Step S3: Use the video features and audio feature vector sets as training data to construct and train a short video background music model based on reinforcement learning;

[0011] Step S4: Based on the reinforcement learning music model, design a sampling method to obtain the music feature vector set;

[0012] Step S5: Decode the background music feature vector set into a MIDI format file to complete the automatic background music for the short video.

[0013] Furthermore, step S1 specifically includes:

[0014] Step S101: Perform single-channel sampling on the short video with a sampling precision of 8 bits, a sampling frame rate of 10 fps, and stretch the resolution to 512×512 to obtain a short video image set;

[0015] Step S102: Construct an adaptive video encoder by inputting the sampled short video image set into the encoder and outputting a depth video feature vector set.

[0016] Furthermore, the adaptive video encoder in step S102 specifically comprises:

[0017] H t =ReLU(conv(v t ,P H )+conv(v t-1 Q H ))

[0018] Z t =tanh(conv(v t ,P Z )+H t ⊙conv(v t-1 Q Z ))

[0019] U t =v t-1 ⊙(H t +1)+Z t ⊙H t

[0020] g t =ReLU(v t F)

[0021] Where v is the sampled video frame sequence, v t For the current video frame, v t-1For the previous video frame, ReLU() and tanh() are activation functions, H t P represents the degree of information retention of the video frames at times t and t-1. H and Q H Z is the video information weight matrix. t P represents the degree of information loss in the video frames at times t and t-1. Z and Q Z U is a weight matrix that has the effect of losing video information weights. t Let g be the feature matrix containing key information of the video frame at time t, ⊙ be the Hadamard product (element-wise multiplication of corresponding positions of two vectors), F be the linear transformation weight matrix, and g be the vector matrix. t Let be the feature vector containing key information of the video frame at time t.

[0022] Furthermore, step S2 includes:

[0023] Step S201: Obtain the MIDI file of the background music corresponding to the short video and trim it to the same length as the short video described in step S1;

[0024] Step S202: Extract the note parameters, instrument parameters, velocity parameters, tempo parameters, and controller parameters from the extracted MIDI file;

[0025] Step S203: Construct an audio encoder, input the audio parameters extracted in step S202, and output multiple sets of audio feature vectors.

[0026] Furthermore, the audio encoder in step S203 specifically comprises:

[0027] The note parameters with values ​​ranging from 00 to 7F are multi-hot encoded and added to the feature vector set;

[0028] The instrument parameters with values ​​ranging from 00 to 7F are one-hot encoded and added to the feature vector set;

[0029] The force parameter, which ranges from 00 to 7F, is one-hot encoded and added to the feature vector set;

[0030] The beat parameters, which range from 00 to 7F, are one-hot encoded and added to the feature vector set;

[0031] The controller parameters, which range from 00 to 7F, are multi-hot encoded and added to the feature vector set.

[0032] Furthermore, step S3 includes:

[0033] Step S301: Model the process of adding background music to short videos as a specific Markov decision process, defining the state, action, and reward;

[0034] Step S302: Based on a specific Markov decision process, use video features and audio feature vector sets as training data to establish a novel short video background music model based on reinforcement learning;

[0035] Step S303: Train the above reinforcement learning model to obtain a set of short video background music experience.

[0036] Furthermore, in step S302, the short video background music model based on reinforcement learning is established as follows:

[0037] Within the reinforcement learning framework, the automatic music accompaniment system is treated as an intelligent agent, and the process of automatically accompaniing music to short videos is modeled as a Markov decision process model:

[0038] The state, actions, and reward function of the index selection process described in step S301 are as follows:

[0039] State: refers to the current state of the agent, i.e., the current set of audio feature vectors. The initial state is defined as follows: the first bit of the note parameter code, instrument parameter code, velocity parameter code, beat parameter code, and controller parameter code is 1, and the remaining bits are 0.

[0040] Intelligent agent: Given a key information vector of a video frame, it outputs corresponding action values ​​via an automatic music-based action value function. The automatic music-based action value function is specifically as follows:

[0041] y t 1 =Leaky_Relu((g t *w1+b1) (i％2) +(g t *w2+b2) ((i+1)％2) )

[0042] y t 2 =sigmoid(g t i *w3+b3)

[0043] A t =[sigmoid(y t 2 *e1 n +b4), sigmoid(y t 2 *e2 1 +b4), sigmoid(y t 2 *e3 1+b4), sigmoid(y t 2 *e4 1 +b4),

[0044] sigmoid(y t 2 *e5 n +b4)]

[0045] Where sigmoid() and Leaky_Relu() are activation functions, g t Let y be a feature vector containing key information of the video frame at time t, where i represents the i-th element of the feature vector. t 1 Let y be the first-level action selection vector at time t, where w1 and w2 represent the weights of the first-level action selection, and b1 and b2 represent the biases of the first-level action selection; t 2 Let w3 and b3 be the second-layer action selection vector at time t, where w3 and b3 represent the weights and biases of the second-layer action selection, respectively; A t Let b4 be the agent's action vector at time t, and e1 be the bias of the third-layer action selection. n e2 1 e3 1 e4 1 e5 n These are the action weights of the agent on the note parameter, instrument parameter, velocity parameter, beat parameter, and controller parameter, respectively, where the superscript 1 and n represent the maximum number of operations the agent can perform on that parameter.

[0046] Action: refers to the modifications made by the agent on the music feature vector set. Note parameter encoding and controller parameter encoding can modify multiple values ​​each time, while instrument parameter encoding, velocity parameter encoding and beat parameter encoding can set at most one one-hot encoding to 1 each time.

[0047] Reward: The reward for each action taken by the agent is described as r. t Let represent the difference between the agent's operation at time t and the original audio feature vector, and the reward function is:

[0048]

[0049] Where, r t s represents the reward value obtained by the agent in the current state. ot Let s represent the audio feature vector at time t in step S203. at Let t represent the audio feature vector selected by the agent at time t, n be the reward calculation length of the agent's continuous operations, i represent the i-th bit of the original audio vector, and j represent the j-th bit of the agent's operation vector;

[0050] After the agent completes each operation, the total reward R for this music playback is calculated using the following formula:

[0051]

[0052] Where R(t) represents the total reward of the current step, k is the total number of actions performed by the agent, and its value depends on the duration of the background music; r t r is the reward obtained by the agent for this operation. t-1 R(0) represents the reward obtained by the agent in the previous operation, and d represents the influence factor of the current operation on the historical reward. When t is 0, R(0) = 0 and r0 = 0.

[0053] Furthermore, step S4 includes:

[0054] Step S401: Input the deep feature set of the video to be set to music into the trained reinforcement learning model, and design an intelligent music sampling algorithm so that the agent selects the best action from the experience set;

[0055] Step S402: Combine the action sequences of the agent to obtain the music feature vector set.

[0056] Furthermore, the specific formula for the intelligent music sampling algorithm in step S401 is as follows:

[0057]

[0058] Where C represents the agent's selection of the optimal action from the experience set, and R... t g represents the total reward for the current step. t Let w be a feature vector containing key information of the video frame at time t. t The weights for the action selection at time t are represented by exp(), which is an exponential function with the natural constant e as the base, max() means to find the maximum value, and select() means to select the target policy with the maximum value.

[0059] The beneficial effects of this invention are:

[0060] (1) This invention uses a large amount of music data from real short videos as training samples, requiring no manual intervention. In contrast, traditional music composition methods based on music theory require composers to have a high level of knowledge of music theory. For users' specific music composition requirements, composers find it difficult to grasp a large amount of music information, resulting in music that is far from the expected result and is not accurate enough. However, this invention proposes an automatic music composition method based on reinforcement learning, which automatically composes music based on massive amounts of short video data, making the generated music more consistent with the video content and more diverse.

[0061] (2) This invention designs a professional encoding method for audio data, performs multi-hot and one-hot encoding on note parameters, instrument parameters, velocity parameters, beat parameters and controller parameters, integrates richer encoding information, can extract and generate deep-level audio feature vectors, make the background music conform to the rules of music theory, and make the background music information more complete and accurate.

[0062] (3) The present invention designs a reasonable reward function and a historical reward value influence factor on the total reward of music generation for each step of the intelligent agent’s operation. The influence factor helps to weaken the negative impact of historical operations on the final short video background music, making the background music more realistic.

[0063] (4) This invention designs a reasonable action value function for reinforcement learning algorithms, enabling the algorithm to select appropriate actions in continuous actions or high-dimensional action spaces, making the agent more fluid in selecting continuous note parameters. Therefore, the action value function of this invention enables the algorithm to run efficiently in model training and actual short video background music, reducing the time cost of short video creation.

[0064] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained from the following description and the foregoing claims. Attached Figure Description

[0065] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0066] Figure 1 This is a schematic diagram of the process of an automatic music addition method for short videos based on reinforcement learning according to the present invention.

[0067] Figure 2 This is a schematic diagram of the audio encoding method of the present invention;

[0068] Figure 3 This is a schematic diagram illustrating the training of the reinforcement learning model used in this invention;

[0069] Figure 4 This is a schematic diagram illustrating the use of a reinforcement learning model to generate background music in this invention.

[0070] Figure 5 This is a schematic diagram illustrating the decoding of the background music feature vector set into audio in this invention. Detailed Implementation

[0071] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be understood that the preferred embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0072] This invention provides a method for automatic music accompaniment in short videos based on reinforcement learning, such as... Figure 1 As shown, it includes the following steps:

[0073] Step S1: Construct an adaptive video encoder to extract video depth features; in this embodiment, step S1 specifically includes:

[0074] Step S101: Perform single-channel sampling on the short video with a sampling precision of 8 bits, a sampling frame rate of 10 fps, and stretch the resolution to 512×512 to obtain a short video image set;

[0075] Step S102: Construct an adaptive video encoder by inputting the sampled short video image set into the encoder and outputting a depth video feature vector set.

[0076] Furthermore, the adaptive video encoder in step S102 specifically comprises:

[0077] H t =ReLU(conv(v t ,P H )+conv(v t-1 Q H ))

[0078] Z t =tanh(conv(v t ,P Z )+H t ⊙conv(v t-1 Q Z ))

[0079] U t =v t-1 ⊙(H t +1)+Z t ⊙H t

[0080] g t =ReLU(v t F)

[0081] Where v is the sampled video frame sequence, v t For the current video frame, v t-1 For the previous video frame, ReLU() and tanh() are activation functions, H t P represents the degree of information retention of the video frames at times t and t-1. H and Q HZ is the video information weight matrix. t P represents the degree of information loss in the video frames at times t and t-1. Z and Q Z U is a weight matrix that has the effect of losing video information weights. t Let g be the feature matrix containing key information of the video frame at time t, ⊙ be the Hadamard product (element-wise multiplication of corresponding positions of two vectors), F be the linear transformation weight matrix, and g be the vector matrix. t Let be the feature vector containing key information of the video frame at time t.

[0082] Step S2: Obtain the MIDI file of the background music for the short video, construct an audio encoder, and extract multiple sets of audio feature vectors from the MIDI format file; in this embodiment, step S2 includes:

[0083] Step S201: Obtain the MIDI file of the background music corresponding to the short video and trim it to the same length as the short video described in step S1;

[0084] Step S202: Extract the note parameters, instrument parameters, velocity parameters, tempo parameters, and controller parameters from the extracted MIDI file;

[0085] Step S203: Construct an audio encoder, input the audio parameters extracted in step S202, and output multiple sets of audio feature vectors. In this embodiment, the audio encoder in step S203 is specifically:

[0086] The note parameters with values ​​ranging from 00 to 7F are multi-hot encoded and added to the feature vector set;

[0087] The instrument parameters with values ​​ranging from 00 to 7F are one-hot encoded and added to the feature vector set;

[0088] The force parameter, which ranges from 00 to 7F, is one-hot encoded and added to the feature vector set;

[0089] The beat parameters, which range from 00 to 7F, are one-hot encoded and added to the feature vector set;

[0090] The controller parameters, which range from 00 to 7F, are multi-hot encoded and added to the feature vector set.

[0091] In this embodiment, as follows Figure 2The notes shown are encoded. The first group of notes, represented as "A4" and "D5" in music, are encoded in this method as follows: the 69th and 74th bits are multi-hot encoded as 1, and the rest as 0; this note has no velocity parameter, and in this encoding method, the velocity parameter is encoded using the default encoding, with the 1st bit being one-hot encoded as 1, and the rest as 0; this note is a quarter note, which in this embodiment is 1 beat, with a tempo of 80 bpm (80 beats per minute), and in this encoding method, the beat parameter is encoded as follows: the 80th bit is one-hot encoded as 1, and the rest as 0;

[0092] The second note is represented as "D5" in music. In this encoding method, the note parameter is encoded as follows: the 74th bit multi-hot is 1, and the rest are 0; the dynamics of this note are "ff", and the dynamics parameter is encoded as follows: the 96th bit one-hot is 1, and the rest are 0; this note is a half note, which is 2 beats in this embodiment, with a tempo of 80 bpm. In this encoding method, the beat parameter is encoded as follows: the 160th bit one-hot is 1, and the rest are 0;

[0093] The third note is represented as "C5" in music. In this encoding method, the note parameter is encoded as follows: the 72nd bit multi-hot is 1, and the rest are 0. There is no velocity parameter. In this encoding method, the velocity parameter is encoded as follows: the 1st bit one-hot is 1, and the rest are 0. This note is an eighth note, which is half a beat in this embodiment, and the tempo is 80. In this encoding method, the beat parameter is encoded as follows: the 40th bit one-hot is 1, and the rest are 0.

[0094] The fourth note is represented as "F4" in the music. In this encoding method, the note parameter is encoded as follows: the 65th bit (multi-hot) is 1, and the rest are 0. The dynamics of this note are "ppp". In this encoding method, the dynamics parameter is encoded as follows: the 20th bit (one-hot) is 1, and the rest are 0. This note is an eighth note, which is half a beat in this embodiment, with a tempo of 80 bpm. In this encoding method, the beat parameter is encoded as follows: the 40th bit (one-hot) is 1, and the rest are 0. In addition, the instrument in this music is an electric piano, and the instrument parameter is encoded as follows: the 5th bit (one-hot) is 1, and the rest are 0. This note does not have controller changes, and the controller parameter is encoded as 0.

[0095] Step S3: Using the video feature and audio feature vector sets as training data, construct and train a short video background music model based on reinforcement learning; in this embodiment, step S3 includes:

[0096] Step S301: Model the process of adding background music to short videos as a specific Markov decision process, defining the state, action, and reward;

[0097] Step S302: Based on a specific Markov decision process, use video features and audio feature vector sets as training data to establish a novel short video background music model based on reinforcement learning;

[0098] Step S303: Train the above reinforcement learning model to obtain a set of short video background music experience.

[0099] Furthermore, in step S302, the short video background music model based on reinforcement learning is established as follows:

[0100] Within the reinforcement learning framework, the automatic music accompaniment system is treated as an intelligent agent, and the process of automatically accompaniing music to short videos is modeled as a Markov decision process model:

[0101] Among them, such as Figure 3 As shown, in this invention, the state, actions, and reward function of the index selection process described in step S301 are as follows:

[0102] State: refers to the current state of the agent, i.e., the current set of audio feature vectors. The initial state is defined as follows: the first bit of the note parameter code, instrument parameter code, velocity parameter code, beat parameter code, and controller parameter code is 1, and the remaining bits are 0.

[0103] Intelligent agent: Given a key information vector of a video frame, it outputs corresponding action values ​​via an automatic music-based action value function. The automatic music-based action value function is specifically as follows:

[0104] y t 1 =Leaky_Relu((g t *w1+b1) (i％2) +(g t *w2+b2) ((i+1)％2) )

[0105] y t 2 =sigmoid(g t i *w3+b3)

[0106] A t =[sigmoid(y t 2 *e1 n +b4), sigmoid(y t 2 *e2 1 +b4), sigmoid(y t 2 *e3 1 +b4), sigmoid(yt 2 *e4 1 +b4),

[0107] sigmoid(y t 2 *e5 n +b4)]

[0108] Where sigmoid() and Leaky_Relu() are activation functions, g t Let y be a feature vector containing key information of the video frame at time t, where i represents the i-th element of the feature vector. t 1 Let y be the first-level action selection vector at time t, where w1 and w2 represent the weights of the first-level action selection, and b1 and b2 represent the biases of the first-level action selection; t 2 Let w3 and b3 be the second-layer action selection vector at time t, where w3 and b3 represent the weights and biases of the second-layer action selection, respectively; A t Let b4 be the agent's action vector at time t, and e1 be the bias of the third-layer action selection. n e2 1 e3 1 e4 1 e5 n These are the action weights of the agent on the note parameter, instrument parameter, velocity parameter, beat parameter, and controller parameter, respectively, where the superscript 1 and n represent the maximum number of operations the agent can perform on that parameter.

[0109] Action: refers to the modifications made by the agent on the music feature vector set. Note parameter encoding and controller parameter encoding can modify multiple values ​​each time, while instrument parameter encoding, velocity parameter encoding and beat parameter encoding can set at most one one-hot encoding to 1 each time.

[0110] Reward: The reward for each action taken by the agent is described as r. t Let represent the difference between the agent's operation at time t and the original audio feature vector, and the reward function is:

[0111]

[0112] Where, r t s represents the reward value obtained by the agent in the current state. ot Let s represent the audio feature vector at time t in step S203. at Let t represent the audio feature vector selected by the agent at time t, n be the reward calculation length of the agent's continuous operations, i represent the i-th bit of the original audio vector, and j represent the j-th bit of the agent's operation vector;

[0113] In this embodiment, the note feature vector of the original audio is [21,23,20,18,20], while the audio feature vector selected by the agent is [22,23,23,17,20]. The reward calculation length n for the agent's continuous operations is set to 2. The reward r1 for the agent's first operation can be calculated as 2; the reward r2 for the agent's second operation is 2 + 0.2 + 0.5 + 4 = 6.7; the reward r3 for the agent's third operation is 4 + 1 + 0.1 + 0.4 = 5.5; the reward r4 for the agent's fourth operation is 0.4 + 0.1 + 0.04 + 2 = 2.54; and the reward r5 for the agent's fifth operation is 2 + 0.2 + 0.1 + 4 = 6.3.

[0114] After the agent completes each operation, the total reward R for this music playback is calculated using the following formula:

[0115]

[0116] Where R(t) represents the total reward of the current step, and k is the total number of operations performed by the agent, the size of which depends on the duration of the background music; t r is the reward obtained by the agent for this operation. t-1 R(0) represents the reward obtained by the agent in the previous operation, and d represents the influence factor of the current operation on the historical reward. When t is 0, R(0) = 0 and r0 = 0.

[0117] In this embodiment, the reward values ​​obtained by the agent after 5 operations are [2, 6.7, 5.5, 2.54, 6.3], and d is set to 0.5. Using this formula, the total reward obtained by the agent for the first operation is R(1) = 0.2, the total reward obtained by the agent for the second operation is R(2) = 0.27, the total reward obtained by the agent for the third operation is R(3) = 0.47, the total reward obtained by the agent for the fourth operation is R(4) = 0.68, and the total reward obtained by the agent for the fifth operation is R(5) = 0.35.

[0118] Step S4: Based on the reinforcement learning music model, design a sampling method to obtain the music feature vector set; such as... Figure 4 As shown, in this embodiment, step S4 includes:

[0119] Step S401: Input the deep feature set of the video to be set to music into the trained reinforcement learning model, and design an intelligent music sampling algorithm so that the agent selects the best action from the experience set;

[0120] Step S402: Combine the action sequences of the agent to obtain the music feature vector set.

[0121] Furthermore, the specific formula for the intelligent music sampling algorithm in step S401 is as follows:

[0122]

[0123] Where C represents the agent's selection of the optimal action from the experience set, and R... t g represents the total reward for the current step. t Let w be a feature vector containing key information of the video frame at time t. t The weights for the action selection at time t are represented by exp(), which is an exponential function with the natural constant e as the base, max() means to find the maximum value, and select() means to select the target policy with the maximum value.

[0124] Step S5: Decode the background music feature vector set into a MIDI format file to complete the automatic background music for the short video;

[0125] In this embodiment, the five musical notes with the feature vector sets [{70:1},{13:1},{80:1},{20:1},{0:1}], [{72:1},{13:1},{0:1},{10:1},{0:1}], [{76:1},{13:1},{0:1},{10:1},{0:1}], [{72:1},{13:1},{0:1},{40:1},{0:1}], and [{75:1},{13:1},{53:1},{40:1},{0:1}] are converted into audio, such as... Figure 5 As shown.

[0126] It should be recognized that embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable storage medium. The method can be implemented using standard programming techniques—including a non-transitory computer-readable storage medium configured with a computer program, wherein such a storage medium causes the computer to operate in a specific and predefined manner—according to the methods and drawings described in the specific embodiments. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if desired, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. Furthermore, for this purpose, the program can run on a programmed application-specific integrated circuit (ASIC).

[0127] Furthermore, the procedures described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by the context. The procedures described herein (or variations and / or combinations thereof) may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that commonly executes on one or more processors. The computer program comprises a plurality of instructions executable by one or more processors.

[0128] Furthermore, the method can be implemented in any suitable type of computing platform, including but not limited to personal computers, minicomputers, mainframes, workstations, networked or distributed computing environments, standalone or integrated computer platforms, or in communication with charged particle tools or other imaging devices, etc. Aspects of the invention can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it is readable by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the processes described herein. Furthermore, the machine-readable code, or portions thereof, can be transmitted via wired or wireless networks. The invention described herein includes these and other different types of non-transitory computer-readable storage media when such media comprises instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. When programmed according to the methods and techniques described in the invention, the invention also includes the computer itself.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for automatically adding background music to short videos based on reinforcement learning, characterized in that: Includes the following steps: Step S1: Construct an adaptive video encoder to extract video depth features; Step S2: Obtain the MIDI file of the background music for the short video, build an audio encoder, and extract multiple sets of audio feature vectors from the MIDI format file; Step S3: Using the video feature and audio feature vector sets as training data, construct and train a short video background music model based on reinforcement learning; Step S3 includes: Step S301: Model the process of adding background music to short videos as a specific Markov decision process, defining the state, action, and reward; Step S302: Based on a specific Markov decision process, use video features and audio feature vector sets as training data to establish a novel short video background music model based on reinforcement learning; Step S303: Train the above reinforcement learning model to obtain a set of short video background music experience; In step S302, the short video background music model based on reinforcement learning is established as follows: Within the reinforcement learning framework, the automatic music accompaniment system is treated as an intelligent agent, and the process of automatically accompaniing music to short videos is modeled as a Markov decision process model: The state, actions, and reward function of the index selection process described in step S301 are as follows: State: refers to the current state of the agent, that is, the current audio feature vector set. The initial state is defined as the first bit of the note parameter code, instrument parameter code, velocity parameter code, beat parameter code, and controller parameter code being 1, and the remaining bits being 0. Intelligent agent: Given a key information vector of a video frame, it outputs corresponding action values ​​via an automatic music-based action value function. The automatic music-based action value function is specifically as follows: Where sigmoid() and Leaky_Relu() are activation functions, g t Let y be a feature vector containing key information of the video frame at time t, where i represents the i-th element of the feature vector. t 1 Let y be the first-level action selection vector at time t, where w1 and w2 represent the weights of the first-level action selection, and b1 and b2 represent the biases of the first-level action selection; t 2 Let w3 and b3 be the second-layer action selection vector at time t, where w3 and b3 represent the weights and biases of the second-layer action selection, respectively; A t Let b4 be the agent's action vector at time t, and e1 be the bias of the third-layer action selection. n e2 1 e3 1 e4 1 e5 n These are the action weights of the agent on the note parameter, instrument parameter, velocity parameter, beat parameter, and controller parameter, respectively, where the superscript 1 and n represent the maximum number of operations the agent can perform on that parameter. Action: refers to the modifications made by the agent on the music feature vector set. Note parameter encoding and controller parameter encoding can modify multiple values ​​each time, while instrument parameter encoding, velocity parameter encoding and beat parameter encoding can set at most one one-hot encoding to 1 each time. Reward: The reward for each action taken by the agent is described as r. t Let represent the difference between the agent's operation at time t and the original audio feature vector, and the reward function is: Where, r t s represents the reward value obtained by the agent in the current state. ot Let s represent the audio feature vector at time t in step S203. at Let t represent the audio feature vector selected by the agent at time t, n be the reward calculation length of the agent's continuous operations, i represent the i-th bit of the original audio vector, and j represent the j-th bit of the agent's operation vector; After the agent completes each operation, the total reward R for this music playback is calculated using the following formula: Where R(t) represents the total reward of the current step, k is the total number of actions performed by the agent, and its value depends on the duration of the background music; r t r is the reward obtained by the agent for this operation. t-1 R(0) represents the reward obtained by the agent in the previous operation, and d represents the influence factor of the current operation on the historical reward. When t is 0, R(0) = 0 and r0 = 0. Step S4: Based on the reinforcement learning music model, design a sampling method to obtain the music feature vector set; Step S5: Decode the background music feature vector set into a MIDI format file to complete the automatic background music for the short video.

2. The method for automatically adding background music to short videos based on reinforcement learning according to claim 1, characterized in that: Step S1 specifically includes: Step S101: Perform single-channel sampling on the short video with a sampling precision of 8 bits, a sampling frame rate of 10 fps, and stretch the resolution to 512×512 to obtain a short video image set; Step S102: Construct an adaptive video encoder by inputting the sampled short video image set into the encoder and outputting a depth video feature vector set.

3. The method for automatic background music selection in short videos based on reinforcement learning according to claim 2, characterized in that: The adaptive video encoder in step S102 is specifically as follows: H t = ReLU(conv(v t ,P H )+conv(v t-1 , Q H )) Z t =tanh(conv(v t , P Z )+H t ⊙conv(v t-1 , Q Z )) U t = v t-1 ⊙(H t +1)+ Z t ⊙H t g t = ReLU(v t F) Where v is the sampled video frame sequence, v t For the current video frame, v t-1 For the previous video frame, ReLU() and tanh() are activation functions, H t P represents the degree of information retention of the video frames at times t and t-1. H and Q H Z is the video information weight matrix. t P represents the degree of information loss in the video frames at times t and t-1. Z and Q Z U is a weight matrix that has the effect of losing video information weights. t Let g be the feature matrix containing key information of the video frame at time t, ⊙ be the Hadamard product (element-wise multiplication of corresponding positions of two vectors), F be the linear transformation weight matrix, and g be the feature matrix containing key information of the video frame at time t. t Let be the feature vector containing key information of the video frame at time t.

4. The method for automatic background music selection in short videos based on reinforcement learning according to claim 1, characterized in that: Step S2 includes: Step S201: Obtain the MIDI file of the background music corresponding to the short video and trim it to the same length as the short video described in step S1; Step S202: Extract the note parameters, instrument parameters, velocity parameters, tempo parameters, and controller parameters from the extracted MIDI file; Step S203: Construct an audio encoder, input the audio parameters extracted in step S202, and output multiple sets of audio feature vectors.

5. The method for automatic background music selection in short videos based on reinforcement learning according to claim 4, characterized in that: The audio encoder in step S203 is specifically: The note parameters with values ​​ranging from 00 to 7F are multi-hot encoded and added to the feature vector set; The instrument parameters with values ​​ranging from 00 to 7F are one-hot encoded and added to the feature vector set; The force parameter, which has a value range of 00 to 7F, is one-hot encoded and added to the feature vector set; The beat parameters, which range from 00 to 7F, are one-hot encoded and added to the feature vector set; The controller parameters, which range from 00 to 7F, are multi-hot encoded and added to the feature vector set.

6. The method for automatic background music selection in short videos based on reinforcement learning according to claim 1, characterized in that: Step S4 includes: Step S401: Input the deep feature set of the video to be set to music into the trained reinforcement learning model, and design an intelligent music sampling algorithm so that the agent selects the best action from the experience set; Step S402: Combine the action sequences of the agent to obtain the music feature vector set.

7. The method for automatic background music selection in short videos based on reinforcement learning according to claim 6, characterized in that: The specific formula for the intelligent music sampling algorithm in step S401 is as follows: Where C represents the agent's selection of the optimal action from the experience set, and R... t g represents the total reward for the current step. t Let w be a feature vector containing key information of the video frame at time t. t The weights for the action selection at time t are represented by exp(), which is an exponential function with the natural constant e as the base, max() means to find the maximum value, and select() means to select the target policy with the maximum value.

8. A short video automatic music accompaniment system based on reinforcement learning, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, it implements the method as described in any one of claims 1-7.

Images