AI-based video and music card point matching method and medium
Through the AI-based video and music point matching method, the spectrum characteristics and motion trajectory model are used to achieve efficient matching of video and music, solving the problems of high cost and low efficiency in the existing technology.
Patent Information
- Application Number
- CN202410549389.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-06
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-05-06
AI Technical Summary
The existing video editing and music matching process is costly and inefficient, requiring a lot of manual operation and expertise.
Using AI-based video and music card point matching method, by obtaining target video and audio, identifying the audio spectrum, extracting spectrum features, using a pre-trained drum point recognition model for drum point recognition, establishing a motion trajectory model, extracting action features, and calculating the difference value , match keyframes and drum time stamps to achieve video and audio card matching.
It improves the efficiency and accuracy of point matching between video and music, reduces costs, reduces manual operations, and improves overall efficiency.
Smart Images

Figure CN118381979B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an AI-based video and music card point matching method and medium. Background Art
[0002] With the rapid development of information technology and artificial intelligence technology, the fields of video editing and music creation are undergoing unprecedented changes. Traditional video editing methods mostly rely on manual operations, which are not only time-consuming and labor-intensive, but also require operators to have certain professional skills. At the same time, matching music with video is often a complicated process, requiring producers to manually adjust the rhythm of audio and video to achieve synchronization between the two, which is costly and inefficient. Summary of the invention
[0003] The present invention provides an AI-based video and music card point matching method and medium, the main purpose of which is to solve the problems of high cost and low efficiency of existing card point video editing.
[0004] To achieve the above purpose, the present invention provides an AI-based video and music card point matching method, comprising:
[0005] Obtain the target video and audio that need to be matched with the card points;
[0006] Identify the frequency spectrum of the target audio and extract frequency spectrum features of the frequency spectrum;
[0007] Using a pre-trained drum beat recognition model to perform drum beat recognition on the target audio according to the frequency spectrum features, and obtaining a drum beat timestamp list;
[0008] Establishing a motion trajectory model according to the target video;
[0009] Extracting features from each action frame of the motion trajectory model to obtain an action feature set;
[0010] Calculate the difference value between adjacent action features in the action feature set, wherein the difference value is calculated using the following formula:
[0011]
[0012] Wherein, C is the covariance matrix of the action feature set, N is the number of action features in the action feature set, and F k represents the kth action feature in the action feature set, D is the difference value, ΔF i The difference between the feature vectors of the adjacent action features in the i-th dimension, ΔF j represents the difference between the feature vectors of the adjacent action features in the jth dimension, n is the dimension of the vector, Cii represents the element of the i-th row and i-th column of the covariance matrix C of the action feature set, C ij The element in the i-th row and j-th column of the covariance matrix C of the action feature set is represented;
[0013] Extracting the next action feature from all adjacent action features whose difference values are greater than or equal to a preset difference value threshold to obtain a key action feature set;
[0014] Perform key frame matching on the target video according to the key action feature set to obtain a key frame timestamp list;
[0015] The target video and the target audio are matched at the card points according to the drum timestamp list and the key frame timestamp list to obtain the card point video.
[0016] Optionally, the identifying the spectrum of the target audio and extracting spectrum features of the spectrum includes:
[0017] The target audio is subjected to Fourier transform using the following formula to obtain frequency domain features:
[0018]
[0019] Wherein, F represents the frequency domain feature, x n represents the nth sampling point in the target audio, N represents the total number of sampling points of the target audio, e is the base of the natural logarithm, j is the imaginary unit, and j 2 = -1;
[0020] Draw a spectrum diagram according to the frequency domain characteristics;
[0021] Extracting spectrum information from the spectrum graph to obtain a spectrum;
[0022] extracting frequency components of the spectrum;
[0023] The frequency components are normalized to obtain frequency spectrum features.
[0024] Optionally, the using a pre-trained drum beat recognition model to perform drum beat recognition on the target audio according to the frequency spectrum features to obtain a drum beat timestamp list includes:
[0025] Performing denoising on the target audio to obtain a denoised audio frequency;
[0026] Partition the denoised audio according to a preset number of partitions to obtain partitioned audio;
[0027] Using the drum beat recognition model to predict the drum beat probability of each partition in the partition audio according to the frequency spectrum features, to obtain the drum beat probability of each partition in the partition audio;
[0028] Using the drum beat recognition model to set a drum beat probability threshold according to a predetermined card point task requirement;
[0029] The drum beat recognition model is used to mark the partitioned audio with drum beats according to the drum beat probability threshold to obtain a drum beat timestamp list.
[0030] Optionally, performing denoising on the target audio to obtain denoised audio includes:
[0031] Performing noise type recognition on the target audio to obtain the noise type;
[0032] Acquire a noise sample of the target audio according to the noise type;
[0033] Calculating the power spectrum of the target audio to obtain an audio power spectrum, and calculating the power spectrum of the noise sample to obtain a noise power spectrum;
[0034] Subtracting the noise power spectrum from the audio power spectrum to obtain a denoised frequency domain signal;
[0035] The denoised frequency domain signal is inversely Fourier transformed using the following formula to obtain the denoised frequency domain signal:
[0036]
[0037] Among them, x n represents the nth sampling point in the target audio, N represents the total number of sampling points of the target audio, F represents the frequency domain feature, e is the base of the natural logarithm, j is the imaginary unit, and j satisfies 2 =-1.
[0038] Optionally, calculating the power spectrum of the target audio to obtain the audio power spectrum includes:
[0039] The spectrum of the target audio is processed by framing to obtain a spectrum frame set
[0040] The frame energy of each frame in the spectrum frame set is calculated using the following formula to obtain the spectrum energy:
[0041]
[0042] Where, E is the energy spectrum, L is the number of samples in the spectrum frame set, x s represents the sth sample in the spectrum frame set, w is the window function, w srepresents the value of the sth sample in the spectrum frame set in the window function;
[0043] A power spectrum is generated according to the spectrum energy to obtain an audio power spectrum.
[0044] Optionally, using the drum beat recognition model to predict the drum beat probability of each partition in the partition audio according to the frequency spectrum feature to obtain the drum beat probability of each partition in the partition audio includes:
[0045] Inputting the frequency spectrum feature into the hidden layer of the drum beat recognition model;
[0046] Partitioning the spectrum feature according to a preset number of partitions to obtain a set of partitioned spectrum features;
[0047] The hidden layer is used to call the following formula to calculate the output of each neuron in the drum beat recognition model according to the partition spectrum feature set to obtain the hidden layer output of each partition of the partition audio:
[0048]
[0049] Wherein, Z is the hidden layer output of any partition in the partitioned audio, n is the number of neurons in the hidden layer, w is the weight parameter of the drum beat recognition model, x i is the ith partition spectrum in the partition spectrum feature set, and b is the bias term of the drum beat recognition model;
[0050] Using an activation function, the hidden layer output of each partition of the partitioned audio is converted into an activation value to obtain the activation value of each partition of the partitioned audio;
[0051] The output layer of the drum beat recognition model is used to call the following formula to calculate the drum beat probability of each partition in the partition audio according to the activation value of each partition of the partition audio:
[0052]
[0053] Wherein, P is the probability of a drum beat in any partition of the partitioned audio, n is the number of neurons in the hidden layer, w is the weight parameter, and a i is the activation value of the i-th partition in the partitioned audio, b is the bias term, and e is the base of the natural logarithm.
[0054] Optionally, establishing a motion trajectory model according to the target video includes:
[0055] Using a computer vision algorithm to capture key points of the target video to obtain key points;
[0056] Creating an action change sequence according to the changes of the key points over time;
[0057] Establishing a skeleton model according to the key points;
[0058] The skeleton model is driven by the action change sequence to obtain a motion trajectory model.
[0059] Optionally, performing key frame matching on the target video according to the key action feature set to obtain a key frame timestamp list includes:
[0060] Matching all corresponding action frames in the motion trajectory model according to all key action features in the key action feature set to obtain a key action frame set;
[0061] Extracting time information of each key action in the key action set to obtain a time information set;
[0062] According to each time information in the time information set, a frame picture of a corresponding time in the target video is intercepted to obtain a key frame timestamp list.
[0063] Optionally, performing card point matching on the target video and the target audio according to the timestamp list and the key frame timestamp list to obtain the card point video includes:
[0064] Get the number of drum beats in the drum beat timestamp list;
[0065] Adaptively adjusting the key frame timestamp list according to the number of drum beats to obtain an adaptive key frame list;
[0066] Aligning the key frames in the target video with the drum beats in the drum beat timestamp list according to the adaptive key frame list to obtain an aligned video;
[0067] The aligned video is fused with the target audio to obtain a card point video.
[0068] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned AI-based video and music card point matching method.
[0069] The embodiment of the present invention obtains the target video and target audio that need to be matched at the card point, identifies the spectrum of the target audio and extracts the spectrum features of the spectrum, thereby improving the efficiency of subsequent drum point recognition, using the pre-trained drum point recognition model to perform drum point recognition on the target audio according to the spectrum features, obtaining a drum point timestamp list, and improving the accuracy of subsequent card point matching, establishing a motion trajectory model according to the target video, extracting features for each action frame of the motion trajectory model, and calculating the difference value between adjacent action features, extracting the next action feature of all adjacent action features whose difference values are greater than or equal to a preset difference value threshold, and obtaining a key action feature set, thereby improving the accuracy and efficiency of key action feature recognition, performing key frame matching on the target video according to the key action feature set, obtaining a key frame timestamp list, and improving the accuracy and efficiency of card point matching, performing card point matching on the target video and the target audio according to the drum point timestamp list and the key frame timestamp list, and obtaining a card point video. Therefore, the AI-based video and music card point matching method and medium proposed in the present invention can solve the problem of high cost and low efficiency of existing card point video editing. BRIEF DESCRIPTION OF THE DRAWINGS
[0070] Figure 1 A schematic diagram of a flow chart of a method for matching video and music card points based on AI provided in one embodiment of the present invention;
[0071] Figure 2 A schematic diagram of a process for performing drum beat recognition on target audio provided by an embodiment of the present invention;
[0072] Figure 3 A schematic diagram of a process for performing key frame matching on a target video provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0073] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0074] The embodiment of the present application provides a method for matching video and music card points based on AI. The execution subject of the method for matching video and music card points based on AI includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, the method for matching video and music card points based on AI can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.
[0075] Reference Figure 1 FIG. 1 is a flow chart of a method for matching video and music points based on AI according to an embodiment of the present invention. In this embodiment, the method for matching video and music points based on AI includes:
[0076] S1. Obtain the target video and target audio that need to be matched with the card points.
[0077] In the embodiment of the present invention, the target video includes but is not limited to a dance video and a transition video.
[0078] In detail, the transition video refers to a video that transitions and smoothly switches between two different scenes or shots through certain techniques or effects.
[0079] In the embodiment of the present invention, the target audio is audio selected by a user to be matched with the target video.
[0080] In detail, the target video and the target audio that need to be matched with the card points are obtained by receiving the video file and the audio file uploaded by the user to obtain the target video and the target audio.
[0081] In the embodiment of the present invention, the card point matching refers to making specific rhythm points in the target video and the target audio coincide with each other to achieve an audio-visual synchronization effect.
[0082] In the embodiment of the present invention, by acquiring the target video and the target audio that need to be matched at the card point, it is convenient to subsequently perform spectrum recognition on the target audio and to subsequently establish a trajectory model according to the target video, thereby improving the efficiency of matching the card points of the target video and the target audio.
[0083] S2. Identify the frequency spectrum of the target audio and extract frequency spectrum features of the frequency spectrum.
[0084] In the embodiment of the present invention, the spectrum is one of the basic properties of sound, describing the information of sound fluctuation. In the spectrum, the frequency can be represented by the height of the vertical bar graph, and the higher the height, the stronger the sound component corresponding to the frequency.
[0085] In the embodiment of the present invention, the identifying the spectrum of the target audio and extracting the spectrum features of the spectrum includes:
[0086] The target audio is subjected to Fourier transform using the following formula to obtain frequency domain features:
[0087]
[0088] Wherein, F represents the frequency domain feature, x n represents the nth sampling point in the target audio, N represents the total number of sampling points of the target audio, e is the base of the natural logarithm, j is the imaginary unit, and j 2 = -1;
[0089] Draw a spectrum diagram according to the frequency domain characteristics;
[0090] Extracting spectrum information from the spectrum graph to obtain a spectrum;
[0091] extracting frequency components of the spectrum;
[0092] The frequency components are normalized to obtain frequency spectrum features.
[0093] In detail, the Fourier transform is a widely used spectrum analysis method, which can convert a one-dimensional time domain signal into a frequency domain signal.
[0094] In detail, the spectrogram is a graph showing the distribution of the audio spectrum, in which the horizontal axis represents the frequency and the vertical axis represents the amplitude or energy of the signal. Each frequency component corresponds to a specific amplitude value on the spectrogram, thus forming an amplitude-frequency distribution graph of the audio signal.
[0095] In detail, the spectrum information includes frequency components of various frequency types in the spectrum graph, the intensity or size of each frequency component, the distribution of the frequency components, the minimum frequency interval of the spectrum graph, the maximum and minimum amplitudes of the audio signal in the spectrum graph, and the frequency range of the spectrum graph.
[0096] In detail, the extracting of the frequency components of the spectrum is performed by checking the peaks or specific frequency ranges in the spectrum, thereby extracting the frequency components in the spectrum. The frequency components represent specific tones or harmonic structures in the target audio.
[0097] In detail, the normalization process is a common technique in data preprocessing, the purpose of which is to adjust the numerical range of data to a specific interval, usually between 0 and 1, to facilitate comparison and weighting between different features, and also to accelerate the convergence speed of the algorithm.
[0098] In the embodiment of the present invention, by performing Fourier transform on the target audio to obtain frequency domain features, the accuracy of the spectrum graph is improved, thereby improving the accuracy of the spectrum.
[0099] In the embodiment of the present invention, by normalizing the frequency components, the comparability of the frequency components is improved, and the efficiency of subsequent drum beat recognition is improved.
[0100] In the embodiment of the present invention, by identifying the spectrum of the target audio, it can help understand the energy distribution of the audio signal at different frequencies, so as to perform a deeper audio analysis. By extracting the spectrum features of the spectrum, the data representation of the target audio is simplified, the interpretability and comparability of the target audio are improved, and the analysis efficiency of the target audio is improved.
[0101] S3. Using a pre-trained drum beat recognition model to perform drum beat recognition on the target audio according to the frequency spectrum features, to obtain a drum beat timestamp list.
[0102] In the embodiment of the present invention, before the drum beat recognition model trained in advance is used to perform drum beat recognition on the target audio according to the frequency spectrum features to obtain a drum beat timestamp list, the method further includes:
[0103] Acquire audio data with drum mark, identify the frequency spectrum of the audio data, and obtain a training frequency spectrum;
[0104] Extracting training spectrum features of the training spectrum;
[0105] Partitioning the training spectrum according to a preset number of partitions to obtain a training partitioned spectrum;
[0106] Using a neural network model to perform drum beat recognition on each partition of the partition spectrum according to the training spectrum features, to obtain a drum beat recognition result;
[0107] Comparing the drum beat recognition result with the standard drum beat annotation preset in the audio data to obtain a drum beat recognition success rate;
[0108] If the drum beat recognition success rate is less than a preset success rate threshold, the weight parameters and bias items of the neural network model are adjusted by using a back propagation algorithm, and then the step of performing drum beat recognition on each partition of the partition spectrum according to the training spectrum features by using the neural network model is re-executed to obtain a drum beat recognition result;
[0109] If the drum beat recognition success rate is greater than or equal to a preset success rate threshold, it is confirmed that the model training is completed and a drum beat recognition model is obtained.
[0110] In the embodiment of the present invention, the use of a neural network model to perform drum beat recognition on each partition of the partitioned spectrum according to the training spectrum features to obtain a drum beat recognition result is to obtain the drum beat recognition result by using forward propagation of the neural network model.
[0111] In detail, the forward propagation is the forward calculation process in the neural network, which is used to calculate the output value of each neuron layer by layer starting from the input data until the final predicted value is obtained. During the forward propagation process, the input data is linearly transformed through the weights and biases of each layer, and nonlinearly transformed through the activation function, and then output to the next layer until it reaches the output layer.
[0112] In the embodiment of the present invention, the back propagation algorithm is currently the most commonly used and most effective algorithm for training neural networks. The training set data is input into the input layer of the ANN, through the hidden layer, and finally reaches the output layer and outputs the result. This is the forward propagation process of the ANN; since there is an error between the output result of the ANN and the actual result, the error between the estimated value and the actual value is calculated, and the error is back-propagated from the output layer to the hidden layer until it is propagated to the input layer; in the process of back propagation, the values of various parameters are adjusted according to the error; the above process is continuously iterated until convergence.
[0113] Specifically, the working principle of the back propagation algorithm is based on the idea of error feedback. It first predicts the result based on the input information, and then takes the difference between the predicted result and the actual result. This difference is the error rate, which is used to indicate the accuracy of network training. Then, through the back propagation algorithm, the weights and biases of the neural network are adjusted according to the error rate, so that the network can fit the model more accurately in each training.
[0114] In the embodiment of the present invention, the weight parameters are the values connecting different neurons (or nodes) in the neural network, which determine how the network responds to a specific input pattern. The weight parameters are a key component in the learning process of the neural network. By adjusting these parameters, the network can gradually learn and recognize the inherent laws and patterns of the input data.
[0115] In detail, at the initial stage of a neural network, the weight parameters are usually randomly initialized. This means that each weight is assigned a random initial value before starting training. This initialization process is crucial to the final performance of the network, because different initialization methods may cause the network to converge to different local optimal solutions.
[0116] In the embodiment of the present invention, the bias term is another key component of the neuron, which is used to adjust the output of the neuron. Unlike the weight parameter, the bias term does not depend on the input data, but provides a fixed offset for each neuron.
[0117] The existence of the bias term allows the activation function of the neuron to still have a non-zero output when the input is zero or very small. This increases the flexibility of the neural network, enabling it to learn and adapt to more complex patterns.
[0118] In the embodiment of the present invention, by using the back propagation algorithm to adjust the weight parameters and bias items of the neural network model, the convergence speed of the drum beat recognition model is improved, and the accuracy of the drum beat recognition model in recognizing drum beats is improved.
[0119] In the embodiment of the present invention, refer to Figure 2 As shown, a schematic diagram of a process of performing drum beat recognition on a target audio is provided in an embodiment of the present invention, wherein the drum beat recognition model trained in advance is used to perform drum beat recognition on the target audio according to the frequency spectrum features to obtain a drum beat timestamp list, including:
[0120] S21, performing denoising processing on the target audio to obtain a denoised audio frequency;
[0121] S22, partitioning the denoised audio according to a preset number of partitions to obtain partitioned audio;
[0122] S23, using the drum beat recognition model to predict the drum beat probability of each partition in the partition audio according to the frequency spectrum feature, to obtain the drum beat probability of each partition in the partition audio;
[0123] S24, using the drum beat recognition model to set a drum beat probability threshold according to a predetermined card point task requirement;
[0124] S25. Mark the partitioned audio with drum beats according to the drum beat probability threshold using the drum beat recognition model to obtain a drum beat timestamp list.
[0125] In the embodiment of the present invention, the denoising process is a method of reducing or eliminating noise components in an audio signal by using signal processing technology to improve the quality and clarity of the audio.
[0126] In detail, the denoising process is performed on the target audio to obtain the denoised audio, including:
[0127] Performing noise type recognition on the target audio to obtain the noise type;
[0128] Acquire a noise sample of the target audio according to the noise type;
[0129] Calculating the power spectrum of the target audio to obtain an audio power spectrum, and calculating the power spectrum of the noise sample to obtain a noise power spectrum;
[0130] Subtracting the noise power spectrum from the audio power spectrum to obtain a denoised frequency domain signal;
[0131] The denoised frequency domain signal is inversely Fourier transformed using the following formula to obtain the denoised frequency domain signal:
[0132]
[0133] Among them, x n represents the nth sampling point in the target audio, N represents the total number of sampling points of the target audio, F represents the frequency domain feature, e is the base of the natural logarithm, j is the imaginary unit, and j satisfies 2 =-1.
[0134] In detail, the noise types can be classified into mechanical noise, fluid dynamic noise and electromagnetic noise according to the sound source; can be divided into continuous noise and intermittent noise according to the noise duration; can be divided into steady-state noise and fluctuating noise according to the noise fluctuation state; can be divided into low-frequency noise, medium-frequency noise and high-frequency noise according to the noise frequency.
[0135] In detail, the acquiring of the noise sample of the target audio according to the noise type is performed by identifying the audio containing the noise type in the target audio to obtain the noise sample, wherein the noise sample is audio containing only noise.
[0136] In detail, the calculating the power spectrum of the target audio to obtain the audio power spectrum includes:
[0137] The spectrum of the target audio is processed by framing to obtain a spectrum frame set
[0138] The frame energy of each frame in the spectrum frame set is calculated using the following formula to obtain the spectrum energy:
[0139]
[0140] Where, E is the energy spectrum, L is the number of samples in the spectrum frame set, x s represents the sth sample in the spectrum frame set, w is the window function, ws represents the value of the sth sample in the spectrum frame set in the window function;
[0141] A power spectrum is generated according to the spectrum energy to obtain an audio power spectrum.
[0142] In detail, the frame processing is to divide the continuous spectrum signal into multiple short frames to reduce the influence of frame boundaries. The length of each frame is usually between 20 milliseconds and 30 milliseconds.
[0143] In detail, the step of calculating the power spectrum of the noise sample to obtain the noise power spectrum is the same as the step of obtaining the audio power spectrum, which will not be repeated here.
[0144] In the embodiment of the present invention, the denoised audio is partitioned according to a preset number of partitions to obtain partitioned audio, which is to partition the denoised audio according to a certain time interval, and the number of partitions is set depending on the level of detail of the audio analysis and the complexity of the audio. The more partitions there are, the more detailed the audio analysis is. On the other hand, the more complex the audio is, the more partitions are needed, which needs to be set according to actual business requirements.
[0145] In the embodiment of the present invention, the step of using the drum beat recognition model to predict the drum beat probability of each partition in the partition audio according to the frequency spectrum feature to obtain the drum beat probability of each partition in the partition audio includes:
[0146] Inputting the frequency spectrum feature into the hidden layer of the drum beat recognition model;
[0147] Partitioning the spectrum feature according to a preset number of partitions to obtain a set of partitioned spectrum features;
[0148] The hidden layer is used to call the following formula to calculate the output of each neuron in the drum beat recognition model according to the partition spectrum feature set to obtain the hidden layer output of each partition of the partition audio:
[0149]
[0150] Wherein, Z is the hidden layer output of any partition in the partitioned audio, n is the number of neurons in the hidden layer, w is the weight parameter, x i is the i-th partition spectrum in the partition spectrum feature set, and b is the bias term;
[0151] Using an activation function, the hidden layer output of each partition of the partitioned audio is converted into an activation value to obtain the activation value of each partition of the partitioned audio;
[0152] The output layer of the drum beat recognition model is used to call the following formula to calculate the drum beat probability of each partition in the partition audio according to the activation value of each partition of the partition audio:
[0153]
[0154] Wherein, P is the probability of a drum beat in any partition of the partitioned audio, n is the number of neurons in the hidden layer, w is the weight parameter, and a i is the activation value of the i-th partition in the partitioned audio, b is the bias term, and e is the base of the natural logarithm.
[0155] In the embodiment of the present invention, the activation function is a function used to introduce nonlinear factors into the neural network model. Its main purpose is to map the input of the neuron to the output end so that the network can learn and approximate complex nonlinear functions. In a neural network, each neuron receives input from the neurons in the previous layer, and performs nonlinear transformation on these inputs through the activation function, and then passes the results to the next layer. Common activation functions include ReLU, sigmoid, and tanh.
[0156] In the embodiment of the present invention, the predetermined checkpoint task requirement is set according to the user's requirement on the number of checkpoints.
[0157] Specifically, the drum beat probability threshold is set according to the predetermined card point task requirement, which means that the more card points the user requires for the target video and the target audio, the lower the drum beat probability threshold is set, so that more drum beats are screened out.
[0158] In the embodiment of the present invention, by performing denoising processing on the target audio, a denoised audio is obtained, thereby improving the clarity of the audio and the accuracy of subsequent drum beat recognition.
[0159] In the embodiment of the present invention, the denoised audio is partitioned according to a preset number of partitions to obtain partitioned audio, thereby improving the accuracy and efficiency of subsequent drum beat probability prediction.
[0160] In the embodiment of the present invention, the noise type is obtained by performing noise type recognition on the target audio, thereby improving the accuracy of obtaining noise samples.
[0161] In the embodiment of the present invention, the power spectrum of the target audio is calculated to obtain the audio power spectrum, and the power spectrum of the noise sample is calculated to obtain the noise power spectrum, thereby improving the efficiency and accuracy of the denoising process.
[0162] In the embodiment of the present invention, the frequency spectrum of the target audio is subjected to frame processing to obtain a frequency spectrum frame set, thereby improving the accuracy of the audio power spectrum.
[0163] In the embodiment of the present invention, the spectral features are partitioned according to a preset number of partitions to obtain a set of partitioned spectral features, thereby improving the accuracy of the hidden layer output of each partition of the partitioned audio.
[0164] In an embodiment of the present invention, by using a pre-trained drum beat recognition model to perform drum beat recognition on the target audio according to the spectral features to obtain a drum beat timestamp list, the accuracy and efficiency of drum beat recognition can be improved, and the drum beat recognition model can be reused, reducing the cost of drum beat recognition.
[0165] S4. Establishing a motion trajectory model according to the target video.
[0166] In the embodiment of the present invention, the motion trajectory model is a simulation model for describing the changes of the character's action in the target video in space over time.
[0167] In the embodiment of the present invention, the step of establishing a motion trajectory model according to the target video includes:
[0168] Using a computer vision algorithm to capture key points of the target video to obtain key points;
[0169] Creating an action change sequence according to the changes of the key points over time;
[0170] Establishing a skeleton model according to the key points;
[0171] The skeleton model is driven by the action change sequence to obtain a motion trajectory model.
[0172] In detail, the computer vision algorithm is a series of technologies and methods in the field of artificial intelligence used to enable computers to understand and interpret image and video data. These algorithms imitate the working method of the human visual system, enabling computers to recognize and process information from visual input.
[0173] In an embodiment of the present invention, creating an action change sequence based on the changes of the key points over time is to continuously track the key points in the target video, identify the change amount of the key points, associate the change amount with time, and obtain an action change sequence containing information about the changes of the key points over time.
[0174] In detail, establishing a skeleton model according to the key points refers to establishing a skeleton model that can replicate the actions of the character in the target video according to the key points such as joints, limbs, torso, etc. representing the character in the target video.
[0175] In detail, using the action change sequence to drive the skeleton model to obtain the motion trajectory model means controlling the skeleton model to make corresponding continuous actions in a time sequence according to the change amount of the key points in the action change sequence, so that the skeleton model can make the same actions as the characters in the target video, thereby obtaining the motion trajectory model.
[0176] In an embodiment of the present invention, by establishing a motion trajectory model based on the target video, the action changes of the characters in the target video can be displayed more clearly, which facilitates the subsequent identification of the key frames of the target video based on the motion trajectory model, thereby improving the efficiency and accuracy of subsequent key frame identification.
[0177] S5. Extract features from each action frame of the motion trajectory model to obtain an action feature set.
[0178] In the embodiment of the present invention, the feature extraction is performed on each action frame of the motion trajectory model to obtain a set of action features, and then the action frames of the motion trajectory model are captured to obtain frame images, and the feature extraction is performed on the frame images to obtain action features.
[0179] In detail, image feature extraction algorithms such as edge detection algorithms may be used to extract features from the frame images.
[0180] In detail, the edge detection algorithm is a basic technology in computer vision, which is used to identify edges from images, that is, places where the brightness changes dramatically in the image, so as to extract the features of the image. Common edge detection algorithms include Sobel operator, Prewitt operator, Canny edge detection and LoG (Laplacian of Gaussian).
[0181] In the embodiment of the present invention, by extracting features from each action frame of the motion trajectory model, an action feature set is obtained, thereby improving the comparability of each action frame and improving the efficiency of subsequent identification of key action features.
[0182] S6. Calculate the difference values between adjacent action features in the action feature set.
[0183] In the embodiment of the present invention, before calculating the difference value between adjacent action features in the action feature set, all action features in the action feature set need to be converted into a vector form.
[0184] In detail, the calculating the difference value between adjacent action features in the action feature set includes:
[0185] The difference value is calculated using the following formula:
[0186]
[0187] Wherein, C is the covariance matrix of the action feature set, N is the number of action features in the action feature set, and F k represents the kth action feature in the action feature set, D is the difference value, ΔF i The difference between the feature vectors of the adjacent action features in the i-th dimension, ΔF j represents the difference between the feature vectors of the adjacent action features in the jth dimension, n is the dimension of the vector, C ii represents the element of the i-th row and i-th column of the covariance matrix C of the action feature set, C ij The element in the i-th row and j-th column of the covariance matrix C of the action feature set is represented;
[0188] In the embodiment of the present invention, by calculating the difference values between adjacent action features in the action feature set, a reference threshold is provided for subsequent key action feature recognition, thereby improving the efficiency and accuracy of subsequent key action feature recognition.
[0189] S7. Extract the next action feature from all adjacent action features whose difference values are greater than or equal to a preset difference value threshold, to obtain a key action feature set.
[0190] In the embodiment of the present invention, the preset difference value threshold needs to be set according to the card point task requirements.
[0191] In detail, the less the number of stuck points between the target video and the target audio is required by the user, the higher the difference value threshold is set, thereby reducing the number of action features that meet the requirement.
[0192] In detail, the method of extracting the latter action feature among all adjacent action features whose difference values are greater than or equal to a preset difference value threshold is because in a pair of adjacent action features, if their difference values are greater than the difference value threshold, it means that the action change amplitude of the latter action feature is large, and it is the key action feature required for the task.
[0193] In an embodiment of the present invention, the latter action feature among all adjacent action features whose difference values are greater than or equal to a preset difference value threshold is extracted to obtain a key action feature set, which can effectively identify the action features with a large amplitude of action change, improve the accuracy of the obtained key action feature set, and improve the efficiency of subsequent movement key frame matching.
[0194] S8. Perform key frame matching on the target video according to the key action feature set to obtain a key frame timestamp list.
[0195] In the embodiment of the present invention, refer to Figure 3 As shown, a schematic diagram of a process of performing key frame matching on a target video is provided in an embodiment of the present invention, wherein the key frame matching is performed on the target video according to the key action feature set to obtain a key frame timestamp list, including:
[0196] S31, matching all corresponding action frames in the motion trajectory model according to all key action features in the key action feature set to obtain a key action frame set;
[0197] S32, extracting the time information of each key action in the key action set to obtain a time information set;
[0198] S33. According to each time information in the time information set, capture a frame image of a corresponding time in the target video to obtain a key frame timestamp list.
[0199] In the embodiment of the present invention, since the key action feature set is obtained by extracting features from each action frame of the motion trajectory model, all corresponding action frames in the motion trajectory model can be matched according to all key action features in the key action feature set to obtain a key action frame set.
[0200] In the embodiment of the present invention, a key frame timestamp list is obtained by intercepting a frame image of a corresponding time in the target video according to each time information in the time information set, thereby improving the accuracy of the obtained key frame timestamp list.
[0201] In the embodiment of the present invention, by performing key frame matching on the target video according to the key action feature set, a key frame timestamp list is obtained, thereby improving the accuracy of subsequent card point matching and improving the efficiency of subsequent card point matching.
[0202] S9. Performing card point matching on the target video and the target audio according to the drum timestamp list and the key frame timestamp list to obtain a card point video.
[0203] In the embodiment of the present invention, performing card point matching on the target video and the target audio according to the timestamp list and the key frame timestamp list to obtain the card point video includes:
[0204] Get the number of drum beats in the drum beat timestamp list;
[0205] Adaptively adjusting the key frame timestamp list according to the number of drum beats to obtain an adaptive key frame list;
[0206] Aligning the key frames in the target video with the drum beats in the drum beat timestamp list according to the adaptive key frame list to obtain an aligned video;
[0207] The aligned video is fused with the target audio to obtain a card point video.
[0208] In the embodiment of the present invention, the adaptively adjusting the key frame timestamp list according to the number of drum beats refers to adjusting the number of key frames in the key frame timestamp list so that it is not greater than the number of drum beats.
[0209] In the embodiment of the present invention, aligning the key frames in the target video with the drum beats in the drum beat timestamp list according to the adaptive key frame list to obtain the aligned video includes:
[0210] Perform key point matching on the drum beat timestamp list and the adaptive key frame list to obtain a key point matching result;
[0211] Obtain the time intervals between adjacent drum beats in the drum beat timestamp list to obtain a time interval set;
[0212] According to the key point matching result and the time interval set, slow motion frame insertion processing and fast motion frame extraction processing are performed on adjacent key frames in the target video to obtain an aligned video.
[0213] In detail, when aligning the key frames of the target video with the drum beats, it is necessary to adjust the time interval between adjacent key frames in the target video to be the same as the time interval between adjacent drum beats of the target audio.
[0214] Specifically, when the time interval between adjacent key frames is shorter than the time interval between corresponding adjacent drum beats, the adjacent key frames need to be slowed down and interpolated to make their time interval the same as the time interval between corresponding adjacent drum beats.
[0215] Furthermore, when the time interval between adjacent key frames is greater than the time interval between corresponding adjacent drum beats, the adjacent key frames need to be fast-forwarded and frame-decreased so that their time interval is the same as the time interval between corresponding adjacent drum beats.
[0216] In detail, the target video may be interpolated using an optical flow method.
[0217] In detail, the optical flow method generates an intermediate frame by calculating pixel changes between adjacent frames to complete interpolation.
[0218] In the embodiment of the present invention, the target video and the target audio are matched at the card points according to the drum timestamp list and the key frame timestamp list to obtain the card point video, thereby improving the efficiency of card point matching.
[0219] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program can implement:
[0220] Obtain the target video and audio that need to be matched with the card points;
[0221] Identify the frequency spectrum of the target audio and extract frequency spectrum features of the frequency spectrum;
[0222] Using a pre-trained drum beat recognition model to perform drum beat recognition on the target audio according to the frequency spectrum features, and obtaining a drum beat timestamp list;
[0223] Establishing a motion trajectory model according to the target video;
[0224] Extracting features from each action frame of the motion trajectory model to obtain an action feature set;
[0225] Calculate the difference value between adjacent action features in the action feature set, wherein the difference value is calculated using the following formula:
[0226]
[0227] Wherein, C is the covariance matrix of the action feature set, N is the number of action features in the action feature set, and F k represents the kth action feature in the action feature set, D is the difference value, ΔF i The difference between the feature vectors of the adjacent action features in the i-th dimension, ΔF j represents the difference between the feature vectors of the adjacent action features in the jth dimension, n is the dimension of the vector, C ii represents the element of the i-th row and i-th column of the covariance matrix C of the action feature set, C ij The element in the i-th row and j-th column of the covariance matrix C of the action feature set is represented;
[0228] Extracting the next action feature from all adjacent action features whose difference values are greater than or equal to a preset difference value threshold to obtain a key action feature set;
[0229] Perform key frame matching on the target video according to the key action feature set to obtain a key frame timestamp list;
[0230] The target video and the target audio are matched at the card points according to the drum timestamp list and the key frame timestamp list to obtain the card point video.
[0231] In the embodiments provided by the present invention, it should be understood that the disclosed methods and media can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.
[0232] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0233] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.
[0234] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.
[0235] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.
[0236] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0237] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.
[0238] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.
Claims
1. A video and music card point matching method based on AI, characterized in that: The method comprises: Obtain the target video and audio that need to be matched with the card points; Identify the frequency spectrum of the target audio and extract frequency spectrum features of the frequency spectrum; Using a pre-trained drum beat recognition model to perform drum beat recognition on the target audio according to the frequency spectrum features, and obtaining a drum beat timestamp list; Establishing a motion trajectory model according to the target video; Extracting features from each action frame of the motion trajectory model to obtain an action feature set; Calculate the difference value between adjacent action features in the action feature set, wherein the difference value is calculated using the following formula: Wherein, C is the covariance matrix of the action feature set, N is the number of action features in the action feature set, and F k represents the kth action feature in the action feature set, D is the difference value, ΔF i The difference between the feature vectors of the adjacent action features in the i-th dimension, ΔF j represents the difference between the feature vectors of the adjacent action features in the jth dimension, n is the dimension of the vector, C ii represents the element of the i-th row and i-th column of the covariance matrix C of the action feature set, C ij The element in the i-th row and j-th column of the covariance matrix C of the action feature set is represented; Extracting the next action feature from all adjacent action features whose difference values are greater than or equal to a preset difference value threshold to obtain a key action feature set; Perform key frame matching on the target video according to the key action feature set to obtain a key frame timestamp list; The target video and the target audio are matched at the card points according to the drum timestamp list and the key frame timestamp list to obtain the card point video.
2. The AI-based video and music card point matching method according to claim 1, characterized in that: The identifying the frequency spectrum of the target audio and extracting the frequency spectrum features of the frequency spectrum includes: The target audio is subjected to Fourier transform using the following formula to obtain frequency domain features: Wherein, F represents the frequency domain feature, x n represents the nth sampling point in the target audio, N represents the total number of sampling points of the target audio, e is the base of the natural logarithm, j is the imaginary unit, and j 2 = -1; Draw a spectrum diagram according to the frequency domain characteristics; Extracting spectrum information from the spectrum graph to obtain a spectrum; extracting frequency components of the spectrum; The frequency components are normalized to obtain frequency spectrum features.
3. The AI-based video and music card point matching method according to claim 1, characterized in that: The method of using the pre-trained drum beat recognition model to perform drum beat recognition on the target audio according to the frequency spectrum features to obtain a drum beat timestamp list includes: Performing denoising on the target audio to obtain a denoised audio frequency; Partition the denoised audio according to a preset number of partitions to obtain partitioned audio; Using the drum beat recognition model to predict the drum beat probability of each partition in the partition audio according to the frequency spectrum features, to obtain the drum beat probability of each partition in the partition audio; Using the drum beat recognition model to set a drum beat probability threshold according to a predetermined card point task requirement; The drum beat recognition model is used to mark the partitioned audio with drum beats according to the drum beat probability threshold to obtain a drum beat timestamp list.
4. The AI-based video and music card point matching method according to claim 3, characterized in that: The step of performing denoising on the target audio to obtain denoised audio includes: Performing noise type recognition on the target audio to obtain the noise type; Acquire a noise sample of the target audio according to the noise type; Calculating the power spectrum of the target audio to obtain an audio power spectrum, and calculating the power spectrum of the noise sample to obtain a noise power spectrum; Subtracting the noise power spectrum from the audio power spectrum to obtain a denoised frequency domain signal; The denoised frequency domain signal is inversely Fourier transformed using the following formula to obtain the denoised frequency domain signal: Among them, x n represents the nth sampling point in the target audio, N represents the total number of sampling points of the target audio, F represents the frequency domain feature, e is the base of the natural logarithm, j is the imaginary unit, and j satisfies 2 =-1.
5. The AI-based video and music card point matching method according to claim 4, characterized in that: The step of calculating the power spectrum of the target audio to obtain the audio power spectrum includes: The spectrum of the target audio is processed by framing to obtain a spectrum frame set The frame energy of each frame in the spectrum frame set is calculated using the following formula to obtain the spectrum energy: Where, E is the energy spectrum, L is the number of samples in the spectrum frame set, x s represents the sth sample in the spectrum frame set, w is the window function, w s represents the value of the sth sample in the spectrum frame set in the window function; A power spectrum is generated according to the spectrum energy to obtain an audio power spectrum.
6. The AI-based video and music card point matching method according to claim 3, characterized in that: The step of using the drum beat recognition model to predict the drum beat probability of each partition in the partition audio according to the frequency spectrum feature to obtain the drum beat probability of each partition in the partition audio includes: Inputting the frequency spectrum feature into the hidden layer of the drum beat recognition model; Partitioning the spectrum feature according to a preset number of partitions to obtain a set of partitioned spectrum features; The hidden layer is used to call the following formula to calculate the output of each neuron in the drum beat recognition model according to the partition spectrum feature set to obtain the hidden layer output of each partition of the partition audio: Wherein, Z is the hidden layer output of any partition in the partitioned audio, n is the number of neurons in the hidden layer, w is the weight parameter of the drum beat recognition model, x i is the ith partition spectrum in the partition spectrum feature set, and b is the bias term of the drum beat recognition model; Using an activation function, the hidden layer output of each partition of the partitioned audio is converted into an activation value to obtain the activation value of each partition of the partitioned audio; The output layer of the drum beat recognition model is used to call the following formula to calculate the drum beat probability of each partition in the partition audio according to the activation value of each partition of the partition audio: Wherein, P is the probability of a drum beat in any partition of the partitioned audio, n is the number of neurons in the hidden layer, w is the weight parameter, and a i is the activation value of the i-th partition in the partitioned audio, b is the bias term, and e is the base of the natural logarithm.
7. The AI-based video and music card point matching method according to claim 1, characterized in that: The step of establishing a motion trajectory model according to the target video includes: Using a computer vision algorithm to capture key points of the target video to obtain key points; Creating an action change sequence according to the changes of the key points over time; Establishing a skeleton model according to the key points; The skeleton model is driven by the action change sequence to obtain a motion trajectory model.
8. The AI-based video and music card point matching method according to claim 1, characterized in that: The step of performing key frame matching on the target video according to the key action feature set to obtain a key frame timestamp list includes: Matching all corresponding action frames in the motion trajectory model according to all key action features in the key action feature set to obtain a key action frame set; Extracting time information of each key action in the key action set to obtain a time information set; According to each time information in the time information set, a frame picture of a corresponding time in the target video is intercepted to obtain a key frame timestamp list.
9. The AI-based video and music card point matching method according to claim 1, characterized in that: The performing card point matching on the target video and the target audio according to the timestamp list and the key frame timestamp list to obtain the card point video includes: Get the number of drum beats in the drum beat timestamp list; Adaptively adjusting the key frame timestamp list according to the number of drum beats to obtain an adaptive key frame list; Aligning the key frames in the target video with the drum beats in the drum beat timestamp list according to the adaptive key frame list to obtain an aligned video; The aligned video is fused with the target audio to obtain a card point video.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the AI-based video and music card point matching method as described in any one of claims 1 to 9 is implemented.
