Data processing method, electronic device, storage medium and computer program product

Abstract

The invention discloses a data processing method and device, a medium and a program product. The method comprises the steps that a first feature sequence of audio sample data and a second feature sequence of video sample data are determined, the first feature sequence represents emotion changes of N audio clips in the audio sample data, and the second feature sequence represents emotion changes of N video clips in the video sample data; the first feature sequence and the second feature sequence are clustered, a first feature dictionary and a second feature dictionary are obtained, the first feature dictionary comprises one or more audio clip sets with similar emotion changes in the audio sample data, and the second feature dictionary comprises one or more video clip sets with similar emotion changes in the video sample data; using the first feature dictionary and the second feature dictionary to obtain a first relationship, the first relationship including a mapping relationship of the first feature sequence and the second feature sequence in a semantic space; and selecting matched audio data for the to-be-matched video data by using the first relationship.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, electronic device, storage medium, and computer program product. Background Technology

[0002] Currently, automatic music video (MV) generation solutions mainly fall into two categories: automatic video editing and automatic music composition. Automatic video editing typically ensures synchronization between music and video by editing the video itself. This method requires the user to first select the music, and then the video is edited accordingly to match the music. Simultaneously, the creator needs to check the generated results to ensure seamless transitions between video segments. Automatic music composition, on the other hand, usually utilizes artificial intelligence technology to provide matching music to the video content.

[0003] However, some of the related video soundtrack solutions suffer from a lack of proper matching between the video and the music. Summary of the Invention

[0004] To address the related technical problems, embodiments of this application provide a data processing method, an electronic device, a storage medium, and a computer program product.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a data processing method, including:

[0007] A first feature sequence of audio sample data and a second feature sequence of video sample data are determined. The first feature sequence characterizes the emotional changes of N audio segments in the audio sample data, and the second feature sequence characterizes the emotional changes of N video segments in the video sample data, where N is an integer greater than or equal to 1.

[0008] Clustering is performed on the first feature sequence and the second feature sequence to obtain a first feature dictionary and a second feature dictionary. The first feature dictionary contains a set of one or more audio segments with similar emotional changes in the audio sample data, and the second feature dictionary contains a set of one or more video segments with similar emotional changes in the video sample data.

[0009] Using the first feature dictionary and the second feature dictionary, a first relation is obtained, which contains the mapping relationship between the first feature sequence and the second feature sequence in the semantic space;

[0010] Using the first relationship, select matching audio data for the video data to be matched.

[0011] This application also provides an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor.

[0012] When the processor runs the computer program, it executes the steps of any of the data processing methods described above.

[0013] This application also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of any of the data processing methods described above.

[0014] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the data processing methods described above.

[0015] The data processing method, electronic device, storage medium, and computer program product provided in this application embodiment determine a first feature sequence of audio sample data and a second feature sequence of video sample data. The first feature sequence represents the emotional changes of N audio segments in the audio sample data, and the second feature sequence represents the emotional changes of N video segments in the video sample data, where N is an integer greater than or equal to 1. The first feature sequence and the second feature sequence are clustered to obtain a first feature dictionary and a second feature dictionary. The first feature dictionary contains a set of one or more audio segments with similar emotional changes in the audio sample data, and the second feature dictionary contains a set of one or more video segments with similar emotional changes in the video sample data. Using the first feature dictionary and the second feature dictionary, a first relation is obtained, which contains the mapping relationship between the first feature sequence and the second feature sequence in the semantic space. Using the first relation, matching audio data is selected for the video data to be matched. The technical solution provided in this application constructs an audio-visual feature dictionary by clustering the features of emotional fluctuations in short-term audio-visual sample data, and establishes a semantic mapping relationship between the features of emotional fluctuations in the audio-visual feature dictionary, so as to better match music that conforms to the trend of emotional changes for the video to be set to music based on the mapping relationship, thereby improving the matching degree between the video to be set to music and the music. Attached Figure Description

[0016] Figure 1 This is a schematic flowchart of a data processing method according to an embodiment of this application;

[0017] Figure 2 This is a schematic diagram illustrating the process of constructing a second feature dictionary according to an embodiment of this application;

[0018] Figure 3 This is a schematic diagram illustrating the process of determining the matching degree of emotional fluctuation synchronization as an application example of this application;

[0019] Figure 4 This is a schematic diagram illustrating the process of determining the matching degree of emotional consistency as an application example of this application;

[0020] Figure 5 This is a schematic diagram of the structure of a data processing device according to an embodiment of this application;

[0021] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0022] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0023] In related video soundtrack solutions, candidate music that matches the video is determined based on the features of single-modal or multi-modal fusion. However, this approach only considers video features and audio features, without considering the relationship between the two features, which leads to a low degree of matching between the video and the music.

[0024] Based on this, in various embodiments of this application, an audio-visual feature dictionary is constructed by measuring the emotional fluctuations of audio-visual sample data in short segments, and a mapping relationship between the potential semantic features of audio-visual features is constructed based on the audio-visual feature dictionary, so as to match the video with music based on the mapping relationship. This helps to better match the video with music that is synchronized with the emotional fluctuations, and improves the accuracy of music matching.

[0025] This application provides a data processing method, such as... Figure 1 As shown, applied to electronic devices, the method includes:

[0026] Step 101: Determine the first feature sequence of the audio sample data and the second feature sequence of the video sample data. The first feature sequence represents the emotional changes of N audio segments in the audio sample data, and the second feature sequence represents the emotional changes of N video segments in the video sample data, where N is an integer greater than or equal to 1.

[0027] Step 102: Cluster the first feature sequence and the second feature sequence to obtain a first feature dictionary and a second feature dictionary. The first feature dictionary contains a set of one or more audio segments with similar emotional changes in the audio sample data, and the second feature dictionary contains a set of one or more video segments with similar emotional changes in the video sample data.

[0028] Step 103: Using the first feature dictionary and the second feature dictionary, obtain a first relation, which contains the mapping relationship between the first feature sequence and the second feature sequence in the semantic space;

[0029] Step 104: Using the first relationship, select matching audio data for the video data to be matched.

[0030] In practical applications, the electronic device may include a device capable of performing steps 101 to 104, such as a server, laptop computer, tablet computer, or desktop computer. This application embodiment does not limit the type of electronic device, as long as it performs its functions.

[0031] In practical applications, before step 101, the electronic device can obtain multimedia data (such as music video data) from the network. When the multimedia data includes both audio and video data, the electronic device can segment the multimedia data, separating audio sample data from the multimedia data to obtain audio sample data and video sample data.

[0032] It should be noted that, since the data formats of the acquired multimedia data may be different, the electronic device can adjust the data formats of the audio sample data and video sample data, and save the adjusted audio sample data and video sample data; wherein, the audio sample data can be saved in MP3 format, and the video sample data can be saved in MP4 format.

[0033] In practical applications, since the emotional changes in audio sample data and video sample data have similar trends in a short period of time, the electronic device can perform feature analysis at the data segment level.

[0034] Specifically, in one embodiment, determining the first feature sequence corresponding to the audio sample data and the second feature sequence corresponding to the video sample data includes:

[0035] For each of the N audio segments, a first feature, a second feature, and a third feature are determined, and these features are fused to obtain a third feature sequence corresponding to the audio segment. The first feature represents the loudness change of the audio segment, the second feature represents the spectral energy change of the audio segment, and the third feature represents the degree of incoherence of the audio segment. The third feature sequences corresponding to the N audio segments are then fused to obtain the first feature sequence.

[0036] For each of the N video segments, a fourth feature and a fifth feature are determined, and the fourth and fifth features are fused to obtain a fourth feature sequence corresponding to the video segment. The fourth feature represents the optical flow motion of the video segment, and the fifth feature represents the shot switching of the video segment. The fourth feature sequences corresponding to the N video segments are fused to obtain the second feature sequence.

[0037] The first feature can be called audio energy, which describes the loudness variation of an audio segment; the second feature can be called spectral flux, which describes the rate of change of the spectral energy of the signal in the audio segment; and the third feature can be called auditory roughness, which describes the degree of psychological and physiological sensation a user experiences when hearing dissonant (i.e., discordant) sounds in an audio segment. The first, second, and third features can be understood as the emotional fluctuations of the audio segment.

[0038] Furthermore, the fourth feature can be called optical flow motion feature, which can describe the changes in the direction and / or amplitude of object movement in a video clip; the fifth feature can be called shot switching frequency, which can describe the frequency of scene changes. The fourth and fifth features can be understood as the emotional fluctuation features of the video clip.

[0039] In practical applications, after dividing the audio sample data into N audio segments, the electronic device can determine the first feature, the second feature, and the third feature of each audio segment. Then, the first feature, the second feature, and the third feature are fused (e.g., spliced ​​or concatenated) to obtain the first feature sequence.

[0040] Here, in the process of determining the features of an audio segment, the electronic device can determine the sub-features of each frame at the frame level, thereby obtaining the features of the audio segment.

[0041] Specifically, in one embodiment, determining the first feature, second feature, and third feature of the audio segment includes:

[0042] Identify one or more first frames associated with an audio segment;

[0043] Feature extraction is performed on each of the one or more first frames to obtain a first sub-feature, a second sub-feature, and a third sub-feature. The first sub-feature represents the loudness change of the first frame, the second feature represents the spectral energy change of the first frame, and the third feature represents the degree of incoherence of the first frame.

[0044] The first sub-feature, the second sub-feature, and the third sub-feature of the one or more first frames are fused to obtain the first feature, the second feature, and the third feature, respectively.

[0045] Here, the electronic device can convert the sampling points of the audio segment in time sequence through a preset first window and a first parameter to obtain one or more first frames. The first window can be understood as a sampling window, for example, with a size of 50ms. The first parameter represents the overlap rate between the first windows, for example, the first parameter is 50%. The first frame can be called an audio frame, and each first frame can contain one or more sampling points.

[0046] In practical applications, for each first frame, the electronic device can extract (or statistically analyze) the amplitude energy of each sampling point in the first frame to obtain the first sub-feature; extract the sum of squares of the differences in spectral energy values ​​between the first frame and the next adjacent first frame to obtain the second sub-feature; and extract the differences in the amplitude fluctuation rate of the audio signal in the first frame to obtain the third sub-feature, wherein the amplitude fluctuation rate is correlated with the distance between peaks or troughs.

[0047] Then, by performing feature fusion processing on the first, second, and third sub-features of each first frame, the electronic device can obtain a feature of an audio segment regarding emotional changes; wherein, the feature of an audio segment regarding emotional changes can be represented as:

[0048] a m =[G e (1),...,G e (q),G f (1),...,G f (q),G rg (1),...,G rg (q)] (1)

[0049] Where q represents the number of first frames associated with audio segment m. For example, with a first window size of 50ms and a first parameter of 50%, a 3s audio segment m is associated with 119 first frames; G e Represented as the first sub-feature of the first frame; G f Represented as the second sub-feature of the first frame; G rg This is represented as the third sub-feature of the first frame.

[0050] For example, when the loudness represented by the first sub-feature changes faster, the spectral energy represented by the second sub-feature changes faster, and the degree of incoordination represented by the third sub-feature is higher, the electronic device can determine that the rhythm of the audio segment is more compact and the emotion is more exciting by concatenating the first sub-feature, the second sub-feature, and the third sub-feature.

[0051] In practical applications, after dividing the video sample data into N video segments, the electronic device can determine the fourth and fifth features of each video segment. Then, the fourth and fifth features are fused (e.g., spliced ​​or concatenated) to obtain the second feature sequence.

[0052] Here, in the process of determining the features of a video segment, the electronic device can determine the sub-features of each frame at the frame level, thereby obtaining the features of the video segment.

[0053] In one embodiment, determining the fourth and fifth features of the video segment includes:

[0054] Identify one or more second frames associated with the video segment;

[0055] Feature extraction is performed on each of the one or more second frames to obtain a fourth sub-feature and a fifth sub-feature. The fourth sub-feature represents the optical flow motion of the second frame, and the fifth sub-feature represents the shot switching of the second frame.

[0056] The fourth and fifth sub-features of the one or more second frames are fused to obtain the fourth feature and the fifth feature, respectively.

[0057] In practical applications, the electronic device can determine one or more second frames using a second parameter, where the second parameter represents the frame rate of the video segment, and the second frame can be referred to as a video frame. For example, in a video segment with a duration of 3 seconds and a second parameter of 24 fps, the electronic device can determine 72 second frames.

[0058] In practical applications, for each second frame, the electronic device can extract the temporal change of the average optical flow vector (also known as the dense optical flow average) of the pixels between the second frame and the next adjacent second frame to obtain the fourth sub-feature; by performing scene detection on the one or more second frames and determining the distance between the two scenes closest to the second frame (which can be understood as the second frame where the scene changes, i.e., the key frame) based on the order in which the second frames appear, the fifth sub-feature is obtained.

[0059] Then, by performing feature fusion processing on the fourth and fifth sub-features of each second frame, the electronic device can obtain a feature of the video segment regarding emotional changes; wherein, the feature of the video segment regarding emotional changes can be represented as:

[0060] a v =[G m (1),...,G m (p),G r (1),...,G r (p)] (2)

[0061] Where p represents the number of second frames associated with video segment v; G m Represented as the fourth sub-feature of the second frame; G r This is represented as the fifth sub-feature of the second frame.

[0062] For example, when the motion amplitude represented by the fourth sub-feature is greater and the scene change represented by the fifth sub-feature is faster, the electronic device can determine that the video clip has a stronger sense of rhythm and more obvious emotional fluctuations by concatenating the fourth and fifth sub-features.

[0063] It should be noted that during the feature extraction process of audio and video segments, the electronic device may consider one or more of the following: audio channel format, sampling rate, video frame rate, and frame width and height, thereby ensuring the integrity of the original data information and reducing the complexity of engineering implementation.

[0064] In practical applications, in step 102, the electronic device can cluster the first feature sequence and the second feature sequence in the feature space to construct the first feature dictionary and the second feature dictionary; wherein, the first feature dictionary may include the one or more audio segment sets (also called audio segment clusters), each audio segment set containing multiple audio segments with similar features related to emotional changes, the first feature dictionary may be called a music feature dictionary, and this application embodiment does not limit this; the second feature dictionary may include the one or more video segment sets (also called video segment clusters), each video segment set containing multiple video segments with similar features related to emotional changes, the second feature dictionary may be called a video feature dictionary, and this application embodiment does not limit this.

[0065] Here, by comparing the first feature sequence and the second feature sequence, the first feature sequence is more discriminative and representative. Therefore, the electronic device can cluster the first feature sequence to construct the first feature dictionary. Then, based on the first feature dictionary and the second feature sequence, the second feature dictionary is constructed. Here, the algorithm of Kmeans++ for randomly initializing cluster centers can be selected, and the number of cluster centers is linearly searched within a certain range to achieve clustering of the first feature sequence. In this way, it is ensured that the first feature dictionary can fully express the features of the audio segment about emotional changes, and at the same time, the evaluation criteria of the clustering results are met.

[0066] For example, such as Figure 2As shown, since the clustering results of audio segments in the feature space are simpler and converge better, the electronic device can perform k-means clustering on the first feature sequence to construct a first feature dictionary. The first feature dictionary contains n sets of audio segments, and the cluster center of each audio segment set is calculated based on the features of all audio segments with regard to emotional changes, where n is an integer greater than or equal to 2. Since audio segments and video segments have co-occurrence properties (which can also be understood as similarity), the electronic device can construct corresponding n sets of video segments based on the n sets of audio segments and the second feature sequence, and then construct a second feature dictionary. The second feature dictionary contains n sets of video segments, and the cluster center of each video segment set is calculated based on the features of all video segments with regard to emotional changes.

[0067] In practical applications, in step 103, the electronic device can transform the first feature sequence and the second feature sequence into mid-level features in the semantic space based on the first feature dictionary and the second feature dictionary (which can be understood as features obtained after feature processing of the sample data), and establish a mapping relationship between the first feature sequence and the second feature sequence through the mid-level features. In this way, the complex feature space distribution caused by directly fusing feature sequences can be avoided, the complexity of establishing the mapping relationship can be reduced, and the description of the sample data can also have good discriminative power.

[0068] Specifically, in one embodiment, obtaining the first relationship using the first feature dictionary and the second feature dictionary includes:

[0069] Determine one or more first cluster centers corresponding to the one or more audio segment sets, and determine one or more second cluster centers corresponding to the one or more video segment sets;

[0070] The first feature sequence is updated using the one or more first cluster centers to obtain the updated first feature sequence;

[0071] The second feature sequence is updated using the one or more second cluster centers to obtain the updated second feature sequence;

[0072] The first relationship is obtained using the updated first feature sequence and the updated second feature sequence.

[0073] In practical applications, each audio segment set corresponds to a first cluster center, which is calculated based on the emotional change features of all audio segments in the set. Additionally, each video segment set corresponds to a second cluster center, which is also calculated based on the emotional change features of all video segments in the set.

[0074] In practical applications, for the first feature sequence, the electronic device can determine the feature subsequence corresponding to each audio segment set, and use the similarity between the feature subsequence and the first cluster center as the mid-level feature, thereby updating the first feature sequence using the mid-level feature.

[0075] Specifically, in one embodiment, updating the first feature sequence using the one or more first cluster centers to obtain an updated first feature sequence includes:

[0076] Using the first feature sequence, a first feature subsequence is determined for the set of one or more audio segments, the first feature subsequence representing the emotional changes of the set of one or more audio segments;

[0077] Using the first feature subsequence and the one or more first cluster centers, a first result is determined, wherein the first result characterizes the similarity between the first feature subsequence and the one or more first cluster centers;

[0078] The first feature subsequence is updated using the first result to obtain the updated first feature subsequence.

[0079] In practical applications, using the first feature dictionary, the electronic device can divide the first feature sequence into one or more first feature subsequences; wherein each audio segment set corresponds to one first feature subsequence, in other words, the first feature subsequence contains features related to emotional changes for each audio segment in the audio segment set. For example, in the case of k audio segment sets, the electronic device can determine k first feature subsequences; wherein the k audio segment sets can be represented as: {A1, A...} i ,...,A k}, where k is an integer greater than or equal to 2.

[0080] Here, for each set of audio segments, the electronic device can use a first feature subsequence to determine the features of each audio segment regarding emotional changes, and sequentially calculate the Euclidean distance between each feature and each of the one or more first cluster centers to obtain the first result, wherein the dimension of the first result is the same as the number of first cluster centers. Then, the electronic device can use the first result as the updated first feature subsequence.

[0081] For example, suppose there is a set A of audio clips. k Includes audio clip X i And audio clip X i The characteristic of emotional change is x. iThe electronic device can calculate x respectively. i The Euclidean distances between the audio segment and all first cluster centers are calculated, and the results are used as the updated audio segment X. i Characteristics of emotional changes i '.

[0082] In practical applications, for the second feature sequence, the electronic device can determine the feature subsequence corresponding to each video segment set, and use the similarity between the feature subsequence and the second cluster center as the mid-level feature, thereby updating the second feature sequence using the mid-level feature.

[0083] Specifically, in one embodiment, updating the second feature sequence using the one or more second cluster centers to obtain the updated second feature sequence includes:

[0084] Using the second feature sequence, a second feature subsequence is determined for the set of one or more video clips, the second feature subsequence representing the emotional changes of the set of one or more video clips;

[0085] Using the second feature subsequence and the one or more second cluster centers, a second result is determined, wherein the second result characterizes the similarity between the second feature subsequence and the one or more second cluster centers;

[0086] The second feature subsequence is updated using the second result to obtain the updated second feature subsequence.

[0087] In practical applications, using the second feature dictionary, the electronic device can divide the second feature sequence into one or more second feature subsequences; wherein each video segment set corresponds to one second feature subsequence, in other words, the second feature subsequence contains features related to emotional changes for each video segment in the video segment set. For example, in the case of k video segment sets, the electronic device can determine k second feature subsequences; wherein the k video segment sets can be represented as: {B1, B...} i ,...,B k}, where k is an integer greater than or equal to 2.

[0088] Here, for each set of video clips, the electronic device can use the second feature subsequence to determine the features of each video clip regarding emotional changes, and sequentially calculate the Euclidean distance between each feature and each of the one or more second cluster centers to obtain the second result, wherein the dimension of the second result is the same as the number of second cluster centers. Then, the electronic device can use the second result as the updated second feature subsequence.

[0089] For example, suppose there is a set B of video clips. k Includes audio segment Y i And video clip Y i The characteristic of emotional changes is y. i The electronic device can calculate y respectively. i The Euclidean distances between the video segment Y and all second cluster centers are calculated, and the results are used as the updated video segment Y. i Characteristics of emotional changes i '.

[0090] In practical applications, to establish the first relationship, the electronic device can use the updated first feature subsequence as the predicted feature vector and the updated second feature subsequence as the predicted feature vector, and employ canonical correlation analysis to determine the mapping relationship between the updated first and second feature subsequences. Although the first and second feature subsequences of the two modes reside in different feature spaces, the electronic device can use the first canonical correlation vector (which can be represented as a) to determine the mapping relationship between them. i ') and the second canonical correlation vector (which can be represented as b) i '), respectively, the first feature subsequence features corresponding to the audio segment set (which can be represented as x) i The first feature subsequence feature corresponding to the set of video segments (which can be represented as y) and the first feature subsequence feature (which can be represented as y) i Mapping them to the same coordinate system makes the first feature subsequence after mapping (which can be represented as) ) and the mapped second feature subsequence (which can be represented as The correlation is greatest for the first feature subsequence. Next, the electronic device can use linear regression to fit the relationship between the mapped first feature subsequence, the first feature subsequence feature, and the second feature subsequence, respectively, and use the resulting residuals as the first feature subsequence feature and the second feature subsequence in the next fitting process, until the first and second canonical correlation vectors of all audio and video segment sets are determined, and then the first relationship is established based on the determined canonical correlation variables; wherein, the first relationship can be expressed as:

[0091] y i '=x i 'w i (3)

[0092] Among them, w i 'Can be represented as x i 'To y i The fitting parameters of '.

[0093] For example, during the testing phase, assume that the first feature subsequence corresponding to the j-th audio segment set is m' j Using formula (3), the electronic device can predict the set of pseudo-video segments corresponding to the audio segments as follows:

[0094] In practical applications, in step 104, based on the first relationship, the electronic device can also consider the emotional consistency between the video data to be matched and the candidate audio data, and select matching audio data for the video data to be matched, so as to take into account the consistency of the user's visual and auditory sensory experience.

[0095] Based on this, in one embodiment, the method may further include:

[0096] A fifth feature sequence corresponding to the video data to be matched is determined, and a sixth feature sequence corresponding to one or more candidate audio data is determined. The fifth feature sequence represents the emotional change of the video data to be matched, and the sixth feature sequence represents the emotional change of the one or more candidate audio data.

[0097] Using the fifth feature sequence and the sixth feature sequence, a third result is obtained, which characterizes the similarity between the video data to be matched and the one or more candidate audio data regarding emotional changes;

[0098] Using the third result and the first relationship, audio data that matches the video data to be matched is selected from the one or more candidate audio data.

[0099] The candidate audio data can be obtained from a relevant database (also known as a music library). This application embodiment does not limit the method of obtaining the candidate audio data.

[0100] In practical applications, before determining the fifth feature sequence and the sixth feature sequence, the electronic device needs to pre-train a first classifier. The first classifier is used to identify emotional changes in video data and / or audio data. The first classifier can also be called an emotion classifier, emotion classification model, or audio-visual emotion recognition model, etc. This application embodiment does not limit this, as long as its function is implemented.

[0101] In practical applications, during the training of the first classifier, the electronic device can preprocess the audio and video sample data separately. Simultaneously, it can perform transfer learning on relevant audio recognition models (also known as music style recognition models) and video emotion recognition models (also known as visual emotion concept recognition models) to establish the relationship between the first classifier and the audio sample data, video sample data, and emotion cognition. Then, using the preprocessed audio and video sample data and the true emotion category, the learning rate and learning rate decay of each layer weight in the first classifier are adjusted to obtain the trained first classifier.

[0102] For example, for audio sample data, the electronic device can compress dual-channel audio sample data into mono to reduce redundant information. Simultaneously, it uses Discrete Fourier Transform to convert the time-domain signal to the frequency domain, i.e., converting the waveform into a spectrum. Then, it divides the spectrum into images of equal duration and uses these images as descriptions of the corresponding audio segments. For video sample data, the electronic device can segment the video sample data into segments of equal duration and extract keyframes from each video segment (which can be understood as frames corresponding to scenes). If a video segment has only one keyframe, that keyframe is used as the description of the video segment. If a video segment has multiple keyframes, the average pixel values ​​at the same position of all keyframes are used as the pixel value at the corresponding position in the new image, and this new image is used as the description of the video segment.

[0103] For example, assuming a pre-trained convolutional neural network model for music style recognition and visual emotion concept recognition is used, after transfer learning is performed on the first classifier, the electronic device can train the first classifier using descriptions of video and audio segments, and correct the emotion prediction results of the first classifier using real emotion categories. Given T audio segments, the electronic device can predict the emotion category of each audio segment using the first classifier. If the most prevalent emotion category exists, that emotion category is used as the emotion category of the music sample data, and the predicted emotion category is corrected using the real emotion category. If multiple emotion categories have the same number of segments, the emotion category with the smallest label is selected as the emotion category of the music sample data, and the predicted emotion category is corrected using the real emotion category. This completes the training of the first classifier.

[0104] In practical applications, to determine the fifth feature sequence, the electronic device can divide the video data to be matched into M video segments; by inputting the M video segments into the first classifier, the electronic device can obtain the fifth feature sequence, which can contain the sentiment category of each of the M video segments. The fifth feature sequence can be represented as: B = [b1, b2, ..., b n ], b n This represents the emotion category of the nth video segment, such as high-spirited, neutral, or grief-stricken.

[0105] In addition, to determine the sixth feature sequence, the electronic device can divide each candidate audio data into M audio segments; by inputting the M audio segments of all candidate audio data into the first classifier, the electronic device can obtain the sixth feature sequence, which may contain the sixth feature subsequence corresponding to the one or more candidate audio data. Each sixth feature subsequence contains the sentiment category of each video segment in the M audio segments, and the sixth feature subsequence can be represented as: A = [a1, a2, ..., a n ], a n This represents the emotional category of the nth audio segment, such as high-pitched, flat, or sorrowful.

[0106] In practical applications, after obtaining the fifth feature sequence and the sixth feature sequence, the electronic device can calculate the Hamming distance between the fifth feature sequence and the sixth feature sequence, and use the calculated Hamming distance as the third result; wherein, the third result can be calculated by the following formula:

[0107]

[0108] In practical applications, after obtaining the third result, the electronic device can determine the emotional consistency between the one or more candidate audio data and the video data to be matched based on the first relationship, and then select the matching audio data based on the emotional consistency (which can also be understood as the synchronicity of emotional fluctuations) and the third result associated with emotional consistency.

[0109] Specifically, in one embodiment, selecting audio data matching the video data to be matched from the one or more candidate audio data using the third result and the first relationship includes:

[0110] Using the seventh feature sequence corresponding to the one or more candidate audio data and the first relationship, an eighth feature sequence corresponding to one or more candidate video data is determined, wherein the seventh feature sequence characterizes the emotional changes of the one or more candidate audio data and the eighth feature sequence characterizes the emotional changes of the one or more candidate video data;

[0111] Using the eighth feature sequence and the ninth feature sequence corresponding to the video data to be matched, a fourth result is obtained. The ninth feature sequence characterizes the emotional changes of the video data to be matched, and the fourth result characterizes the similarity between the video data to be matched and the one or more candidate video data regarding emotional changes.

[0112] Using the third and fourth results, audio data that matches the video data to be matched is selected from the one or more candidate audio data.

[0113] In practical applications, in order to determine the seventh feature sequence, the electronic device can divide each candidate audio data into L audio segments; by extracting features from each audio segment, the seventh feature subsequence corresponding to each candidate audio data is obtained, and then the seventh feature sequence is obtained. The seventh feature sequence can contain the features of each audio segment in the L audio segments regarding emotional changes.

[0114] Next, using formula (3), the electronic device can determine the eighth feature subsequence corresponding to each candidate video data, thereby obtaining the eighth feature sequence. The eighth feature subsequence can contain features of emotional changes in each of the L video segments. The eighth feature subsequence can be understood as a pseudo-video feature sequence obtained by fitting based on the first relationship. In addition, the electronic device can divide the video data to be matched into L video segments; by extracting features from each video segment, the ninth feature sequence is obtained. The ninth feature sequence can contain features of emotional changes in each of the L video segments.

[0115] Here, by matching each eighth feature subsequence in the eighth feature sequence with the ninth feature sequence, the electronic device can determine the similarity between the video data to be matched and the one or more candidate video data regarding emotional changes, i.e., the third result; wherein, cosine distance can be used to measure the similarity between the video data to be matched and each candidate video data:

[0116]

[0117] Where A represents the ninth feature sequence, A = {A} i |A i ∈R m}, i = 1, 2, ..., n; B represents the eighth feature subsequence, B = {B i |B i ∈R m}, i = 1, 2, ..., n.

[0118] In practical applications, the electronic device can perform linear fusion processing on the third result and the fourth result to obtain a fifth result corresponding to each candidate audio data. The fifth result represents the degree of matching between the video to be matched and the candidate audio data.

[0119] For example, suppose the i-th video segment v of the video data v to be matched is... i The characteristics of emotional changes are: The i-th video segment v i The fifth characteristic sequence is ve i The i-th audio segment m of candidate audio data m i The eighth feature subsequence of the corresponding candidate audio data is The i-th audio segment m i The corresponding sixth feature subsequence is me i Using formulas (4) and (5), the electronic device can obtain the matching score between the candidate audio data and the video data to be matched, i.e., the fifth result; wherein, the fifth result can be obtained by the following formula:

[0120]

[0121] Wherein, α1 represents the weight of the similarity between the video data to be matched and the candidate audio data regarding emotional changes in the linear fusion, and can be the confidence level of the candidate video data obtained by fitting the candidate audio data; α2 represents the similarity between the video data to be matched and the candidate audio data regarding emotional changes, and can be the proportion of emotional classification in the candidate audio data.

[0122] In practical applications, the electronic device can sequentially determine the matching degree between each candidate audio data and the video to be matched using formula (6), and arrange the matching degree in descending order, thereby selecting the candidate audio data with the highest matching degree as the audio data that matches the video data to be matched.

[0123] The data processing method provided in this application embodiment determines a first feature sequence of audio sample data and a second feature sequence of video sample data. The first feature sequence represents the emotional changes of N audio segments in the audio sample data, and the second feature sequence represents the emotional changes of N video segments in the video sample data, where N is an integer greater than or equal to 1. The first feature sequence and the second feature sequence are clustered to obtain a first feature dictionary and a second feature dictionary. The first feature dictionary contains a set of one or more audio segments with similar emotional changes in the audio sample data, and the second feature dictionary contains a set of one or more video segments with similar emotional changes in the video sample data. A first relation is obtained using the first feature dictionary and the second feature dictionary. The first relation contains the mapping relationship between the first feature sequence and the second feature sequence in the semantic space. The first relation is then used to select matching audio data for the video data to be matched. The technical solution provided in this application embodiment constructs an audio-video feature dictionary by clustering the features of emotional fluctuations in short segments of audio-video sample data. Through the audio-video feature dictionary, a mapping relationship is established between the features of emotional fluctuations in the semantic space, so as to better match music that conforms to the emotional change trend for the video to be matched, thereby improving the matching degree between the video to be matched and the music.

[0124] The following section provides a more detailed description of this application with reference to application examples.

[0125] In the application examples of this application, for the task of adding background music to videos, based on the similar trend of the emotional fluctuations of the audio and video in short clips of music videos, the abstract semantics and underlying features are comprehensively considered and the relationship between audio and video is established so as to match music that matches the emotional flow of the video to be set to music.

[0126] Specifically, such as Figure 3As shown, in the process of modeling low-order feature relationships between audio and video, firstly, the video sample set (i.e., the audio sample data mentioned above) and the audio sample set (i.e., the video sample data mentioned above) are preprocessed respectively. Then, video feature engineering is used to extract the video optical flow motion (i.e., the fourth feature mentioned above) and shot switching frequency (i.e., the fifth feature mentioned above) of the video segments as video emotional fluctuation features (i.e., the fourth feature sequence mentioned above). At the same time, audio feature engineering is used to extract the audio energy (i.e., the first feature mentioned above), spectral flux (i.e., the second feature mentioned above), and auditory roughness (i.e., the third feature mentioned above) of the audio segments as audio emotional fluctuation features (i.e., the third feature sequence mentioned above). Secondly, through the clustering of audio emotional fluctuation features, an audio feature dictionary (i.e., the first feature dictionary mentioned above) and a video feature dictionary (i.e., the second feature dictionary mentioned above) are constructed respectively to provide structural information of the feature space for subsequent audio-video relationship modeling. Next, based on the distribution of video and audio emotional fluctuation features in the feature space, mid-level features are extracted (i.e., the first and second results mentioned above). Canonical correlation analysis is then used to learn the mapping relationship between the mid-level features of the audio and video segments, thereby establishing a relational model (i.e., the first relation mentioned above). During the use of the relational model, pseudo-video features (i.e., the eighth feature sequence mentioned above) can be fitted using the musical features of the candidate music (i.e., the seventh feature sequence mentioned above). Then, the similarity between the pseudo-video features and the real video features of the video to be matched with music (i.e., the ninth feature sequence mentioned above) is calculated (i.e., the fourth result mentioned above), and the calculated similarity is used as the matching degree of the emotional fluctuation synchronization of the audio and video segments.

[0127] In the above process, by extracting the similarity between the audio emotional fluctuation features and the cluster centers of each cluster in the feature dictionary (i.e., the first and second cluster centers mentioned above) as mid-level features, it is possible to simplify the complex feature space distribution caused by the concatenation of audio and video emotional fluctuation features, as well as the problem that it is difficult to directly approximate video features with music features through a relational model.

[0128] In addition, such as Figure 4As shown, in the process of high-order semantic space modeling of audio and video, firstly, the emotion classifier is transferred to a pre-trained visual emotion concept recognition model and music style recognition model. Then, the hyperparameters such as the learning rate and learning rate decay of each layer weight of the emotion classifier are adjusted using pre-processed video and music sample sets to establish a mapping relationship between audio and video data and emotional cognition. Secondly, a majority voting strategy is used to correct the prediction results of the emotion classifier, making the prediction results closer to the true emotion that the music or video intends to express. Next, the similarity between the audio emotion sequence of the candidate music (i.e., the sixth feature sequence mentioned above) and the video emotion sequence of the video to be paired with the music (i.e., the fifth feature sequence mentioned above) is calculated (i.e., the third result mentioned above), and the calculated similarity is used as the matching degree of emotional consistency.

[0129] It should be noted that the Hamming distance between audio and video sentiment sequences can be calculated using the XOR of two vectors. However, in order to maintain the same monotonicity as the similarity of audio and video emotional fluctuation features, this scheme calculates the similarity of audio and video sentiment sequences based on the modified Hamming distance.

[0130] Here, after obtaining the matching degree of emotional fluctuation synchronization and emotional consistency of audio and video clips, the confidence scores of these scores are used as weights to linearly fuse them, resulting in a matching degree between candidate music and the video to be set to music. Then, the music selection results for the video to be set to music are output based on the matching degree of the candidate music.

[0131] In the application examples of this application, music is selected for the video to be set to music based on the similarity of the emotional fluctuations of audio and video and the similarity of the emotional sequences. This ensures that the music selection results take into account the synchronization between the video's camera movements and the amplitude of the movement of the screen content and the speed, rhythm, and intensity of the music, while also ensuring the consistency of the user's visual and auditory sensory experience.

[0132] To implement the method of the embodiments of this application, the embodiments of this application also provide a data processing apparatus, disposed on an electronic device, such as... Figure 5 As shown, the device includes:

[0133] The first determining unit 501 is used to determine a first feature sequence of audio sample data and a second feature sequence of video sample data. The first feature sequence characterizes the emotional changes of N audio segments in the audio sample data, and the second feature sequence characterizes the emotional changes of N video segments in the video sample data, where N is an integer greater than or equal to 1.

[0134] Clustering unit 502 is used to cluster the first feature sequence and the second feature sequence to obtain a first feature dictionary and a second feature dictionary. The first feature dictionary contains a set of one or more audio segments with similar emotional changes in the audio sample data, and the second feature dictionary contains a set of one or more video segments with similar emotional changes in the video sample data.

[0135] The second determining unit 503 is used to obtain a first relation using the first feature dictionary and the second feature dictionary, wherein the first relation contains the mapping relationship between the first feature sequence and the second feature sequence in the semantic space;

[0136] Selection unit 504 is used to select matching audio data for the video data to be matched using the first relationship.

[0137] In one embodiment, the first determining unit 501 is configured to, for each of the N audio segments, determine a first feature, a second feature, and a third feature of the audio segment, and perform a fusion process on the first feature, the second feature, and the third feature to obtain a third feature sequence corresponding to the audio segment. The first feature represents the loudness change of the audio segment, the second feature represents the spectral energy change of the audio segment, and the third feature represents the degree of incoherence of the audio segment. The third feature sequence corresponding to the N audio segments is then fused to obtain a first feature sequence. For each of the N video segments, a fourth feature and a fifth feature of the video segment are determined, and the fourth feature and the fifth feature are then fused to obtain a fourth feature sequence corresponding to the video segment. The fourth feature represents the optical flow motion of the video segment, and the fifth feature represents the shot switching of the video segment. The fourth feature sequence corresponding to the N video segments is then fused to obtain a second feature sequence.

[0138] In one embodiment, the first determining unit 501 is configured to determine one or more first frames associated with an audio segment; perform feature extraction on each of the one or more first frames to obtain a first sub-feature, a second sub-feature, and a third sub-feature, wherein the first sub-feature characterizes the loudness change of the first frame, the second feature characterizes the spectral energy change of the first frame, and the third feature characterizes the degree of incoherence of the first frame; and perform fusion processing on the first sub-feature, the second sub-feature, and the third sub-feature of the one or more first frames to obtain the first feature, the second feature, and the third feature, respectively.

[0139] The first determining unit 501 is used to determine one or more second frames associated with the video segment; to extract features from each of the one or more second frames to obtain a fourth sub-feature and a fifth sub-feature, wherein the fourth sub-feature represents the optical flow motion of the second frame and the fifth sub-feature represents the shot switching of the second frame; and to fuse the fourth sub-feature and the fifth sub-feature of the one or more second frames to obtain the fourth feature and the fifth feature respectively.

[0140] In one embodiment, the second determining unit 503 is configured to determine one or more first cluster centers corresponding to the one or more audio segment sets, and determine one or more second cluster centers corresponding to the one or more video segment sets; update the first feature sequence using the one or more first cluster centers to obtain an updated first feature sequence; update the second feature sequence using the one or more second cluster centers to obtain an updated second feature sequence; and obtain the first relationship using the updated first feature sequence and the updated second feature sequence.

[0141] In one embodiment, the second determining unit 503 is configured to: use the first feature sequence to determine a first feature subsequence of the one or more audio segment sets, the first feature subsequence representing the emotional changes of the one or more audio segment sets; use the first feature subsequence and the one or more first cluster centers to determine a first result, the first result representing the similarity between the first feature subsequence and the one or more first cluster centers; and use the first result to update the first feature subsequence to obtain an updated first feature subsequence.

[0142] The second determining unit 503 is configured to use the second feature sequence to determine a second feature subsequence of the one or more video clip sets, the second feature subsequence representing the emotional changes of the one or more video clip sets; use the second feature subsequence and the one or more second cluster centers to determine a second result, the second result representing the similarity between the second feature subsequence and the one or more second cluster centers; and use the second result to update the second feature subsequence to obtain an updated second feature subsequence.

[0143] In one embodiment, the selection unit 504 is further configured to determine a fifth feature sequence corresponding to the video data to be matched, and to determine a sixth feature sequence corresponding to one or more candidate audio data, wherein the fifth feature sequence characterizes the emotional changes of the video data to be matched, and the sixth feature sequence characterizes the emotional changes of the one or more candidate audio data; using the fifth feature sequence and the sixth feature sequence, a third result is obtained, wherein the third result characterizes the similarity between the video data to be matched and the one or more candidate audio data regarding emotional changes; using the third result and the first relationship, audio data matching the video data to be matched is selected from the one or more candidate audio data.

[0144] In one embodiment, the selection unit 504 is configured to: determine one or more candidate video data using the one or more candidate audio data and the first relationship; determine a seventh feature sequence corresponding to the one or more candidate video data and determine an eighth feature sequence corresponding to the video data to be matched, wherein the seventh feature sequence characterizes the emotional changes of the one or more candidate video data and the eighth feature sequence characterizes the emotional changes of the video data to be matched; obtain a fourth result using the seventh feature sequence and the eighth feature sequence, wherein the fourth result characterizes the similarity between the video data to be matched and the one or more candidate video data regarding emotional changes; and select audio data matching the video data to be matched from the one or more candidate audio data using the third result and the fourth result.

[0145] In practical applications, the first determining unit 501, the clustering unit 502, the second determining unit 503, and the selection unit 504 can be implemented by a processor in a data processing device.

[0146] It should be noted that the data processing apparatus provided in the above embodiments is only illustrated by the division of the above-described program units. In practical applications, the above processing can be assigned to different program units as needed, that is, the internal structure of the apparatus can be divided into different program units to complete all or part of the processing described above. In addition, the data processing apparatus and data processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0147] Based on the hardware implementation of the above program modules, and in order to implement the data processing method of the embodiments of this application, the embodiments of this application also provide an electronic device, such as... Figure 6 As shown, the electronic device 600 includes:

[0148] The communication interface 601 enables interaction with other devices;

[0149] The processor 602 is connected to the communication interface 601 to enable interaction with other devices and to execute the methods provided by one or more of the above-mentioned technical solutions when running a computer program;

[0150] The computer program is stored in memory 603.

[0151] Specifically, the processor 602 is configured to determine a first feature sequence of audio sample data and a second feature sequence of video sample data, wherein the first feature sequence characterizes the emotional changes of N audio segments in the audio sample data, and the second feature sequence characterizes the emotional changes of N video segments in the video sample data, where N is an integer greater than or equal to 1; cluster the first feature sequence and the second feature sequence to obtain a first feature dictionary and a second feature dictionary, wherein the first feature dictionary contains a set of one or more audio segments with similar emotional changes in the audio sample data, and the second feature dictionary contains a set of one or more video segments with similar emotional changes in the video sample data; use the first feature dictionary and the second feature dictionary to obtain a first relation, wherein the first relation contains the mapping relationship between the first feature sequence and the second feature sequence in the semantic space; and use the first relation to select matching audio data for the video data to be matched.

[0152] In one embodiment, the processor 602 is configured to: determine a first feature, a second feature, and a third feature for each of N audio segments; fuse the first feature, the second feature, and the third feature to obtain a third feature sequence corresponding to the audio segment; the first feature characterizes the loudness change of the audio segment, the second feature characterizes the spectral energy change of the audio segment, and the third feature characterizes the degree of incoherence of the audio segment; fuse the third feature sequences corresponding to the N audio segments to obtain a first feature sequence; determine a fourth feature and a fifth feature for each of N video segments; fuse the fourth feature and the fifth feature to obtain a fourth feature sequence corresponding to the video segment; the fourth feature characterizes the optical flow motion of the video segment, and the fifth feature characterizes the shot switching of the video segment; fuse the fourth feature sequences corresponding to the N video segments to obtain a second feature sequence.

[0153] In one embodiment, the processor 602 is configured to determine one or more first frames associated with an audio segment; extract features from each of the one or more first frames to obtain a first sub-feature, a second sub-feature, and a third sub-feature, wherein the first sub-feature characterizes the loudness change of the first frame, the second feature characterizes the spectral energy change of the first frame, and the third feature characterizes the degree of incoherence of the first frame; and fuse the first sub-feature, the second sub-feature, and the third sub-feature of the one or more first frames to obtain the first feature, the second feature, and the third feature, respectively.

[0154] In one embodiment, the processor 602 is configured to determine one or more second frames associated with a video segment; extract features from each of the one or more second frames to obtain a fourth sub-feature and a fifth sub-feature, wherein the fourth sub-feature characterizes the optical flow motion of the second frame and the fifth sub-feature characterizes the shot switching of the second frame; and fuse the fourth sub-feature and the fifth sub-feature of the one or more second frames to obtain the fourth feature and the fifth feature, respectively.

[0155] In one embodiment, the processor 602 is configured to determine one or more first cluster centers corresponding to the one or more audio segment sets, and determine one or more second cluster centers corresponding to the one or more video segment sets; update the first feature sequence using the one or more first cluster centers to obtain an updated first feature sequence; update the second feature sequence using the one or more second cluster centers to obtain an updated second feature sequence; and obtain the first relationship using the updated first feature sequence and the updated second feature sequence.

[0156] In one embodiment, the processor 602 is configured to: use the first feature sequence to determine a first feature subsequence of the one or more audio segment sets, the first feature subsequence representing the emotional changes of the one or more audio segment sets; use the first feature subsequence and the one or more first cluster centers to determine a first result, the first result representing the similarity between the first feature subsequence and the one or more first cluster centers; and use the first result to update the first feature subsequence to obtain an updated first feature subsequence.

[0157] In one embodiment, the processor 602 is configured to use the second feature sequence to determine a second feature subsequence of the one or more video clip sets, the second feature subsequence representing the emotional changes of the one or more video clip sets; use the second feature subsequence and the one or more second cluster centers to determine a second result, the second result representing the similarity between the second feature subsequence and the one or more second cluster centers; and use the second result to update the second feature subsequence to obtain an updated second feature subsequence.

[0158] In one embodiment, the processor 602 is further configured to determine a fifth feature sequence corresponding to the video data to be matched, and to determine a sixth feature sequence corresponding to one or more candidate audio data, wherein the fifth feature sequence characterizes the emotional changes of the video data to be matched, and the sixth feature sequence characterizes the emotional changes of the one or more candidate audio data; using the fifth feature sequence and the sixth feature sequence, a third result is obtained, wherein the third result characterizes the similarity between the video data to be matched and the one or more candidate audio data regarding emotional changes; and using the third result and the first relationship, audio data matching the video data to be matched is selected from the one or more candidate audio data.

[0159] In one embodiment, the processor 602 is configured to: determine one or more candidate video data using the one or more candidate audio data and the first relationship; determine a seventh feature sequence corresponding to the one or more candidate video data and determine an eighth feature sequence corresponding to the video data to be matched, wherein the seventh feature sequence characterizes the emotional changes of the one or more candidate video data and the eighth feature sequence characterizes the emotional changes of the video data to be matched; obtain a fourth result using the seventh feature sequence and the eighth feature sequence, wherein the fourth result characterizes the similarity between the video data to be matched and the one or more candidate video data regarding emotional changes; and select audio data matching the video data to be matched from the one or more candidate audio data using the third result and the fourth result.

[0160] It should be noted that the specific processing procedures of the processor 602 and the communication interface 601 can be understood with reference to the above method.

[0161] Of course, in practical applications, the various components in electronic device 600 are coupled together through bus system 604. It can be understood that bus system 604 is used to realize the connection and communication between these components. In addition to a data bus, bus system 604 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 6 The general designated all buses as Bus System 604.

[0162] The memory 603 in this embodiment is used to store various types of data to support the operation of the electronic device 600. Examples of such data include any computer program used to operate on the electronic device 600.

[0163] The methods disclosed in the embodiments of this application can be applied to the processor 602, or implemented by the processor 602. The processor 602 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 602 or by instructions in the form of software. The processor 602 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 602 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the memory 603. The processor 602 reads the information in the memory 603 and combines its hardware to complete the steps of the aforementioned method.

[0164] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0165] It is understood that the memory (memory 603) in this embodiment of the application can be volatile memory or non-volatile memory, or both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be disk storage or magnetic tape storage. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0166] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 603 storing a computer program, which can be executed by the processor 602 of the electronic device 600 to complete the steps described in the aforementioned data processing method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disk, or CD-ROM; the magnetic surface memory may be a disk storage device or a magnetic tape storage device.

[0167] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by a first processor 602 of an electronic device 600 to complete the steps described in the aforementioned data processing method.

[0168] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0169] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0170] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A data processing method, characterized by, The method comprises: determining a first feature sequence of audio sample data and a second feature sequence of video sample data, the first feature sequence representing emotion changes of N audio segments in the audio sample data, and the second feature sequence representing emotion changes of N video segments in the video sample data, N being an integer greater than or equal to 1; clustering the first feature sequence and the second feature sequence to obtain a first feature dictionary and a second feature dictionary, the first feature dictionary containing one or more audio segment sets with similar emotion changes in the audio sample data, and the second feature dictionary containing one or more video segment sets with similar emotion changes in the video sample data; using the first feature dictionary and the second feature dictionary to obtain a first relationship, the first relationship containing a mapping relationship between the first feature sequence and the second feature sequence in a semantic space; using the first relationship to select matching audio data for to-be-matched video data.

2. The method of claim 1, wherein, The method comprises: determining a first feature sequence of audio sample data and a second feature sequence of video sample data, the first feature sequence representing emotion changes of N audio segments in the audio sample data, and the second feature sequence representing emotion changes of N video segments in the video sample data, N being an integer greater than or equal to 1; clustering the first feature sequence and the second feature sequence to obtain a first feature dictionary and a second feature dictionary, the first feature dictionary containing one or more audio segment sets with similar emotion changes in the audio sample data, and the second feature dictionary containing one or more video segment sets with similar emotion changes in the video sample data; 3. The method of claim 2, wherein, using the first feature dictionary and the second feature dictionary to obtain a first relationship, the first relationship containing a mapping relationship between the first feature sequence and the second feature sequence in a semantic space; using the first relationship to select matching audio data for to-be-matched video data. The method comprises: determining a first feature sequence of audio sample data and a second feature sequence of video sample data, the first feature sequence representing emotion changes of N audio segments in the audio sample data, and the second feature sequence representing emotion changes of N video segments in the video sample data, N being an integer greater than or equal to 1; clustering the first feature sequence and the second feature sequence to obtain a first feature dictionary and a second feature dictionary, the first feature dictionary containing one or more audio segment sets with similar emotion changes in the audio sample data, and the second feature dictionary containing one or more video segment sets with similar emotion changes in the video sample data; using the first feature dictionary and the second feature dictionary to obtain a first relationship, the first relationship containing a mapping relationship between the first feature sequence and the second feature sequence in a semantic space; using the first relationship to select matching audio data for to-be-matched video data. The method comprises: determining a first feature sequence of audio sample data and a second feature sequence of video sample data, the first feature sequence representing emotion changes of N audio segments in the audio sample data, and the second feature sequence representing emotion changes of N video segments in the video sample data, N being an integer greater than or equal to 1; clustering the first feature sequence and the second feature sequence to obtain a first feature dictionary and a second feature dictionary, the first feature dictionary containing one or more audio segment sets with similar emotion changes in the audio sample data, and the second feature dictionary containing one or more video segment sets with similar emotion changes in the video sample data; using the first feature dictionary and the second feature dictionary to obtain a first relationship, the first relationship containing a mapping relationship between the first feature sequence and the second feature sequence in a semantic space; using the first relationship to select matching audio data for to-be-matched video data. The method comprises: determining a first feature sequence of audio sample data and a second feature sequence of video sample data, the first feature sequence representing emotion changes of N audio segments in the audio sample data, and the second feature sequence representing emotion changes of N video segments in the video sample data, N being an integer greater than or equal to 1; clustering the first feature sequence and the second feature sequence to obtain a first feature dictionary and a second feature dictionary, the first feature dictionary containing one or more audio segment sets with similar emotion changes in the audio sample data, and the second feature dictionary containing one or more video segment sets with similar emotion changes in the video sample data; using the first feature dictionary and the second feature dictionary to obtain a first relationship, the first relationship containing a mapping relationship between the first feature sequence and the second feature sequence in a semantic space; using the first relationship to select matching audio data for to-be-matched video data. The fourth and fifth sub-features of the one or more second frames are fused to obtain the fourth feature and the fifth feature, respectively.

4. The method of claim 1, wherein, The step of obtaining the first relationship using the first feature dictionary and the second feature dictionary includes: Determine one or more first cluster centers corresponding to the one or more audio segment sets, and determine one or more second cluster centers corresponding to the one or more video segment sets; The first feature sequence is updated using the one or more first cluster centers to obtain the updated first feature sequence; The second feature sequence is updated using the one or more second cluster centers to obtain the updated second feature sequence; The first relationship is obtained using the updated first feature sequence and the updated second feature sequence.

5. The method of claim 4, wherein, The step of updating the first feature sequence using the one or more first cluster centers to obtain the updated first feature sequence includes: Using the first feature sequence, a first feature subsequence is determined for the set of one or more audio segments, the first feature subsequence representing the emotional changes of the set of one or more audio segments; Using the first feature subsequence and the one or more first cluster centers, a first result is determined, wherein the first result characterizes the similarity between the first feature subsequence and the one or more first cluster centers; The first feature subsequence is updated using the first result to obtain the updated first feature subsequence; The step of updating the second feature sequence using the one or more second cluster centers to obtain the updated second feature sequence includes: Using the second feature sequence, a second feature subsequence is determined for the set of one or more video clips, the second feature subsequence representing the emotional changes of the set of one or more video clips; Using the second feature subsequence and the one or more second cluster centers, a second result is determined, wherein the second result characterizes the similarity between the second feature subsequence and the one or more second cluster centers; The second feature subsequence is updated using the second result to obtain the updated second feature subsequence.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: A fifth feature sequence corresponding to the video data to be matched is determined, and a sixth feature sequence corresponding to one or more candidate audio data is determined. The fifth feature sequence represents the emotional change of the video data to be matched, and the sixth feature sequence represents the emotional change of the one or more candidate audio data. Using the fifth feature sequence and the sixth feature sequence, a third result is obtained, which characterizes the similarity between the video data to be matched and the one or more candidate audio data regarding emotional changes; Using the third result and the first relationship, audio data that matches the video data to be matched is selected from the one or more candidate audio data.

7. The method of claim 6, wherein, The step of selecting audio data that matches the video data to be matched from the one or more candidate audio data using the third result and the first relationship includes: Using the seventh feature sequence corresponding to the one or more candidate audio data and the first relationship, an eighth feature sequence corresponding to one or more candidate video data is determined, wherein the seventh feature sequence characterizes the emotional changes of the one or more candidate audio data and the eighth feature sequence characterizes the emotional changes of the one or more candidate video data; Using the eighth feature sequence and the ninth feature sequence corresponding to the video data to be matched, a fourth result is obtained. The ninth feature sequence characterizes the emotional changes of the video data to be matched, and the fourth result characterizes the similarity between the video data to be matched and the one or more candidate video data regarding emotional changes. Using the third and fourth results, audio data that matches the video data to be matched is selected from the one or more candidate audio data.

8. An electronic device, comprising: include: The processor and the memory used to store computer programs that can run on the processor. When the processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 7.

9. A storage medium having stored thereon a computer program, characterized in that When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images