An audio-video matching method and system based on federated transfer learning

Abstract

An audio-video matching method and system based on federal transfer learning, the method disclosed by the application trains a pre-training model based on a large amount of emotional data, and a personal user can obtain an audio-video matching model suitable for his own needs based on federal transfer learning on the pre-training model using a small amount of personal data. In the training process of the application, data is isolated, the risk of data leakage that may exist in audio-video matching is solved, and the copyright and privacy of the user are protected. At the same time, the quality of the model is guaranteed to be transmitted without loss, and negative transfer does not occur.

Description

Technical Field

[0001] This invention belongs to the field of audio and video processing technology, and more specifically, relates to an audio and video matching method and system based on federated transfer learning. Background Technology

[0002] Audio-video matching refers to the process of combining or matching audio and video according to a certain standard or purpose. Audio-video matching has many applications, such as in movies, TV series, advertisements, games, and education. The quality of audio-video matching directly affects the viewer's perception and emotional experience; therefore, audio-video matching is an important technical issue.

[0003] Traditional audio-video matching methods primarily rely on manual editing or content-based analysis. While manual editing can ensure matching quality, it is time-consuming, labor-intensive, and unsuitable for large-scale and personalized applications. Content-based analysis extracts features from audio and video, such as rhythm, tempo, hue, and scene, and then performs matching based on algorithms or rules. Although this method offers automation and speed, it neglects the emotional connections between audio and video, failing to reflect the deeper meaning and value of audio-video matching.

[0004] In real-world applications, owners of large amounts of audio and video resources, such as enterprise users, often cannot train models to meet the needs of all users. Meanwhile, individual users, with only a small amount of personal data, cannot address the large-scale data requirements of deep learning training. Furthermore, individual users are concerned about the leakage of their privacy data, making it difficult for existing technical solutions to meet the needs of individual users.

[0005] Chinese patent CN112348063A discloses a model training method and apparatus based on federated transfer learning in the Internet of Things (IoT). The method includes: a data holder initiating a query to a local server for available machine learning training models; the local server querying whether there is an available machine learning training model that matches the data holder's request. However, this method is not suitable for voice-over video matching scenarios and does not solve the problem of limited computing power for individual users. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides an audio-video matching method and system based on emotional features. The method trains a pre-trained model using a large amount of emotional data. Individual users can then use federated transfer learning on this pre-trained model, employing only a small amount of personal data, to obtain an audio-video matching model suitable for their needs. Furthermore, due to the characteristics of federated learning, the user's privacy information is completely stored locally, eliminating any risk of privacy leakage.

[0007] The present invention adopts the following technical solution.

[0008] An audio / video matching method based on federated transfer learning includes:

[0009] Step 1, data preprocessing: preprocess the audio and video separately, converting the audio into a spectrogram and the video into a 16-layer RGB image.

[0010] Step 2: Construct neural network models. The audio network is mainly composed of ResNet18, and the video network is mainly composed of 3D-ResNet50.

[0011] Step 3: Use cross-entropy loss to train the classification task and obtain the parameters of the pre-trained federated model;

[0012] Step 4: The user downloads the pre-trained federated model parameters from the server to their local machine, performs forward propagation on the pre-trained federated model parameters based on the local data to obtain the gradient, and then encrypts and sends the gradient back.

[0013] Step 5: The server updates the model parameters based on the gradient results returned by the client;

[0014] Step 6: The user downloads the updated model from the server, repeats steps 4 and 5, until the user-preset number of iterations is reached, and the model training is completed.

[0015] Step 7: Use the trained model to perform cross-modal audio and video matching.

[0016] Preferably, in step 1, the audio preprocessing includes the following steps:

[0017] 1) For an audio signal with a sampling rate of sr and a duration of t seconds, perform windowing; add a 10ms Hamming window to the signal every 25ms to obtain w(n);

[0018]

[0019] 2) Perform a Fast Fourier Transform on the windowed signal to obtain the amplitude spectrum W of the Mel filter. k ;

[0020] W k =FFT(w(n))

[0021] 5) Perform discrete cosine transform on the amplitude spectrum and take the logarithm of the result to obtain the logarithmic Mel filter bank coefficients M(m) of the audio signal;

[0022]

[0023] In the formula, k represents the number of sampling points, and N represents the total number of sampling points;

[0024] 6) Scale the spectrogram to 256x100 to obtain the audio preprocessing results.

[0025] Preferably, in step 1, the video preprocessing includes the following steps:

[0026] 1) For the first 64 seconds of the video, perform uniform frame extraction at a fixed frequency; if the video length is less than 64 seconds, fill the subsequent images with pure black content; for a video with an HxW frame, a 16-layer RGB image of 16x3xHxW can be obtained.

[0027] 2) Scale the extracted frame image from HxW to 224x224 to obtain a 16x3x224x224 video preprocessing result.

[0028] Preferably, in step 2, the input to the audio network is the 256x100 spectrogram obtained in step 1, which is processed by the ResNet18 network to obtain a 256-dimensional feature vector.

[0029] The input to the video network is a 16x3x224x224 RGB image with 16 layers obtained in step 1. After processing by the 3D-Resnet50 network, a 2048-dimensional feature vector is obtained, and then a 256-dimensional feature vector is obtained through a fully connected layer.

[0030] Preferably, in step 3, 32 samples in a batch are audio input and 32 samples are video input. After forward propagation based on the network structure in step 2, a 32x256 audio feature vector and a 32x256 video feature vector are obtained. These are then concatenated at the sample level to obtain a 64x256 fused feature vector. The network weights are updated by calculating the cross-entropy loss between the data label results and the fused feature vector to obtain the pre-trained model.

[0031] Preferably, in step 4, the gradient is encrypted using the following steps:

[0032] 1) Randomly select two prime numbers p and q that are greater than 1, and p and q satisfy pq and (p-1)(q-1).

[0033] Coprime;

[0034] 2) Let n = pq, and λ be the least common multiple of p-1 and q-1;

[0035] 3) Randomly select an integer g between 1 and the square of n;

[0036] 4) Order If μ does not exist, then select p and q again and repeat the above steps until μ exists;

[0037] 5) For all gradients m that need to be backpropagated, calculate c = g m ·r n mod n 2 The encrypted text 'c' is obtained and sent to the server.

[0038] Preferably, in step 5, the model parameters are updated using stochastic gradient descent.

[0039]

[0040] Where W represents the network parameters that need to be updated, L represents the forward propagation loss, and η represents the learning rate.

[0041] An audio-video matching system based on federated transfer learning includes a preprocessing module, a neural network construction module, a training module, and an audio-video cross-modal matching module.

[0042] The preprocessing module preprocesses the audio and video separately, converting the audio into a 256x100 spectrogram and the video into a 16x3x224x224 RGB image with 16 layers.

[0043] The neural network construction module builds neural network models, with the audio network primarily using ResNet18 and the video network primarily using 3D-ResNet50.

[0044] The training module trains a pre-trained model using cross-entropy loss for classification tasks, resulting in a pre-trained model. The server updates the model parameters based on the gradient results returned by the client, resulting in a new model.

[0045] The audio-video cross-modal matching module performs cross-matching of video and audio using cosine similarity.

[0046] The beneficial effects of this invention are that, compared with existing technologies, it isolates data during training, thus resolving the potential data leakage risks in audio and video matching and protecting user copyright and privacy. Simultaneously, it ensures lossless transmission of model quality, preventing negative transfer. Since current datasets in the audio and video sentiment matching field are relatively small, federated transfer performs better than independent training of the model by the user. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the audio / video matching method of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this invention. The embodiments described in this application are merely some embodiments of this invention, and not all embodiments. Based on the spirit of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of this invention.

[0049] An audio / video matching method based on federated transfer learning, such as Figure 1 As shown, it includes the following steps:

[0050] Step 1: Data preprocessing. Audio and video are preprocessed separately. Audio is converted into a 256x100 spectrogram, and video is converted into a 16x3x224x224 RGB image with 16 layers.

[0051] The audio preprocessing in step 1 includes the following steps:

[0052] 1. For audio with a sampling rate of sr (default sampling rate 48000Hz) and a duration of t seconds, perform windowing. Apply a 10ms Hamming window to the signal every 25ms to obtain w(n);

[0053]

[0054] 2. Perform a Fast Fourier Transform on the windowed signal to obtain the amplitude spectrum W of the Mel filter. k ;

[0055] W k =FFT(w(n))

[0056] 3. Perform a discrete cosine transform on the amplitude spectrum and take the logarithm of the result to obtain the logarithmic Mel filter bank coefficients M(m) of the audio signal. All M(m) constitute the Mel spectrum of the signal, with dimensions of 256×100t;

[0057]

[0058] In the formula, k represents the number of sampling points, and N represents the total number of sampling points;

[0059] 4. Scale the spectrogram to 256x100 to obtain the audio preprocessing result.

[0060] The video preprocessing in step 1 includes the following steps:

[0061] 1. For the first 64 seconds of the video, perform uniform frame skipping at a rate of 1 frame every 4 seconds. If the video length is less than 64 seconds, fill the subsequent images with pure black content. For a video with an HxW aspect ratio, this will result in a 16x3xHxW 16-layer RGB image.

[0062] 2. Scale the extracted frames from HxW to 224x224, resulting in a 16x3x224x224 video preprocessing result.

[0063] Step 2: Construct neural network models. The main audio network is ResNet18, and the main video network is 3D-ResNet50.

[0064] In step 2, the input to the audio network is the 256x100 spectrogram obtained in step 1, which is processed by the ResNet18 network to obtain a 256-dimensional feature vector.

[0065] In step 2, the input to the video network is a 16x3x224x224 RGB image with 16 layers obtained in step 1. After processing by the 3D-Resnet50 network, a 2048-dimensional feature vector is obtained, and then a 256-dimensional feature vector is obtained through a fully connected layer.

[0066] Let the weight parameters of the fully connected layer be W, and the result of 3DResnet50 be Input. Then the 256-dimensional feature vector of the video is:

[0067] Feature = W·Input

[0068] Step 3: Train the pre-trained model parameters. Use cross-entropy loss to train the classification task and obtain the pre-trained federated model parameters.

[0069] In step 3, the pre-trained model parameters are trained using a batch size of 64. To fuse audio and video features into the same feature space, 32 samples in the batch are audio input and 32 samples are video input. After forward propagation based on the network structure in step 2, a 32x256 audio feature vector and a 32x256 video feature vector are obtained. These are concatenated along the batch dimension to obtain a 64x256 fused feature vector. The network weights are updated by calculating the cross-entropy loss between the data label results and the fused feature vector to obtain the pre-trained model parameters. Let N be the batch size; M be the number of class labels; y ic Let y be a sign function; if the class label of sample i is c, then y ic =1, otherwise y ic =0; p ic If we express the probability values ​​of the network output, then the expression for the loss function L is:

[0070]

[0071] Step 4: The user downloads the pre-trained federated model parameters from the server to their local machine, performs forward propagation on the pre-trained federated model parameters based on the local data to obtain the gradient, and then encrypts and sends the gradient back.

[0072] In step 4, to prevent the encrypted data from being cracked, the following steps are used for gradient encryption.

[0073] 1. Randomly select two large prime numbers (prime numbers greater than 1,000,000,000), p and q, such that pq and (p-1)(q-1) are coprime.

[0074] 2. Let n = pq, and λ be the least common multiple of p-1 and q-1.

[0075] 3. Randomly select an integer g between 1 and the square of n.

[0076] 4. Order If μ does not exist, then reselect p and q and repeat the above steps until μ exists.

[0077] 5. For all gradients m that need to be backpropagated, calculate c = g m ·r n mod n 2 The encrypted text 'c' is obtained and sent to the server.

[0078] Step 5: The server updates the model parameters based on the gradient results returned by the client, and performs federated learning.

[0079] In step 5, the model parameters are updated using stochastic gradient descent, where W is the network parameter to be updated, L is the forward propagation loss, and η is the learning rate.

[0080]

[0081] Step 6: The user downloads the updated model from the server and repeats steps 4 and 5 until the user's preset iteration termination condition is met, thus completing model training.

[0082] In step 6, the user sets the termination condition according to the expected model performance, performs multiple transfer learning, gradient encryption backpropagation, federated training to update the model, and repeats the training until the model performance reaches the user's expected accuracy or loss value, then training stops and the model is obtained.

[0083] Step 7: Perform cross-modal audio and video matching.

[0084] In step 7, for the audio to be matched, a 256x100 network input is obtained using the method described in step 1. The neural network model trained in the above steps is used for feature extraction to obtain a 256-dimensional feature vector. The cosine similarity is calculated with all videos in the video library, and the video with the highest similarity is taken as the matching result.

[0085] In step 7, for the video to be matched, a 16x3x224x224 network input is obtained through the method described in step 1. The neural network model trained in the above steps is used to extract features to obtain a 256-dimensional feature vector. The cosine similarity is calculated with all audio in the audio library, and the audio with the highest similarity is taken as the matching result.

[0086] The formula for cosine similarity is:

[0087]

[0088] After the above steps, the user obtains an audio / video matching neural network model, which can be used to perform emotion-based matching and retrieval of any audio / video file. Throughout the process, user data is stored locally, and gradient uploads are performed via encrypted transmission, eliminating any risk of privacy breaches.

[0089] An audio-video matching system based on emotion features includes a preprocessing module, a neural network construction module, a training module, and an audio-video cross-modal matching module.

[0090] The preprocessing module preprocesses the audio and video separately, converting the audio into a 256x100 spectrogram and the video into a 16x3x224x224 RGB image with 16 layers.

[0091] The neural network construction module builds neural network models. The main audio network is ResNet18, and the main video network is 3D-ResNet50.

[0092] The training module trains a pre-trained model. It uses cross-entropy loss to train the model for a classification task, resulting in a pre-trained model. The server updates the model parameters based on the gradient results returned by the client, resulting in a new model.

[0093] The audio-video cross-modal matching module performs cross-matching of video and audio using cosine similarity.

[0094] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0095] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0096] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0097] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the claims of the present invention.

Claims

1. An audio-video matching method based on federated transfer learning, characterized in that, Comprise: Step 1, data preprocessing, audio and video are preprocessed respectively, audio is converted into frequency spectrum, and video is converted into 16-layer RGB image; Step 2, construct neural network model, audio network main body is Resnet18; Video network main body is 3D-Resnet50; Step 3, use cross entropy loss to train classification task, obtain pre-training federal model parameters; The pre-training model parameters are trained using a batch size of 64, 32 samples in the batch are audio inputs, and 32 samples are video inputs. After forward propagation based on the network structure in step 2, 32x256 audio feature vectors and 32x256 video feature vectors are obtained. The batch dimension is spliced to obtain a 64x256 fusion feature vector. The network weights are updated by calculating the cross-entropy loss between the data label result and the fusion feature vector to obtain the pre-training model parameters. Let N be the batch size, and M be the number of category labels. is a symbol function, when the category label of the sample is c, then , otherwise ; The probability value of the network output is expressed as P(y|x), and the expression of the loss function L is: ； N is the total number of sampling points; Step 4, user downloads pre-training federal model parameters from server to local, carries out forward propagation based on local data to obtain gradient, and returns the encrypted gradient; Step 5, server updates model parameters based on the gradient returned by the client; Step 6, user downloads the updated model from the server, repeats step 4 and step 5 until the user reaches the preset iteration number, and completes the model training; Step 7, use the trained model to perform audio and video cross-modal matching.

2. The audio and video matching method based on federal transfer learning according to claim 1, characterized in that: In step 1, the audio preprocessing includes the following steps: 1) for audio with sampling rate sr and time length t seconds, windowing; every 25 ms Hamming window is added to the signal, and w(n) is obtained; 2) Fast Fourier Transform is performed on the windowed signal to obtain the Mel filter amplitude spectrum W k ; Discrete cosine transform is performed on the amplitude spectrum, and the result is taken as logarithm to obtain the logarithmic Mel filter bank coefficient M(m) of the audio signal; Where k is the sampling point, and N is the total number of sampling points; The frequency spectrum is scaled to 256x100 to obtain the audio preprocessing result.

3. The audio and video matching method based on federal transfer learning according to claim 1, characterized in that: In step 1, the video preprocessing includes the following steps: For the first 64 seconds of video images, uniform frame extraction is performed at a fixed frequency; If the video length is less than 64 seconds, the latter images are filled with black content; For the video with picture HxW, 16-layer RGB image of 16x3xHxW is obtained; The frame extraction image is scaled from HxW to 224x224 to obtain the video preprocessing result of 16x3x224x224.

4. The audio and video matching method based on federal transfer learning according to claim 1 or 3, characterized in that: In step 2, the input of the audio network is the 256x100 frequency spectrum obtained in step 1, and the 256-dimensional feature vector is obtained after the Resnet18 network processing; The input of the video network is the 16-layer RGB image of 16x3x224x224 obtained in step 1, and the 2048-dimensional feature vector is obtained after the 3D-Resnet50 network processing, and then a fully connected layer is used to obtain the 256-dimensional feature vector.

5. The audio and video matching method based on federal transfer learning according to claim 1, characterized in that: In step 4, the gradient is encrypted by the following steps: 1) randomly select two prime numbers greater than 1000000000, p and q, and p and q satisfy pq and (p-1)(q-1) are coprime; 2) let n=pq, and λ is the least common multiple of p-1 and q-1; 3) Randomly select an integer g between 1 and n squared; 4) Let μ = p / q If μ does not exist, then reselect p and q and repeat the above steps until μ exists; 5) For all gradients m that need to be returned, compute c = m * r , get the ciphertext c, and send it to the server.

6. The audio-video matching method based on federal transfer learning according to claim 1, characterized in that: In step 5, the model parameters are updated by using the stochastic gradient descent method, where W is the network parameter to be updated, L is the forward propagation loss, and η is the learning rate.

7. An audio-video matching system based on federal transfer learning, used to implement the audio-video matching method based on federal transfer learning according to any one of claims 1-6, comprising a preprocessing module, a neural network construction module, a training module, and an audio-video cross-modal matching module, characterized in that: The preprocessing module pre-processes the audio and video respectively, converts the audio into a 256x100 spectrum graph, and converts the video into 16 RGB images of 3x224x224 layers; The neural network construction module constructs a neural network model, and the audio network main body is Resnet18; the video network main body is 3D-Resnet50; The training module trains the pre-trained model, uses the cross-entropy loss to train the classification task, and obtains the pre-trained model; the server updates the model parameters based on the gradient results returned by the client, and obtains a new model; The audio-video cross-modal matching module cross-matches the video and audio by using the cosine similarity.

8. A terminal comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is used to operate according to the instructions to perform the steps of the method according to any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method according to any one of claims 1-6.

Images