Audio and video mutual retrieval method based on user click behaviors

Abstract

The invention discloses an audio and video mutual retrieval method based on user click behaviors. The method comprises the following steps: preprocessing input audio and video data; Sending the preprocessed audio data into a deep convolutional neural network to obtain an audio representation vector and attention weight distribution; Sending the preprocessed video key frame into a deep convolutional neural network to obtain a key frame representation vector, and sequentially sending the key frame representation vector into a time sequence processing network based on an attention mechanism to obtain a representation vector of the video and attention weight distribution; Calculating the similarity of the audio and video representation vectors and sorting the audio and video according to the similarity; performing Annotating according to the attention weight distribution to provide explainable basis for sorting; Calculating the loss function through a user click behavior, and carrying outmodel training by adopting a backward propagation method; And carrying out retrieval matching on audios and videos in the media library based on the trained model. According to the method and the device, matched audios and videos in the media library can be retrieved under the condition of giving videos and audios.

Description

technical field [0001] The invention relates to data retrieval technology, in particular to a method for mutual retrieval of audio and video based on user click behavior. Background technique [0002] With the rapid development of the Internet industry, especially the mobile Internet industry, a large amount of audio and video content has been produced, and how to deal with these audio and video contents has become an urgent problem to be solved. In the past, finding matching audio for video or finding matching video for audio and editing it into a complete work was often only the needs of practitioners in the audio and video industry. For professionals, they can rely on their exposure and professional understanding of a large amount of audio and video content to organically combine audio and video. But even so, people's memory for audio and video content is still limited, and the size of the audio and video candidate library is limited by human memory, so it is difficult t...

AI Technical Summary

Problems solved by technology

Although this method has good interpretability, it only uses one or more classification results to match, the basis is relatively monotonous, and the effect is often not good

Images