A person recognition method and system based on a multi-frame audio-video fusion network

Abstract

The present invention proposes a method and system for character recognition based on a multi-frame audio-video fusion network, which is characterized in that it includes: a visual feature fusion step, decoding a video to be recognized, obtaining continuous K frames of the video, and extracting the continuous K frames For the face features of each frame in , all the face features are weighted and fused to obtain multi-frame visual features, K is a positive integer; the voiceprint feature fusion step extracts the voiceprint features of each frame in the continuous K frames, using The time recurrent neural network fuses all the voiceprint features to obtain multi-frame voiceprint features; the audio and video feature fusion step uses the fully connected layer to fuse the multi-frame visual features and the multi-frame voiceprint features, and uses the classification loss to constrain the fusion process , the multi-frame audio-video fusion feature is obtained, and the person recognition is performed according to the multi-frame audio-video fusion feature.

Description

technical field [0001] The present invention relates to the field of character recognition, and in particular to a character recognition method and system based on a multi-frame audio-video fusion network. Background technique [0002] Person recognition in video mainly uses the intrinsic or extrinsic attributes of the person to determine its identity. At present, the commonly used method is to use the biological characteristics of the human body, such as human face, voiceprint, etc., to identify the identity of the person. The corresponding algorithms include face recognition algorithm, voiceprint recognition algorithm and so on. Mainstream face recognition algorithms use convolutional neural networks to learn a mapping from raw face images to identity-invariant features from large-scale face datasets. Researchers often carefully design different loss functions, such as bigram loss, triplet loss, and center loss, to constrain the mapping process from images to features. ...

AI Technical Summary

Problems solved by technology

At present, the person recognition algorithm based on audio and video fusion can make full use of the information of face features and voiceprint features to determine the identity of the person, but the fusion algorithm fails to solve the problem of the decline in the discrimination of visual features in low-quality situations

[0004] When the inventor was conducting research on person recognition in the field of network audio and video monitoring, he found the following defects in the existing technology: one is that the algorithm based on single modality is difficult to solve the practical problems of complex network audio and video monitoring, and the face recognition algorithm works at low The degradation of high-quality images is serious, and the recognition accuracy of voiceprint recognition algorithms is also limited; second, in the field of network audio and video surveillance, there are often a large number of difficult-to-recognize images

Directly extracting face features from these difficult-to-recognition pictures will lead to a decrease in the ability to distinguish features, which will affect the effect of subsequent fusion features

Images