Speaker recognition method, apparatus, electronic device, medium, and program product

Abstract

The application provides a speaker recognition method and device, electronic equipment, medium and program product. The speaker recognition method comprises the following steps: performing scene detection on a to-be-identified video, dividing the to-be-identified video into multiple video segments based on the result of the scene detection; for each video segment, separating audio data and video frames in the video segment; extracting face features of the video frames and audio features of the audio data; for multiple video frames of scene switching in the multiple video segments, extracting face depth features of the multiple video frames, calculating distances between face depth features of adjacent video frames in the multiple video frames, and obtaining cross-scene distance features; and identifying a speaker from a face included in the video segment based on the cross-scene distance features, the face features and the audio features. The speaker recognition is performed scene by scene, the face correlation is realized in combination with the cross-scene features, and the problem of poor speaker recognition accuracy caused by scene switching is overcome.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio and video processing, and particularly relates to a speaker recognition method and device, electronic equipment, medium and program product. BACKGROUND

[0002] Speaker recognition refers to automatically recognizing different speakers appearing in an audio or video, so as to divide the audio or video into multiple segments according to different speakers. In a human conversation scene, quickly and accurately detecting a speaker and extracting the voice or video of the specified speaker have high application value in multimedia content creation, video customization, lip adjustment in different languages and other scenes.

[0003] In speaker recognition in a multi-person conversation video, related technologies usually regard the entire video as a whole, and recognize the speaker in the video through extraction and matching of visual features and audio features. The present inventors have found in research that in a multi-person conversation scene, the speaker frequently switches within a certain time or the speaker moves, resulting in poor continuity and integrity of the video. The way of recognizing the speaker by regarding the entire video as a whole has poor accuracy.

[0004] Therefore, there is an urgent need to provide a multi-person conversation video speaker recognition scheme with high accuracy. SUMMARY

[0005] The present application provides a speaker recognition method, device, electronic equipment, medium and program product, which divides a video into segments through scene detection, and recognizes the speaker in each segment, thereby overcoming the influence of scene switching on speaker recognition, using visual features extracted in the same scene and cross-scene dimensions, and matching the results with audio features to recognize the speaker, thereby improving the accuracy of recognition.

[0006] In a first aspect, the present application provides a speaker recognition method, comprising:

[0007] performing scene detection on a to-be-recognized video, and dividing the to-be-recognized video into multiple video segments based on the results of the scene detection;

[0008] for each video segment in the multiple video segments, separating audio data and video frames in the video segment;

[0009] extracting face features of the video frames and audio features of the audio data;

[0010] for multiple video frames of scene switching in the multiple video segments, extracting face depth features of the multiple video frames, and calculating distances between face depth features of adjacent video frames in the multiple video frames to obtain cross-scene distance features;

[0011] identify a speaker from the faces contained in the video segments based on the cross-scene distance feature, the face feature and the audio feature.

[0012] Optionally, for the multiple video frames of scene switching in the multiple video segments, face depth features of the multiple video frames are extracted, and distances of the face depth features of adjacent video frames in the multiple video frames are calculated to obtain the cross-scene distance feature, including:

[0013] For the multiple video frames of scene switching in the multiple video segments, face depth features of the multiple video frames and direction gradient histogram features of the face boxes are extracted;

[0014] The cross-scene fusion features of the video frames are obtained by fusing the face depth features and the direction gradient histogram features of the same video frame;

[0015] Distances of the cross-scene fusion features of adjacent video frames in the multiple video frames are calculated to obtain the cross-scene distance feature.

[0016] Optionally, the face feature is used to describe features of a face box corresponding to a face contained in a video frame, and a speaker is identified from the faces contained in the video segments based on the cross-scene distance feature, the face feature and the audio feature, including:

[0017] Based on the cross-scene distance feature, a connection relationship of the same face box in adjacent video frames of scene switching in the multiple video segments is determined, and based on the face feature, a connection relationship of the same face box in adjacent video frames in the same video segment is determined; the connection relationship of the face box is used to describe a positional relationship of the same face box in different video frames;

[0018] For each face box in the video segment, the face feature of the face box is obtained from the face feature of the video frame containing the face box according to the connection relationship of the face box, and the face feature matrix of the face box is obtained by splicing the face features of the face box in the video frames containing the face box;

[0019] A speaker is identified from the faces contained in the video segments based on the face feature matrix and the audio feature.

[0020] Optionally, based on the cross-scene distance feature, the connection relationship of the same face box in adjacent video frames of scene switching in the multiple video segments is determined, including:

[0021] The cross-scene distance features of adjacent video frames of scene switching in the multiple video segments are subjected to Hungarian matching to obtain the connection relationship of the same face box in adjacent video frames of scene switching in the multiple video segments.

[0022] Optionally, the face feature is used to describe a feature of a face frame corresponding to a face contained in a video frame, and before the speaker is identified from the face contained in the video segment based on the cross-scene distance feature, the face feature and the audio feature, the method further comprises:

[0023] For each face frame detected in the video segment, a lip key point of the face frame is extracted from a video frame corresponding to the face frame in the video segment;

[0024] Based on the lip key point of the face frame in the video frame corresponding to the face frame in the video segment, a lip offset of the face frame in the video segment is calculated;

[0025] The face frame with a lip offset less than a preset threshold in the video segment is removed, and the remaining face frame is used as a candidate face frame to identify the speaker from the candidate face frame.

[0026] Optionally, the audio feature of the audio data is extracted, comprising:

[0027] The audio data is input into a multi-language speech representation model to obtain the audio feature of the audio data; and a training set of the multi-language speech representation model comprises audio samples in multiple languages.

[0028] In a second aspect, the present application provides a speaker identification device, comprising:

[0029] A scene detection module is configured to perform scene detection on a to-be-identified video, and divide the to-be-identified video into a plurality of video segments based on the result of the scene detection;

[0030] An audio-video separation module is configured to separate audio data and video frames in each video segment of the plurality of video segments;

[0031] A first feature extraction module is configured to extract a face feature of the video frame and extract an audio feature of the audio data;

[0032] A second feature extraction module is configured to extract a face depth feature of a plurality of video frames in which a scene is switched in the plurality of video segments, and calculate a distance between face depth features of adjacent video frames in the plurality of video frames to obtain a cross-scene distance feature;

[0033] A speaker identification module is configured to identify a speaker from a face contained in the video segment based on the cross-scene distance feature, the face feature and the audio feature.

[0034] In a third aspect, the present application provides an electronic device, comprising a memory and at least one processor;

[0035] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the head-mounted display to perform the speaker recognition method provided in the first aspect of the present application.

[0036] In a fourth aspect, the present application provides a computer-readable storage medium, and the computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the speaker recognition method provided in the first aspect of the present application is implemented.

[0037] In a fifth aspect, the present application provides a computer program product, and the computer program product comprises a computer program, and when a processor executes the computer program, the speaker recognition method provided in the first aspect of the present application is implemented.

[0038] The speaker recognition method, device, electronic device, medium and program product provided by the present application divide the to-be-recognized video into multiple video segments through the detection result of the to-be-recognized video scene, separate the audio data and video frames of the video segments, and extract the face features in the video frames and the audio features in the audio data. Meanwhile, for multiple video frames across scenes, such as the last frame of a previous video segment and the first frame of a subsequent video segment in adjacent video segments, the face tracking at scene switching is implemented through the distance of the face depth features of the front and rear frames in different scenes. In the unit of a video segment in the same scene, the speaker in the video segment is recognized in combination with the face features extracted from each video frame in the video segment, the corresponding audio features and the result of cross-scene face tracking, and thus the speaker recognition in the entire video is implemented. The speaker recognition is performed scene by scene, the influence of scene switching on speaker recognition is overcome, the visual features extracted in the two dimensions of the same scene and the cross scene are utilized, the result of matching with the audio features is used to implement speaker recognition, and the accuracy of recognition is improved. BRIEF DESCRIPTION OF DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] Figure 1 FIG. 1 is a flowchart of the main process of a speaker recognition method;

[0041] Figure 2 FIG. 2 is a flowchart of a speaker recognition method provided by an embodiment of the present application;

[0042] Figure 3 FIG. 3 is a flowchart of another speaker recognition method provided by an embodiment of the present application;

[0043] Figure 4A schematic diagram of a process for processing adjacent video frames with different scenes by an embodiment of the present application;

[0044] Figure 5 A schematic diagram of a connection relationship of face frames in different video frames provided by an embodiment of the present application;

[0045] Figure 6 A schematic diagram of a speaker recognition network model provided by an embodiment of the present application;

[0046] Figure 7 A flowchart of another speaker recognition method provided by an embodiment of the present application;

[0047] Figure 8 A flowchart of another speaker recognition method provided by an embodiment of the present application;

[0048] Figure 9 A structural schematic diagram of a speaker recognition device provided by an embodiment of the present application;

[0049] Figure 10 A structural schematic diagram of an electronic device provided by an embodiment of the present application.

[0050] Through the above-mentioned drawings, the specific embodiments of the present application have been shown, and more detailed descriptions will be given hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application by any means, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0051] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments are not meant to represent all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the appended claims.

[0052] The wide spread of multimedia short videos promotes the development and expansion of social media platforms. Retrieving the required or related multimedia content from a large amount of audio and video information has great application value in content expression, material creation, and social interaction activities. Speaker recognition is a key technology in audio and video information retrieval. For example, in a multi-person dialogue video, extracting the audio and video of the speaker is an important basic technology for processing multi-person dialogue videos.

[0053] Figure 1 A flowchart of the main process of a speaker recognition method, such as Figure 1As shown, for a video of a to-be-identified speaker (referred to as a to-be-identified video), audio data in the to-be-identified video is first separated; MFCC (Mel Frequency Cepstrum Coefficient) features of the audio data are extracted, and two-dimensional features composed of the MFCC features and time are taken as audio features; the entire to-be-identified video is input into a deep learning network model, or the to-be-identified video is divided into a plurality of single-speaker videos such as video 1 to video n in FIG. 1 by face clustering, and face features of continuous video frames of the single-speaker videos are extracted; and speaker recognition is implemented by matching results of the face features and the audio features. Figure 1 The speaker recognition method has the following defects: because the total number of speakers is unknown, it is difficult to divide the single-speaker videos, and the detection error is large; when facing a scene in which speakers frequently switch, the MFCC features cannot be accurately extracted; when facing a scene in which speakers frequently switch and the speakers move, the single-speaker video analysis is performed on the entire video, and the segmentation accuracy is poor due to the frequent switching of the scene, resulting in poor speaker recognition accuracy.

[0054] The speaker recognition method has the following defects: because the total number of speakers is unknown, it is difficult to divide the single-speaker videos, and the detection error is large; when facing a scene in which speakers frequently switch, the MFCC features cannot be accurately extracted; when facing a scene in which speakers frequently switch and the speakers move, the single-speaker video analysis is performed on the entire video, and the segmentation accuracy is poor due to the frequent switching of the scene, resulting in poor speaker recognition accuracy.

[0055] To solve the problem of poor speaker recognition accuracy in a multi-person dialogue video, the present application provides a speaker recognition method, which first performs scene detection on a to-be-identified video, divides the to-be-identified video into a plurality of video segments by using the results of the scene detection, and the scenes of adjacent video segments are different; subsequently, feature extraction is performed on the video segments, audio data and video frames in the video segments are first separated, audio features of the audio data and face features in the video frames are extracted; meanwhile, for adjacent video frames across scenes, face tracking across scenes is implemented by the distance between face deep features before and after scene switching, and speaker recognition is implemented by the results of the face tracking across scenes, the face features extracted in the video segments, and the audio features; by performing speaker recognition in scenes and face tracking across scenes, the influence of scene switching on speaker recognition is effectively overcome, and the accuracy of speaker recognition is improved.

[0056] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0057] Figure 2A flowchart of a speaker recognition method provided by an embodiment of the present application is shown in FIG. 1. The speaker recognition method can be executed by an electronic device with corresponding data processing capability, such as a speaker recognition device, which can be a computer, a server, or other electronic device. The speaker recognition method includes the following steps: Figure 2

[0058] In step S201, scene detection is performed on the video to be recognized, and the video to be recognized is divided into multiple video segments based on the result of the scene detection.

[0059] The video to be recognized can be a multi-person conversation video, and the video to be recognized includes multiple faces and multiple speakers.

[0060] Scene detection determines whether multiple consecutive video frames in the video to be recognized belong to the same scene by comparing the changes between the multiple video frames or by evaluating the similarity between the multiple video frames.

[0061] The result of the scene detection can include the numbers of the video frames where the scene switching occurs, so that the video to be recognized is divided into multiple video segments by the numbers. Each video segment includes video frames of the same scene, and adjacent video segments correspond to different scenes.

[0062] For example, the video to be recognized includes 10 video frames, i.e., f1 to f10, and the numbers of the video frames where the scene switching occurs are 3, 5, and 8 in sequence. Then, the video to be recognized can be divided into four video segments, the first video segment includes f1 and f2, the second video segment includes f3 and f4, the third video segment includes f5, f6, and f7, and the fourth video segment includes f8, f9, and f10.

[0063] The result of the scene detection can also include the numbers of the maximum video frames where the scene does not switch. For example, the video to be recognized includes the video frames f1 to f10 in the foregoing embodiment, and the numbers recorded in the result of the scene detection are also 3, 5, and 8 in sequence. Then, the video to be recognized can be divided into four video segments, the first video segment includes f1, f2, and f3, the second video segment includes f4 and f5, the third video segment includes f6, f7, and f8, and the fourth video segment includes f9 and f10.

[0064] For example, the scene detection on the video to be recognized can be performed based on a deep learning algorithm to automatically identify different scenes in the video to be recognized, such as different scenes caused by the entry and exit of personnel, the movement of the camera, and the like.

[0065] Any scene detection algorithm can be used to perform scene detection on the video to be recognized, such as a machine learning algorithm, a histogram comparison method, and the like, which are not limited in the present application. ​

[0066] In step S202, audio data and video frames in each video segment are separated.

[0067] After the video to be recognized is divided into multiple video segments by scene detection, audio data and video frames in each video segment are separated, i.e., audio-video separation is performed on the video segment to obtain audio data and video frames corresponding to the video segment.

[0068] The audio data in each video segment can be separated by using a video editing tool, software or online tool to obtain audio data and video frames corresponding to the video segment.

[0069] In step S203, facial features of the video frames are extracted and audio features of the audio data are extracted.

[0070] For each video frame in each video segment, facial features of the video frame are extracted; and for audio data corresponding to each video segment, audio features of the audio data are extracted.

[0071] The extracted facial features of the video frames are feature expressions of face IDs (Identification, identity) and are biological features of faces.

[0072] When the facial features are extracted, a face frame detection algorithm can be used to identify face frames in the video frames, and then facial features of faces in the identified face frames, such as facial key point features, are extracted.

[0073] Any facial feature extractor can be used to extract facial features of faces appearing in the video frames. The facial feature extractor can be an ArcFace model, a VGGFace model, a FaceNet model, a Mobile FaceNet model, etc.

[0074] For example, the facial features can include MIOU (Mean Intersection over Union, mean intersection over union) of normalized and aligned face frames. The same faces between video frames can be identified by the MIOU features.

[0075] The audio features of the audio data can be MFCC features or wav2vec features.

[0076] For consecutive video frames of the same scene, i.e., consecutive multiple video frames in the same video segment, the MIOU feature is used as a distinguishing signal for face ID to realize face tracking in different video frames; for consecutive video frames of different scenes, i.e., consecutive multiple video frames in adjacent video segments, in addition to extracting the MIOU feature, the face depth feature of the same face before and after the scene switching needs to be considered to realize more accurate face tracking.

[0077] In step S204, for the multiple video frames of the scene switching in the multiple video segments, the face depth features of the multiple video frames are extracted, and the distance of the face depth features of adjacent video frames in the multiple video frames is calculated to obtain the cross-scene distance feature.

[0078] The face depth feature is used to describe the depth information of the face frame, and can be represented by the depth of key points on the face, such as key points on the mouth, nose, eyes, etc.

[0079] For example, the face depth feature can be extracted by a Mobile FaceNet model.

[0080] The distance of the face depth feature can be the cosine distance of the face depth feature.

[0081] For adjacent video frames of different scenes, in addition to extracting the face feature of the video frame, the face depth feature of the video frame before and after the scene switching also needs to be extracted.

[0082] For adjacent video frames of scene switching, i.e., the last video frame of the previous video segment and the first video frame of the next video segment, the distance of the face depth features of the two video frames is calculated to obtain the cross-scene distance feature.

[0083] Through the cross-scene distance feature, face tracking during scene switching can be realized, and the connection relationship of the face frames extracted in the adjacent video frames of the scene switching is established, and the connected face frames in different video frames correspond to the same face.

[0084] In step S205, based on the cross-scene distance feature, the face feature, and the audio feature, a speaker is identified from the faces contained in the video segment.

[0085] Specifically, based on the extracted face feature and the cross-scene distance feature, the tracking of the face in the adjacent video frames of the same scene and the cross-scene is realized to obtain the connection relationship of the same face frame in different video frames; for each face frame, the face feature sequence corresponding to the face frame is separated from the face features extracted from each video frame in the video segment; based on the face feature sequence corresponding to each face frame extracted in the video segment, the matching result of the audio feature is matched respectively to determine the face frame of the speaker from the face frame extracted in the video segment, i.e., to identify the speaker.

[0086] The speaker recognition method provided by the embodiment divides the to-be-identified video into multiple video segments through the result of the to-be-identified video scene detection, separates the audio data and the video frames of the video segments, and extracts the face features in the video frames and the audio features in the audio data. Meanwhile, for multiple video frames across scenes, such as the last frame of a previous video segment and the first frame of a subsequent video segment in adjacent video segments, the face tracking at the scene switching is implemented through the distance of the face depth features of the front and rear frames in different scenes. The speaker in the video segment is identified in units of the video segment in the same scene, in combination with the face features extracted from each video frame in the video segment, the corresponding audio features, and the result of the cross-scene face tracking, and the speaker recognition in the entire video is implemented. The speaker recognition is performed scene by scene, the influence of the scene switching on the speaker recognition is overcome, the visual features extracted in the two dimensions of the same scene and the cross scene are matched with the result of the audio features to implement the speaker recognition, and the accuracy of the recognition is improved.

[0087] Figure 3 The flowchart of another speaker recognition method provided by the embodiment of the application is shown in Figure 2 The steps S203 to S205 are further refined based on the embodiment shown in FIG. 2. Figure 3 The speaker recognition method provided by the embodiment can specifically include the following steps:

[0088] Step S301, scene detection is performed on a to-be-identified video, and the to-be-identified video is divided into multiple video segments based on the result of the scene detection.

[0089] Step S302, for each video segment in the multiple video segments, audio data and video frames in the video segment are separated.

[0090] Before separating the audio data in the video segment, the background sound in the video segment can be removed first to avoid the influence of the background sound on the expression of the audio data features.

[0091] Step S303, the audio data in the video segment is input into a multi-language speech representation model to obtain audio features of the audio data; a training set of the multi-language speech representation model includes multi-language audio samples.

[0092] In order to meet the multi-language application problem, a multi-language speech representation model is introduced for the coding of the audio data. The multi-language speech representation model has the coding capability of cross-language audio data, and the multi-language audio samples can be obtained through a large public corpus.

[0093] The audio feature can be a wav2vec feature. Compared with traditional MFCC features, the wav2vec feature can better adapt to a multilingual scenario, and speaker recognition of a multi-person dialogue video in different languages can be implemented.

[0094] In step S304, face features of each video frame are extracted.

[0095] The face features include features of a face frame such as a number, a position, and the like, face key point features, and MIOU features of the face frame after alignment in adjacent video frames. The MIOU features can be used to implement face tracking in video frames of the same scene, that is, face tracking in video segmentation.

[0096] The face frame in the video frame can be extracted by a face frame detection algorithm to obtain information such as a position and a number of the face frame. A face key point feature of a face in the face frame is extracted by using a face key detection network model such as an MTCNN, a RetinaFace, or the like.

[0097] In step S305, face depth features and histogram of oriented gradient (HOG) features of a face frame of a scene switching video frame in the multiple video segments are extracted.

[0098] The face depth features and the HOG features of the face frame of the scene switching video frame are extracted.

[0099] In step S306, the face depth features and the HOG features of the same video frame are fused to obtain a cross-scene fusion feature of the video frame.

[0100] The face depth features and the HOG features of the same video frame are fused to obtain a cross-scene fusion feature of the video frame.

[0101] In step S307, a distance of the cross-scene fusion features of adjacent video frames in the multiple video frames is calculated to obtain a cross-scene distance feature.

[0102] Specifically, a distance of the cross-scene fusion features of the adjacent video frames of the scene switching can be calculated, such as a cosine distance, an Euclidean distance, or the like, to obtain the cross-scene distance feature.

[0103] In step S308, a connection relationship of the same face frame in the adjacent video frames of the scene switching in the multiple video segments is determined based on the cross-scene distance feature, and a connection relationship of the same face frame in the adjacent video frames of the same video segment is determined based on the face features.

[0104] The connection relationship of the face boxes is used to describe the positional relationship of the face boxes of the same face in different video frames. The face boxes of the same face are the face boxes of the same face, such as the face boxes with the same number.

[0105] For adjacent video frames of the same scene, i.e., video frames in the same video segment, the connection relationship of the face boxes of the same face represented by the adjacent video frames can be determined by the MIOU of the aligned face boxes in the adjacent video frames in the same video segment.

[0106] For adjacent video frames of different scenes, i.e., adjacent video frames in different video segments, the connection relationship of the face boxes of the same face represented by the adjacent video frames can be determined by the result of the cross-scene distance feature matching of the adjacent video frames.

[0107] Optionally, the connection relationship of the face boxes of the same face in the adjacent video frames of the scene switching in the plurality of video segments is determined based on the cross-scene distance features, and includes:

[0108] The cross-scene distance features of the adjacent video frames of the scene switching in the plurality of video segments are subjected to Hungarian matching to obtain the connection relationship of the face boxes of the same face in the adjacent video frames of the scene switching in the plurality of video segments.

[0109] Specifically, the connection relationship of the face boxes of the same face represented by the adjacent video frames can be determined based on the result of the Hungarian matching of the cross-scene distance features of the adjacent video frames.

[0110] The Hungarian matching algorithm can determine the face boxes matched by two video frames, i.e., the face boxes representing the same face, based on the cross-scene distance features of the two video frames.

[0111] The cross-scene face tracking by using the Hungarian matching improves the comprehensiveness and accuracy of the tracking.

[0112] Figure 4 The schematic diagram of the processing procedure of the adjacent video frames of different scenes by the embodiments of the present application is shown in FIG. 1. Figure 4 As shown in FIG. 1, the video to be recognized is divided into M video segments by scene detection. Figure 4Take M as an even number as an example) video segments, namely video segment 1 to video segment M, for adjacent video segments such as the i-th video segment and the i+1-th video segment, the last video frame (abbreviated as the last frame) of the i-th video segment and the first video frame (abbreviated as the first frame) of the i+1-th video segment are adjacent video frames, for each video frame in the adjacent video frames, a face frame in the video frame is extracted by a face detector; for the extracted face frame, a face ID feature and a face depth feature in the face frame are extracted by a face feature extractor, and a HOG feature of the face frame is extracted by a HOG feature extractor; after feature fusion of the extracted face ID feature, face depth feature and HOG feature of the face frame, a distance metric matrix of the adjacent video frames, namely a cross-scene distance feature, is obtained, and a connection relationship of the face frame in the adjacent video frames is obtained through Hungarian matching of the cross-scene distance feature, so as to realize cross-scene face tracking.

[0113] By introducing the face depth feature and the HOG feature, face tracking in scenes such as face occlusion, side face and scene transition can be realized, and the stability of face tracking is improved.

[0114] Exemplarily, Figure 5 A schematic diagram of the connection relationship of the face frame in different video frames provided by the embodiment of the present application is shown as follows, Figure 5 As shown in the figure, the i-th video segment includes N video frames, the last video frame of the i-th video segment, namely the Nth frame, and the first video frame of the i+1-th video segment, namely the 1st frame, are adjacent video frames with different scenes, and the connection relationship of the face frames in the two video frames, namely the connection relationship of the face frames 51 to 53, is established by using the cross-scene distance feature, and the face frames connected by straight lines are face frames of the same person, corresponding to the same face; the first video frame and the second video frame, namely the 2nd frame, of the i+1-th video segment are adjacent video frames with the same scene, and the connection relationship of the face frames in the two video frames, namely the connection relationship of the face frames 52 and 54, is established by using the MIOU feature, and the connection relationship of the face frames is continuously extended in sequence, so as to realize the connection relationship of the face frames in the entire video segment or even the entire video to be recognized.

[0115] In step S309, for each face frame in the video segment, the face feature of the face frame is obtained from the face features of the video frames containing the face frame according to the connection relationship of the face frame, and the face features of the face frame in the video frames containing the face frame are spliced to obtain the face feature matrix of the face frame.

[0116] The face feature matrix of the face frame is used to describe the change of the face feature of the face frame in the video segment, including the change of the face mouth key point.

[0117] Specifically, for a face frame identified in a video segment, face features of the face frame in each video frame containing the face frame in the video segment are extracted in sequence according to a connection relationship of the face frame, such as face key point features, the face key points including multiple key points on a mouth, and face feature matrices of the face frame are obtained by splicing the extracted face features of the face frame in sequence according to video frames or time.

[0118] In step S310, a speaker is identified from the face contained in the video segment based on the face feature matrix and the audio feature.

[0119] Specifically, for each face frame detected in a video segment, a face feature matrix of the face frame and an audio feature of audio data corresponding to the video segment are input into a speaker identification network model, and a face frame of a speaker is identified from the face frame contained in the video segment by the speaker identification network model. Since different face frames correspond to different faces, the face frame of the speaker can identify the speaker.

[0120] The speaker identification network model can analyze a lip movement change feature of a face in the face frame based on the input face feature matrix of the face frame, and then determine whether the face in the face frame is the speaker by matching the lip movement change feature with the audio feature.

[0121] Figure 6 A schematic diagram of the speaker identification network model provided by the embodiments of the present application is shown in FIG. 1. Figure 6 The speaker identification network model can include an audio encoder, a face encoder, and a speaker identification layer. The audio encoder is configured to encode the input audio data of the video segment to obtain an audio feature. The face encoder is configured to encode each face frame contained in each video frame of the video segment to obtain a face feature matrix of each face frame. The speaker identification layer is configured to determine a face frame of a speaker from each face frame detected in the video segment based on the input audio feature and face feature matrix. The speaker identification network model can calculate a probability that each face frame detected in the video segment is the face frame of the speaker based on the audio feature and face feature matrix, and determine the face frame with the highest probability as the face frame of the speaker.

[0122] To expand the training set during training of the speaker identification network model, the speaker identification network model can be trained using a contrastive learning framework. The model is trained by maximizing the consistency between positive samples and minimizing the consistency between negative samples, and the model parameters are constantly optimized.

[0123] In the training set collection stage of the speaker recognition network model, multiple video samples can be obtained from public video materials, such as news broadcasts, which can include multiple languages. The video samples are subjected to scene detection, and the video samples are divided into multiple video segment samples; the audio data and video frames in the video segment samples are separated; the positive samples in the training samples can be obtained by manually separating the audio data and video frames from the video segment samples. The audio data in the negative samples can be obtained by translating the audio data of the positive samples into other languages, or by replacing the audio data in the positive samples with the audio data in other positive samples.

[0124] In the present embodiment, the audio features of the audio data are extracted using a multi-language speech representation model, which realizes speech feature expression in a multi-language scene, and can realize speaker recognition in a cross-language multi-person dialogue video, thereby expanding the application range. In addition to the biological features of the face, the depth features and HOG features of the face are combined to realize cross-scene face tracking, effectively overcoming the problem of poor face tracking accuracy in scenes such as large-scale face movement, face occlusion, etc., thereby improving the stability of face tracking and the accuracy of speaker recognition.

[0125] Figure 7 Another flowchart of a speaker recognition method provided by the present embodiment is provided, which is based on the foregoing embodiments and adds a step of filtering face frames based on face key points before step S205 or step S310, as shown in Figure 7 The speaker recognition method provided by the present embodiment can further include the following steps before identifying the speaker from the face included in the video segment:

[0126] In step S701, for each face frame detected in the video segment, the lip key points of the face frame are extracted from the video frame corresponding to the face frame in the video segment.

[0127] For each face frame detected in the video segment, the position of the lip key points of the face frame is extracted from the video frame including the face frame in the video segment.

[0128] If the position of the lip key points is included in the face features extracted in the foregoing step, the position of the lip key points can be extracted from the face features of the face frame.

[0129] The lip key points include multiple key points located on the lips, such as the key points on the outer edge line of the upper lip and the key points on the outer edge line of the lower lip, and can also include the key points on the inner edge line of the upper lip and the key points on the inner edge line of the lower lip.

[0130] Step S702, based on the lip key points of the face frame in the video frame corresponding to the face frame in the video segment, the lip offset of the face frame in the video segment is calculated.

[0131] Specifically, the offset of one or more lip key points or the overall lip key point of a certain face frame in the video segment in the time sequence can be calculated, and the lip offset of the face frame in the video segment is obtained based on the statistical value of the calculated offset, such as the average value, the maximum value, etc.

[0132] For example, the lip offset of the face frame in the video segment can be the offset of the lip key point with the largest offset of the face frame in the video segment.

[0133] Step S703, eliminating the face frame in the video segment with a lip offset less than a preset threshold, and taking the remaining face frame as a candidate face frame to identify a speaker from the candidate face frame.

[0134] If the lip offset of the face frame in the video segment is less than the preset threshold, the face frame is deleted.

[0135] After deleting the face frame in the video segment with a lip offset less than the preset threshold, the remaining face frame is regarded as a candidate face frame to identify a speaker from the candidate face frame of the video segment.

[0136] By screening the face frame through the lip offset before identifying the speaker, the number of face frames required for identification is reduced, and the efficiency of speaker identification is improved.

[0137] Figure 8 Another flowchart of a speaker identification method provided by an embodiment of the present application is shown in FIG. 8. Figure 8 The speaker identification method mainly includes the following steps:

[0138] Step S801, scene detection: performing scene detection on the provided video to be identified, and dividing the video to be identified into multiple video segments based on the result of the scene detection.

[0139] Step S802, audio and video separation: for each video segment, the audio data (abbreviated as audio) and the video frame in the video segment are separated.

[0140] Step S803, voice separation: separating the background sound from the audio data to separate the voice and the background sound.

[0141] Step S804, audio feature extraction: extracting the features of the separated voice to obtain the audio features, such as wav2vec features.

[0142] Step S805, face detection and tracking: face detection is performed on the video frame to obtain a face frame in the video frame and a face key point feature in the face frame; a face tracking algorithm is used to determine a connection relationship of the face frame.

[0143] Specifically, the MIOU feature in the aligned face frame can be used to establish the connection relationship of the face frame between the same scene video frames, and the MIOU feature, the face depth feature and the HOG feature in the face frame can be used to establish the connection relationship of the face frame between different scene video frames.

[0144] Step S806, key point screening: the offset of the lip key point in the face frame is used to screen the face frame in units of video segments.

[0145] Step S807, speaker recognition: for the remaining face frame after screening, i.e., the candidate face frame, a time sequence, i.e., a face feature matrix of the face frame, is obtained based on the face key point feature in the face frame, and the speaker is recognized from the remaining face frame after screening in the video segment.

[0146] Further, after recognizing the speaker in the video, the speaker in the video can be labeled, i.e., the time period corresponding to different speakers is labeled, so that the video can be edited based on the preset speaker input by the user to obtain a video segment in which only the preset speaker is speaking, i.e., a video segment corresponding to the time period of the preset speaker is obtained by editing the video.

[0147] Further, after recognizing the speaker in the video, the audio in the video can also be replaced with the translated audio, and the lip movement of the corresponding face in the video can be modified based on the speaker corresponding to each segment in the translated audio to avoid inconsistency between the translated audio and the lip movement in the video.

[0148] Figure 9 A structure schematic diagram of a speaker recognition device provided by an embodiment of the present application is shown in FIG. 9. Figure 9 As shown in FIG. 9, the speaker recognition device includes a scene detection module 910, an audio-video separation module 920, a first feature extraction module 930, a second feature extraction module 940 and a speaker recognition module 950.

[0149] The scene detection module 910 is configured to perform scene detection on a to-be-identified video, and divide the to-be-identified video into a plurality of video segments based on a result of the scene detection; the audio-video separation module 920 is configured to separate, for each video segment in the plurality of video segments, audio data and video frames in the video segment; the first feature extraction module 930 is configured to extract face features of the video frames and extract audio features of the audio data; the second feature extraction module 940 is configured to extract face depth features of a plurality of video frames that are scene-switching in the plurality of video segments, and calculate distances of the face depth features of adjacent video frames in the plurality of video frames to obtain cross-scene distance features; and the speaker recognition module 950 is configured to recognize a speaker from faces contained in the video segments based on the cross-scene distance features, the face features and the audio features.

[0150] Optionally, the second feature extraction module 940 comprises:

[0151] a depth feature and HOG extraction unit configured to extract face depth features and histogram of oriented gradient (HOG) features of a face bounding box for a plurality of video frames that are scene-switching in the plurality of video segments; a feature fusion unit configured to fuse the face depth features and the HOG features of a same video frame to obtain cross-scene fusion features of the video frame; and a distance calculation unit configured to calculate distances of the cross-scene fusion features of adjacent video frames in the plurality of video frames to obtain the cross-scene distance features.

[0152] Optionally, the face features are used to describe features of a face bounding box corresponding to a face contained in a video frame, and the speaker recognition module 950 comprises:

[0153] a cross-scene connection relationship determination unit configured to determine a connection relationship of a same face bounding box in adjacent video frames that are scene-switching in the plurality of video segments based on the cross-scene distance features, wherein the connection relationship of the face bounding box is used to describe a positional relationship of the same face bounding box in different video frames; a same-scene connection relationship determination unit configured to determine a connection relationship of the same face bounding box in adjacent video frames in a same video segment based on the face features; a face bounding box feature obtaining unit configured to, for each face bounding box in the video segment, obtain face features of the face bounding box from the face features of video frames containing the face bounding box according to the connection relationship of the face bounding box, and splice the face features of the face bounding box in the video frames containing the face bounding box to obtain a face feature matrix of the face bounding box; and a speaker recognition unit configured to recognize a speaker from faces contained in the video segments based on the face feature matrix and the audio features.

[0154] Optionally, the cross-scene connection relationship determination unit is specifically configured to:

[0155] Hungarian matching is performed on the cross-scene distance features of adjacent video frames with scene switching in the multiple video segments to obtain the connection relationship of the same face bounding box in adjacent video frames with scene switching in the multiple video segments.

[0156] Optionally, the facial features are used to describe the features of the face bounding boxes corresponding to the faces contained in the video frame. The speaker recognition device further includes a face bounding box filtering module, used for:

[0157] Before identifying the speaker from the faces contained in the video segments based on the cross-scene distance features, the facial features, and the audio features, for the face bounding boxes detected in each video segment, the lip key points of the face bounding boxes are extracted from the video frames corresponding to the face bounding boxes in the video segment; based on the lip key points of the face bounding boxes in the video frames corresponding to the face bounding boxes in the video segment, the lip offset of the face bounding box in the video segment is calculated; face bounding boxes with lip offsets less than a preset threshold in the video segment are removed, and the remaining face bounding boxes are used as candidate face bounding boxes to identify the speaker from the candidate face bounding boxes.

[0158] Optionally, the first feature extraction module 930 is specifically used for: extracting audio features from the audio data.

[0159] The audio data is input into a multilingual speech representation model to obtain the audio features of the audio data; the training set of the multilingual speech representation model includes audio samples from multiple languages.

[0160] The facial model correction device provided in this application embodiment can be used to execute the technical solution of the facial model correction method provided in any of the above embodiments of this application. Its implementation principle and technical effect are similar, and will not be repeated here.

[0161] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 10 As shown, the electronic device of this embodiment may include: at least one processor 1001; and a memory 1002 communicatively connected to the at least one processor; wherein the memory 1002 stores instructions executable by the at least one processor 1001, the instructions being executed by the at least one processor 1001 to cause the electronic device to perform the method as described in any of the above embodiments.

[0162] Optionally, the memory 1002 can be either standalone or integrated with the processor 1001.

[0163] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the foregoing embodiments, and will not be repeated here.

[0164] The embodiment of the present application further provides a computer readable storage medium, wherein computer execution instructions are stored in the computer readable storage medium, and when the computer execution instructions are executed by a processor, the method provided in any of the foregoing embodiments can be implemented.

[0165] The embodiment of the present application further provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the method provided in any of the foregoing embodiments.

[0166] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other manners. For example, the above-described device embodiments are merely illustrative, and the division of the modules is merely a logical function division, and there can be another division manner in actual implementation, for example, a plurality of modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0167] The integrated modules in the form of software function modules described above can be stored in a computer readable storage medium. The software function modules described above are stored in a storage medium, and include a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute part of the steps of the methods described in the embodiments of the present application.

[0168] It should be understood that the processor described above can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like. The steps of the methods disclosed in the present application can be directly embodied as the execution of the hardware processor, or the combination of hardware and software modules in the processor. The memory can include a high-speed memory, and can also include a nonvolatile memory, for example, at least one disk memory, and can also be a U disk, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0169] The storage medium described above can be implemented by any type of volatile or nonvolatile storage devices or a combination thereof, such as a static random access memory, an electrically erasable programmable read-only memory, an erasable programmable read-only memory, a programmable read-only memory, a read-only memory, a magnetic memory, a flash memory, a magnetic disk or an optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0170] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside within an application-specific integrated circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or a vehicle's sentry-mode control device.

[0171] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0172] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0173] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods provided in the various embodiments of this application.

[0174] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0175] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method of speaker recognition, characterized by, The method comprises the following steps: scene detection is performed on a to-be-identified video, and the to-be-identified video is divided into a plurality of video segments based on a result of the scene detection; for each video segment in the plurality of video segments, audio data and video frames in the video segment are separated; facial features of the video frames are extracted, and audio features of the audio data are extracted; for a plurality of video frames in which scenes switch in the plurality of video segments, facial depth features of the plurality of video frames are extracted, and a distance between facial depth features of adjacent video frames in the plurality of video frames is calculated to obtain a cross-scene distance feature; based on the cross-scene distance feature, the facial features and the audio features, a speaker is identified from faces contained in the video segments; for a plurality of video frames in which scenes switch in the plurality of video segments, facial depth features of the plurality of video frames are extracted, and a distance between facial depth features of adjacent video frames in the plurality of video frames is calculated to obtain a cross-scene distance feature, comprising: for a plurality of video frames in which scenes switch in the plurality of video segments, facial depth features and histogram of oriented gradients features of a face box of the plurality of video frames are extracted; the facial depth features and the histogram of oriented gradients features of the same video frame are fused to obtain a cross-scene fusion feature of the video frame; a distance between cross-scene fusion features of adjacent video frames is calculated to obtain the cross-scene distance feature.

2. The method of claim 1, wherein, The facial features are used to describe features of a face box corresponding to a face contained in a video frame, and based on the cross-scene distance feature, the facial features and the audio features, a speaker is identified from faces contained in the video segments, comprising: based on the cross-scene distance feature, a connection relationship of the same face box in adjacent video frames in which scenes switch in the plurality of video segments is determined, and based on the facial features, a connection relationship of the same face box in adjacent video frames in a same video segment is determined; the connection relationship of the face box is used to describe a positional relationship of the same face box in different video frames; for each face box in the video segments, the facial features of the face box are obtained from the facial features of the video frames containing the face box according to the connection relationship of the face box, and the facial features of the face box in the video frames containing the face box are spliced to obtain a facial feature matrix of the face box; based on the facial feature matrix and the audio features, a speaker is identified from faces contained in the video segments.

3. The method of claim 1, wherein, Based on the cross-scene distance feature, a connection relationship of the same face box in adjacent video frames in which scenes switch in the plurality of video segments is determined, comprising: Hungarian matching is performed on the cross-scene distance features of the adjacent video frames in which scenes switch in the plurality of video segments to obtain the connection relationship of the same face box in the adjacent video frames in which scenes switch in the plurality of video segments.

4. The method of claim 1, wherein, The facial features are used to describe features of a face box corresponding to a face contained in a video frame, and before the speaker is identified from the faces contained in the video segments based on the cross-scene distance feature, the facial features and the audio features, the method further comprises: For each face bounding box detected in each video segment, extract lip key points of the face bounding box from a video frame corresponding to the face bounding box in the video segment; Based on the lip key points of the face bounding box in the video frame corresponding to the face bounding box in the video segment, calculate the lip offset of the face bounding box in the video segment; Remove the face bounding box in the video segment with a lip offset less than a preset threshold, and take the remaining face bounding boxes as candidate face bounding boxes to identify a speaker from the candidate face bounding boxes.

5. The method of claim 1, wherein, Extract audio features of the audio data, including: Input the audio data into a multi-language speech representation model to obtain audio features of the audio data; the training set of the multi-language speech representation model includes audio samples in multiple languages.

6. A speaker recognition apparatus characterized by comprising: Comprise: A scene detection module for performing scene detection on a video to be identified and dividing the video to be identified into multiple video segments based on the results of the scene detection; An audio-video separation module for separating audio data and video frames in each video segment in the multiple video segments; A first feature extraction module for extracting face features of the video frames and extracting audio features of the audio data; A second feature extraction module for extracting face depth features of multiple video frames in which scenes are switched in the multiple video segments and calculating distances between face depth features of adjacent video frames in the multiple video frames to obtain cross-scene distance features; A speaker identification module for identifying a speaker from faces included in the video segments based on the cross-scene distance features, the face features, and the audio features; The second feature extraction module comprises: A depth feature and HOG extraction unit for extracting face depth features and histogram of oriented gradient features of face bounding boxes of the multiple video frames in which scenes are switched in the multiple video segments; a feature fusion unit for fusing the face depth features and the histogram of oriented gradient features of the same video frame to obtain cross-scene fusion features of the video frame; and a distance calculation unit for calculating distances between cross-scene fusion features of adjacent video frames in the multiple video frames to obtain the cross-scene distance features.

7. An electronic device, comprising: Comprise: A memory and at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to cause the electronic device to perform the method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer execution instructions, and when the processor executes the computer execution instructions, the method of any one of claims 1-5 is implemented.

9. A computer program product, characterised in that, The computer program is executed by the processor to implement the method of any one of claims 1-5.

Images