Speaker recognition method and device, electronic equipment, medium and program product
By performing scene detection and feature extraction on multi-person dialogue videos, using cross-scene face depth features and audio features matching, the problem of low recognition accuracy of speakers in multi-person dialogue videos is solved, and higher recognition accuracy is achieved.
Patent Information
- Application Number
- CN202411098519.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-09
AI Technical Summary
In multi-person conversation videos, it is difficult for the prior art to accurately identify the speaker, especially when the speaker switches frequently or moves, resulting in poor recognition accuracy.
By performing scene detection of the recognized video, it is divided into multiple video segments, the audio and video features of each segment are extracted, and the cross-scene face depth features and audio features are used to match to identify the speaker.
It improves the accuracy of speaker recognition, overcomes the impact of scene switching on recognition, and enhances the ability to accurately extract the speaker's identity in multi-person dialogue videos.
Smart Images

Figure CN119007731A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of audio and video processing technology, and in particular to a speaker recognition method, device, electronic device, medium and program product. Background Art
[0002] Speaker recognition refers to automatically identifying different speakers in an audio or video clip, so as to segment the audio or video into multiple segments according to different speakers. In a conversation scenario, it can quickly and accurately detect the speaker and extract the voice or video of the specified speaker, which has high application value in multimedia content creation, video customization, lip shape adjustment in different languages, and other scenarios.
[0003] When identifying speakers in a multi-person conversation video, related technologies usually treat the entire video as a whole, and identify the speakers in the video by extracting and matching visual features and audio features. The inventors of this application found during their research that in a multi-person conversation scenario, speakers frequently switch or move within a certain period of time, resulting in poor continuity and integrity of the video. The method of treating the entire video as a whole for speaker recognition has poor accuracy.
[0004] Therefore, there is an urgent need to provide a solution for speaker recognition in multi-person conversation videos with high accuracy. Summary of the invention
[0005] The present application provides a speaker recognition method, device, electronic device, medium and program product, which implement video segmentation through scene detection and perform speaker recognition segment by segment, thus overcoming the influence of scene switching on speaker recognition, utilizing visual features extracted from two dimensions, the same scene and across scenes, and matching the results with audio features to implement speaker recognition, thus improving the recognition accuracy.
[0006] In a first aspect, the present application provides a speaker recognition method, comprising:
[0007] Performing scene detection on the video to be identified, and dividing the video to be identified into multiple video segments based on the result of the scene detection;
[0008] For each video segment in the multiple video segments, separate and obtain audio data and video frames in the video segment;
[0009] Extracting facial features of the video frame and extracting audio features of the audio data;
[0010] For a plurality of video frames of scene switching in the plurality of video segments, extracting face depth features of the plurality of video frames, and calculating distances between face depth features of adjacent video frames in the plurality of video frames to obtain cross-scene distance features;
[0011] Based on the cross-scene distance feature, the facial feature and the audio feature, a speaker is identified from the faces included in the video segment.
[0012] Optionally, for a plurality of video frames of scene switching in the plurality of video segments, extracting face depth features of the plurality of video frames, and calculating distances between face depth features of adjacent video frames in the plurality of video frames to obtain cross-scene distance features, including:
[0013] For a plurality of video frames of scene switching in the plurality of video segments, extracting face depth features and directional gradient histogram features of face frames of the plurality of video frames;
[0014] Fusing the face depth feature and the directional gradient histogram feature of the same video frame to obtain a cross-scene fusion feature of the video frame;
[0015] The distances of the cross-scene fusion features of adjacent video frames in the multiple video frames are calculated to obtain the cross-scene distance features.
[0016] Optionally, the facial features are used to describe features of a facial frame corresponding to a face included in a video frame, and based on the cross-scene distance features, the facial features, and the audio features, identifying a speaker from the faces included in the video segment includes:
[0017] Determine, based on the cross-scene distance feature, a connection relationship of the same face frame in adjacent video frames of scene switching in the multiple video segments, and determine, based on the face feature, a connection relationship of the same face frame in adjacent video frames in the same video segment; the connection relationship of the face frame is used to describe the position relationship of the same face frame in different video frames;
[0018] For each face frame in the video segment, according to the connection relationship of the face frame, obtain the face feature of the face frame from the face features of the video frame containing the face frame, splice the face features of the face frame in the video frame containing the face frame, and obtain the face feature matrix of the face frame;
[0019] Based on the facial feature matrix and the audio features, a speaker is identified from the faces included in the video segment.
[0020] Optionally, determining, based on the cross-scene distance feature, a connection relationship of the same face frame in adjacent video frames of scene switching in the multiple video segments includes:
[0021] Hungarian matching is performed on cross-scene distance features of adjacent video frames of scene switching in the multiple video segments to obtain connection relationships of the same face frames in adjacent video frames of scene switching in the multiple video segments.
[0022] Optionally, the facial features are used to describe features of a facial frame corresponding to a face included in a video frame. Before identifying a speaker from the faces included in the video segment based on the cross-scene distance features, the facial features, and the audio features, the method further includes:
[0023] For a face frame detected in each video segment, extracting lip key points of the face frame from a video frame corresponding to the face frame in the video segment;
[0024] Calculate the lip offset of the face frame in the video segment based on the lip key points of the face frame in the video frame corresponding to the face frame in the video segment;
[0025] The face frames whose lip offsets in the video segment are less than a preset threshold are eliminated, and the remaining face frames are used as candidate face frames to identify the speaker from the candidate face frames.
[0026] Optionally, extracting audio features of the audio data includes:
[0027] The audio data is input into a multilingual speech representation model to obtain audio features of the audio data; the training set of the multilingual speech representation model includes audio samples in multiple languages.
[0028] In a second aspect, the present application provides a device for speaker recognition, comprising:
[0029] A scene detection module, used to perform scene detection on the video to be identified, and divide the video to be identified into multiple video segments based on the result of the scene detection;
[0030] An audio and video separation module, used for separating each video segment in the multiple video segments to obtain audio data and video frames in the video segment;
[0031] A first feature extraction module, used to extract facial features of the video frame and audio features of the audio data;
[0032] A second feature extraction module is used to extract face depth features of a plurality of video frames of scene switching in the plurality of video segments, and calculate the distance between the face depth features of adjacent video frames in the plurality of video frames to obtain a cross-scene distance feature;
[0033] A speaker recognition module is used to recognize a speaker from the faces included in the video segment based on the cross-scene distance feature, the face feature and the audio feature.
[0034] In a third aspect, the present application provides an electronic device, including: a memory and at least one processor;
[0035] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the head-mounted display to perform the speaker recognition method provided in the first aspect of the present application.
[0036] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the speaker recognition method provided in the first aspect of the present application is implemented.
[0037] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the speaker recognition method provided in the first aspect of the present application.
[0038] The speaker recognition method, device, electronic device, medium and program product provided by the present application divide the video to be recognized into multiple video segments according to the result of the detection of the video scene to be recognized, separate the audio data and video frames of the video segments, and extract the facial features in the video frames and the audio features in the audio data; at the same time, for multiple video frames across scenes, such as the last frame of the previous video segment and the first frame of the next video segment in adjacent video segments, the distance of the facial depth features of the previous and next frames under different scenes is used to realize face tracking when the scene switches; taking the video segments of the same scene as the unit, the speaker in the video segment is identified by combining the facial features extracted from each video frame in the video segment, the corresponding audio features and the results of cross-scene face tracking, thereby realizing speaker recognition in the entire video. By performing speaker recognition scene by scene, the influence of scene switching on speaker recognition is overcome, and speaker recognition is realized by using the visual features extracted from the same scene and cross-scene dimensions and the results of matching with the audio features, thereby improving the accuracy of recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0040] Figure 1 A flowchart of the main process of the speaker identification method;
[0041] Figure 2 A flowchart of a speaker recognition method provided in an embodiment of the present application;
[0042] Figure 3 A flowchart of another speaker recognition method provided in an embodiment of the present application;
[0043] Figure 4A schematic diagram of a process of processing adjacent video frames with different scenes according to an embodiment of the present application;
[0044] Figure 5 A schematic diagram of the connection relationship between face frames in different video frames provided in an embodiment of the present application;
[0045] Figure 6 A schematic diagram of a speaker recognition network model provided in an embodiment of the present application;
[0046] Figure 7 A flowchart of another speaker recognition method provided in an embodiment of the present application;
[0047] Figure 8 A flowchart of another speaker recognition method provided in an embodiment of the present application;
[0048] Fig. 9 A schematic diagram of the structure of a speaker recognition device provided in an embodiment of the present application;
[0049] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0050] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0051] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0052] The widespread dissemination of multimedia short videos has promoted the development and growth of social media platforms. Retrieving the required or relevant multimedia content from massive amounts of audio and video information has great application value in content expression, material creation, and social interaction. Speaker recognition is a key technology in audio and video information retrieval. Taking multi-person conversation videos as an example, extracting the speaker's audio and video from multi-person conversation videos is an important basic technology for processing multi-person conversation videos.
[0053] Figure 1 This is a flowchart of the main process of the speaker recognition method, such as Figure 1As shown in FIG. 1 , for a video of a speaker to be identified (referred to as a video to be identified), first separate the audio data in the video to be identified; extract the MFCC (Mel Frequency Cepstrum Coefficient) features of the audio data, and use the two-dimensional features composed of the MFCC features and time as audio features; input the entire video to be identified into a deep learning network model, or use face clustering or other methods to segment the video to be identified into multiple single-person speaking videos, such as Figure 1 From video 1 to video n in the video, facial features of multiple consecutive video frames of a single person speaking video are extracted; speaker recognition is achieved through the matching results of facial features and audio features.
[0054] The above speaker recognition method has the following defects: since the total number of speakers is unknown, it is difficult to segment a single-person speaking video, and the detection error is large; when faced with scenes where the speaker frequently switches, the MFCC features may not be accurately extracted; when faced with scenes such as frequent speaker switching and speaker movement, the single-person speaking video analysis is performed on the entire video, and the segmentation accuracy is poor due to the frequent scene switching, resulting in poor speaker recognition accuracy.
[0055] In order to solve the problem of poor speaker recognition accuracy in multi-person conversation videos, the present application provides a speaker recognition method, which first performs scene detection on a video to be recognized, and uses the result of the scene detection to divide the video to be recognized into multiple video segments, where adjacent video segments have different scenes; subsequently, feature extraction is performed on the video segments, and audio data and video frames in the video segments are first separated, and audio features of the audio data and facial features in the video frames are extracted; at the same time, for adjacent video frames across scenes, cross-scene tracking of faces is achieved through the distance of facial depth features before and after scene switching, and speaker recognition is achieved through the results of cross-scene face tracking, facial features extracted from video segments, and audio features; by performing speaker recognition by scene and face tracking across scenes, the influence of scene switching on speaker recognition is effectively overcome, and the accuracy of speaker recognition is improved.
[0056] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0057] Figure 2A flow chart of a speaker recognition method provided in an embodiment of the present application is provided. The speaker recognition method can be executed by an electronic device with corresponding data processing capabilities, such as a speaker recognition device. The speaker recognition device can be a computer, a server or other electronic devices, such as Figure 2 As shown, the speaker recognition method comprises the following steps:
[0058] Step S201 : performing scene detection on a video to be identified, and dividing the video to be identified into a plurality of video segments based on a result of the scene detection.
[0059] The video to be identified may be a multi-person conversation video, which includes multiple faces and multiple speakers.
[0060] Scene detection determines whether multiple video frames belong to the same scene by comparing the changed parts between multiple consecutive video frames in the video to be identified or by evaluating the similarity between the multiple video frames.
[0061] The result of scene detection may include the numbers of the video frames where scene switching occurs, so that the video to be identified is divided into multiple video segments through these numbers, the scenes of the video frames included in each video segment are the same, and the scenes corresponding to adjacent video segments are different.
[0062] For example, taking the video to be identified as including 10 video frames, namely f1 to f10, the video frames where scene switching occurs are numbered 3, 5 and 8 respectively, then the video to be identified can be divided into four video segments, the first video segment includes f1 and f2, the second video segment includes f3 and f4, the third video segment includes f5, f6 and f7, and the fourth video segment includes f8, f9 and f10.
[0063] The result of scene detection may also include the number of the largest video frame in which the scene has not been switched. Taking the video to be identified in the aforementioned embodiment including video frames f1 to f10 as an example, the numbers recorded in the result of scene detection are also 3, 5 and 8, respectively, and the video to be identified can be divided into four video segments, the first video segment includes f1, f2 and f3, the second video segment includes f4 and f5, the third video segment includes f6, f7 and f8, and the fourth video segment includes f9 and f10.
[0064] Exemplarily, based on a deep learning algorithm, scene detection can be performed on the video to be identified, and different scenes in the video to be identified can be automatically identified, such as different scenes caused by the entry and exit of people, movement of the camera, etc.
[0065] Any scene detection algorithm can be used to perform scene detection on the video to be identified, such as a machine learning algorithm, a histogram comparison method, etc., and this application does not limit this.
[0066] Step S202: For each video segment in the multiple video segments, separate and obtain audio data and video frames in the video segment.
[0067] After dividing the video to be identified into multiple video segments through scene detection, for each video segment, the audio data and video frames in the video segment are separated, that is, the audio and video separation is performed on the video segment to obtain the audio data and video frames corresponding to the video segment.
[0068] Video editing tools, software or online tools can be used to separate the audio data in each video segment to obtain the audio data and video frames corresponding to the video segment.
[0069] Step S203: extracting facial features of the video frame and extracting audio features of the audio data.
[0070] For each video frame in each video segment, facial features in the video frame are extracted; for audio data corresponding to each video segment, audio features of the audio data are extracted.
[0071] The facial features extracted from the video frames are characteristic expressions of face ID (Identification), which are biological features of the face.
[0072] When extracting facial features, a face frame in a video frame may be first identified using a face frame detection algorithm, and then facial features of the face in the identified face frame, such as facial key point features, may be extracted.
[0073] Any facial feature extractor may be used to extract facial features of faces appearing in video frames. The facial feature extractor may be an ArcFace model, a VGGFace model, a FaceNet model, a Mobile FaceNet model, etc.
[0074] For example, the face feature may include the MIOU (Mean Intersection over Union) of the normalized aligned face frame. The same face between video frames can be identified through the MIOU feature.
[0075] The audio features of the audio data may be MFCC features or wav2vec features.
[0076] For continuous video frames of the same scene, that is, multiple continuous video frames in the same video segment, MIOU features are used as the distinguishing signal of face ID to achieve face tracking in different video frames; for continuous video frames of different scenes, that is, multiple continuous video frames in adjacent video segments, in addition to extracting MIOU features, it is also necessary to consider the facial depth features of the same face before and after scene switching to achieve more accurate face tracking.
[0077] Step S204: extracting facial depth features of the multiple video frames of scene switching in the multiple video segments, and calculating distances between facial depth features of adjacent video frames in the multiple video frames to obtain cross-scene distance features.
[0078] Among them, the face depth feature is used to describe the depth information of the face frame, and the depth representation of key points on the face can be used, such as key points on the mouth, nose, eyes and other parts.
[0079] Exemplarily, the deep features of the face can be extracted by the Mobile FaceNet model.
[0080] The distance of the facial depth feature may be a cosine distance of the facial depth feature.
[0081] For adjacent video frames with different scenes, in addition to extracting the facial features of the video frames, it is also necessary to extract the facial depth features in the video frames before and after the scene switch.
[0082] For adjacent video frames of scene switching, that is, the last video frame of the previous video segment and the first video frame of the next video segment, the distance between the facial depth features of the two video frames is calculated to obtain a cross-scene distance feature.
[0083] Through the cross-scene distance feature, face tracking can be achieved during scene switching, and the connection relationship between the face frames extracted from adjacent video frames of scene switching can be established. The face frames connected in different video frames correspond to the same face.
[0084] Step S205 , identifying a speaker from the faces included in the video segment based on the cross-scene distance feature, the face feature and the audio feature.
[0085] Specifically, based on the extracted facial features and cross-scene distance features, faces in adjacent video frames in the same scene and across scenes are tracked to obtain the connection relationship of the same face frame in different video frames; for each face frame, a facial feature sequence corresponding to the face frame is separated from the facial features extracted from each video frame in the video segment; based on the facial feature sequences corresponding to each face frame extracted from the video segment and the matching results with the audio features, the face frame of the speaker is determined from the face frame extracted from the video segment, that is, the speaker is recognized.
[0086] The speaker recognition method provided in this embodiment divides the video to be recognized into multiple video segments according to the result of the detection of the video scene to be recognized, separates the audio data and video frames of the video segments, and extracts the facial features in the video frames and the audio features in the audio data; at the same time, for multiple video frames across scenes, such as the last frame of the previous video segment and the first frame of the next video segment in adjacent video segments, the distance of the facial depth features of the previous and next frames under different scenes is used to realize face tracking when the scene switches; taking the video segments of the same scene as the unit, the speaker in the video segment is identified by combining the facial features extracted from each video frame in the video segment, the corresponding audio features and the results of cross-scene face tracking, thereby realizing speaker recognition in the entire video. By performing speaker recognition scene by scene, the influence of scene switching on speaker recognition is overcome, and speaker recognition is realized by using the extracted visual features of the same scene and cross-scene dimensions and the results of matching with the audio features, thereby improving the accuracy of recognition.
[0087] Figure 3 A flow chart of another speaker recognition method provided in an embodiment of the present application. Figure 2 The embodiment shown in the figure further refines step S203 to step S205. Figure 3 As shown, the speaker recognition method provided in this embodiment may specifically include the following steps:
[0088] Step S301 : performing scene detection on a video to be identified, and dividing the video to be identified into a plurality of video segments based on a result of the scene detection.
[0089] Step S302: For each video segment in the multiple video segments, separate and obtain audio data and video frames in the video segment.
[0090] Before separating the audio data in the video segment, the background sound in the video segment may be removed to avoid the influence of the background sound on the characteristic expression of the audio data.
[0091] Step S303: input the audio data in the video segment into a multilingual speech representation model to obtain audio features of the audio data; the training set of the multilingual speech representation model includes multilingual audio samples.
[0092] In order to meet the multi-language applicability problem, a multi-language speech representation model is introduced to encode audio data. The multi-language speech representation model has the ability to encode cross-language audio data, and its multi-language audio samples can be obtained through a large public corpus.
[0093] The audio feature may be a wav2vec feature. Compared with the traditional MFCC feature, the wav2vec feature can better adapt to multi-language scenarios and can realize speaker recognition of multi-person conversation videos in different languages.
[0094] Step S304: extracting facial features from each of the video frames.
[0095] The facial features include the features of the face frame such as number and position, the features of the facial key points, and the MIOU features of the aligned face frames in adjacent video frames. The MIOU features can be used to achieve face tracking in video frames of the same scene, that is, to achieve face tracking in video segments.
[0096] The face frame detection algorithm can be used to extract the face frame in the video frame to obtain the position, number and other information of the face frame. The face key point features of the face in the face frame can be extracted by using the face key detection network model, such as MTCNN, RetinaFace and other models.
[0097] Step S305 : extracting face depth features and directional gradient histogram features of face frames from a plurality of video frames of scene switching in the plurality of video segments.
[0098] For adjacent video frames of scene switching, face depth features of face frames contained in each of the adjacent video frames and Histogram of Oriented Gradients (HOG) features of the face frames are extracted.
[0099] Step S306, fusing the face depth feature and the oriented gradient histogram feature of the same video frame to obtain a cross-scene fusion feature of the video frame.
[0100] The facial depth features and HOG features extracted from the same video frame are fused to obtain a feature matrix, i.e., cross-scene fusion features.
[0101] Step S307 , calculating the distance between the cross-scene fusion features of adjacent video frames in the multiple video frames to obtain the cross-scene distance feature.
[0102] Specifically, the distances of cross-scene fusion features in adjacent video frames of scene switching, such as cosine distance, Euclidean distance, etc., can be calculated to obtain cross-scene distance features.
[0103] Step S308, determining the connection relationship of the same face frames in adjacent video frames of scene switching in the multiple video segments based on the cross-scene distance features, and determining the connection relationship of the same face frames in adjacent video frames in the same video segment based on the face features.
[0104] The connection relationship of the face frames is used to describe the position relationship of the same face frame in different video frames. The same face frame is a face frame of the same face, such as a face frame with the same number.
[0105] For adjacent video frames of the same scene, that is, video frames in the same video segment, the connection relationship between face frames representing the same face in the adjacent video frames can be determined through the MIOU of the aligned face frames in the adjacent video frames in the same video segment.
[0106] For adjacent video frames in different scenes, that is, adjacent video frames in different video segments, the connection relationship between face frames representing the same face in the adjacent video frames can be determined through the results of cross-scene distance feature matching of the adjacent video frames.
[0107] Optionally, determining, based on the cross-scene distance feature, a connection relationship of the same face frame in adjacent video frames of scene switching in the multiple video segments includes:
[0108] Hungarian matching is performed on cross-scene distance features of adjacent video frames of scene switching in the multiple video segments to obtain connection relationships of the same face frames in adjacent video frames of scene switching in the multiple video segments.
[0109] Specifically, the connection relationship between the face frames representing the same face in the adjacent video frames may be determined based on the result of the Hungarian matching of the cross-scene distance features of the adjacent video frames.
[0110] The Hungarian matching algorithm can determine the matching face frames of the two video frames, that is, the face frames representing the same face, based on the cross-scene distance features of the two video frames.
[0111] Hungarian matching is used for cross-scene face tracking, which improves the comprehensiveness and accuracy of tracking.
[0112] Figure 4 FIG. 1 is a schematic diagram of a process of processing adjacent video frames of different scenes according to an embodiment of the present application. Figure 4 As shown, the video to be identified is divided into M ( Figure 4For example, M is an even number) video segments, namely, video segment 1 to video segment M, for adjacent video segments such as the i-th video segment and the i+1-th video segment, the last video frame of the i-th video segment (abbreviated as the last frame) and the first video frame of the i+1-th video segment (abbreviated as the first frame) are adjacent video frames, and for each video frame in the adjacent video frames, a face detector is used to extract a face frame in the video frame; for the extracted face frame, a face feature extractor is used to extract a face ID feature and a face depth feature in the face frame, and a HOG feature extractor is used to extract a HOG feature of the face frame; after feature fusion of the extracted face ID feature, face depth feature and HOG feature of the face frame, the cosine distance is calculated to obtain a distance metric matrix of the adjacent video frames, namely, a cross-scene distance feature, and a connection relationship between the face frames in the adjacent video frames is obtained by Hungarian matching of the cross-scene distance feature, thereby realizing cross-scene face tracking.
[0113] By introducing facial depth features and HOG features, face tracking can be achieved in scenarios such as face occlusion, side face, and transition, thereby improving the stability of face tracking.
[0114] For example, Figure 5 A schematic diagram of the connection relationship between face frames in different video frames provided in an embodiment of the present application, such as Figure 5 As shown, the i-th video segment includes N video frames. The last video frame of the i-th video segment, i.e., the N-th frame, and the first video frame in the i+1-th video segment, i.e., the 1st frame, are adjacent video frames with different scenes. The cross-scene distance feature is used to establish a connection relationship between the face frames in the two video frames, i.e., the connection relationship between face frame 51 and face frame 53. The face frames connected by a straight line are the same face frame and correspond to the same face. The first video frame and the second video frame, i.e., the 2nd frame, in the i+1-th video segment are adjacent video frames with the same scene. The MIOU feature is used to establish a connection relationship between the face frames in the two video frames, i.e., the connection relationship between face frame 52 and face frame 54. And so on. By continuously extending the connection relationship of the face frames, the connection relationship of the face frames in the entire video segment or even the entire video to be identified can be achieved.
[0115] Step S309, for each face frame in the video segment, according to the connection relationship of the face frame, obtain the facial features of the face frame from the facial features of the video frame containing the face frame, splice the facial features of the face frame in the video frame containing the face frame, and obtain the facial feature matrix of the face frame.
[0116] The facial feature matrix of the face frame is used to describe the changes in the facial features of the face frame in the video segment, including the changes in the key points of the face mouth.
[0117] Specifically, for a face frame identified in a certain video segment, facial features of the face frame in each video frame containing the face frame in the video segment are extracted in sequence according to the connection relationship of the face frame, such as facial key point features, where the facial key points include multiple key points on the mouth. The extracted facial features of the face frame are spliced in the order of video frames or time to obtain a facial feature matrix of the face frame.
[0118] Step S310: identifying a speaker from the faces included in the video segment based on the facial feature matrix and the audio features.
[0119] Specifically, for each face frame detected in the video segment, the facial feature matrix of the face frame and the audio features of the audio data corresponding to the video segment are input into the speaker recognition network model, and the speaker recognition network model identifies the speaker's face frame from the face frames contained in the video segment. Since different face frames correspond to different faces, the speaker can be identified by identifying the speaker's face frame.
[0120] The speaker recognition network model can analyze the lip shape change characteristics of the face in the input face frame based on the facial feature matrix of the face frame, and then determine whether the face in the face frame is the speaker through the matching results of the lip shape change characteristics and the audio features.
[0121] Figure 6 A schematic diagram of a speaker recognition network model provided in an embodiment of the present application, such as Figure 6 As shown, the speaker recognition network model may include an audio encoder, a face encoder, and a speaker recognition layer; the audio encoder is used to encode the audio data of the input video segment to obtain audio features; the face encoder is used to encode each face frame contained in each video frame in the video segment to obtain a face feature matrix of each face frame; the speaker recognition layer is used to determine the speaker's face frame from each face frame detected in the video segment based on the input audio features and face feature matrix. The speaker recognition network model can calculate the probability that each face frame detected in the video segment is the speaker's face frame based on the audio features and face feature matrix, and determine the face frame with the highest probability as the speaker's face frame.
[0122] In order to expand the training set when training the speaker recognition network model, the contrastive learning framework can be used to train the speaker recognition network model. By maximizing the consistency between positive samples and minimizing the consistency between negative samples, the model is trained and the model parameters are continuously optimized.
[0123] During the training set collection phase of the speaker recognition network model, multiple video samples can be obtained from public video materials, such as news broadcasts, and the multiple video samples can include multiple languages. Scene detection is performed on the video samples to divide the video samples into multiple video segmentation samples; audio data and video frames in the video segmentation samples are separated; and positive samples in the training samples can be obtained from the audio data and video frames separated from the video segmentation samples manually. The audio data in the negative samples can be obtained by translating the audio data of the positive samples into other languages, or by replacing the audio data in other positive samples with the audio data in the positive samples.
[0124] In this embodiment, a multilingual speech representation model is used to extract audio features of audio data, thereby realizing speech feature expression in multilingual scenarios, and realizing speaker recognition in cross-language multi-person conversation videos, thereby expanding the scope of application. In addition to the biological features of the face, the depth features and HOG features of the face are combined to realize cross-scene face tracking, effectively overcoming the problem of poor face tracking accuracy in scenarios such as large-scale face movement, such as changing from a front face to a side face, and face occlusion, thereby improving the stability of face tracking and the accuracy of speaker recognition.
[0125] Figure 7 A flowchart of another speaker recognition method provided in an embodiment of the present application is shown. Based on the above embodiment, this embodiment adds a step of selecting a face frame based on face key points before step S205 or step S310, such as Figure 7 As shown, the speaker recognition method provided in this embodiment may further include the following steps before recognizing the speaker from the face included in the video segment:
[0126] Step S701: for a face frame detected in each video segment, extract lip key points of the face frame from a video frame corresponding to the face frame in the video segment.
[0127] For each face frame detected by the video segment, the position of the lip key point of the face frame is extracted from the video frame of the video segment including the face frame.
[0128] If the facial features extracted in the aforementioned step include the position of the lip key points, then the step here can be adjusted to extract the lip key points from the facial features extracted from the face frame.
[0129] The lip key points include multiple key points located on the lips, such as key points on the outer edge line of the upper lip, key points on the outer edge line of the lower lip, etc., and may also include key points on the inner edge line of the upper lip and key points on the inner edge line of the lower lip.
[0130] Step S702: Calculate the lip offset of the face frame in the video segment based on the lip key points of the face frame in the video frame corresponding to the face frame in the video segment.
[0131] Specifically, the offset of one or more lip key points or the overall lip key points specified by a face frame in the video segment in the time series can be calculated, and the lip offset of the face frame in the video segment can be obtained based on the statistical values of the calculated offsets, such as the average value, the maximum value, etc.
[0132] Exemplarily, the lip offset of the face frame in the video segment may be the offset of the lip key point of the face frame with the largest offset in the video segment.
[0133] Step S703, removing face frames whose lip offsets are less than a preset threshold in the video segment, and using the remaining face frames as candidate face frames to identify the speaker from the candidate face frames.
[0134] If the lip offset of the face frame in the video segment is less than a preset threshold, the face frame is deleted.
[0135] After deleting the face frames whose lip offset is less than a preset threshold in the video segment, the remaining face frames are regarded as candidate face frames to identify the speaker from the candidate face frames in the video segment.
[0136] The face frames are screened by lip offset before speaker recognition, which reduces the number of face frames to be recognized and improves the efficiency of speaker recognition.
[0137] Figure 8 A flowchart of another speaker recognition method provided in an embodiment of the present application is shown as follows: Figure 8 As shown, the speaker recognition method mainly includes the following steps:
[0138] Step S801, scene detection: performing scene detection on the video to be identified provided by the user, and dividing the video to be identified into multiple video segments based on the result of the scene detection.
[0139] Step S802, audio and video separation: for each video segment, separate the audio data (abbreviated as audio) and video frame in the video segment.
[0140] Step S803, voice separation: for audio data, separate the background color therein to achieve separation of the human voice and the background sound.
[0141] Step S804, audio feature extraction: extract features of the separated human voice to obtain audio features, such as wav2vec features.
[0142] Step S805, face detection and tracking: perform face detection on the video frame to obtain the face frame in the video frame and the key features of the face in the face frame; use the face tracking algorithm to determine the connection relationship of the face frame.
[0143] Specifically, the connection relationship between face frames in the same scene can be established through the MIOU features in the aligned face frames, and the connection relationship between face frames in different scenes can be established through the MIOU features, face depth features and HOG features in the face frames.
[0144] Step S806, key point screening: using the offset of the lip key point in the face frame, the face frame is screened in units of video segments.
[0145] Step S807, speaker recognition: for the remaining face frames after screening in the video segmentation, i.e., candidate face frames, a time series is obtained based on the facial key point features in the face frames, i.e., a facial feature matrix of the face frames, and the speaker is identified from the remaining face frames after screening in the video segmentation.
[0146] Furthermore, after identifying the speaker in the video, the speaker can be annotated in the video, that is, the time periods corresponding to different speakers can be annotated, so that the video can be edited based on the preset speaker input by the user to obtain a video clip in which only the preset speaker is speaking, that is, a video clip of the time period corresponding to the preset speaker can be edited from the video.
[0147] Furthermore, after identifying the speaker in the video, the audio in the video can be replaced with the translated audio, and the lip shape of the corresponding face in the video can be modified based on the speakers corresponding to each segment in the translated audio to avoid inconsistency between the translated audio and the lip shape in the video.
[0148] Fig. 9 A schematic diagram of the structure of a speaker recognition device provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the speaker recognition device includes: a scene detection module 910 , an audio and video separation module 920 , a first feature extraction module 930 , a second feature extraction module 940 and a speaker recognition module 950 .
[0149] The scene detection module 910 is used to perform scene detection on the video to be identified, and divide the video to be identified into multiple video segments based on the result of the scene detection; the audio and video separation module 920 is used to separate the audio data and video frames in each video segment in the multiple video segments; the first feature extraction module 930 is used to extract the facial features of the video frames and the audio features of the audio data; the second feature extraction module 940 is used to extract the facial depth features of the multiple video frames with scene switching in the multiple video segments, and calculate the distance between the facial depth features of adjacent video frames in the multiple video frames to obtain a cross-scene distance feature; the speaker recognition module 950 is used to identify the speaker from the faces contained in the video segments based on the cross-scene distance feature, the facial feature and the audio feature.
[0150] Optionally, the second feature extraction module 940 includes:
[0151] A depth feature and HOG extraction unit is used to extract the face depth features and directional gradient histogram features of the face frames of the multiple video frames of scene switching in the multiple video segments; a feature fusion unit is used to fuse the face depth features and the directional gradient histogram features of the same video frame to obtain the cross-scene fusion features of the video frame; a distance calculation unit is used to calculate the distance of the cross-scene fusion features of adjacent video frames in the multiple video frames to obtain the cross-scene distance features.
[0152] Optionally, the facial features are used to describe features of a facial frame corresponding to a face included in a video frame. The speaker recognition module 950 includes:
[0153] A cross-scene connection relationship determination unit is used to determine the connection relationship of the same face frame in adjacent video frames of scene switching in the multiple video segments based on the cross-scene distance features; the connection relationship of the face frames is used to describe the positional relationship of the same face frame in different video frames; a same-scene connection relationship determination unit is used to determine the connection relationship of the same face frame in adjacent video frames in the same video segment based on the face features; a face frame feature acquisition unit is used to obtain the facial features of each face frame in the video segment from the facial features of the video frame containing the face frame according to the connection relationship of the face frame, and splice the facial features of the face frame in the video frame containing the face frame to obtain the facial feature matrix of the face frame; a speaker recognition unit is used to recognize the speaker from the faces contained in the video segment based on the facial feature matrix and the audio features.
[0154] Optionally, the cross-scenario connection relationship determination unit is specifically used to:
[0155] Hungarian matching is performed on cross-scene distance features of adjacent video frames of scene switching in the multiple video segments to obtain connection relationships of the same face frames in adjacent video frames of scene switching in the multiple video segments.
[0156] Optionally, the facial features are used to describe features of a face frame corresponding to a face included in a video frame, and the speaker recognition device further includes a face frame screening module, which is used to:
[0157] Before identifying the speaker from the faces included in the video segment based on the cross-scene distance feature, the facial feature and the audio feature, for the face frame detected in each video segment, the lip key points of the face frame are extracted from the video frame corresponding to the face frame in the video segment; based on the lip key points of the face frame in the video frame corresponding to the face frame in the video segment, the lip offset of the face frame in the video segment is calculated; the face frames in the video segment whose lip offsets are less than a preset threshold are eliminated, and the remaining face frames are used as candidate face frames to identify the speaker from the candidate face frames.
[0158] Optionally, when extracting the audio features of the audio data, the first feature extraction module 930 is specifically used to:
[0159] The audio data is input into a multilingual speech representation model to obtain audio features of the audio data; the training set of the multilingual speech representation model includes audio samples in multiple languages.
[0160] The face model correction device provided in the embodiment of the present application can be used to execute the technical solution of the face model correction method provided in any of the above embodiments of the present application. The implementation principle and technical effect are similar, and this embodiment will not be repeated here.
[0161] Fig.10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.10 As shown, the electronic device of this embodiment may include: at least one processor 1001; and a memory 1002 communicatively connected to the at least one processor; wherein the memory 1002 stores instructions executable by the at least one processor 1001, and the instructions are executed by the at least one processor 1001 so that the electronic device executes the method described in any of the above embodiments.
[0162] Optionally, the memory 1002 may be independent or integrated with the processor 1001 .
[0163] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the aforementioned embodiments and will not be described in detail here.
[0164] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the method provided in any of the aforementioned embodiments can be implemented.
[0165] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method provided in any of the above embodiments when executed by a processor.
[0166] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.
[0167] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application.
[0168] It should be understood that the above-mentioned processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The memory may include high-speed memory, and may also include non-volatile memory, such as at least one disk memory, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.
[0169] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.
[0170] An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in a dedicated integrated circuit. Of course, the processor and the storage medium can also exist as discrete components in the electronic device or the control device of the vehicle sentry mode.
[0171] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0172] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0173] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods provided in each embodiment of the present application.
[0174] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0175] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A speaker recognition method, characterized in that: include: Performing scene detection on the video to be identified, and dividing the video to be identified into multiple video segments based on the result of the scene detection; For each video segment in the multiple video segments, separate and obtain audio data and video frames in the video segment; Extracting facial features of the video frame and extracting audio features of the audio data; For a plurality of video frames of scene switching in the plurality of video segments, extracting face depth features of the plurality of video frames, and calculating distances between face depth features of adjacent video frames in the plurality of video frames to obtain cross-scene distance features; Based on the cross-scene distance feature, the facial feature and the audio feature, a speaker is identified from the faces included in the video segment.
2. The method according to claim 1, characterized in that For a plurality of video frames of scene switching in the plurality of video segments, extracting face depth features of the plurality of video frames, and calculating distances between face depth features of adjacent video frames in the plurality of video frames to obtain cross-scene distance features, including: For a plurality of video frames of scene switching in the plurality of video segments, extracting face depth features and directional gradient histogram features of face frames of the plurality of video frames; Fusing the face depth feature and the directional gradient histogram feature of the same video frame to obtain a cross-scene fusion feature of the video frame; The distances of the cross-scene fusion features of adjacent video frames in the multiple video frames are calculated to obtain the cross-scene distance features.
3. The method according to claim 1 or 2, characterized in that: The facial features are used to describe features of a facial frame corresponding to a face included in a video frame, and based on the cross-scene distance features, the facial features, and the audio features, identifying a speaker from a face included in the video segment includes: Determine, based on the cross-scene distance feature, a connection relationship of the same face frame in adjacent video frames of scene switching in the multiple video segments, and determine, based on the face feature, a connection relationship of the same face frame in adjacent video frames in the same video segment; the connection relationship of the face frame is used to describe the position relationship of the same face frame in different video frames; For each face frame in the video segment, according to the connection relationship of the face frame, obtain the face feature of the face frame from the face features of the video frame containing the face frame, splice the face features of the face frame in the video frame containing the face frame, and obtain the face feature matrix of the face frame; Based on the facial feature matrix and the audio features, a speaker is identified from the faces included in the video segment.
4. The method according to claim 3, characterized in that Determining, based on the cross-scene distance feature, a connection relationship between the same face frames in adjacent video frames of scene switching in the multiple video segments, comprising: Hungarian matching is performed on cross-scene distance features of adjacent video frames of scene switching in the multiple video segments to obtain connection relationships of the same face frames in adjacent video frames of scene switching in the multiple video segments.
5. The method according to claim 1 or 2, characterized in that: The facial features are used to describe features of a facial frame corresponding to a face included in a video frame. Before identifying a speaker from a face included in the video segment based on the cross-scene distance features, the facial features, and the audio features, the method further includes: For a face frame detected in each video segment, extracting lip key points of the face frame from a video frame corresponding to the face frame in the video segment; Calculate the lip offset of the face frame in the video segment based on the lip key points of the face frame in the video frame corresponding to the face frame in the video segment; The face frames whose lip offsets in the video segment are less than a preset threshold are eliminated, and the remaining face frames are used as candidate face frames to identify the speaker from the candidate face frames.
6. The method according to claim 1 or 2, characterized in that: Extracting audio features of the audio data includes: The audio data is input into a multilingual speech representation model to obtain audio features of the audio data; the training set of the multilingual speech representation model includes audio samples in multiple languages.
7. A speaker recognition device, characterized in that: include: A scene detection module, used to perform scene detection on the video to be identified, and divide the video to be identified into multiple video segments based on the result of the scene detection; An audio and video separation module, used for separating each video segment in the multiple video segments to obtain audio data and video frames in the video segment; A first feature extraction module, used to extract facial features of the video frame and audio features of the audio data; A second feature extraction module is used to extract face depth features of a plurality of video frames of scene switching in the plurality of video segments, and calculate the distance between the face depth features of adjacent video frames in the plurality of video frames to obtain a cross-scene distance feature; A speaker recognition module is used to recognize a speaker from the faces included in the video segment based on the cross-scene distance feature, the face feature and the audio feature.
8. An electronic device, characterized in that: include: memory and at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to perform the method according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 6 is implemented.
10. A computer program product, characterized in that The method comprises a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Multi-feature fusion-based deep learning face recognition method
CN107578007A
Gait recognition method and system based on deep neural network and machine vision
CN110909612A
Video mirror dividing method and device, electronic equipment and computer readable storage medium
CN111641869A
Face automatic exposure optimization method, storage medium, electronic equipment and system
CN113766260A
Multi-mode speaker identity recognition method and device and equipment
CN113920560A
Cited By
Speaking object detection method and device, electronic equipment and storage medium
CN119810884A
