Speaker identification method, device, computer equipment and storage medium

Abstract

The present application relates to a speaker identification method, apparatus, computer equipment, computer-readable storage medium, and computer program product. The method comprises: acquiring N frames of to-be-processed images from a video captured by a camera over a period of time, each frame of the to-be-processed image including at least one participant in the same conference room; for each frame of the to-be-processed image, determining an analysis area on the to-be-processed image based on the camera's lens field of view when capturing the to-be-processed image, the angle of arrival of the acoustic signal, and the extended length of the preset analysis area, and performing face detection on the analysis area to obtain all faces contained in the analysis area; and determining, based on all faces contained in the analysis area on each frame of the to-be-processed image and a pre-trained classification model, from at least one participant the participant who is currently speaking during the period of time. The above method is not affected by environmental noise and has higher stability.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a speaker identification method, apparatus, computer equipment, storage medium, and computer program product. Background Art

[0002] A video conference is a meeting where participants from multiple locations engage in face-to-face conversations over the internet. This method of conferencing offers significant convenience for multi-site office environments and greatly improves work efficiency. When a conference has many participants, being able to identify the speaker and capture a close-up video of them allows other participants to clearly see the speaker's expressions and movements.

[0003] In traditional technology, the sound wave signals transmitted by each participant in the venue can be collected, and the arrival angle of the sound wave signals corresponding to each participant can be calculated based on the sound wave signals. The participant whose sound wave signal arrival angle is closest to the preset threshold is regarded as the participant who is speaking.

[0004] However, there may be noisy conditions in the venue, which may affect the above recognition results. Therefore, the accuracy of the above method is not stable enough. Summary of the Invention

[0005] Based on this, it is necessary to provide a speaker identification method, apparatus, computer device, computer-readable storage medium and computer program product that can improve the stability of recognition results in response to the above technical problems.

[0006] In a first aspect, the present application provides a speaker identification method. The method comprises:

[0007] N frames of unprocessed images are obtained from videos captured by a camera over a period of time, each frame of the image to be processed includes at least one participant in the same conference room; for each frame of the image to be processed, an analysis area is determined on each frame of the image to be processed based on the lens field of view, the arrival angle of the acoustic signal, and the extended length of the preset analysis area when the camera captures the image to be processed, and face detection is performed on the analysis area to obtain all faces contained in the analysis area on each frame of the image to be processed; based on all faces contained in the analysis area on each frame of the image to be processed and a pre-trained classification model, a participant who is speaking within a period of time is determined from at least one participant.

[0008] In a second aspect, the present application also provides a speaker identification device. The device includes:

[0009] The acquisition module is used to obtain N frames of unprocessed images from videos captured by a camera over a period of time, where each frame of the image to be processed includes at least one participant in the same conference room. The detection module is used to determine an analysis area on each frame of the image to be processed based on the lens field of view, the arrival angle of the acoustic signal, and the extended length of the preset analysis area when the camera captures the image to be processed, and perform face detection on the analysis area to obtain all the faces contained in the analysis area on each frame of the image to be processed. The recognition module is used to determine, from at least one participant, the participant who is currently speaking within a period of time based on all the faces contained in the analysis area on each frame of the image to be processed and a pre-trained classification model.

[0010] In some embodiments, the acquisition module is specifically configured to:

[0011] Extract the target video within a preset time length before the current moment from the video captured by the camera within a period of time; extract N frames of images from the image frames contained in the target video, and use the extracted N frames of images as N frames of images to be processed.

[0012] In some embodiments, the detection module is specifically configured to:

[0013] According to the lens field of view angle and the arrival angle of the acoustic wave signal when the camera collects the image to be processed, the center line of the analysis area is determined on the image to be processed; according to the center line and the extension length, the boundary line of the analysis area is determined on the image to be processed, and the analysis area is determined based on the boundary line of the analysis area.

[0014] In some embodiments, the identification module is specifically configured to:

[0015] For each frame of the image to be processed, all faces contained in the analysis area on the image to be processed are filtered to obtain filtered faces, and the filtering process includes at least one of the following: quality filtering or mouth occlusion filtering; based on the filtered faces contained in the analysis area on each frame of the image to be processed, a face sequence corresponding to each face is obtained; based on the face sequence corresponding to each face and a pre-trained classification model, a participant who is speaking within a period of time is determined from at least one participant.

[0016] In some embodiments, the filtering process includes quality filtering, and the identification module is specifically configured to:

[0017] For each of the faces contained in the analysis area, calculate at least one of the clarity, brightness, and posture of the face; based on at least one of the clarity, brightness, and posture of the face, determine whether the face meets the quality requirements and obtain a quality judgment result; based on the quality judgment results corresponding to each face, determine the face after filtering.

[0018] In some embodiments, the filtering process includes mouth occlusion filtering, and the recognition module is specifically configured to:

[0019] All faces contained in the analysis area are input into the mouth occlusion detection model to obtain filtered faces.

[0020] In some embodiments, the identification module is specifically configured to:

[0021] The face sequence corresponding to each face is input into a pre-trained classification model to obtain the confidence level of each face as the speaker; the participant corresponding to the face with the highest confidence level is determined as the participant currently speaking.

[0022] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the following steps are performed:

[0023] N frames of unprocessed images are obtained from videos captured by a camera over a period of time, each frame of the image to be processed includes at least one participant in the same conference room; for each frame of the image to be processed, an analysis area is determined on each frame of the image to be processed based on the lens field of view, the arrival angle of the acoustic signal, and the extended length of the preset analysis area when the camera captures the image to be processed, and face detection is performed on the analysis area to obtain all faces contained in the analysis area on each frame of the image to be processed; based on all faces contained in the analysis area on each frame of the image to be processed and a pre-trained classification model, a participant who is speaking within a period of time is determined from at least one participant.

[0024] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0025] N frames of unprocessed images are obtained from videos captured by a camera over a period of time, each frame of the image to be processed includes at least one participant in the same conference room; for each frame of the image to be processed, an analysis area is determined on each frame of the image to be processed based on the lens field of view, the arrival angle of the acoustic signal, and the extended length of the preset analysis area when the camera captures the image to be processed, and face detection is performed on the analysis area to obtain all faces contained in the analysis area on each frame of the image to be processed; based on all faces contained in the analysis area on each frame of the image to be processed and a pre-trained classification model, a participant who is speaking within a period of time is determined from at least one participant.

[0026] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0027] N frames of unprocessed images are obtained from videos captured by a camera over a period of time, each frame of the image to be processed includes at least one participant in the same conference room; for each frame of the image to be processed, an analysis area is determined on each frame of the image to be processed based on the lens field of view, the arrival angle of the acoustic signal, and the extended length of the preset analysis area when the camera captures the image to be processed, and face detection is performed on the analysis area to obtain all faces contained in the analysis area on each frame of the image to be processed; based on all faces contained in the analysis area on each frame of the image to be processed and a pre-trained classification model, a participant who is speaking within a period of time is determined from at least one participant.

[0028] The above-mentioned speaker identification method, device, computer equipment, storage medium and computer program product obtain N frames of unprocessed images from the video captured by the camera over a period of time. For each frame of the unprocessed image, the analysis area is determined on each frame of the unprocessed image based on the lens field of view angle, the arrival angle of the sound wave signal and the extended length of the preset analysis area when the camera captures the unprocessed image. Subsequently, only the analysis area needs to be processed, which reduces the amount of calculation and improves the recognition efficiency. After determining the analysis area, face detection is performed on the analysis area to obtain all the faces contained in the analysis area on each frame of the unprocessed image; based on all the faces contained in the analysis area on each frame of the unprocessed image and the pre-trained classification model, the participant who is speaking within a period of time is determined from at least one participant. The above-mentioned method is not affected by environmental noise and has higher stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of an application system of a speaker identification method according to an embodiment;

[0030] Figure 2 1 is a flow chart of a speaker identification method according to an embodiment;

[0031] Figure 3 A schematic diagram of determining a center line in one embodiment;

[0032] Figure 4 A schematic diagram of determining an analysis area in one embodiment;

[0033] Figure 5 is a flowchart of a speaker identification method according to another embodiment;

[0034] Figure 6 is a flowchart of a speaker identification method in yet another embodiment;

[0035] Figure 7 is a structural block diagram of a speaker identification device in one embodiment;

[0036] Figure 8 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0038] The speaker identification method provided in the embodiment of the present application can be applied to Figure 1 In the system shown. Figure 1 The system shown includes: multiple terminals participating in the video conference, each terminal is connected via a network. Figure 1 Only two terminals are shown, but it is understood that the number of terminals may be greater than two. Figure 1 It does not constitute a limitation to the embodiments of the present application. The terminal is placed in the venue, and a camera and a microphone array are installed on the terminal. The microphone array can be used to receive sound wave signals and calculate the angle of arrival of the sound wave signals based on the received sound wave signals. The camera can be used to shoot videos in the venue. The terminal can execute the speaker identification method provided in the embodiment of the present application based on the angle of arrival of the sound wave signals output by the microphone array and the video output by the camera. After determining the participant who is speaking, a close-up video of the participant who is speaking can be collected and displayed. The close-up video can also be sent to other terminals. After receiving the close-up video, the other terminals can also display the close-up video, so that all participants can see the expressions and movements of the participants who are speaking in dialect more clearly, thereby enhancing the immersive feeling of the participants.

[0039] For example, see Figure 1 As shown, the terminals participating in the video conference are terminal 1 and terminal 2. Terminal 1 is placed in venue 1, and terminal 2 is placed in venue 2. There are four participants in venue 1, namely participant A, participant B, participant C, and participant D, and there are four participants in venue 2, namely participant E, participant F, participant G, and participant H. Terminal 1 is used to identify the participant who is speaking in venue 1, and terminal 2 is used to identify the participant who is speaking in venue 2. Assuming that during a video conference, terminal 1 recognizes that participant B in venue 1 is speaking, terminal 1 can subsequently capture a close-up video of participant B and display it. It can also send the close-up video to other terminals. After receiving the close-up video, the other terminals can also display the close-up video, allowing all participants to see participant B's expressions and movements more clearly.

[0040] It should be noted that: after the terminal recognizes the participant who is speaking, collecting a close-up video of the participant who is speaking is only an example, and the recognition result can also be applied to other purposes, which is not limited in this embodiment of the present application.

[0041] The terminal can be any device integrated with a camera and microphone array. Exemplary terminals include personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices can include smart watches, smart bracelets, and head-mounted devices. The terminal form factor is not limited in the embodiments of this application. The specific process of the speaker identification method is described below with reference to specific examples.

[0042] In one embodiment, Figure 2 As shown, a speaker identification method is provided, which can be applied to Figure 1 Any terminal in the present application embodiment is applied to Figure 1 Taking the terminal 1 in FIG. 1 as an example, the speaker identification method includes the following steps:

[0043] S202: Obtain N frames of images to be processed from videos captured by a camera within a period of time, where each frame of the image to be processed includes at least one participant in the same conference room.

[0044] The camera may capture video at a preset video frame rate. For example, assuming the preset video frame rate is 30 fps, the camera may capture video at a speed of 30 frames per second.

[0045] In some embodiments, the video is composed of frames of images, and the terminal 1 can use the current moment as a reference and use the most recently received N frames of images as N frames of images to be processed.

[0046] Among them, since the terminal 1 is placed in the conference room 1, considering that the purpose of the embodiment of the present application is to identify the participant who is speaking from among many participants, the camera can be aimed at the participants in the conference room 1, so that the participants will be included in the image captured by the camera. The participant who is speaking can be identified based on the image captured by the camera. The number of participants in the conference room 1 can be 1 or more than 1. Figure 1 It is indicated that there are 4 participants in venue 1, but this does not constitute a limitation to the embodiment of the present application.

[0047] S204. For each frame of the image to be processed, an analysis area is determined on each frame of the image to be processed based on the field of view of the camera lens when the image to be processed is captured, the arrival angle of the acoustic signal, and the extended length of the preset analysis area. Face detection is performed on the analysis area to obtain all faces contained in the analysis area on each frame of the image to be processed.

[0048] Among them, the video is composed of frames of images. During the process of capturing the video, the camera can use the same lens field of view to capture each frame of image, or it can adjust the lens field of view regularly or according to pre-set rules. The embodiments of the present application do not limit this.

[0049] Among them, the camera can capture images and the microphone array can receive sound wave signals synchronously. For example, each time the camera captures an image, it can send a synchronization signal to the microphone array. After receiving the synchronization signal, the microphone array starts to receive the sound wave signal and calculates the angle of arrival (Direction of Arrival, DOA) of the sound wave signal based on the received sound wave signal. It should be noted that the above-mentioned method of achieving synchronization is only an example. Other methods can also be used to achieve the synchronization of the camera capturing images and the microphone array receiving sound wave signals. The embodiments of this application will not be repeated here.

[0050] A microphone array and signal processing system can be pre-built. Given the known speed of sound and the spacing between array elements, the time difference between the sound signals received by the two microphones can be calculated. Once this time difference is obtained, the angle of arrival of the sound signal can be calculated using a corresponding formula. For specific formulas, refer to the prior art and are not limited in this embodiment.

[0051] In some embodiments, after N frames of images to be processed are acquired through S102, for each frame of the image to be processed, the lens field of view angle when the camera captures the image, and the sound wave signal arrival angle calculated by the microphone array based on the synchronously received sound wave signal can be acquired. Then, based on the acquired lens field of view angle and sound wave signal arrival angle, a center line is determined on the image to be processed, and an analysis area is determined based on the center line. In the subsequent processing process, only the analysis area needs to be processed, which reduces the terminal calculation amount and improves the recognition efficiency.

[0052] In some embodiments, for each frame of image to be processed, after determining the analysis area, face detection can be performed on the analysis area through methods such as convolutional neural network, three-dimensional recognition, skin texture analysis, local binary pattern histogram, etc. to obtain all faces contained in the analysis area.

[0053] S206 : Determine, from at least one conference participant, a participant who is currently speaking within a period of time based on all faces contained in the analysis area of ​​each frame of the image to be processed and a pre-trained classification model.

[0054] In some embodiments, after acquiring N frames of images to be processed in S201, each frame of the images to be processed is processed in S204. In other words, faces contained in the corresponding analysis area of ​​the N frames of images to be processed can be obtained. Thus, for each face, a corresponding face sequence can be obtained. Based on the face sequence corresponding to each face and a pre-trained classification model, a participant who is currently speaking within a period of time can be determined from at least one participant.

[0055] For example, assume that terminal 1 obtains 10 frames of images to be processed through S202 and obtains the faces contained in the analysis area of ​​each frame of the image to be processed through S204. Assume that the faces contained in the analysis area of ​​each frame of the image to be processed are: the face of participant A, the face of participant B, the face of participant C, and the face of participant D. Since there are 10 frames of images to be processed, the face sequence of participant A, the face sequence of participant B, the face sequence of participant C, and the face sequence of participant D can be obtained through overlap calculation. Each participant's face series includes 10 facial images. Based on the face sequence of participant A, the face sequence of participant B, the face sequence of participant C, and the face sequence of participant D and a pre-trained classification model, the participant who is currently speaking within a period of time can be determined from at least one participant.

[0056] Among them, when training the classification model, a training set can be constructed first. For example, multiple images and speakers manually labeled for the multiple images can be used as a training sample. After the training set is constructed, the initial model can be trained based on the training set to finally obtain a classification model.

[0057] In some embodiments, after obtaining a face sequence corresponding to a face, the face sequence can be input into a classification model, which can then output a confidence score that the face is the speaker. Based on the confidence scores corresponding to each face, the participant who was currently speaking within a certain period of time can be identified from at least one participant.

[0058] It should be noted that the method provided in the embodiment of the present application can be executed periodically. For example, it can be executed every 30 seconds, or triggered by the user. The embodiment of the present application is not limited to this.

[0059] The speaker identification method provided in the embodiment of the present application obtains N frames of to-be-processed images from a video captured by a camera over a period of time. For each frame of to-be-processed image, an analysis area is determined on each frame of to-be-processed image based on the lens field of view angle, the arrival angle of the sound wave signal, and the extended length of the preset analysis area when the camera captures the to-be-processed image. Subsequently, only the analysis area needs to be processed, which reduces the amount of calculation and improves the recognition efficiency. After determining the analysis area, face detection is performed on the analysis area to obtain all faces contained in the analysis area on each frame of to-be-processed image; based on all faces contained in the analysis area on each frame of to-be-processed image and a pre-trained classification model, a participant who is speaking within a period of time is determined from at least one participant. The above method is not affected by environmental noise and has higher stability.

[0060] In some embodiments, the step of acquiring N frames of images to be processed from videos captured by a camera over a period of time includes:

[0061] Extract the target video within a preset time length before the current moment from the video captured by the camera within a period of time; extract N frames of images from the image frames contained in the target video, and use the extracted N frames of images as N frames of images to be processed.

[0062] Among them, the preset duration can be set by the user through the user interface, or it can be a default value, and the embodiment of the present application does not limit this.

[0063] Among them, the target video within the preset time length before the current moment can be the target video within the most recent preset time length based on the current moment. For example, the current moment is 12:30:03, the preset time length is 1.5 seconds, and the target video is the video between 12:30:01.5 seconds and 12:30:03 seconds.

[0064] Among them, one frame of image can be extracted from the target video every other frame, so that N frames of image can be obtained. For example, assuming that the video frame rate used by the camera when capturing the video is 30fps, then 45 frames of image can be captured in 1.5 seconds. Among these 45 frames of image, one frame of image can be extracted every other frame, so 22 frames of image (extracted from the second frame) or 23 frames of image (extracted from the first frame) can be obtained. The extracted 22 frames of image can be used as the images to be processed, or the extracted 23 frames of image can be used as the images to be processed.

[0065] In the above embodiment, an implementation method for obtaining N frames of images to be processed is provided. Considering that in video scenes, there are many similarities between adjacent image frames and the information provided is limited, the above embodiment evenly extracts N frames of images from the target video as N frames of images to be processed, thereby ensuring the validity of the image information and reducing the amount of computation in the subsequent processing process.

[0066] In some embodiments, the step of determining the analysis area on each frame of the image to be processed based on the field of view angle of the camera lens during acquisition, the arrival angle of the acoustic wave signal, and the extended length of the preset analysis area includes:

[0067] According to the lens field of view angle and the arrival angle of the acoustic wave signal when the camera collects the image to be processed, the center line of the analysis area is determined on the image to be processed; according to the center line and the extended length of the analysis area, the boundary line of the analysis area is determined on the image to be processed, and the analysis area is determined based on the boundary line of the analysis area.

[0068] In some embodiments, X_DOA is calculated by the following formula:

[0069] (α-30) / β*X=X_DOA

[0070] Among them, α is the arrival angle of the sound wave signal, β is the field of view of the lens, and X is the width of the image to be processed.

[0071] In some embodiments, after obtaining X_DOA, the centerline can be determined based on X_DOA. Figure 3 As shown, Figure 3 The width of the image to be processed is X, and the line L that is X_DOA away from the left side of the image to be processed can be used as the center line.

[0072] The extension length may be set by the user through the user interface or may be a default value, which is not limited in the present embodiment. An exemplary extension length may be X / 6.

[0073] In some embodiments, after the center line is determined, the line to the left of the center line and the distance from the center line to the center line by the above-mentioned extension length can be used as the left boundary line of the analysis area, and the line to the right of the center line and the distance from the center line to the center line by the above-mentioned extension length can be used as the right boundary line of the analysis area. The analysis area can be determined based on the left boundary line and the right boundary line. For example, see Figure 4 As shown, assuming that the extension length is X / 6, after determining the center line L, the line M on the left side of the center line and at a distance of X / 6 from the center line can be used as the left boundary line, and the line M on the right side of the center line and at a distance of X / 6 from the center line can be used as the right boundary line, and the area O formed by the left boundary line and the right boundary line can be used as the analysis area.

[0074] In the above embodiment, an implementation method for determining the analysis area is provided, and only the analysis area needs to be processed subsequently, which reduces the amount of calculation and improves the recognition efficiency.

[0075] In some embodiments, see Figure 5As shown, the step of determining a participant who is currently speaking within a period of time from at least one participant based on all faces contained in the analysis area of ​​each frame of the image to be processed and a pre-trained classification model includes:

[0076] S501: For each frame of the image to be processed, filter all faces contained in the analysis area of ​​the image to be processed to obtain filtered faces, where the filtering includes at least one of the following: quality filtering or mouth occlusion filtering;

[0077] Considering that the speaker's face generally has high clarity and brightness, and is oriented in a normal direction, we can first filter out all faces within the analysis area that have low clarity and brightness, or face orientations that are abnormal. For ease of explanation, this filtering process is referred to as quality filtering in this embodiment.

[0078] Among them, considering that speakers generally do not cover their mouths during speeches, mouth occlusion filtering can be performed on the faces remaining after quality filtering.

[0079] It should be noted that the filtering process involved in this step can include only quality filtering, only mouth occlusion filtering, or both quality and mouth occlusion filtering. When the filtering process includes both quality and mouth occlusion filtering, for each frame of the image to be processed, quality filtering can be performed first, followed by mouth occlusion filtering. Alternatively, mouth occlusion filtering can be performed first, followed by quality filtering. This embodiment of the present application is not limited to this.

[0080] It should be noted that after acquiring N frames of images to be processed, filtering is performed on each frame of the images to be processed, so that filtered faces contained in the analysis areas of the N frames of images to be processed can be obtained.

[0081] S502 : Obtain a face sequence corresponding to each face according to the faces after mouth occlusion filtering contained in the analysis area of ​​each frame of the image to be processed.

[0082] Among them, through S501, the filtered faces contained in the analysis areas of the N frames of images to be processed can be obtained, and the overlap detection can be performed on the analysis areas of the N frames of images to be processed to obtain a face sequence corresponding to each face.

[0083] For example, after the aforementioned quality filtering and mouth occlusion filtering, the analysis area of ​​each of the N frames to be processed contains four faces. For each face, overlap detection can be used to find the face in the analysis area of ​​each frame to be processed. Since there are N frames to be processed, N faces can be found, and these N faces can form a face sequence corresponding to the face. This operation is repeated for each face, resulting in a face sequence corresponding to each of the four faces.

[0084] S503: Based on the face sequence corresponding to each face and the pre-trained classification model, determine a participant who is speaking within a period of time from at least one participant.

[0085] In some embodiments, the facial sequence corresponding to each face can be input into a pre-trained classification model, which will output a classification result for the corresponding face, indicating whether the corresponding face is a speaker or a silent person. The classification model will also output a confidence score. The participant corresponding to the face classified as a speaker can be determined as the participant currently speaking. If multiple faces are classified as speakers, it is assumed that there may be multiple speakers in the same time period. The confidence scores corresponding to each of the multiple faces are then obtained, and the participant corresponding to the face with the highest confidence score is determined as the participant currently speaking.

[0086] In the above embodiment, before using the classification model to identify the speaker, quality filtering and mouth occlusion filtering are used to filter out faces that are obviously not speakers, and the remaining faces are then identified using the classification model, which reduces model pressure and improves recognition efficiency.

[0087] In some embodiments, when the filtering process includes quality filtering, the step of filtering all faces contained in the analysis area of ​​the image to be processed to obtain filtered faces includes:

[0088] For each of the faces contained in the analysis area, calculate at least one of the clarity, brightness, and posture of the face; based on at least one of the clarity, brightness, and posture of the face, determine whether the face meets the quality requirements and obtain a quality judgment result; based on the quality judgment results corresponding to each face, determine the face after filtering.

[0089] Among them, for each face, the variance of the Laplace operator of the face can be calculated, and the variance of the Laplace operator can be used as the clarity of the face; the mean of the grayscale image of the face can be calculated, and the mean can be used as the brightness of the face; the Euler angle of the face can be calculated, and the Euler angle can be used as the posture of the face.

[0090] In some embodiments, the user can pre-set a clarity threshold, a brightness threshold, and a posture threshold. For each face, after obtaining the clarity, face brightness, and face posture corresponding to the face, it is determined whether the clarity corresponding to the face is greater than or equal to the clarity threshold, whether the face brightness is greater than or equal to the brightness threshold, and whether the face posture is greater than or equal to the posture threshold. If one of the judgment results is yes, or two of the judgment results are yes, or all three judgment results are yes, it is determined that the face meets the quality requirements. In this case, the quality judgment result is passed.

[0091] In some embodiments, after obtaining the quality judgment results corresponding to all faces contained in the analysis area, faces with failed quality judgment results can be eliminated, and the remaining faces can be used as filtered faces.

[0092] In the above embodiment, a method for implementing quality filtering is provided. Quality filtering can filter out faces that do not meet at least one of the speaker's requirements in terms of clarity, brightness, and facial posture, thereby avoiding the waste of computing resources caused by subsequently inputting these faces that are not displayed as speakers into the classification model.

[0093] In some embodiments, when the filtering process includes mouth occlusion filtering, the step of filtering all faces contained in the analysis area of ​​the image to be processed to obtain filtered faces includes:

[0094] All faces contained in the analysis area are input into the mouth occlusion detection model to obtain filtered faces.

[0095] In some embodiments, a training set can be pre-constructed. Any image, along with a manually annotated label indicating whether the mouth is occluded, can be used as a sample in the training set. An initial model can be trained based on the training set, ultimately yielding a mouth occlusion detection model. All faces within the analysis area can be input into the mouth occlusion detection model, which then outputs a result indicating whether the mouth is occluded for each face. Faces with occluded mouths are removed, and the remaining faces serve as filtered faces.

[0096] In the above embodiment, the implementation method of the mouth occlusion filtering is provided, which can filter out faces with occluded mouths, thereby avoiding the waste of computing resources caused by subsequently inputting these faces that are not speakers into the classification model.

[0097] In one embodiment, see Figure 6 As shown, a speaker identification method is provided, comprising:

[0098] S601: Extract target videos within a preset time period before the current moment from videos captured by a camera within a period of time.

[0099] S602 : Extract N frames of images from the image frames included in the target video, and use the extracted N frames of images as N frames of images to be processed.

[0100] S603: For each frame of the image to be processed, determine the center line of the analysis area on the image to be processed based on the camera's field of view and the angle of arrival of the acoustic wave signal when the camera captured the image to be processed. Determine the boundary line of the analysis area on the image to be processed based on the center line and the extended length. The analysis area is determined based on the boundary line of the analysis area. Face detection is then performed on the analysis area to obtain all faces within the analysis area.

[0101] S604: For each frame of the image to be processed, for each of all faces within the analysis area, calculate at least one of facial clarity, facial brightness, and facial posture; determine whether the face meets quality requirements based on at least one of the facial clarity, facial brightness, and facial posture, and obtain a quality judgment result; and determine a quality-filtered face based on the quality judgment result corresponding to each face. The quality-filtered face is input into a mouth occlusion detection model to obtain a filtered face.

[0102] S605 : Obtain a face sequence corresponding to each face according to the filtered faces contained in the analysis area of ​​each frame of the image to be processed.

[0103] S606: Input the face sequence corresponding to each face into a pre-trained classification model to obtain the confidence level of each face as the speaker.

[0104] S607: Determine the participant corresponding to the face with the highest confidence as the participant who is speaking.

[0105] The speaker identification method provided in the embodiment of the present application obtains N frames of to-be-processed images from a video captured by a camera over a period of time. For each frame of to-be-processed image, an analysis area is determined on each frame of to-be-processed image based on the lens field of view angle, the arrival angle of the sound wave signal, and the extended length of the preset analysis area when the camera captures the to-be-processed image. Subsequently, only the analysis area needs to be processed, which reduces the amount of calculation and improves the recognition efficiency. After determining the analysis area, face detection is performed on the analysis area to obtain all faces contained in the analysis area of ​​each frame of to-be-processed image; based on all faces contained in the analysis area on each frame of to-be-processed image and a pre-trained classification model, a participant who is speaking within a period of time is determined from at least one participant. The above method is not affected by environmental noise and has higher stability.

[0106] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0107] Based on the same inventive concept, embodiments of the present application also provide a speaker identification device for implementing the aforementioned speaker identification method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more speaker identification device embodiments provided below can be found in the above-described limitations of the speaker identification method and will not be further elaborated here.

[0108] In one embodiment, Figure 7 As shown, a speaker identification device is provided, comprising:

[0109] The acquisition module 701 is used to obtain N frames of unprocessed images from the video captured by the camera over a period of time, where each frame of the image to be processed includes at least one participant in the same conference room; the detection module 702 is used to determine the analysis area on each frame of the image to be processed based on the lens field of view, the arrival angle of the acoustic signal, and the extended length of the preset analysis area when the camera captures the image to be processed, and perform face detection on the analysis area to obtain all the faces contained in the analysis area on each frame of the image to be processed; the recognition module 703 is used to determine the participant who is speaking within a period of time from at least one participant based on all the faces contained in the analysis area on each frame of the image to be processed and a pre-trained classification model.

[0110] In some embodiments, the acquisition module 701 is specifically configured to:

[0111] Extract the target video within a preset time length before the current moment from the video captured by the camera within a period of time; extract N frames of images from the image frames contained in the target video, and use the extracted N frames of images as N frames of images to be processed.

[0112] In some embodiments, the detection module 702 is specifically configured to:

[0113] According to the lens field of view angle and the arrival angle of the acoustic wave signal when the camera collects the image to be processed, the center line of the analysis area is determined on the image to be processed; according to the center line and the extension length, the boundary line of the analysis area is determined on the image to be processed, and the analysis area is determined based on the boundary line of the analysis area.

[0114] In some embodiments, the identification module 703 is specifically configured to:

[0115] For each frame of the image to be processed, all faces contained in the analysis area on the image to be processed are filtered to obtain filtered faces, and the filtering process includes at least one of the following: quality filtering or mouth occlusion filtering; based on the filtered faces contained in the analysis area on each frame of the image to be processed, a face sequence corresponding to each face is obtained; based on the face sequence corresponding to each face and a pre-trained classification model, a participant who is speaking within a period of time is determined from at least one participant.

[0116] In some embodiments, the filtering process includes quality filtering, and the identification module 703 is specifically configured to:

[0117] For each of the faces contained in the analysis area, calculate at least one of the clarity, brightness, and posture of the face; based on at least one of the clarity, brightness, and posture of the face, determine whether the face meets the quality requirements and obtain a quality judgment result; based on the quality judgment results corresponding to each face, determine the face after filtering.

[0118] In some embodiments, the filtering process includes mouth occlusion filtering, and the recognition module 703 is specifically configured to:

[0119] All faces contained in the analysis area are input into the mouth occlusion detection model to obtain filtered faces.

[0120] In some embodiments, the identification module 703 is specifically configured to:

[0121] The face sequence corresponding to each face is input into a pre-trained classification model to obtain the confidence level of each face as the speaker; the participant corresponding to the face with the highest confidence level is determined as the participant currently speaking.

[0122] Each module in the speaker identification device described above can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0123] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, memory, communication interface, display screen, camera, microphone array and input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a speaker identification method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0124] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0125] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0126] N frames of unprocessed images are obtained from videos captured by a camera over a period of time, each frame of the image to be processed includes at least one participant in the same conference room; for each frame of the image to be processed, an analysis area is determined on each frame of the image to be processed based on the lens field of view, the arrival angle of the acoustic signal, and the extended length of the preset analysis area when the camera captures the image to be processed, and face detection is performed on the analysis area to obtain all faces contained in the analysis area on each frame of the image to be processed; based on all faces contained in the analysis area on each frame of the image to be processed and a pre-trained classification model, a participant who is speaking within a period of time is determined from at least one participant.

[0127] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0128] Extract the target video within a preset time length before the current moment from the video captured by the camera within a period of time; extract N frames of images from the image frames contained in the target video, and use the extracted N frames of images as N frames of images to be processed.

[0129] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0130] According to the lens field of view angle and the arrival angle of the acoustic wave signal when the camera collects the image to be processed, the center line of the analysis area is determined on the image to be processed; according to the center line and the extension length, the boundary line of the analysis area is determined on the image to be processed, and the analysis area is determined based on the boundary line of the analysis area.

[0131] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0132] For each frame of the image to be processed, all faces contained in the analysis area on the image to be processed are filtered to obtain filtered faces, and the filtering process includes at least one of the following: quality filtering or mouth occlusion filtering; based on the filtered faces contained in the analysis area on each frame of the image to be processed, a face sequence corresponding to each face is obtained; based on the face sequence corresponding to each face and a pre-trained classification model, a participant who is speaking within a period of time is determined from at least one participant.

[0133] In one embodiment, the filtering process includes quality filtering, and when the processor executes the computer program, the processor further implements the following steps:

[0134] For each of the faces contained in the analysis area, calculate at least one of the clarity, brightness, and posture of the face; based on at least one of the clarity, brightness, and posture of the face, determine whether the face meets the quality requirements and obtain a quality judgment result; based on the quality judgment results corresponding to each face, determine the face after filtering.

[0135] In one embodiment, the filtering process includes mouth occlusion filtering, and the processor, when executing the computer program, further implements the following steps:

[0136] All faces contained in the analysis area are input into the mouth occlusion detection model to obtain filtered faces.

[0137] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0138] The face sequence corresponding to each face is input into a pre-trained classification model to obtain the confidence level of each face as the speaker; the participant corresponding to the face with the highest confidence level is determined as the participant currently speaking.

[0139] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0140] N frames of unprocessed images are obtained from videos captured by a camera over a period of time, each frame of the image to be processed includes at least one participant in the same conference room; for each frame of the image to be processed, an analysis area is determined on each frame of the image to be processed based on the lens field of view, the arrival angle of the acoustic signal, and the extended length of the preset analysis area when the camera captures the image to be processed, and face detection is performed on the analysis area to obtain all faces contained in the analysis area on each frame of the image to be processed; based on all faces contained in the analysis area on each frame of the image to be processed and a pre-trained classification model, a participant who is speaking within a period of time is determined from at least one participant.

[0141] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0142] Extract the target video within a preset time length before the current moment from the video captured by the camera within a period of time; extract N frames of images from the image frames contained in the target video, and use the extracted N frames of images as N frames of images to be processed.

[0143] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0144] According to the lens field of view angle and the arrival angle of the acoustic wave signal when the camera collects the image to be processed, the center line of the analysis area is determined on the image to be processed; according to the center line and the extension length, the boundary line of the analysis area is determined on the image to be processed, and the analysis area is determined based on the boundary line of the analysis area.

[0145] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0146] For each frame of the image to be processed, all faces contained in the analysis area on the image to be processed are filtered to obtain filtered faces, and the filtering process includes at least one of the following: quality filtering or mouth occlusion filtering; based on the filtered faces contained in the analysis area on each frame of the image to be processed, a face sequence corresponding to each face is obtained; based on the face sequence corresponding to each face and a pre-trained classification model, a participant who is speaking within a period of time is determined from at least one participant.

[0147] In one embodiment, the filtering process includes quality filtering, and the computer program, when executed by a processor, further implements the following steps:

[0148] For each of the faces contained in the analysis area, calculate at least one of the clarity, brightness, and posture of the face; based on at least one of the clarity, brightness, and posture of the face, determine whether the face meets the quality requirements and obtain a quality judgment result; based on the quality judgment results corresponding to each face, determine the face after filtering.

[0149] In one embodiment, the filtering process includes mouth occlusion filtering, and the computer program, when executed by the processor, further implements the following steps:

[0150] All faces contained in the analysis area are input into the mouth occlusion detection model to obtain filtered faces.

[0151] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0152] N frames of unprocessed images are obtained from videos captured by a camera over a period of time, each frame of the image to be processed includes at least one participant in the same conference room; for each frame of the image to be processed, an analysis area is determined on each frame of the image to be processed based on the lens field of view, the arrival angle of the acoustic signal, and the extended length of the preset analysis area when the camera captures the image to be processed, and face detection is performed on the analysis area to obtain all faces contained in the analysis area on each frame of the image to be processed; based on all faces contained in the analysis area on each frame of the image to be processed and a pre-trained classification model, a participant who is speaking within a period of time is determined from at least one participant.

[0153] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0154] Extract the target video within a preset time length before the current moment from the video captured by the camera within a period of time; extract N frames of images from the image frames contained in the target video, and use the extracted N frames of images as N frames of images to be processed.

[0155] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0156] According to the lens field of view angle and the arrival angle of the acoustic wave signal when the camera collects the image to be processed, the center line of the analysis area is determined on the image to be processed; according to the center line and the extension length, the boundary line of the analysis area is determined on the image to be processed, and the analysis area is determined based on the boundary line of the analysis area.

[0157] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0158] For each frame of the image to be processed, all faces contained in the analysis area on the image to be processed are filtered to obtain filtered faces, and the filtering process includes at least one of the following: quality filtering or mouth occlusion filtering; based on the filtered faces contained in the analysis area on each frame of the image to be processed, a face sequence corresponding to each face is obtained; based on the face sequence corresponding to each face and a pre-trained classification model, a participant who is speaking within a period of time is determined from at least one participant.

[0159] In one embodiment, the filtering process includes quality filtering, and the computer program, when executed by a processor, further implements the following steps:

[0160] For each of the faces contained in the analysis area, calculate at least one of the clarity, brightness, and posture of the face; based on at least one of the clarity, brightness, and posture of the face, determine whether the face meets the quality requirements and obtain a quality judgment result; based on the quality judgment results corresponding to each face, determine the face after filtering.

[0161] In one embodiment, the filtering process includes mouth occlusion filtering, and the computer program, when executed by the processor, further implements the following steps:

[0162] All faces contained in the analysis area are input into the mouth occlusion detection model to obtain filtered faces.

[0163] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0164] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] The above embodiments merely illustrate several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A speaker identification method, characterized in that: The method comprises: Obtain N frames of images to be processed from the video captured by the camera within a period of time, where each frame of the image to be processed includes at least one participant in the same conference room; For each frame of the image to be processed, an analysis area is determined on each frame of the image to be processed based on the lens field of view angle, the arrival angle of the acoustic wave signal, and the extension length of the preset analysis area when the camera captures the image to be processed, and face detection is performed on the analysis area to obtain all faces contained in the analysis area on each frame of the image to be processed; Determining, from the at least one participant, a participant who is speaking during the period of time based on all faces contained in the analysis area of ​​each frame of the image to be processed and a pre-trained classification model; The step of determining an analysis area on each frame of the image to be processed according to the field of view angle of the camera when the camera captures the image to be processed, the arrival angle of the acoustic wave signal, and the extended length of the preset analysis area comprises: Determining a center line of an analysis area on the image to be processed based on a lens field of view angle and an arrival angle of an acoustic wave signal when the camera captures the image to be processed; According to the center line and the extension length, a boundary line of an analysis area is determined on the image to be processed, and the analysis area is determined based on the boundary line of the analysis area.

2. The method according to claim 1, characterized in that The step of obtaining N frames of images to be processed from videos collected by a camera within a period of time includes: Extract the target video within a preset time period before the current moment from the video captured by the camera over a period of time; N frames of images are extracted from the image frames included in the target video, and the extracted N frames of images are used as the N frames of images to be processed.

3. The method according to claim 1, characterized in that The determining, from the at least one participant, a participant who is speaking within the period of time based on all faces contained in the analysis area of ​​each frame of the image to be processed and a pre-trained classification model, includes: For each frame of the image to be processed, filtering all faces contained in the analysis area of ​​the image to be processed to obtain filtered faces, wherein the filtering includes at least one of the following: quality filtering or mouth occlusion filtering; Obtaining a face sequence corresponding to each face according to the filtered faces contained in the analysis area of ​​each frame of the image to be processed; Based on the face sequence corresponding to each face and the pre-trained classification model, a participant who is speaking within the period of time is determined from the at least one participant.

4. The method according to claim 3, characterized in that The filtering process includes quality filtering, filtering all faces contained in the analysis area on the image to be processed to obtain filtered faces, including: For each of all the faces contained in the analysis area, calculating at least one of the clarity of the face, the brightness of the face, and the posture of the face; Determining whether the face meets quality requirements based on at least one of the clarity of the face, the brightness of the face, and the posture of the face, and obtaining a quality judgment result; Based on the quality judgment results corresponding to each face, the filtered face is determined.

5. The method according to claim 3, characterized in that The filtering process includes mouth occlusion filtering, and filtering all faces contained in the analysis area on the image to be processed to obtain filtered faces includes: All human faces contained in the analysis area are input into the mouth occlusion detection model to obtain filtered human faces.

6. The method according to claim 3, characterized in that The determining, based on the face sequence corresponding to each face and the pre-trained classification model, a participant who is speaking from the at least one participant includes: Inputting the face sequence corresponding to each face into the pre-trained classification model to obtain the confidence level of each face as the speaker; The participant corresponding to the face with the highest confidence score is determined as the participant who is speaking.

7. A speaker identification device, characterized in that: The device comprises: An acquisition module is used to acquire N frames of images to be processed from the video captured by the camera within a period of time, where each frame of the image to be processed includes at least one participant in the same conference room; a detection module for determining, for each frame of the image to be processed, an analysis area on each frame of the image to be processed based on the field of view of the camera when the image to be processed is acquired, the angle of arrival of the acoustic wave signal, and the extension length of a preset analysis area, and performing face detection on the analysis area to obtain all faces contained in the analysis area on each frame of the image to be processed; a recognition module configured to determine, from the at least one participant, a participant who is speaking during the period of time based on all faces contained in an analysis area of ​​each frame of the image to be processed and a pre-trained classification model; The detection module is specifically used to: Determining a center line of an analysis area on the image to be processed based on a lens field of view angle and an arrival angle of an acoustic wave signal when the camera captures the image to be processed; According to the center line and the extension length, a boundary line of an analysis area is determined on the image to be processed, and the analysis area is determined based on the boundary line of the analysis area.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images