A conference speaker tracking method, device, computer equipment and storage medium

By monitoring the lips area of ​​the meeting participants to determine the speech status, the error problem under the influence of noise in the prior art is solved, and quick and accurate speech-person identification is achieved.

CN112633219BActive Publication Date: 2024-12-03SHENZHEN HAOLI INTELLIGENT TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202011609884.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-30
Publication Date
2024-12-03
Estimated Expiration
2040-12-30

AI Technical Summary

Technical Problem

The prior art is susceptible to noise when tracking conference speakers, resulting in errors and cannot accurately determine the spokesperson.

Method used

By obtaining the face image information of the participants, calculating the lips area, and monitoring in real time whether the preset area threshold is reached. If it is reached, it will be determined as the target person and obtain their audio and video information for playback.

Benefits of technology

Quickly and accurately identify the conference speakers, avoid noise interference, and improve the accuracy of speaker identification.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112633219B_ABST
    Figure CN112633219B_ABST
Patent Text Reader

Abstract

The present invention discloses a conference speaker tracking method, device, computer equipment and storage medium, the method comprising: obtaining video information containing facial image information of each participant, and obtaining the lip area of ​​each participant based on the facial image information of each participant; real-time monitoring of the lip area of ​​each participant, and determining whether there is a participant whose lip area reaches a preset area threshold; if there is a participant whose lip area reaches the preset area threshold, the participant whose lip area reaches the preset area threshold is taken as a target person; obtaining audio information of the target person, playing the audio information of the target person and projecting the video information of the target person. The present invention can quickly determine whether the corresponding participant is in a speaking state by monitoring and judging the lip area of ​​each participant, thereby quickly and accurately determining the conference speaker.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer application technology, and in particular to a conference speaker tracking method, device, computer equipment and storage medium. Background Art

[0002] Currently, the existing technology uses the speaker's audio and audio timestamp to track and identify the conference speaker, but this method is easily affected by noise, resulting in errors when tracking the speaker, and thus the conference speaker cannot be accurately identified. Summary of the invention

[0003] The embodiments of the present invention provide a conference speaker tracking method, apparatus, computer equipment and storage medium, aiming to quickly and accurately determine the conference speaker.

[0004] In a first aspect, an embodiment of the present invention provides a conference speaker tracking method, comprising:

[0005] Acquire video information including facial image information of each participant, and acquire the lip area of ​​each participant based on the facial image information of each participant;

[0006] Monitor the lip area of ​​each participant in real time and determine whether there are participants whose lip area reaches a preset area threshold;

[0007] If there is a participant whose lip area reaches the preset area threshold, the participant whose lip area reaches the preset area threshold is taken as the target person;

[0008] The audio information of the target person is obtained, and the audio information of the target person is played and the video information of the target person is projected.

[0009] In a second aspect, an embodiment of the present invention provides a conference speaker tracking device, comprising:

[0010] A video information acquisition unit, used to acquire video information including facial image information of each participant, and acquire the lip area of ​​each participant based on the facial image information of each participant;

[0011] A real-time detection unit, used to monitor the lip area of ​​each participant in real time and determine whether there is a participant whose lip area reaches a preset area threshold;

[0012] A first target determination unit, configured to, if there is a participant whose lip area reaches a preset area threshold, take the participant whose lip area reaches the preset area threshold as a target person;

[0013] The first playback and projection unit is used to obtain the audio information of the target person, play the audio information of the target person, and project the video information of the target person.

[0014] In a third aspect, an embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the conference speaker tracking method as described in the first aspect when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the conference speaker tracking method as described in the first aspect is implemented.

[0016] The embodiment of the present invention provides a conference speaker tracking method, device, computer equipment and storage medium, the method comprising: obtaining video information containing facial image information of each participant, and obtaining the lip area of ​​each participant based on the facial image information of each participant; real-time monitoring of the lip area of ​​each participant, and determining whether there is a participant whose lip area reaches a preset area threshold; if there is a participant whose lip area reaches the preset area threshold, the participant whose lip area reaches the preset area threshold is taken as a target person; obtaining audio information of the target person, playing the audio information of the target person and projecting the video information of the target person. The embodiment of the present invention can quickly determine whether the corresponding participant is in a speaking state by monitoring and judging the lip area of ​​each participant, thereby quickly and accurately determining the conference speaker. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0018] Figure 1 A flowchart of a conference speaker tracking method provided by an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of a sub-process of step S101 in a conference speaker tracking method provided by an embodiment of the present invention;

[0020] Figure 3 A schematic block diagram of a conference speaker tracking device provided by an embodiment of the present invention;

[0021] Figure 4 A sub-schematic block diagram of a video information acquisition unit 301 in a conference speaker tracking device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0024] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0025] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0026] See below Figure 1 , Figure 1 A flowchart of a conference speaker tracking method provided by an embodiment of the present invention specifically includes steps S101 to S104.

[0027] S101, obtaining video information including facial image information of each participant, and obtaining the lip area of ​​each participant based on the facial image information of each participant;

[0028] S102, monitoring the lip area of ​​each participant in real time, and determining whether there is a participant whose lip area reaches a preset area threshold;

[0029] S103: If there is a participant whose lip area reaches the preset area threshold, the participant whose lip area reaches the preset area threshold is taken as a target person;

[0030] S104: Acquire the audio information of the target person, play the audio information of the target person, and project the video information of the target person.

[0031] In this embodiment, when tracking and determining the conference speaker, the lip area of ​​the participant is used to determine whether the corresponding participant is speaking. When the lip area of ​​the participant reaches a preset area threshold, it can be determined that the corresponding participant is in a speaking state. At this time, the audio information of the participant (i.e., the target person) can be obtained, and the audio information of the participant can be played and the video information of the participant can be projected to inform the other participants.

[0032] This embodiment can quickly determine whether the corresponding participant is in a speaking state by monitoring and judging the lip area of ​​each participant, thereby quickly and accurately determining the conference speaker.

[0033] It is understandable that each participant in this embodiment can obtain video information and audio information through a conference terminal, or obtain video information through a camera installed around the current position of the participant and obtain audio information through a microphone installed around the participant. It should also be noted that the preset area threshold described in this embodiment can be set based on each participant without opening their mouth. In other words, when a participant opens his mouth, it is considered that the lip area of ​​the participant reaches the preset area threshold.

[0034] In one embodiment, if Figure 2 As shown, the step S101 includes: steps S201 to S205.

[0035] S201, for each participant, performing denoising and corrosion preprocessing on the facial image information;

[0036] S202, obtaining pixel coordinates in the preprocessed face image information, and gray-processing the face image information using the following formula to obtain a gray-scale face image:

[0037] I(x,y)=1 / 3*I_R(x,y)+1 / 3*I_G(x,y)+1 / 3*I_B(x,y)

[0038] Where I(x, y) is the coordinate of pixel I, I_R(x, y) is the R channel value of pixel I, I_G(x, y) is the G channel value of pixel I, and I_B(x, y) is the B channel value of pixel I;

[0039] S203, performing local threshold processing on the grayscale face image by using image binarization processing to obtain a lip region in the grayscale face image;

[0040] S204, removing noise points in the lip area by using a dilation-erosion algorithm, and determining the lip contour in the lip area by using a minimum circumscribed rectangle;

[0041] S205: Calculate the minimum circumscribed rectangle area, and use the calculation result as the lip area.

[0042] In this embodiment, in the video information of each participant, the facial image information of each participant can be obtained based on the video information, and a series of processing is performed on the facial image information to extract the lip area of ​​each participant. Specifically, the facial image information is first pre-processed, and the pre-processed facial image information is gray-scaled to obtain a corresponding gray-scale facial image, and then the gray-scale facial image is locally thresholded to obtain a lip area in the gray-scale facial image, and the lip area is denoised to determine the final lip area according to the minimum circumscribed rectangle.

[0043] In one embodiment, the step S101 further includes:

[0044] Determine the left and right corners of the mouth coordinates in the lip area using a corner detection algorithm;

[0045] Calculate the mouth corner lengths of the left and right mouth corners according to the abscissas in the left mouth corner coordinates and the abscissas in the right mouth corner coordinates, and compare the mouth corner lengths with the length of the minimum circumscribed rectangle;

[0046] If the length of the mouth corner is greater than the length of the minimum circumscribed rectangle, the product of the length of the mouth corner and the width of the minimum circumscribed rectangle is taken as the lip area;

[0047] If the length of the mouth corner is less than or equal to the length of the minimum circumscribed rectangle, the area of ​​the minimum circumscribed rectangle is used as the lip area.

[0048] Because when the minimum circumscribed rectangle is used to determine the lip contour within the lip area, the minimum circumscribed rectangle formed may not include the two coordinate points of the left corner of the lips and the right corner of the lips, this embodiment determines the coordinates of the left corner of the mouth and the right corner of the mouth through a corner point detection algorithm, thereby obtaining the distance between the left corner of the mouth and the right corner of the mouth (i.e., the length of the corner of the mouth), compares the length of the corner of the mouth with the length of the minimum circumscribed rectangle, and calculates the lip area based on the larger distance.

[0049] In one embodiment, the step S103 includes:

[0050] When there is a participant whose lip area reaches the preset area threshold, the participant whose lip area reaches the preset area threshold is taken as a candidate target person;

[0051] Continue to monitor the lip area of ​​the candidate target person;

[0052] The duration ratio of the lip area of ​​the candidate target person reaching the preset area threshold within the preset time threshold is obtained, and if the duration ratio is greater than the preset ratio threshold, the candidate target person is taken as the target person.

[0053] In this embodiment, even if the lip area of ​​a participant reaches the preset area threshold at a certain moment, it cannot be determined that the participant is speaking. It is also possible that the participant has only performed the action of opening his mouth, such as yawning, sneezing, etc. Therefore, in order to avoid the above situation, this embodiment continues to monitor the participant whose lip area reaches the preset area threshold (i.e., the candidate target person) to determine whether the participant is in a speaking state. Specifically, since a person's lips are in an open and closed state when speaking, that is, the lips will not always be in an open state, but may also be in a closed state. Therefore, during the continuous monitoring of the lip area of ​​the candidate target person, the lip area will not always reach the preset area threshold. Therefore, this embodiment obtains the time ratio of the lip area of ​​the candidate target person to reach the preset area threshold within the preset time threshold, and compares the time ratio with the preset ratio threshold. Only when the time ratio is greater than the preset ratio threshold, the participant is regarded as the target person.

[0054] For example, the preset ratio threshold is set to 1 / 2, and the lip area of ​​candidate target person A reaches the preset area threshold at time t1. At this time, the lip area of ​​candidate target person A continues to be monitored for 5 seconds (that is, the preset time threshold), and the duration ratio of the lip area of ​​candidate target person A reaching the preset area threshold within these 5 seconds is obtained as 4 / 5 (greater than 1 / 2), that is, within 5 seconds, there are 4 seconds when the lip area of ​​candidate target person A reaches the preset area threshold. Then, it can be determined that the candidate target person A is speaking, and candidate target person A is taken as the target person.

[0055] In one embodiment, the step of obtaining the time ratio of the lip area of ​​the candidate target person to a preset area threshold within a preset time threshold, and taking the candidate target person as the target person if the time ratio is greater than the preset ratio threshold, includes:

[0056] Obtaining the time ratio of the lip area of ​​the candidate target person reaching a preset area threshold within a preset time threshold;

[0057] Determining whether the duration ratio is , and determining whether the audio information of the candidate target person is detected;

[0058] If the duration ratio is greater than a preset ratio threshold and the audio information of the candidate target person is detected, the candidate target person is taken as the target person.

[0059] In this embodiment, when the ratio of the duration of the lip area of ​​the candidate target person reaching the preset area threshold within the preset time threshold is greater than the preset ratio threshold, the audio information of the candidate target person is detected to avoid the state where the candidate target person is just in the state of opening his mouth but not making any sound. That is to say, if the ratio of the duration of the lip area of ​​a candidate target person reaching the preset area threshold within the preset time threshold is greater than the preset ratio threshold, but the audio information of the candidate target person is not detected, then it will be determined that the candidate target person is not speaking.

[0060] In one embodiment, the conference speaker tracking method further includes:

[0061] When there are multiple participants whose lip areas reach the preset area threshold at the same time, all the participants whose lip areas reach the preset area threshold are regarded as target persons;

[0062] The audio and video information of each participant is obtained, and the video information of each target person is projected using multi-window split-screen projection, and the audio information of each target person is played in sequence according to the preset speaking rules.

[0063] In this embodiment, when there are multiple participants whose lip areas reach the preset area threshold at the same time, it means that these participants are all speaking. In order to avoid interference between the participants who are speaking, the preset speaking rules are used to make the participants speak in order. The preset speaking rules can be formulated according to the identity level of the target person. For example, in this meeting, the identity level of target person A is higher than that of target person B, so the audio information of target person A is played first; or it can be determined according to the time when the lip area of ​​the target person reaches the preset area threshold for the first time. For example, if the lip area of ​​target person A reaches the preset area threshold for the first time earlier than the lip area of ​​target person B, then the audio information of target person A is played first, and then the audio information of target person B is played.

[0064] Of course, when there are multiple target persons, although the playback order is set for the audio information of each target person, the video information of all target persons can be projected in split screen, that is, the video information of all target persons is projected in split screen, and the audio information of each target person is played in sequence according to the preset speech rules. Preferably, when playing the audio information of the current target person, the video information of the current target person is enlarged, that is, the projection window of the current target person is enlarged to highlight the current target person.

[0065] In one embodiment, the step S104 includes:

[0066] Use sound source localization technology to locate the acquired audio information;

[0067] When the positioning result is consistent with the position information of the target person, the audio information of the target person is played and the video information of the target person is projected.

[0068] In this embodiment, after the audio information of the target person is obtained, the obtained audio information is located by sound source localization technology to re-determine the position of the speaker corresponding to the audio information, so as to avoid errors in the process of obtaining the audio information of the target person, or to avoid confusion caused by too much collected audio information when there are multiple target persons, thereby causing the audio information finally played to be inconsistent with the audio information of the target person who is speaking.

[0069] In another embodiment, if there is a participant whose lip area reaches a preset area threshold, gesture detection is performed on the participant whose lip area reaches the preset area threshold, and if it is detected that the gesture amplitude of the participant whose lip area reaches the preset area threshold reaches the preset gesture action threshold, the participant whose lip area reaches the preset area threshold and whose gesture amplitude reaches the preset gesture action threshold is taken as the target person, and then the audio information of the target person is obtained, the audio information of the target person is played, and the video information of the target person is projected.

[0070] Figure 3 A schematic block diagram of a conference speaker tracking device 300 provided in an embodiment of the present invention, the device 300 includes:

[0071] The video information acquisition unit 301 is used to acquire video information including facial image information of each participant, and acquire the lip area of ​​each participant based on the facial image information of each participant;

[0072] The real-time detection unit 302 is used to monitor the lip area of ​​each participant in real time and determine whether there is a participant whose lip area reaches a preset area threshold;

[0073] The first target determination unit 303 is used to, if there is a participant whose lip area reaches a preset area threshold, take the participant whose lip area reaches the preset area threshold as a target person;

[0074] The first playback and projection unit 304 is used to obtain the audio information of the target person, play the audio information of the target person, and project the video information of the target person.

[0075] In one embodiment, if Figure 4 As shown, the video information acquisition unit 301 includes:

[0076] The preprocessing unit 401 is used to perform denoising and corrosion preprocessing on the face image information of each participant;

[0077] The grayscale processing unit 402 is used to obtain pixel coordinates in the preprocessed face image information, and grayscale the face image information using the following formula to obtain a grayscale face image:

[0078] I(x,y)=1 / 3*I_R(x,y)+1 / 3*I_G(x,y)+1 / 3*I_B(x,y)

[0079] Where I(x, y) is the coordinate of pixel I, I_R(x, y) is the R channel value of pixel I, I_G(x, y) is the G channel value of pixel I, and I_B(x, y) is the B channel value of pixel I;

[0080] A binarization processing unit 403 is used to perform local threshold processing on the grayscale face image by using image binarization processing to obtain a lip area in the grayscale face image;

[0081] The contour determining unit 404 is used to remove noise points in the lip area by using a dilation-erosion algorithm, and determine the lip contour in the lip area by using a minimum circumscribed rectangle;

[0082] The area calculation unit 405 is used to calculate the minimum circumscribed rectangle area and use the calculation result as the lip area.

[0083] In one embodiment, the video information acquisition unit 301 further includes:

[0084] A corner point detection unit, used for determining the coordinates of the left corner of the mouth and the right corner of the mouth in the lip area by using a corner point detection algorithm;

[0085] A comparison unit, used for calculating the mouth corner lengths of the left mouth corner and the right mouth corner according to the abscissa in the left mouth corner coordinates and the abscissa in the right mouth corner coordinates, and comparing the mouth corner lengths with the length of the minimum circumscribed rectangle;

[0086] A first area determination unit, configured to use the product of the length of the mouth corner and the width of the minimum circumscribed rectangle as the lip area if the length of the mouth corner is greater than the length of the minimum circumscribed rectangle;

[0087] The second area determination unit is configured to use the area of ​​the minimum circumscribed rectangle as the lip area if the length of the mouth corner is less than or equal to the length of the minimum circumscribed rectangle.

[0088] In one embodiment, the first target determination unit 303 includes:

[0089] A candidate target determination unit, configured to, when there is a participant whose lip area reaches a preset area threshold, determine the participant whose lip area reaches the preset area threshold as a candidate target person;

[0090] A candidate monitoring unit, used to continue monitoring the lip area of ​​the candidate target person;

[0091] The first ratio acquisition unit is used to obtain the time ratio of the lip area of ​​the candidate target person to reach a preset area threshold within a preset time threshold, and if the time ratio is greater than the preset ratio threshold, the candidate target person is used as the target person.

[0092] In one embodiment, the first ratio acquisition unit includes:

[0093] A second ratio acquisition unit is used to acquire the time ratio of the lip area of ​​the candidate target person reaching a preset area threshold within a preset time threshold;

[0094] A judging unit, used to judge whether the duration ratio is greater than a preset ratio threshold, and to judge whether the audio information of the candidate target person is detected;

[0095] As a unit, if the duration ratio is greater than a preset ratio threshold and the audio information of the candidate target person is detected, the candidate target person is used as the target person.

[0096] In one embodiment, the conference speaker tracking device 300 further includes:

[0097] A second target determination unit is used to, when there are multiple conference participants whose lip areas reach the preset area threshold at the same time, take all of the multiple conference participants whose lip areas reach the preset area threshold as target persons;

[0098] The split-screen projection unit is used to obtain the audio information and video information of each participant, and use multi-window split-screen projection to project the video information of each target person, and play the audio information of each target person in turn according to the preset speaking rules.

[0099] In one embodiment, the first playback and projection unit 304 includes:

[0100] A positioning unit, used to locate the acquired audio information using sound source localization technology;

[0101] The second playback and projection unit plays the audio information of the target person and projects the video information of the target person when the positioning result is consistent with the position information of the target person.

[0102] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0103] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0104] The embodiment of the present invention also provides a computer device, which may include a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps provided in the above embodiment may be implemented. Of course, the computer device may also include various network interfaces, power supplies and other components.

[0105] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

[0106] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A conference speaker tracking method, It is characterized in that include: Acquire video information including facial image information of each participant, and acquire the lip area of ​​each participant based on the facial image information of each participant; Monitor the lip area of ​​each participant in real time and determine whether there are participants whose lip area reaches a preset area threshold; If there is a participant whose lip area reaches the preset area threshold, the participant whose lip area reaches the preset area threshold is taken as the target person; Acquire the audio information of the target person, play the audio information of the target person, and project the video information of the target person; The step of obtaining video information including facial image information of each participant, and obtaining the lip area of ​​each participant based on the facial image information of each participant, includes: For each participant, performing denoising and corrosion preprocessing on the facial image information; Obtain pixel coordinates in the preprocessed face image information, and grayscale the face image information using the following formula to obtain a grayscale face image: I(x,y)=1 / 3*I_R(x,y)+1 / 3*I_G(x,y)+1 / 3*I_B(x,y) Where I(x, y) is the coordinate of pixel I, I_R(x, y) is the R channel value of pixel I, I_G(x, y) is the G channel value of pixel I, and I_B(x, y) is the B channel value of pixel I; Performing local threshold processing on the grayscale face image by using image binarization processing to obtain a lip region in the grayscale face image; removing noise points in the lip region by using a dilation-erosion algorithm, and determining the lip contour in the lip region by using a minimum circumscribed rectangle; Calculate the minimum circumscribed rectangle area, and use the calculation result as the lip area; The step of acquiring video information including facial image information of each participant, and extracting the lip area of ​​each participant based on the facial image information of each participant, further includes: Determine the left and right corners of the mouth coordinates in the lip area using a corner detection algorithm; Calculate the mouth corner lengths of the left and right mouth corners according to the abscissas in the left mouth corner coordinates and the abscissas in the right mouth corner coordinates, and compare the mouth corner lengths with the length of the minimum circumscribed rectangle; If the length of the mouth corner is greater than the length of the minimum circumscribed rectangle, the product of the length of the mouth corner and the width of the minimum circumscribed rectangle is taken as the lip area; If the length of the mouth corner is less than or equal to the length of the minimum circumscribed rectangle, the area of ​​the minimum circumscribed rectangle is used as the lip area; If there is a participant whose lip area reaches the preset area threshold, the participant whose lip area reaches the preset area threshold is taken as a target person, including: If there is a participant whose lip area reaches the preset area threshold, gesture detection is performed on the participant whose lip area reaches the preset area threshold. If it is detected that the gesture amplitude of the participant whose lip area reaches the preset area threshold reaches the preset gesture action threshold, the participant whose lip area reaches the preset area threshold and whose gesture amplitude reaches the preset gesture action threshold is taken as the target person.

2. The conference speaker tracking method according to claim 1, It is characterized in that If there is a participant whose lip area reaches the preset area threshold, the participant whose lip area reaches the preset area threshold is taken as a target person, including: When there is a participant whose lip area reaches the preset area threshold, the participant whose lip area reaches the preset area threshold is taken as a candidate target person; Continue to monitor the lip area of ​​the candidate target person; The duration ratio of the lip area of ​​the candidate target person reaching the preset area threshold within the preset time threshold is obtained, and if the duration ratio is greater than the preset ratio threshold, the candidate target person is taken as the target person.

3. The conference speaker tracking method according to claim 2, It is characterized in that The step of obtaining the time ratio of the lip area of ​​the candidate target person to a preset area threshold within a preset time threshold, and taking the candidate target person as the target person if the time ratio is greater than the preset ratio threshold, includes: Obtaining the time ratio of the lip area of ​​the candidate target person reaching a preset area threshold within a preset time threshold; Determining whether the duration ratio is greater than a preset ratio threshold, and determining whether the audio information of the candidate target person is detected; If the duration ratio is greater than a preset ratio threshold and the audio information of the candidate target person is detected, the candidate target person is taken as the target person.

4. The conference speaker tracking method according to claim 1, It is characterized in that Also includes: When there are multiple participants whose lip areas reach the preset area threshold at the same time, all the participants whose lip areas reach the preset area threshold are regarded as target persons; The audio and video information of each participant is obtained, and the video information of each target person is projected using multi-window split-screen projection, and the audio information of each target person is played in sequence according to the preset speaking rules.

5. The conference speaker tracking method according to claim 1, It is characterized in that The acquiring the audio information of the target person, playing the audio information of the target person, and projecting the video information of the target person include: Use sound source localization technology to locate the acquired audio information; When the positioning result is consistent with the position information of the target person, the audio information of the target person is played and the video information of the target person is projected.

6. A conference speaker tracking device, It is characterized in that include: A video information acquisition unit, used to acquire video information including facial image information of each participant, and acquire the lip area of ​​each participant based on the facial image information of each participant; A real-time detection unit, used to monitor the lip area of ​​each participant in real time and determine whether there is a participant whose lip area reaches a preset area threshold; A first target determination unit, configured to, if there is a participant whose lip area reaches a preset area threshold, take the participant whose lip area reaches the preset area threshold as a target person; If there is a participant whose lip area reaches the preset area threshold, the participant whose lip area reaches the preset area threshold is taken as a target person, including: If there is a participant whose lip area reaches the preset area threshold, gesture detection is performed on the participant whose lip area reaches the preset area threshold. If it is detected that the gesture amplitude of the participant whose lip area reaches the preset area threshold reaches the preset gesture action threshold, the participant whose lip area reaches the preset area threshold and whose gesture amplitude reaches the preset gesture action threshold is taken as the target person; A first playback and projection unit, used to obtain the audio information of the target person, play the audio information of the target person, and project the video information of the target person; The video information acquisition unit comprises: A preprocessing unit, used for performing denoising and corrosion preprocessing on the face image information of each participant; The grayscale processing unit is used to obtain pixel coordinates in the preprocessed face image information, and grayscale the face image information using the following formula to obtain a grayscale face image: I(x,y)=1 / 3*I_R(x,y)+1 / 3*I_G(x,y)+1 / 3*I_B(x,y) Where I(x, y) is the coordinate of pixel I, I_R(x, y) is the R channel value of pixel I, I_G(x, y) is the G channel value of pixel I, and I_B(x, y) is the B channel value of pixel I; A binarization processing unit, configured to perform local threshold processing on the grayscale face image using image binarization processing to obtain a lip region in the grayscale face image; A contour determination unit, used to remove noise points in the lip area by using a dilation-erosion algorithm, and determine the lip contour in the lip area by using a minimum circumscribed rectangle; An area calculation unit, used for calculating the area of ​​the minimum circumscribed rectangle and using the calculation result as the lip area; The video information acquisition unit also includes: A corner point detection unit, used for determining the coordinates of the left corner of the mouth and the right corner of the mouth in the lip area by using a corner point detection algorithm; A comparison unit, used for calculating the mouth corner lengths of the left mouth corner and the right mouth corner according to the abscissa in the left mouth corner coordinates and the abscissa in the right mouth corner coordinates, and comparing the mouth corner lengths with the length of the minimum circumscribed rectangle; A first area determination unit, configured to use the product of the length of the mouth corner and the width of the minimum circumscribed rectangle as the lip area if the length of the mouth corner is greater than the length of the minimum circumscribed rectangle; The second area determination unit is configured to use the area of ​​the minimum circumscribed rectangle as the lip area if the length of the mouth corner is less than or equal to the length of the minimum circumscribed rectangle.

7. A computer device, It is characterized in that The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the conference speaker tracking method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the conference speaker tracking method according to any one of claims 1 to 5 is implemented.

Citation Information

Patent Citations

  • Spokesman detecting system, spokesman detecting method and audio / video conference system

    CN103581608A

  • Method for monitoring call making and receiving behaviors of driver

    CN104573724A

  • A method, device and mobile terminal for automatically taking a picture

    CN104683680A