Speaker positioning method and system in video conference scene

Abstract

According to the spokesman positioning method and system in the video conference scene, the spokesman positioning accuracy and stability in the video conference scene are well improved, meanwhile, the situation that an area array pickup needs a plurality of pickups is avoided, and the product cost is effectively controlled. The method comprises the following steps of: 1, judging whether a speaker speaks or not, and if so, positioning the sound direction of the speaker; step 2, roughly positioning, and converting the sound orientation of the spokesman to the horizontal coordinate of the image; step 3, through face key point extraction, determining a probability value of speaking of each person; and step 4, accurately positioning, and determining the pixel position of the spokesman in the image by integrating the position information and the face key point features. The system comprises a linear array pickup module, a rough positioning module, a face key point extraction module and an accurate positioning module.

Description

TECHNICAL FIELD

[0001] The present application relates to a speaker positioning method and system in a video conference scenario. BACKGROUND

[0002] The current speaker positioning method in a video conference scenario is generally one of the following two methods: Method one: based on the mapping relationship between the face array microphone and the image, the sound source is positioned by the face array microphone, as shown in the Chinese patent with application number 201810434927.4; this method relies on a large number of microphones, and the product cost is relatively high.

[0003] Method two: based on image recognition algorithm, the change condition of each person's face and mouth is analyzed continuously for multiple frames, so as to determine the pixel position of the speaker in the image, as shown in the Chinese patent with application number 202111564524.X; this method has relatively unstable detection effect, and the mouth feature under different camera shooting angles is unstable, which is easily affected by the shooting angle, shooting distance, etc. SUMMARY

[0004] The purpose of the present application is to overcome the above-mentioned deficiencies in the prior art, and to provide a speaker positioning method and system in a video conference scenario, which is reasonable in design, low in cost and stable in detection effect.

[0005] The technical scheme adopted by the present application to solve the above-mentioned problems is: a speaker positioning method in a video conference scenario, comprising the following steps: Step one: determine whether there is a speaker; if so, locate the sound direction of the speaker; Step two: rough positioning, convert the sound direction of the speaker to the image horizontal coordinate; Step three: determine the probability value of each person's speech by face key point extraction; Step four: accurate positioning, determine the pixel position of the speaker in the image by combining position information and face key point features.

[0006] The step one of the present application specifically comprises the following steps: (1) determine whether there is a speaker; when the sound intensity continuously exceeds the threshold value and meets the human voice spectrum characteristics, it is considered that there is a speaker, and the following steps are entered; otherwise, it is considered that there is no speaker, and the next determination of whether there is a speaker is entered; (2) calculate the sound direction O_TD based on the sound arrival time difference and the sound direction O_SP based on the sound pressure amplitude ratio; According to the sound direction O_TD and the sound direction O_SP, the final sound direction O=a×O_TD+b×O_SP is calculated, wherein a and b are weighting coefficients.

[0007] a and b are dynamically adjusted according to the sound signal-to-noise ratio, and a+b=1 is satisfied.

[0008] The step two specifically comprises the following steps: (1) m points are taken in the image picture and sound is emitted, and the sound direction of the m points and the horizontal coordinates of the sound source in the image are recorded, and the m sets of relations are represented as (O0, X0), (O1, X1),..., (O m-1 , X m-1 ), wherein X0, X m-1 are the horizontal coordinates of the image; (2) based on the above step, the sound direction O of the speaker and the two closest values in the queue [O0, O1,..., O m-1 ] are found, and O i , O j are represented, the corresponding horizontal coordinates of the image are X i , X j , O i ≤O≤O j , and the horizontal coordinates X of the image corresponding to the sound direction O of the speaker are obtained according to the following formula: X=(O j -O) ÷(O j -O i )×X i +(O-O i ) ÷(O j -O i )×X j .

[0009] In the step two (1) of the application, the sound direction and image horizontal coordinate relation queue is sorted from small to large, so that O0 m-1 , X0 m-1 .

[0010] The step three specifically comprises the following steps: (1) all faces in the image are detected by using a deep learning target detection algorithm, and a face frame of each person is given; The intersection-over-union of the face frames of the front and back frames is calculated, and the best matching is obtained by using a two-part graph matching algorithm, so as to track the face, and a unique ID value is assigned to each face.

[0011] (2) based on the face frame given in step 1, the face key point information is calculated, and the distance between the upper and lower lips is further calculated; (3) the variance P m of the distance between the upper and lower lips in a continuous period of time is calculated for each face.

[0012] Step four of this invention specifically includes the following steps: (1) Calculate the probability distribution P of the sound source at different horizontal positions based on the speaker's horizontal coordinate X. s ; (2) Calculate the probability value P of each face speaking. s ×P m ; (3) The face with the largest P value is selected as the final speaker position.

[0013] This invention P s The calculation formula is as follows: When |XZ|≤D, P s = e -α(X-Z)×(X-Z) When |XZ|>D, P s =0; Where Z represents the horizontal coordinate of the face center point, α is an empirical value used to set the probability decreasing trend from the center to the sides, and D is the maximum error distance of coarse positioning.

[0014] A speaker location system for video conferencing scenarios, used to implement the aforementioned speaker location method for video conferencing scenarios, is characterized by comprising: The microphone module is used to locate the speaker's voice direction; The coarse positioning module is used to convert the speaker's voice location to the horizontal coordinates of the image; The facial landmark extraction module is used to determine the probability value of each person speaking. The precise positioning module is used to determine the speaker's pixel position in the image by combining the position information given by the microphone module and the facial key point features.

[0015] The microphone module of this invention consists of N microphones of the same specifications arranged horizontally at equal intervals; Compared with the prior art, the present invention has the following advantages and effects: The present invention greatly improves the accuracy and stability of speaker positioning in video conferencing scenarios, and unlike area array microphones, it does not require many microphones, thus effectively controlling product costs. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the overall workflow of the speaker location method according to an embodiment of the present invention.

[0017] Figure 2 This is a schematic diagram of the speaker positioning system according to an embodiment of the present invention. Detailed Implementation

[0018] The present invention will be further described in detail below through embodiments. The following embodiments are explanations of the present invention, and the present invention is not limited to the following embodiments.

[0019] I. For example Figure 1 As shown, the speaker location method in a video conferencing scenario according to an embodiment of the present invention includes the following steps: Step one: The microphone module determines if someone is speaking; if so, it locates the speaker's approximate azimuth. This step specifically includes the following steps: (1) Determine if someone is speaking; when the sound intensity continuously exceeds the threshold and conforms to the characteristics of the human voice spectrum, it is considered that someone is speaking and proceeds to the next step; otherwise, it is considered that no one is speaking and proceeds to the next judgment on whether someone is speaking.

[0020] (2) Calculate the sound direction O_TD based on the sound arrival time difference TDOA; Environmental factors such as temperature and humidity affect the speed of sound, and the speaker's face orientation affects the direction of sound propagation, leading to errors and fluctuations in the localization results. Further calculation of the sound orientation O_SP based on the sound pressure amplitude ratio is used to improve the stability of sound orientation localization. Based on the sound location O_TD and sound location O_SP, the final sound location O is calculated as O = a × O_TD + b × O_SP, where a and b are weighting coefficients, which are dynamically adjusted according to the sound signal-to-noise ratio to satisfy a + b = 1.

[0021] Step two, rough positioning, converting the speaker's voice location to the horizontal coordinates of the image. This step specifically includes the following steps: (1) Calibrate the system. This operation only needs to be calibrated and recorded once after the system is produced. The system does not need to be calibrated when it is working in real time.

[0022] Select m points in the image and emit sound from them. These m points should be as dispersed as possible. Record the relationship between the sound location of these m points and the horizontal coordinate of the sound source in the image, as provided by the microphone module. These m sets of relationships are represented by (O0,X0), (O1,X1), ..., (O...). m-1 X m-1 ) indicates that X0...X m-1 The horizontal coordinates of the image; For ease of subsequent calculations, the queue of sound location and image horizontal coordinate relationships is sorted from smallest to largest, ensuring that it simultaneously satisfies O0. <O1<...<O m-1 ,X0 <X1<...<X m-1 The further to the left of the sound source in the image, the smaller the sound's location.

[0023] (2) Based on the above steps, find the speaker's vocal location O and the queue [O0, O1, ... Om-1 The two closest values ​​in O[] are represented by O[] i O j This indicates that its corresponding horizontal coordinate is X. i X j Satisfying O i ≤O≤O j Then, the horizontal coordinate X of the image corresponding to the speaker's vocal orientation O can be obtained using the following formula: X=(O j -O) ÷(O j -O i )×X i +(OO i ) ÷(O j -O i )×X j .

[0024] Step 3: Determine the probability value of each person speaking by extracting facial landmarks. This step specifically includes the following steps: (1) Use deep learning object detection algorithm to detect all faces in the image and give the face bounding box of each person; The intersection-union ratio (IUU) of face bounding boxes in consecutive frames is calculated, and the best match is obtained using a bipartite graph matching algorithm, thereby tracking the face and assigning a unique ID value to each face.

[0025] (2) Based on the face bounding box given in step 1, calculate the key points of the face and further calculate the distance between the upper and lower lips.

[0026] (3) Calculate the variance P of the distance between the upper and lower lips for each face over a continuous period of time. m This value is a probability value used to represent each person's speech, calculated based on image recognition methods, and represents the variance P of the distance between the upper and lower lips. m The larger the value, the greater the probability that the person will speak. Step four: Accurately locate the speaker. Combining the location information from the microphone module with facial landmark features, determine the speaker's pixel position in the image. This step specifically includes the following steps: (1) Calculate the probability distribution P of the sound source at different horizontal positions based on the speaker's horizontal coordinate X. s The calculation formula is as follows: When |XZ|≤D, P s = e -α(X-Z)×(X-Z) When |XZ|>D, P s =0; Where Z represents the horizontal coordinate of the face center point, α is an empirical value used to set the probability decreasing trend from the center to the sides, and D is the maximum error distance of the microphone module's coarse positioning. (2) Calculate the probability value P of each face speaking. s ×P m This value is a probability value for each person to speak, calculated by combining graphical and audio statistics.

[0027] (3) The face with the largest P value is selected as the final speaker position.

[0028] II. Figure 2 As shown, the speaker positioning system in a video conferencing scenario according to an embodiment of the present invention includes: The linear array microphone module is used to locate the speaker's approximate azimuth angle. The linear array microphone module consists of N microphones of the same specifications arranged horizontally at equal intervals. Specifically, it can consist of 4, 6, 8, etc. The more microphones there are, the higher the accuracy, precision, and stability, but the higher the cost. The coarse positioning module is used to convert the speaker's voice location to the horizontal coordinates of the image; The facial landmark extraction module is used to determine the probability value of each person speaking. The precise positioning module is used to determine the speaker's pixel position in the image by combining the position information given by the linear array microphone module and the facial key point features.

[0029] Furthermore, it should be noted that the specific embodiments described in this specification may differ in the shape and name of their components, etc. The above description is merely illustrative of the structure of the present invention. All equivalent or simple variations made based on the structure, features, and principles described in this patent concept are included within the protection scope of this patent. Those skilled in the art can make various modifications or additions to the described specific embodiments or use similar methods to substitute them, as long as they do not deviate from the structure of the present invention or exceed the scope defined by the claims, all of which should fall within the protection scope of this invention.

Claims

1. A method for locating a speaker in a video conference scenario, characterized in that, It comprises the following steps: Step one, judge whether there is a speech, if so, locate the sound direction of the speaker; Step two, rough positioning, convert the sound direction of the speaker to the image horizontal coordinate; Step three, through face key point extraction, determine the probability value of each person's speech; Step four, accurate positioning, determine the pixel position of the speaker in the image by integrating the position information and face key point features.

2. The method of claim 1, wherein, The step one comprises the following steps: (1) judge whether there is a speech; when the sound intensity lasts more than the threshold value and meets the human voice spectrum characteristics, it is considered that there is a speech, and the following steps are entered; otherwise, it is considered that there is no speech, and the next judgment of whether there is a speech is entered; (2) calculate the sound direction O_TD based on the sound arrival time difference and the sound direction O_SP based on the sound pressure amplitude ratio; According to the sound direction O_TD and the sound direction O_SP, the final sound direction O=a×O_TD+b×O_SP is calculated, wherein a and b are weighting coefficients.

3. The speaker positioning method in the video conference scene according to claim 2, characterized in that, a and b are dynamically adjusted according to the sound signal-to-noise ratio, and a+b=1 is satisfied.

4. The speaker positioning method in the video conference scene according to claim 2 or 3, characterized in that, The step two comprises the following steps: (1) In the image screen, take m points and send out sound, record the m points sound direction and the relation of sound source in the image horizontal coordinate, this m group relation is expressed by (O0, X0), (O1, X1)…, (O m-1 , X m-1 ) , wherein X0…X m-1 is the image horizontal coordinate; (2) Based on the above steps, find the speaker's sound direction O and the closest two values in the queue [O0, O1,... O m-1 ], denoted as O i , O j , whose corresponding image horizontal coordinates are X i , X j , satisfying O i ≤O≤O j , the image horizontal coordinate X corresponding to the speaker's sound direction O is obtained according to the following formula: X = (O j -O) ÷ (O j -O i ) × X i +(O-O i ) ÷ (O j -O i ) × X j .

5. The method of claim 4, wherein, The sound direction and the image horizontal coordinate relationship queue are sorted from small to large in the step two (1), so that O0O1... O m-1 X0X1... X m-1 .

6. The method of claim 4 or 5, wherein, The step three comprises the following steps: (1) use a deep learning target detection algorithm to detect all faces in the image and give a face frame for each person; Calculate the intersection-over-union of the face frames of the previous and subsequent frames, and use a two-part graph matching algorithm to obtain the best match, thereby tracking the face and assigning a unique ID value to each face; (2) based on the face frame given in step 1, calculate the face key point information and further calculate the distance between the upper and lower lips; (3) calculating the variance P of the upper-lower lip distance for each face in a continuous period of time m .

7. The speaker positioning method in the video conference scene according to claim 6, characterized in that, The step four comprises the following steps: (1) According to the horizontal coordinate X of the speaker, the probability distribution P of the sound source at different horizontal positions is calculated s ; (2) Calculate the probability value P of each face speaking P = P s x P m ; (3) take the face with the maximum P value as the final speaker position.

8. The method of claim 7, wherein, P s The calculation formula is as follows: P = 1 when |X-Z| < D s = e -α(X-Z)×(X-Z) P = 0 when |X-Z| > D s = 0 Wherein Z represents the horizontal coordinate of the face center point, α is an empirical value for setting the probability decline trend from the center to both sides, and D is the maximum error distance of rough positioning.

9. A talker positioning system in a video conference scenario for implementing the talker positioning method in a video conference scenario according to any one of claims 1 to 8, characterized by It comprises: A microphone module for positioning the sound direction of the speaker; A rough positioning module for converting the sound direction of the speaker to the image horizontal coordinate; A face key point extraction module for determining the probability value of each person's speech; An accurate positioning module for determining the pixel position of the speaker in the image by integrating the position information and face key point features given by the microphone module.

10. The talker positioning system in a video conference scenario according to claim 9, characterized in that, The microphone module is composed of N microphones of the same specification arranged at equal intervals horizontally.

Images