Speaker tracking and positioning method based on machine vision, network device and storage medium

Abstract

The application provides a speaker tracking positioning method based on machine vision, network equipment and a storage medium, wherein the method comprises: acquiring a scene picture containing a person opening his mouth; obtaining a face region of interest and a mouth key point coordinate through the scene picture; performing normalization processing on the face region of interest and the mouth key point coordinate; taking an intersection of the area of the normalized face region of interest and the area of a buffered face region of interest; placing the face region of interest with an intersection area greater than a set threshold a in a same queue; and when the number of pictures placed in the same queue exceeds a set threshold b, calculating a variance of the normalized mouth key point coordinate, and determining whether the current detected person is speaking through the variance size. The embodiment of the application performs speaker tracking based on machine vision, is not affected by noise in a scene, and can identify a speaker, even a main speaker and a secondary speaker, even if multiple people are speaking at the same time.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication equipment, and particularly relates to a speaker tracking and positioning method based on machine vision, network equipment and a storage medium. BACKGROUND

[0002] In a conference scenario, the tracking of a speaker is traditionally based on manual positioning, which is slow and inefficient, and it is difficult to determine the speaker in the case of multiple people. Real-time positioning and tracking of a speaker and automatic close-up display of the speaker are good user experiences for a conference.

[0003] To the best of the inventors' knowledge, existing automatic speaker tracking is performed by collecting a sound source signal of a speaker through a microphone, positioning a target speaker by analysis, and tracking the speaker by controlling the position of a pan-tilt camera. However, the influence of noise in a conference scenario cannot be eliminated, and in the case where a speaker is far away from the microphone, if multiple people speak at the same time, the sound source positioning error is large and the speaker cannot be identified. SUMMARY

[0004] To overcome the problems in the prior art, the present application provides a speaker tracking and positioning method based on machine vision, network equipment and a storage medium.

[0005] According to a first aspect of an embodiment of the present application, a speaker tracking and positioning method based on machine vision includes the following steps.

[0006] Obtaining a scene picture containing a person opening his mouth;

[0007] Obtaining a face region of interest and a mouth key point coordinate from the scene picture;

[0008] Performing normalization processing on the face region of interest and the mouth key point coordinate;

[0009] Taking an intersection of the normalized face region of interest and a cached face region of interest in terms of area;

[0010] Placing the normalized face region of interest and the cached face region of interest with an intersection area greater than a set threshold a in a same queue;

[0011] When the number of pictures of the normalized face region of interest in the same queue with the cached face region of interest exceeds a set threshold b, calculating a variance of a mouth key point coordinate corresponding to the normalized face, and determining whether the current detected person is speaking by the size of the variance.

[0012] Preferably, the obtaining of the face region of interest and the mouth key point coordinate from the scene picture includes the following steps.

[0013] identifying, by a human body detection model, a plurality of human body regions of interest and a confidence value corresponding to each human body region of interest from the scene picture;

[0014] identifying, by a face detection model, a plurality of face regions of interest and a confidence value corresponding to each face region of interest from the human body region of interest;

[0015] obtaining, by a face key point detection model, a mouth key point coordinate and a confidence value corresponding to each mouth key point from the face region of interest.

[0016] Further, it is determined whether there is a tracked main speaker at present;

[0017] If there is no tracked main speaker, and it is determined that the number of new speakers is one, the speaker is the main speaker;

[0018] If there is no tracked main speaker, and it is determined that the number of new speakers is two or more, the probability of the current detected person speaking is calculated according to the confidence values obtained when the human body detection model, the face detection model, and the face key point detection model are used to determine whether the current detected person is speaking, and the two persons with the highest probability are respectively regarded as the main speaker and the secondary speaker.

[0019] Further, if there is a tracked main speaker, the intersection of the area of the current detected face region of interest and the face region of interest of the tracked main speaker is obtained;

[0020] If the intersection area is greater than a set threshold c, and it is determined that the current detected person is speaking, the current detected person is the main speaker;

[0021] If the intersection area is greater than the set threshold c, it is determined that the current detected person is not speaking, and the time of not speaking exceeds a set time t, it is determined that there is no longer a tracked main speaker, and the main speaker is reconfirmed;

[0022] If the intersection area is greater than the set threshold c, it is determined that the current detected person is not speaking, but the time of not speaking does not exceed the set time t, the current detected person is the main speaker.

[0023] Further, in the case where the intersection area is greater than the set threshold c, it is determined that the current detected person is not speaking, but the time of not speaking does not exceed the set time t, the current detected person is the main speaker, if the probability of the current detected person speaking is calculated according to the confidence values obtained when the human body detection model, the face detection model, and the face key point detection model are used to determine whether the current detected person is speaking, the person with the highest probability is regarded as the secondary speaker.

[0024] Further, the probability of the current speaker is calculated according to the confidence value of the human shape detection model, the human face detection model and the human face key point detection model, including:

[0025] The confidence average value is calculated based on the confidence value of the human shape region of interest, the human face region of interest and the mouth key point coordinates of the current speaker in the current frame picture.

[0026] If there are multiple frames of the scene picture of the current speaker, the confidence average value is obtained for each frame and then the confidence average value is taken.

[0027] Preferably, the mouth key point coordinates are normalized, including:

[0028] The n mouth key point coordinates (x i , y i ) are obtained.

[0029] The maximum point x max and the minimum point x min on the x-axis, and the maximum point y max and the minimum point y min on the y-axis of the n mouth key point coordinates are obtained.

[0030] The normalized mouth key point coordinates (x mi , y mi ) are calculated.

[0031] The normalization formula is

[0032] x mi represents the normalized mouth key point x-axis coordinate, y mi represents the normalized mouth key point y-axis coordinate; x i and y i represent the i-th key point of the mouth x-axis and y-axis coordinates; x min and x max represent the minimum value and the maximum value of the x-axis coordinate; y min and y max represent the minimum value and the maximum value of the y-axis coordinate.

[0033] Further, the variance of the normalized mouth key point coordinates corresponding to the face is calculated, including:

[0034] The average difference value

[0035] The variance of the n normalized mouth key point coordinates on the y-axis is calculated.

[0036] Further, the current detected person is determined to be speaking by judging the variance size, comprising:

[0037] whether the variance Δd is greater than a set threshold e, and if yes, the current detected person is determined to be speaking, and if no, the current detected person is determined not to be speaking.

[0038] The second aspect of the present application provides a network device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the program to perform the above-mentioned speaker tracking and positioning method based on machine vision.

[0039] The third aspect of the present application provides a storage medium having computer program instructions stored thereon, and the program instructions are executed by a processor to implement the above-mentioned speaker tracking and positioning method based on machine vision.

[0040] The technical solution provided by the embodiments of the present application can include the following beneficial effects:

[0041] The embodiments of the present application track the speaker based on machine vision, are not affected by noise in the scene, and can identify the speaker even if multiple people are speaking at the same time, and even the main speaker and the secondary speaker.

[0042] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of the application, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the application.

[0044] Figure 1 is a schematic flowchart of the first embodiment of the present application;

[0045] Figure 2 is a schematic flowchart of the second embodiment of the present application. DETAILED DESCRIPTION

[0046] The exemplary embodiments will be described in detail herein with reference to the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0047] To solve the problems in the background art, the first embodiment of the present application provides a speaker tracking and positioning method based on machine vision, as shown in Figure 1 comprising:

[0048] 101: Obtain a picture containing a person opening his mouth;

[0049] Obtain a picture containing a person opening his mouth through a camera.

[0050] 102: Obtain a face region of interest and mouth key point coordinates through the picture;

[0051] The face region of interest in the picture can be obtained through face recognition, of course, the human body region of interest can be obtained first, then the face region of interest is obtained from the human body region of interest, and the mouth key point coordinates are obtained from the face region of interest.

[0052] 103: Normalize the face region of interest and the mouth key point coordinates;

[0053] In order to facilitate the training and recognition of the face or mouth key point detection model, the extracted face region of interest can be normalized through linear amplification.

[0054] In this embodiment, the mouth key point coordinates are normalized, including:

[0055] Obtain n mouth key point coordinates (x i , y i );

[0056] Obtain the maximum point x max and the minimum point x min on the x-axis, and the maximum point y max and the minimum point y min on the y-axis in the n mouth key point coordinates;

[0057] Calculate the normalized mouth key point coordinates (x mi , y mi );

[0058] Wherein, the normalization formula

[0059] x mi represents the normalized mouth key point x-axis coordinate, y mi represents the normalized mouth key point y-axis coordinate; x i and y i represent the i-th key point of the mouth x-axis and y-axis coordinates; x min and x max represent the minimum value and the maximum value of the x-axis coordinate; y min and y max represent the minimum value and the maximum value of the y-axis coordinate.

[0060] 104: intersect the normalized face region of interest with the area of the cached face region of interest;

[0061] Here, the cached face region of interest can be a previously stored speaker, speaker, or can be a speaker who is speaking at the moment.

[0062] 105: place the normalized face region of interest with an intersection area greater than a set threshold a in the same queue as the cached face region of interest;

[0063] The intersection area greater than the set threshold a indicates that the normalized face region of interest of the current detected person and the system cached face region of interest are most likely the same person, and thus are placed in the same queue. The queue here can be understood as being classified into a category, such as being placed in the same storage space. The normalized face region of interest with an intersection area less than or equal to the set threshold a can be stored separately.

[0064] 106: when the number of pictures of the normalized face region of interest in the same queue as the cached face region of interest exceeds a set threshold b, calculate the variance of the mouth key point coordinates corresponding to the normalized face, and determine whether the current detected person is speaking by the size of the variance.

[0065] When the number of pictures in the same category exceeds the set threshold b, the probability of being the same person is greatly increased, avoiding detection error interference.

[0066] As an embodiment of the present application for calculating the variance of the mouth key point coordinates corresponding to the normalized face, comprising:

[0067] Calculate the average difference of n normalized mouth key point coordinates on the y-axis

[0068] Calculate the variance of n normalized mouth key point coordinates on the y-axis

[0069] As an embodiment of the present application for determining whether the current detected person is speaking by the size of the variance, comprising:

[0070] Determine whether the variance Δd is greater than a set threshold e, if greater, determine that the current detected person is speaking; if less than or equal to, determine that the current detected person is not speaking.

[0071] In order to avoid false detection caused by the current detected person opening his mouth, etc., several frames can be continuously detected to determine the size of the variance Δd, and if they are all greater than the set threshold e, it is determined that the current detected person is speaking.

[0072] Embodiment two of the present application, as shown in Figure 2 based on embodiment one, comprising:

[0073] 201: Obtain a scene picture containing a person opening his mouth;

[0074] 202: Obtain a face region of interest and a mouth key point coordinate from the scene picture;

[0075] Obtaining a face region of interest and a mouth key point coordinate from the scene picture comprises:

[0076] Identifying a plurality of human body regions of interest and a confidence value corresponding to each human body region of interest from the scene picture by a human body detection model;

[0077] Identifying a face region of interest and a confidence value corresponding to each face region of interest from the human body region of interest by a face detection model;

[0078] Obtaining a mouth key point coordinate and a confidence value corresponding to each mouth key point from the face region of interest by a face key point detection model.

[0079] The face region of interest obtains a face key point coordinate by the face key point detection model, wherein the face key point coordinate includes a mouth key point coordinate.

[0080] 203: Normalize the face region of interest and the mouth key point coordinate;

[0081] 204: Take an intersection of the normalized face region of interest and an area of a cached face region of interest;

[0082] 205: Place the normalized face region of interest and the cached face region of interest with an intersection area greater than a set threshold a in a same queue;

[0083] 206: When a picture number of the normalized face region of interest in the same queue with the cached face region of interest exceeds a set threshold b, calculate a variance of a mouth key point coordinate corresponding to the normalized face, and determine whether the current detected person is speaking by the variance;

[0084] 207: Determine whether a main speaker being tracked exists;

[0085] In order to more effectively and stably track the main speaker in real time, it is determined whether the main speaker being tracked exists.

[0086] 20711: If the main speaker being tracked does not exist and the number of newly added speakers is determined to be one, the speaker is the main speaker;

[0087] 20712: If there is no tracked main speaker and the number of new speakers is two or more, the confidence values obtained by using the human shape detection model, the human face detection model and the human face key point detection model to determine whether the current detected person is speaking are used to calculate the probability that the current detected person is speaking, and the two persons with the highest probabilities are determined as the main speaker and the secondary speaker, respectively.

[0088] 20721: If there is a tracked main speaker, the intersection of the face region of interest of the current detected person and the face region of interest of the tracked main speaker is obtained.

[0089] 207211: If the intersection area is greater than a set threshold value c and it is determined that the current detected person is speaking, the current detected person is the main speaker.

[0090] 207212: If the intersection area is greater than the set threshold value c, it is determined that the current detected person is not speaking, and the time of not speaking exceeds a set time t, it is determined that there is no longer a tracked main speaker, and the main speaker is reconfirmed.

[0091] 207213: If the intersection area is greater than the set threshold value c, it is determined that the current detected person is not speaking, but the time of not speaking does not exceed the set time t, the current detected person is the main speaker.

[0092] 207214: If the intersection area is greater than the set threshold value c, it is determined that the current detected person is not speaking, but the time of not speaking does not exceed the set time t, the current detected person is the main speaker, and there is a new speaker, the confidence values obtained by using the human shape detection model, the human face detection model and the human face key point detection model to determine whether the current detected person is speaking are used to calculate the probability that the current detected person is speaking, and the person with the highest probability is determined as the secondary speaker.

[0093] In the embodiment, the confidence values obtained by using the human shape detection model, the human face detection model and the human face key point detection model to determine whether the current detected person is speaking are used to calculate the probability that the current person is speaking, including: calculating the average value of the confidence values based on the human shape region of interest, the face region of interest and the mouth key point coordinates of the current frame picture of the current speaker. For example, the human shape region of interest confidence value of the current frame picture of the current speaker is 96%, the face region of interest confidence value is 94%, and the mouth key point coordinate confidence value is 98%, and the average confidence value is (96%+94%+98%) / 3=96%. It should be noted that the mouth key point is usually not one key point, and therefore the mouth key point coordinate confidence value here can be the average value of the key point coordinate confidence values.

[0094] The second aspect of the present application provides a network device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the program, the processor executes the method for speaker tracking and positioning based on machine vision.

[0095] The third aspect of the present application provides a storage medium, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method for speaker tracking and positioning based on machine vision.

[0096] The machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device, and can contain or store information such as executable instructions, data, and the like. For example, the machine-readable storage medium can be a RAM (Random Access Memory), a volatile memory, a non-volatile memory, a flash memory, a storage drive (such as a hard disk drive), a solid-state disk, any type of storage disk (such as an optical disk, a DVD, and the like), or similar storage medium, or a combination thereof.

[0097] The system, device, module, or unit illustrated in the above embodiments can be specifically implemented by a computer chip or an entity, or by a product with certain functions. A typical implementation device is a computer, and the specific form of the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an e-mail device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0098] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, a disk storage, a CD-ROM, an optical storage, and the like) containing computer-usable program code.

[0099] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and a combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the flowcharts and / or block diagrams. Figure 1one or more processes and / or blocks Figure 1 an apparatus for performing the functions specified in the flowchart or multiple flows and / or blocks.

[0100] Also, these computer program instructions can be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flowchart or flowcharts Figure 1 one or more processes and / or blocks Figure 1 an apparatus for performing the functions specified in the flowchart or multiple flows and / or blocks.

[0101] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart or flowcharts Figure 1 one or more processes and / or blocks Figure 1 an apparatus for performing the functions specified in the flowchart or multiple flows and / or blocks.

[0102] The application described above is merely preferred embodiments of the application and is not intended to limit the application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the application should be included in the scope of the claims of the application.

[0103] It should be understood that the application is not limited to the precise construction that has been described above and illustrated in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application is limited only by the claims that follow.

[0104] The above merely preferred embodiments of the application and is not intended to limit the application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the application should be included in the scope of the claims of the application.

Claims

1. A machine vision-based speaker tracking and positioning method, characterized by, The method comprises: obtaining a picture of a scene in which a person opens his mouth; obtaining a face region of interest and a mouth key point coordinate from the picture of the scene; normalizing the face region of interest and the mouth key point coordinate; taking an intersection of the normalized face region of interest and an area of a cached face region of interest; placing the normalized face region of interest and the cached face region of interest in the same queue when the intersection area is greater than a set threshold a; calculating a variance of the mouth key point coordinate corresponding to the normalized face when the number of pictures of the normalized face region of interest in the same queue as the cached face region of interest exceeds a set threshold b, and determining whether the current detected person is speaking according to the variance; determining whether a main speaker being tracked exists; if the main speaker being tracked exists, taking an intersection of an area of the face region of interest of the current detected person and the face region of interest of the main speaker being tracked; if the intersection area is greater than a set threshold c and it is determined that the current detected person is speaking, the current detected person is the main speaker; if the intersection area is greater than the set threshold c, it is determined that the current detected person is not speaking, and the time of not speaking exceeds a set time t, it is determined that the main speaker being tracked no longer exists, and the main speaker is reconfirmed; if the intersection area is greater than the set threshold c, it is determined that the current detected person is not speaking, but the time of not speaking does not exceed the set time t, the current detected person is the main speaker; if the intersection area is greater than the set threshold c, it is determined that the current detected person is not speaking, but the time of not speaking does not exceed the set time t, and a new speaker exists, the probability of the current detected person being the speaker is calculated according to the confidence values obtained when the human body detection model, the face detection model and the face key point detection model are used to determine whether the current detected person is speaking, and the person with the highest probability is taken as a secondary speaker. 2.The machine vision-based speaker tracking and positioning method according to claim 1, wherein, The method for obtaining the face region of interest and the mouth key point coordinate from the picture of the scene comprises: identifying a plurality of human body regions of interest and a confidence value corresponding to each human body region of interest from the picture of the scene by using a human body detection model; identifying a face region of interest and a confidence value corresponding to each face region of interest from the human body region of interest by using a face detection model; obtaining a mouth key point coordinate and a confidence value corresponding to each mouth key point from the face region of interest by using a face key point detection model.

3. The method according to claim 2, wherein if there is no main speaker being tracked, and the number of new speakers is one, the speaker is taken as the main speaker; if there is no main speaker being tracked, and the number of new speakers is two or more, the probability of the current detected person speaking is calculated according to the confidence values obtained when the human body detection model, the face detection model and the face key point detection model are used to determine whether the current detected person is speaking, and the two persons with the highest probabilities are taken as the main speaker and the secondary speaker respectively.

4. The method according to claim 3, wherein According to the confidence value obtained by judging the human shape detection model, the human face detection model and the human face key point detection model used by the current detected person when speaking, the probability of the current human speaker is calculated, including: The confidence average value is calculated based on the confidence value of the human shape region of interest, the human face region of interest and the mouth key point coordinates of the current speaker in the current frame picture.

5. The machine vision-based speaker tracking and positioning method according to claim 1, characterized in that, The mouth key point coordinates are normalized, including: obtaining n mouth key point coordinates ; obtaining a maximum point on an x-axis of the n mouth key point coordinates and a minimum point , a maximum point on a y-axis and a minimum point ; calculating normalized mouth key point coordinates ; wherein the normalization formula is ; ; denotes the normalized mouth keypoint x-axis coordinate, denotes the normalized mouth keypoint y-axis coordinate; and denotes the ith mouth x-axis and y-axis coordinate keypoint; and denotes the minimum and maximum values of the x-axis coordinate; and denotes the minimum and maximum values of the y-axis coordinate.

6. The machine vision-based speaker tracking and positioning method according to claim 5, characterized in that, The variance of the normalized mouth key point coordinates corresponding to the human face is calculated, including: for calculating the average difference in the y-axis of the n normalized mouth key point coordinates ; calculating a variance of the n normalized mouth key point coordinates on the y-axis .

7. The machine vision-based speaker tracking and positioning method according to claim 6, characterized in that, Whether the current detected person is speaking is determined by the variance size, including: Judging variance whether greater than a set threshold e, if greater than, judging that the current detected person is speaking; if less than or equal to, judging that the current detected person is not speaking.

8. A network device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program comprises instructions for causing the processor to perform the method of any one of claims 1 to 7. The processor executes the program to execute the machine vision-based speaker tracking and positioning method according to any one of claims 1-7.

9. A storage medium having stored thereon computer program instructions, characterized in that, The program instructions are executed by the processor to implement the machine vision-based speaker tracking and positioning method according to any one of claims 1-7.

Images