A method, device and storage medium based on face tracking control

Abstract

The present invention discloses a control method, device and storage medium based on face tracking. By comprehensively considering the weights of face distance and size, the user with the closest distance and the largest face is selected as the target face, which can ensure that the pan-tilt head is preferentially aimed at the most relevant and important speaker. The pan-tilt head is also automatically controlled to rotate according to the judgment results of lip movement and conference relevance, which can ensure that the pan-tilt head is always aimed at the current speaker. By performing face detection, lip movement detection and conference relevance analysis on the edge device, the dependence on cloud resources is reduced, the delay can be reduced, and the pan-tilt head can respond quickly and accurately aim at the speaker. At the same time, combined with multi-model collaboration, multiple levels of information processing and analysis can be performed, and corresponding control decisions can be better made according to real-time conditions.

Description

Technical Field

[0001] The present invention belongs to the technical field of interactive automatic control, and in particular relates to a control method, device and storage medium based on face tracking. Background Art

[0002] The pan / tilt control is to automatically adjust the direction and angle of the pan / tilt according to the tracking results so that it is aimed at the target face. The pan / tilt control can be achieved by controlling the rotation and pitch motion of the pan / tilt, usually using servo motors and feedback sensors to achieve precise control of the pan / tilt.

[0003] Existing methods usually use traditional microphone arrays to identify the sound source, adjust the direction of the microphone array, and then use the camera to identify the user to adjust the pickup angle, or use wake-up words to infer the user's direction, so that the microphone array can directionally pick up the user's voice. However, there are some problems with the above methods. For example, by identifying the sound source to adjust the direction of the microphone array, in actual usage scenarios, it is easy to cause the device to rotate frequently, resulting in inconsistency and instability in sound pickup, which reduces the sound pickup effect in noisy environments. The sound source may be interrupted, missed or confused, thereby reducing the accuracy and effectiveness of functions such as speech recognition and speech enhancement. If only wake-up words are used, it is easy to be disturbed by environmental noise, affecting the accuracy of command recognition. Summary of the invention

[0004] In view of the defects existing in the above-mentioned prior art, the present invention provides a face tracking control method, which comprises the following steps:

[0005] S1. Deploy a face detection model, a lip movement detection model, a conference correlation analysis model, and a face tracking algorithm on the edge device, and initialize the parameters of the pan-tilt control system, which includes a camera and a microphone;

[0006] S2. Send the image captured by the camera to the edge device, detect the face in the image based on the face detection model deployed on the edge device, track the face based on the face tracking algorithm, and obtain the face position and size output by the tracker;

[0007] S3. For each tracked face, calculate its distance from the camera and the size of the face, evaluate each face according to the weight of distance and size, and select the user with the closest distance and the largest face as the target face;

[0008] S4, sending the target face image to a lip movement detection model deployed on an edge device to determine whether the target face has lip movement;

[0009] S5. If the target face has lip movements, the current conversation information of the target face is sent to the conference relevance analysis model deployed on the edge device to determine whether the current conversation information is related to the conference;

[0010] S6. If the correlation meets a preset threshold, the pan / tilt is controlled to rotate, the position of the target face is tracked, and the direction of the camera and the microphone array is dynamically adjusted;

[0011] S7. Repeat steps S2 to S6 to achieve real-time face tracking and pan / tilt control.

[0012] The face detection model deployed on the edge device is used to detect the face in the image, including:

[0013] Load the pre-trained face detection model and face detection model parameter weights;

[0014] Adjust the size of the input image, including image width and height, and normalize and standardize the image;

[0015] Input the preprocessed image into the face detection model for forward propagation to obtain the model's prediction result for the input image;

[0016] Extract the position and confidence of the face frame, and remove redundant face frames based on the non-maximum suppression (NMS) algorithm;

[0017] For each retained face frame, extract a specific face feature vector from the face region in the image, wherein the feature vector is used to represent the attributes of each face;

[0018] Outputs a list of detected face locations and a list of facial features.

[0019] The camera collects at least one image and detects the face in the image, including:

[0020] Input image sequence I=[I1,I2,...,In], n is greater than or equal to 1, initial face frame B0=(x0,y0,w0,h0), where (x0,y0) represents the angular coordinates of the initial position, w0 represents the width, and h0 represents the height;

[0021] Output the face position and size sequence B=[(x1,y1,w1,h1),(x2,y2,w2,h2),...,(xn,yn,wn,hn)], where the sequence includes the position and size of each face in the image.

[0022] Wherein the method further comprises,

[0023] For image sequence I, if n = 1, the initial face frame B0 is used as the face position and size of the current image;

[0024] If n ≥ 2, use the face position and size of the previous image as the initial position and size of the current image, take the initial face position and size of the current image as input, use the pre-trained face detector model for face detection, and get the face position and size list D = [(xd, yd, wd, hd)] of the current image. For each detected face position and size (xd, yd, wd, hd) ∈ D, use tracker T to update the face tracking, take the face position and size of the current image as input, and get the face position and size (xt, yt, wt, ht) predicted by tracker T based on the information of previous and current images.

[0025] The target face selection includes:

[0026] For each tracked face position and size sequence B=[(x1,y1,w1,h1),(x2,y2,w2,h2),...,(xn,yn,wn,hn)]

[0027] The distance between the face and the camera is calculated using the Euclidean distance formula based on the camera position (cx, cy) and the face position (x, y):

[0028] distance=sqrt((x-cx)^2+ (y-cy)^2)

[0029] Calculate the size of the face based on the width w and height h of the face:

[0030] size=(w+h) / 2

[0031] And select the closest and largest face as the target face based on distance, size and weight:

[0032] score=wd*distance+ws*size

[0033] Where wd represents the distance weight and ws represents the size weight.

[0034] The determination of whether the target face has lip movement includes:

[0035] Preprocess the target face image, including resizing and normalization;

[0036] Input the preprocessed target face image into the convolutional neural network (CNN) lip movement detection model and obtain the output of the model;

[0037] If the confidence of the model output is greater than a preset threshold, it is determined that lip movement has occurred; otherwise, it is determined that no lip movement has occurred.

[0038] The step of determining whether the current conversation information is related to the conference includes:

[0039] If the target face has lip movements, obtain the current conversation information;

[0040] The current conversation information is sent to the conference relevance analysis model deployed on the edge device. The output of the conference relevance analysis model is relevance_score, which indicates the relevance of the current conversation information to the conference.

[0041] Preprocess the current conversation information, including word segmentation, stop word removal, and stemming;

[0042] Input the preprocessed current conversation information into the conference relevance analysis model and obtain the model's output relevance_score;

[0043] A threshold is determined based on relevance_score. If the relevance_score is greater than the threshold, the event is considered relevant to the conference; otherwise, it is considered irrelevant.

[0044] According to the set priority rules, the priority order is determined, including:

[0045] Judge in order of priority. If the person associated with the current conversation exists and meets the conditions, then his priority is the highest. If the person associated with the current conversation does not exist or does not meet the conditions, then judge whether the person with lip movement exists and meets the conditions. Finally, if the person with lip movement does not exist or does not meet the conditions, then judge whether the person with the largest face on the screen exists and meets the conditions.

[0046] The control of the pan / tilt rotation includes:

[0047] For each detected face, a selection box is created based on the face location information;

[0048] Get the coordinates of the four corner points of the selection box, recorded as (x1, y1), (x2, y2), (x3, y3), (x4, y4);

[0049] The following formula calculates the selection box area:

[0050] area=0.5*|(x1*y2+x2*y3+x3*y4+x4*y1)-(x2*y1+x3*y2+x4*y3+x1*y4)|

[0051] Calculate the center point coordinates (cx, cy) of the target marquee, which is the marquee with the largest area among all marquees:

[0052] cx=(x1+x2+x3+x4) / 4

[0053] cy=(y1+y2+y3+y4) / 4

[0054] Calculate the distance from the center of the target selection box to the center of the image:

[0055] distance=sqrt((cx- mage_center_x)²+(cy-image_center_y)²)

[0056] Among them, image_center_x, image_center_y represent the coordinates of the center point of the image;

[0057] Calculate the angle between the target selection box and the gimbal:

[0058] angle=atan2((cy-image_center_y), (cx-image_center_x)) * (180 / pi)

[0059] Determine whether the angle is greater than 90 degrees. If angle>90, ignore the face and do not select it.

[0060] If angle≤90, the pan / tilt is controlled to rotate according to the distance from the center of the target selection box to the center of the image, so that the microphone of the pan / tilt faces the center of the target selection box.

[0061] The present invention also provides a face tracking control device, comprising:

[0062] at least one processor; and

[0063] at least one memory including computer program code,

[0064] The at least one memory and the computer program code are configured to, together with the at least one processor, enable the device to execute the control method for face tracking.

[0065] In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the control method for face tracking is implemented.

[0066] The present invention comprehensively considers the weights of face distance and size, and selects the user with the closest distance and the largest face as the target face, thereby ensuring that the pan-tilt head is aimed at the most relevant and important speaker first. The pan-tilt head is automatically controlled to rotate according to the judgment results of lip movement and conference relevance, thereby ensuring that the pan-tilt head is always aimed at the current speaker. By performing face detection, lip movement detection, and conference relevance analysis on the edge device, the dependence on cloud resources is reduced, the delay can be reduced, and the pan-tilt head can respond quickly and accurately aim at the speaker. At the same time, combined with multi-model collaboration, multiple levels of information processing and analysis can be performed, and corresponding control decisions can be better made according to real-time conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0068] Figure 1 The flowchart shows a face tracking control method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0071] It should be understood that although the terms first, second, third, etc. may be used to describe ... in the embodiments of the present invention, these ... should not be limited to these terms. These terms are only used to distinguish .... For example, without departing from the scope of the embodiments of the present invention, the first ... may also be referred to as the second ..., and similarly, the second ... may also be referred to as the first ....

[0072] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0073] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0074] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the commodity or device including the elements.

[0075] The PTZ control system usually consists of a camera and a microphone. The camera is responsible for capturing images in the conference room, while the microphone is used to collect the speaker's voice. The goal of the PTZ control system is to automatically adjust the direction and angle of the PTZ according to the speaker's position and movement, so that the microphone can always be aimed at the speaker.

[0076] PTZ control refers to automatically adjusting the direction and angle of the PTZ to align it with the target face based on the results of face tracking. PTZ control can be achieved by controlling the rotation and pitch motion of the PTZ. Servo motors and feedback sensors are usually used to achieve precise control of the PTZ.

[0077] like Figure 1 As shown, the present invention discloses a method based on face tracking control, the method comprising:

[0078] S1. Deploy a face detection model, a lip movement detection model, a conference correlation analysis model, and a face tracking algorithm on the edge device, and initialize the parameters of the pan-tilt control system, which includes a camera and a microphone.

[0079] Edge computing is a distributed computing model that pushes computing and data processing to the edge of the network, close to the data source and terminal devices, so as to process data more quickly and efficiently and provide real-time computing capabilities. The goal of edge computing is to reduce latency, reduce network bandwidth consumption, and enhance the privacy and security of sensitive data by moving computing tasks from the traditional centralized cloud to the edge of the network.

[0080] Among them, the face detection model is a computer vision model used to detect faces in images or videos. It can identify and locate the face area in the image and provide the location and bounding box information of the face. The lip movement detection model is used to detect lip movements in face images or videos. By analyzing the morphology and movement of the lips, it can be determined whether lip movements occur, that is, whether the speaker is speaking. The conference relevance analysis model is used to analyze the speaker's conversation content to determine whether it is related to the theme or topic of the current meeting. The model can perform semantic understanding and relevance judgment on the speaker's speech through natural language processing techniques such as text classification or sentiment analysis. The face tracking algorithm is an algorithm for tracking faces in consecutive frames. It continuously tracks faces by detecting and matching facial feature points or bounding boxes in image sequences. The face tracking algorithm can update the speaker's location information in real time when the camera moves or the speaker moves.

[0081] In one embodiment, the conference relevance analysis combining semantic features and knowledge graph can more accurately capture the relevance between text and conference topics, including:

[0082] First, build a knowledge graph in the conference domain, including collecting concepts, entities, attributes and relationships related to conferences, and building a domain ontology model; classify concepts and entities, such as conference topics, conference venues, conference participants, etc.; define various semantic relationships between entities, such as "held in", "organized by...", "involving topics", etc.; use knowledge graph construction tools, such as Protégé, to complete the modeling and storage of knowledge graphs.

[0083] In a certain embodiment, it also includes entity recognition and association of the target text, specifically including named entity recognition of the target text, identifying entities such as names of people, places, and institutions related to the meeting; matching and associating the identified entities with concepts and entities in the knowledge graph; if the entity can be matched with concepts directly related to the meeting theme, venue, participants, etc. in the knowledge graph, then the entity is considered to be related to the meeting.

[0084] Furthermore, it also includes conference relevance score calculation, specifically defining the relevance score rel(x, ei) between each entity and the conference theme; the relevance rel(x, ei) = 1 / dist(ei, conference theme concept) can be defined based on the distance between the entity and the conference theme concept in the knowledge graph; the importance weight w(ei) of the entity ei can be defined, which can be calculated based on factors such as the type of entity and the frequency of occurrence; then the conference relevance score of the target text x is:

[0085] Relevance score = ∑rel(x,ei)*w(ei).

[0086] In a certain embodiment, whether the target text is relevant to the meeting is determined based on the meeting relevance score calculated as above; further, a threshold is pre-set, wherein if the score is higher than the threshold, it is considered relevant, otherwise it is irrelevant; at the same time, the relevant entities and their importance scores are output to provide a basis for subsequent processing.

[0087] The conference relevance analysis combined with the knowledge graph can better understand the semantic association between text content and conference topics, and improve the accuracy of relevance judgment. The rich semantic information in the knowledge graph can support the analysis of conference relevance from multiple perspectives, which helps to improve the efficiency and effectiveness of conference analysis.

[0088] In a certain embodiment, it also includes continuously collecting meeting feedback and user evaluations, adjusting the parameters and algorithms of the correlation analysis model; improving the accuracy and interpretability of the analysis by expanding the coverage and knowledge depth of the knowledge graph, and enhancing the personalization and intelligence of the meeting correlation analysis.

[0089] S2. Send the image captured by the camera to the edge device, detect the face in the image based on the face detection model deployed on the edge device, track the face based on the face tracking algorithm, and obtain the face position and size output by the tracker.

[0090] The face detection model deployed on the edge device detects the face in the image, including:

[0091] Load the pre-trained face detection model and face detection model parameter weights;

[0092] Adjust the size of the input image, including image width and height, and normalize and standardize the image;

[0093] Input the preprocessed image into the face detection model for forward propagation to obtain the model's prediction result for the input image;

[0094] Extract the position and confidence of the face frame, and remove redundant face frames based on the non-maximum suppression (NMS) algorithm;

[0095] For each retained face frame, extract a specific face feature vector from the face region in the image, wherein the feature vector is used to represent the attributes of each face;

[0096] Outputs a list of detected face locations and a list of facial features.

[0097] Based on deep learning methods, deep convolutional neural network (CNN) models can be used to extract features from face images. These feature vectors learn low-level and high-level features that can represent the appearance and structure of the face during the training process.

[0098] It is also possible to extract features from facial images based on feature extraction based on traditional image processing and machine learning methods, using traditional image processing techniques and machine learning algorithms, such as Local Binary Patterns (LBP), facial key point detection, Principal Component Analysis (PCA), etc.

[0099] The camera collects at least one image and detects the face in the image, including:

[0100] Input image sequence I=[I1,I2,...,In], n is greater than or equal to 1, initial face frame B0=(x0,y0,w0,h0), where (x0,y0) represents the angular coordinates of the initial position, w0 represents the width, and h0 represents the height;

[0101] Output the face position and size sequence B=[(x1,y1,w1,h1),(x2,y2,w2,h2),...,(xn,yn,wn,hn)], where the sequence includes the position and size of each face in the image.

[0102] Wherein, the method further comprises:

[0103] For image sequence I, if n = 1, the initial face frame B0 is used as the face position and size of the current image;

[0104] If n ≥ 2, use the face position and size of the previous image as the initial position and size of the current image, take the initial face position and size of the current image as input, use the pre-trained face detector model for face detection, and get the face position and size list D = [(xd, yd, wd, hd)] of the current image. For each detected face position and size (xd, yd, wd, hd) ∈ D, use tracker T to update the face tracking, take the face position and size of the current image as input, and get the face position and size (xt, yt, wt, ht) predicted by tracker T based on the information of previous and current images.

[0105] The face detection model and face tracking algorithm are deployed on the edge device, which can improve real-time performance and reduce communication latency with the cloud server. Using edge computing, face detection and tracking can be performed locally, reducing dependence on cloud resources and processing real-time data streams without waiting for a response from the cloud.

[0106] You can choose a face detection model suitable for edge devices, such as a lightweight convolutional neural network (such as MobileNet, ShuffleNet, etc.) or a compressed model (such as Tiny Face Detector). For face tracking algorithms, you can choose an algorithm suitable for edge devices (such as an algorithm based on Kalman filtering) or use a lightweight deep learning tracker.

[0107] S3. For each tracked face, calculate its distance from the camera and the size of the face, evaluate each face according to the weights of distance and size, and select the user with the closest distance and the largest face as the target face.

[0108] Among them, selecting the target face includes:

[0109] For each tracked face position and size sequence B=[(x1,y1,w1,h1),(x2,y2,w2,h2),...,(xn,yn,wn,hn)]

[0110] The distance between the face and the camera is calculated using the Euclidean distance formula based on the camera position (cx, cy) and the face position (x, y):

[0111] distance=sqrt((x-cx)^2+ (y-cy)^2)

[0112] Calculate the size of the face based on the width w and height h of the face:

[0113] size=(w+h) / 2

[0114] And select the closest and largest face as the target face based on distance, size and weight:

[0115] score=wd*distance+ws*size

[0116] Where wd represents the distance weight and ws represents the size weight.

[0117] S4. Send the target face image to the lip movement detection model deployed on the edge device to determine whether the target face has lip movement.

[0118] Among them, judging whether the target face has lip movement includes:

[0119] Preprocess the target face image, including resizing and normalization;

[0120] Input the preprocessed target face image into the convolutional neural network (CNN) lip movement detection model and obtain the output of the model;

[0121] If the confidence of the model output is greater than a preset threshold, it is determined that lip movement has occurred; otherwise, it is determined that no lip movement has occurred.

[0122] S5. If the target face produces lip movements, the current conversation information of the target face is sent to the conference relevance analysis model deployed on the edge device to determine whether the current conversation information is related to the conference.

[0123] Wherein, judging whether the current conversation information is related to the conference includes:

[0124] If the target face has lip movements, obtain the current conversation information;

[0125] The current conversation information is sent to the conference relevance analysis model deployed on the edge device. The output of the conference relevance analysis model is relevance_score, which indicates the relevance of the current conversation information to the conference.

[0126] Preprocess the current conversation information, including word segmentation, stop word removal, and stemming;

[0127] Input the preprocessed current conversation information into the conference relevance analysis model and obtain the model's output relevance_score;

[0128] A threshold is determined based on relevance_score. If the relevance_score is greater than the threshold, the event is considered relevant to the conference; otherwise, it is considered irrelevant.

[0129] Among them, the priority order is also determined according to the set priority rules, including:

[0130] Judge in order of priority. If the person associated with the current conversation exists and meets the conditions, then his priority is the highest. If the person associated with the current conversation does not exist or does not meet the conditions, then judge whether the person with lip movement exists and meets the conditions. Finally, if the person with lip movement does not exist or does not meet the conditions, then judge whether the person with the largest face on the screen exists and meets the conditions.

[0131] S6. If the correlation meets a preset threshold, the pan / tilt is controlled to rotate, the position of the target face is tracked, and the directions of the camera and microphone array are dynamically adjusted.

[0132] Among them, controlling the rotation of the pan / tilt includes:

[0133] For each detected face, a selection box is created based on the face location information;

[0134] Get the coordinates of the four corner points of the selection box, recorded as (x1, y1), (x2, y2), (x3, y3), (x4, y4);

[0135] The following formula calculates the selection box area:

[0136] area=0.5*|(x1*y2+x2*y3+x3*y4+x4*y1)-(x2*y1+x3*y2+x4*y3+x1*y4)|

[0137] Calculate the center point coordinates (cx, cy) of the target marquee, which is the marquee with the largest area among all marquees:

[0138] cx=(x1+x2+x3+x4) / 4

[0139] cy=(y1+y2+y3+y4) / 4

[0140] Calculate the distance from the center of the target selection box to the center of the image:

[0141] distance=sqrt((cx- mage_center_x)²+(cy-image_center_y)²)

[0142] Among them, image_center_x, image_center_y represent the coordinates of the center point of the image;

[0143] Calculate the angle between the target selection box and the gimbal:

[0144] angle=atan2((cy-image_center_y), (cx-image_center_x)) * (180 / pi)

[0145] Determine whether the angle is greater than 90 degrees. If angle>90, ignore the face and do not select it.

[0146] If angle≤90, the pan / tilt is controlled to rotate according to the distance from the center of the target selection box to the center of the image, so that the microphone of the pan / tilt faces the center of the target selection box.

[0147] Pointing the microphone directly at the center of the target selection box ensures that the microphone is as close to the target sound source as possible, thereby improving the quality of voice capture. Doing so can reduce the impact of ambient noise and other interfering sounds on the voice signal, making the collected voice clearer and more accurate. By pointing the microphone at the center of the target selection box, the accuracy of voice localization can be improved. This is very important for application scenarios that require the location of the sound source, such as meeting recording, speech recognition, voice commands, etc. Accurate voice localization can help the system better recognize and understand voice commands or voice content.

[0148] In addition, the impact of echo and reflection on speech signals can be reduced. The farther the microphone is from the center of the target selection box, the more likely it is that echo and reflection will occur, resulting in reduced speech quality and difficulty in speech recognition. If a microphone array is used, placing the microphone directly at the center of the target selection box can maximize the performance of the microphone array. The microphone array can implement beamforming through a reasonable signal processing algorithm, thereby enhancing the signal strength of the target sound source, suppressing noise and interference, and improving speech clarity and recognition accuracy.

[0149] S7. Repeat steps S2 to S6 to achieve real-time face tracking and pan / tilt control.

[0150] Among them, the gimbal can first determine the approximate direction of the target according to the target selection box center point calculation algorithm, and the azimuth angle of the target is θ_target and the elevation angle is φ_target.

[0151] The gimbal starts to rotate to point the camera in the general direction of the target. The rotation control can be implemented using a PID controller or other control algorithms. The current azimuth angle of the gimbal is θ_gimbal and the elevation angle is φ_gimbal.

[0152] During the rotation process, the rotation speed of the gimbal is gradually reduced according to the difference between the current azimuth and elevation and the target azimuth and elevation. This allows the gimbal to slow down when approaching the target direction to achieve more accurate positioning. The adjustment of the rotation speed can be achieved using a speed control algorithm.

[0153] During the pan / tilt rotation, the image can be continuously captured and recorded. This can be achieved through the linkage control of the camera and the pan / tilt, ensuring that the image data is continuously acquired during the pan / tilt rotation. And by analyzing and processing the continuously captured images, the precise position information of the target in the image can be obtained. Based on this information, the azimuth and elevation angles of the pan / tilt can be fine-tuned to accurately aim the camera at the target.

[0154] According to the face tracking result, the position of the current speaker is determined, and according to the azimuth angle θ_face and the elevation angle φ_face of the position, a control algorithm such as a PID controller is used to adjust the direction of the microphone array so that one or more microphones point to the position of the current speaker.

[0155] By combining face detection models with edge computing, real-time face tracking, target user selection, and lip movement detection can be achieved on edge devices, which can improve the real-time performance and response speed of the system. Such a system can be applied to various scenarios, such as video conferencing, human-computer interaction, etc.

[0156] The present invention comprehensively considers the weights of face distance and size, and selects the user with the closest distance and the largest face as the target face, thereby ensuring that the pan-tilt head is aimed at the most relevant and important speaker first. The pan-tilt head is automatically controlled to rotate according to the judgment results of lip movement and conference relevance, thereby ensuring that the pan-tilt head is always aimed at the current speaker. By performing face detection, lip movement detection, and conference relevance analysis on the edge device, the dependence on cloud resources is reduced, the delay can be reduced, and the pan-tilt head can respond quickly and accurately aim at the speaker. At the same time, combined with multi-model collaboration, multiple levels of information processing and analysis can be performed, and corresponding control decisions can be better made according to real-time conditions.

[0157] The present invention also provides a face tracking control device, comprising:

[0158] at least one processor; and

[0159] at least one memory including computer program code,

[0160] The at least one memory and the computer program code are configured to, together with the at least one processor, enable the device to execute the control method for face tracking.

[0161] In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the control method for face tracking is implemented.

[0162] It should be noted that the computer-readable medium disclosed in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0163] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0164] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0165] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0166] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0167] The above introduces the preferred embodiments of the present invention, which is intended to make the spirit of the present invention clearer and easier to understand, but is not intended to limit the present invention. All modifications, substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection outlined by the claims attached to the present invention.

Claims

1. A face tracking control method, comprising: S1. Deploy a face detection model, a lip movement detection model, a conference correlation analysis model, and a face tracking algorithm on the edge device, and initialize the parameters of the pan-tilt control system, which includes a camera and a microphone; S2. Send the image captured by the camera to the edge device, detect the face in the image based on the face detection model deployed on the edge device, track the face based on the face tracking algorithm, and obtain the face position and size output by the tracker; S3. For each tracked face, calculate its distance from the camera and the size of the face, evaluate each face according to the weight of distance and size, and select the user with the closest distance and the largest face as the target face; S4, sending the target face image to a lip movement detection model deployed on an edge device to determine whether the target face has lip movement; S5. If the target face has lip movements, the current conversation information of the target face is sent to the conference relevance analysis model deployed on the edge device to determine whether the current conversation information is related to the conference; S6. If the correlation meets a preset threshold, the pan / tilt is controlled to rotate, the position of the target face is tracked, and the direction of the camera and the microphone array is dynamically adjusted; S7, repeating steps S2 to S6 to achieve real-time face tracking and pan / tilt control; The face detection model deployed on the edge device is used to detect the face in the image, including: Load the pre-trained face detection model and face detection model parameter weights; Adjust the size of the input image, including image width and height, and normalize and standardize the image; Input the preprocessed image into the face detection model for forward propagation to obtain the model's prediction result for the input image; Extract the position and confidence of the face frame, and remove redundant face frames based on the non-maximum suppression (NMS) algorithm; For each retained face frame, extract a specific face feature vector from the face region in the image, wherein the feature vector is used to represent the attributes of each face; Output a list of detected face locations and a list of facial features; The camera collects at least one image and detects the face in the image, including: Input image sequence I = [I1, I2, ..., In], n is greater than or equal to 1, initial face frame B0 = (x0, y0, w0, h0), where (x0, y0) represents the angular coordinates of the initial position, w0 represents the width, and h0 represents the height; Output face position and size sequence B = [(x1, y1, w1, h1), (x2, y2, w2, h2), ..., (xn, yn, wn, hn)], the sequence includes the position and size of each face in the image; Wherein, for the picture sequence I, if n=1, the initial face frame B0 is used as the face position and size of the current picture; If n ≥ 2, use the face position and size of the previous image as the initial position and size of the current image, take the initial face position and size of the current image as input, use the pre-trained face detection model for face detection, and obtain the face position and size list D = [(xd, yd, wd, hd)] of the current image. For each detected face position and size (xd, yd, wd, hd) ∈ D, use tracker T to update face tracking, take the face position and size of the current image as input, and obtain the face position and size (xt, yt, wt, ht) predicted by tracker T based on the information of previous and current images.

2. The face tracking control method according to claim 1, characterized in that: Select the target face including: For each tracked face position and size sequence B=[(x1,y1,w1,h1),(x2,y2,w2,h2),...,(xn,yn,wn,hn)] The distance between the face and the camera is calculated using the Euclidean distance formula based on the camera position (cx, cy) and the face position (x, y): distance=sqrt((x-cx)^2+(y-cy)^2) Calculate the size of the face based on the width w and height h of the face: size=(w+h) / 2 And select the closest and largest face as the target face based on distance, size and weight: score=wd*distance+ws*size Where wd represents the distance weight and ws represents the size weight.

3. The face tracking control method according to claim 1, characterized in that: Judging whether the target face has lip movement includes: Preprocess the target face image, including resizing and normalization; Input the preprocessed target face image into the convolutional neural network (CNN) lip movement detection model and obtain the output of the model; If the confidence of the model output is greater than a preset threshold, it is determined that lip movement has occurred; otherwise, it is determined that no lip movement has occurred.

4. The face tracking control method according to claim 1, characterized in that: Determining whether the current conversation information is related to the conference includes: If the target face has lip movements, obtain the current conversation information; The current conversation information is sent to the conference relevance analysis model deployed on the edge device. The output of the conference relevance analysis model is relevance_score, which indicates the relevance of the current conversation information to the conference. Preprocess the current conversation information, including word segmentation, stop word removal, and stemming; Input the preprocessed current conversation information into the conference relevance analysis model and obtain the model's output relevance_score; A threshold is determined based on relevance_score. If relevance_score is greater than the threshold, it is determined to be relevant to the meeting; otherwise, it is determined to be irrelevant.

5. The face tracking control method according to claim 1, characterized in that: Control the rotation of the PTZ, including: For each detected face, a selection box is created based on the face location information; Get the coordinates of the four corner points of the selection box, recorded as (x1, y1), (x2, y2), (x3, y3), (x4, y4); The following formula calculates the selection box area: area=0.5*|(x1*y2+x2*y3+x3*y4+x4*y1)-(x2*y1+x3*y2+x4*y3+x1*y4)| Calculate the center point coordinates (cx, cy) of the target marquee, which is the marquee with the largest area among all marquees: cx=(x1+x2+x3+x4) / 4 cy=(y1+y2+y3+y4) / 4 Calculate the distance from the center of the target selection box to the center of the image: distance=sqrt((cx-mage_center_x)2+(cy-image_center_y)2) Among them, image_center_x, image_center_y represent the coordinates of the center point of the image; Calculate the angle between the target selection box and the gimbal: angle=atan2((cy-image_center_y),(cx-image_center_x))* (180 / pi) Determine whether the angle is greater than 90 degrees. If angle>90, ignore the face and do not select it. If angle≤90, the pan / tilt is controlled to rotate according to the distance from the center of the target selection box to the center of the image, so that the microphone of the pan / tilt faces the center of the target selection box.

6. A face tracking control device, comprising: at least one processor; as well as at least one memory including computer program code, The at least one memory and the computer program code are configured to, together with the at least one processor, enable the apparatus to perform the method according to any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images