Method and system for tracking speaker based on panoramic and close-up cameras

Abstract

The invention provides a method for tracking a speaker based on a panoramic camera and a close-up camera. The method comprises the steps that the panoramic camera shoots all people into a panoramic picture; based on a sound source of a speaker and / or analyzing facial actions of all people in the panoramic picture, positioning face position information of the speaker in the panoramic picture; based on the position relation between the panoramic camera and the close-up camera and the face position information of the speaker, the angle and the focal length of the close-up camera are determined, and close-up shooting is carried out on the speaker; the face position of the speaker in the close-up picture is analyzed, and the close-up camera is finely adjusted, so that the speaker is centered in the close-up picture. The position of the speaker is positioned by combining face detection and voice recognition technologies, and the close-up camera picture is triggered to track the close-up picture of the speaker, so that the conference control operation is avoided, and the intelligence of the conference terminal is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of video surveillance, and more particularly to a method and system for tracking a speaker based on panoramic and close-up cameras. Background Art

[0002] In a video conference, when the leader is speaking, the camera needs to track the specific speaker and display it on the main screen. The main methods of tracking the speaker's position are to set PTZ preset positions. In a meeting, practice in advance and switch the camera to the corresponding preset close-up position; use face detection + face recognition technology to switch the camera to the close-up position of the face to be identified; and use voice recognition technology.

[0003] Among them, through the PTZ preset position, in the conference scene, in order to ensure that the speaker's close-up can be displayed to all participants in a timely manner, it is necessary to set and practice the PTZ (pan, tilt, zoom) camera preset position in advance. In addition, in formal scenes, dedicated staff are required to perform conference control operations and set corresponding preset positions to achieve the function of switching to the speaker's close-up.

[0004] Although face detection + face recognition technology can realize intelligent tracking of speakers, it requires facial information to be uploaded in advance and requires dedicated personnel to operate during the meeting, which to some extent affects the fluency and convenience of the meeting.

[0005] In addition, although voice recognition technology can identify the speaker's voice, it cannot accurately locate the speaker's position. Therefore, in a multi-person meeting scenario, it may not be possible to accurately track a specific speaker.

[0006] Although existing video conferencing technologies have met basic conference needs to a certain extent, they still have certain limitations in terms of the camera tracking specific speakers. Moreover, the above technologies all require certain conference control operations, which increases labor costs and technical requirements, and cannot fully realize the intelligence of conference terminals, which reduces the efficiency of meetings to a certain extent. Summary of the invention

[0007] The present invention provides a method and system for tracking speakers based on panoramic and close-up cameras, which realizes automatic tracking of speakers and intelligent adjustment of close-up images in a conference system.

[0008] In a first aspect, the present invention provides a method for tracking a speaker based on a panoramic and close-up camera, comprising:

[0009] The panoramic camera captures everyone in the panoramic picture;

[0010] Locating the face position information of the speaker in the panoramic picture based on the speaker's sound source and / or analyzing the facial movements of all persons in the panoramic picture;

[0011] Based on the positional relationship between the panoramic camera and the close-up camera and the face position information of the speaker, the angle and focal length of the close-up camera are determined to take a close-up shot of the speaker;

[0012] The face position of the speaker in the close-up picture is analyzed, and the close-up camera is fine-tuned to center the speaker in the close-up picture.

[0013] In a second aspect, the present invention provides a system for tracking a speaker based on a panoramic and close-up camera, comprising:

[0014] Panoramic camera, used to capture all people in the panoramic picture;

[0015] A positioning module, used to locate the face position information of the speaker in the panoramic picture based on the speaker's sound source and / or analyzing the facial movements of all persons in the panoramic picture;

[0016] A determination module, configured to determine an angle and a focal length of the close-up camera based on a positional relationship between the panoramic camera and the close-up camera and facial position information of the speaker;

[0017] Close-up camera, used to take close-up shots of speakers;

[0018] The fine-tuning module is used to analyze the face position of the speaker in the close-up picture and fine-tune the close-up camera to center the speaker in the close-up picture.

[0019] The beneficial technical effects of the present invention are:

[0020] First, in video conferencing, the panoramic fixed-focus camera of the binocular camera (close-up + panoramic) can capture panoramic images to obtain the facial information of all participants without uploading facial information in advance, thereby improving the efficiency of tracking the speaker; second, by combining face detection and voice recognition technology, the function of accurately locating the speaker's position can be realized; third, the built-in position conversion algorithm can calculate the PTZ value of the close-up focus, which simplifies the steps of advance setting and rehearsal and improves the intelligence of the conference terminal. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0022] Figure 1 A flow chart of a method for tracking a speaker based on a panoramic + close-up camera provided in an embodiment of the present invention;

[0023] Figure 2-4 Flowchart of three methods for determining speaker's face position information provided by embodiments of the present invention;

[0024] Figure 5-6 A panoramic top view of a panoramic camera and a panoramic picture provided by an embodiment of the present invention;

[0025] Figure 7 Schematic diagram of a system for tracking speakers based on panoramic + close-up cameras provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. SUMMARY OF THE INVENTION

[0028] In the present invention, the panoramic camera captures the faces of all participants in the conference room. The speaker's position is locked in the panoramic image by locating the sound source or identifying the speaker's facial movement features. After determining the speaker's facial position information, the pan, tilt and zoom values ​​of the panoramic camera are calculated. According to the positional relationship between the panoramic camera and the close-up camera, the PTZ offset of the close-up camera is calculated to obtain the PTZ value of the close-up camera.

[0029] The close-up camera adjusts the angle and focal length based on the calculated PTZ value and takes a close-up shot of the speaker. The speaker's face position is detected again in the close-up image, and the camera is fine-tuned to center the speaker in the close-up image. The panoramic camera captures everyone's face, so the speaker can be tracked even if the speaker moves. The speaker's face position information is accurately confirmed through multiple methods (sound source localization, facial movement feature recognition, and correlation between audio and facial movements).

[0030] There may be errors when switching from full view to close-up, so fine-tuning is done in the close-up image to place the speaker more accurately in the center of the image.

[0031] The present invention realizes automatic, high-precision, real-time tracking and shooting of speakers through the collaborative work of panoramic cameras and close-up cameras, combined with a variety of technical means, solving the problems of cumbersome manual operation, inaccurate positioning, large switching errors, and inability to track moving speakers in traditional conference cameras, greatly improving the quality of video conferencing and user experience.

[0032] Exemplary Methods

[0033] like Figure 1 As shown, this implementation provides a flow chart of a method for speaker tracking based on a close-up camera and a panoramic camera.

[0034] Step S101: The panoramic camera captures all persons in the panoramic picture.

[0035] In this embodiment, there are two types of cameras, one is a close-up camera, specifically a close-up PTZ camera, which is used to take close-up photos of the speaker in the meeting. The other is a panoramic camera, specifically a panoramic fixed-focus camera, which is used to capture the faces of all people in the meeting into a panoramic picture. There is no need to manually upload the face information of the participants to the system. The panoramic camera automatically obtains all the face information, which facilitates the subsequent system to recognize and locate the face of the speaker.

[0036] Step S102: locating the face position information of the speaker in the panoramic picture based on the speaker's sound source and / or analyzing the facial movements of all persons in the panoramic picture.

[0037] In this step, three methods are provided to locate the position information of the face in the panoramic picture.

[0038] 1. Lock the face position information of the speaker in the panoramic image according to the speaker's sound source, such as Figure 2 shown.

[0039] S201: Detecting faces of all persons in the panoramic picture, and determining face position information of each person in the panoramic picture.

[0040] Detect all faces in the panoramic image and determine the face position information of each person in the panoramic image. First, use face detection technology to scan the panoramic image and detect the positions of all faces. Then, mark each detected face with a rectangular frame and record the position information of these rectangular frames in the panoramic image. The recorded information includes the size (height and width) and coordinates of the rectangular frame. The coordinates can be expressed as the coordinates of the upper left corner, lower right corner, or center point of the rectangular frame. The size and coordinates of these rectangular frames are recorded as the position information of each face.

[0041] S202: Divide the sound source area of ​​the corresponding range in the panoramic image according to the sound source of the speaker. Use a multi-microphone array to capture the speaker's sound signal, and calculate the sound source direction angle according to the time difference when each microphone receives the speaker's sound, such as Figure 5 The sound source angle α is shown. Figure 5It is shown that the sound source direction angle is converted into the x-coordinate of the panoramic picture by a multi-microphone array composed of at least two horizontally placed microphones, and an area with a fixed width △x is defined as the sound source area (x-△x, x+△x) with the x-coordinate as the center. Figure 5 Medium yellow area.

[0042] Normally, the participants are arranged horizontally in the panoramic picture, and the speaker's position can be determined more accurately by only locating the x-coordinate of the speaker in the horizontal direction in the panoramic picture. Therefore, the x-coordinate of the speaker in the horizontal direction in the panoramic picture can be obtained by a multi-microphone array consisting of at least two horizontally placed microphones. However, for the classroom scene, in a stepped classroom with multiple rows of students, the students are arranged vertically and horizontally in the panoramic picture. Therefore, it is necessary not only to locate the x-coordinate of the speaker in the horizontal direction in the panoramic picture, but also to locate the y-coordinate of the speaker in the horizontal direction in the panoramic picture in order to more accurately determine the speaker's position. Therefore, a multi-microphone array consisting of at least four microphones arranged vertically and horizontally can be used to obtain the x-coordinate of the speaker in the horizontal direction and the y-coordinate in the vertical direction in the panoramic picture.

[0043] The microphones arranged on the left and right sides in the horizontal direction convert the direction angle of the sound source into the x coordinate in the panoramic image, and define an area with a fixed width △x as the width of the sound source area (x-△x, x+△x) with the x coordinate as the center. The microphones arranged on the left and right sides in the vertical direction convert the direction angle of the sound source into the y coordinate in the panoramic image, and define an area with a fixed width △y as the height of the sound source area (y-△y, y+△y) with the y coordinate as the center.

[0044] S203: searching for faces located in the sound source area according to the face position information of each person. Searching for faces whose frames are mostly located in the sound source area according to the size and coordinates of the frames in the face position information of each person.

[0045] S204: Determine the face position information located in the sound source area as the face position information of the speaker.

[0046] For example, Figure 5 As shown in the figure, the yellow area is the determined sound source area, and only the face in the red box is located in the sound source area. The other blue boxes are not located in the sound source area, and the face corresponding to the red box is the speaker.

[0047] If there is only one face in the sound source area, then method 1 directly determines that the face is the speaker's face. If there are multiple faces in the sound source area, method 1 cannot provide a more accurate judgment and needs to be screened in combination with other information to accurately lock the speaker's face.

[0048] Method 2: Lock the speaker's position based on the speaker's sound source and analyze the facial movements of everyone in the panoramic image, such as Figure 3 shown.

[0049] S301: Pre-train the model to learn the motion change features that distinguish between speaking and non-speaking states. Collect a large amount of facial image data, which contains facial expressions and motion change features of people in speaking and non-speaking states. Use computer vision technology and machine learning algorithms to extract facial features related to speech. For example, mouth movements, lip shape changes, facial muscle movements, etc. Select a suitable model architecture (such as convolutional neural network, CNN) and train it to enable it to distinguish between speaking and non-speaking states.

[0050] S302: Facial features of each person in the panoramic image are collected in time series, and the facial image of each person is input into a trained model, and the model outputs a sign indicating whether each person is speaking.

[0051] The facial images of each person are continuously collected in the panoramic picture to form facial feature data arranged in time sequence. This can capture the dynamic changes of facial movements. The facial images of each person are input into the pre-trained model in time sequence. Based on the input image sequence, the model outputs a flag indicating whether each person is speaking (for example, the speaking state flag is 1, and the non-speaking state flag is 0).

[0052] S303: Divide the panoramic picture into corresponding sound source areas according to the sound source of the speaker.

[0053] Use a multi-microphone array to capture the speaker's voice signal, and calculate the sound source direction angle by the time difference when each microphone receives the speaker's voice, such as Figure 5 The sound source angle α is indicated in the figure. The sound source direction angle α is converted into the x coordinate of the panoramic image. With the x coordinate as the center, an area with a fixed width △x is defined as the sound source area (x-△x, x+△x), as follows: Figure 5 Medium yellow area.

[0054] S304: In the panoramic picture, a person whose face position information is located in the sound source area and is marked as a speaking state is determined as a speaker.

[0055] Check the face position information of each person to determine which faces are located in the sound source area. For faces located in the sound source area, check their speaking state flags. Only those marked as speaking are considered speakers. The face position information of faces located in the sound source area and whose speaking state flag is 1 is determined as the face position information of the speaker.

[0056] Method 3: Synchronously analyze audio and face video to locate the speaker in the panoramic image, such as Figure 4 shown.

[0057] S401: Start recording the audio and the video of the facial movements of each person in the panoramic picture at the same time. Ensure that the audio recording and the video recording start at the same time so that the time series data can be accurately aligned in subsequent analysis. Usually, a high-precision clock or synchronization signal is used to achieve synchronous recording.

[0058] S402: Synchronously extract the facial action features of each person's video and the sound waveform features of the audio. Use a facial detection and tracking algorithm (such as Dlib, OpenCV, or a deep learning model) to extract the facial action features of each person. These features may include mouth movements, eye blinks, head postures, etc. These features need to be extracted in time series to capture dynamic changes.

[0059] Use audio processing algorithms (such as short-time Fourier transform, STFT) to extract the time-frequency features of the sound. These features can include spectrum, energy, pitch, etc. The sound waveform features also need to be extracted in time series in order to align with the video features.

[0060] S403: In the same time series, analyzing the correlation between each person's facial features and sound waveform features.

[0061] Ensure that facial action features and sound waveform features are aligned on the same time axis. Time alignment can be achieved by using timestamps or synchronization signals. Use statistical methods or machine learning algorithms to analyze the correlation between each person's facial features and sound waveform features. For example, the correlation coefficient between facial actions (such as mouth opening and closing) and sound fluctuations (such as volume changes) can be calculated.

[0062] S404: Find out the face with relevant features as the speaker, and determine the face position information in the panoramic picture.

[0063] Find the face with the highest correlation between facial features and sound waveform features. This indicates that the facial movements of the face are synchronized with the sound waveform features and may be the speaker. Mark the face with the highest correlation in the panoramic image and determine its face position information. This step can be achieved using the facial position information (such as bounding box) extracted previously.

[0064] For example, suppose a panoramic camera and microphone are installed in a conference room, and the system records audio and video at the same time. There are three people in the conference room: A, B, and C. By synchronously analyzing their facial movements and voice waveforms, it is determined who is speaking.

[0065] When the meeting starts, the system starts recording audio and video at the same time. Each frame of video has a corresponding timestamp, and the audio signal also has a timestamp to ensure audio and video synchronization.

[0066] Use facial detection algorithms to detect faces in each frame and track their facial movements. For example, detect the opening and closing of the mouth, lip movements, etc.

[0067] Facial action features can be represented as a time series, for example, the change in mouth opening and closing over time.

[0068] For example, suppose that within a certain time window (such as 1 second), the mouth opening and closing degrees of A, B, and C are extracted as follows:

[0069] A: 0.1, 0.4, 0.6, 0.3, 0.2

[0070] B: 0.0,0.0,0.0,0.0,0.0

[0071] C: 0.2, 0.3, 0.4, 0.5, 0.4

[0072] Use audio processing algorithms to extract sound waveform features, such as changes in volume over time. Sound waveform features can be represented as a time series. For example, within the same time window, the volume changes are: 0.15, 0.35, 0.55, 0.25, 0.20.

[0073] Perform correlation analysis on facial action features and sound waveform features. For example, calculate the Pearson correlation coefficient between mouth opening and closing degree and sound volume.

[0074] For A:

[0075] Mouth opening: 0.1, 0.4, 0.6, 0.3, 0.2

[0076] Volume: 0.15, 0.35, 0.55, 0.25, 0.20

[0077] Calculate the correlation coefficient and assume that the value obtained is 0.98 (high correlation).

[0078] For B:

[0079] Mouth opening: 0.0,0.0,0.0,0.0,0.0

[0080] Volume: 0.15, 0.35, 0.55, 0.25, 0.20

[0081] Calculate the correlation coefficient, assuming the resulting value is 0.0 (no correlation).

[0082] For C:

[0083] Mouth opening: 0.2, 0.3, 0.4, 0.5, 0.4

[0084] Volume: 0.15, 0.35, 0.55, 0.25, 0.20

[0085] Calculate the correlation coefficient and assume that the value obtained is 0.85 (high correlation).

[0086] According to the results of correlation analysis, the mouth movement features of A and C are highly correlated with the sound waveform features, while the features of B are not correlated.

[0087] Looking further at the correlation coefficient values, A's correlation coefficient (0.98) is higher than C's (0.85), so A is more likely to be the speaker.

[0088] The face position information of A in the panoramic picture is determined, and the person is marked as the speaker.

[0089] The above method three can also realize the positioning of the speaker without using a microphone array, and can be applied to occasions with fewer participants.

[0090] Method 4: Locate the speaker in the panoramic image based on the speaker's sound source and synchronously analyze the audio and face video.

[0091] Method 4 combines the solution of method 3 with the speaker's sound source localization to further accurately determine the speaker.

[0092] Use a multi-microphone array to capture the speaker's voice signal, and calculate the sound source direction angle by the time difference when each microphone receives the speaker's voice, such as Figure 5 The sound source angle α is indicated in the figure. The sound source direction angle α is converted into the x coordinate of the panoramic image. With the x coordinate as the center, an area with a fixed width △x is defined as the sound source area (x-△x, x+△x), as follows: Figure 5 If there are multiple participants in the sound source area, a speaker related to the collected audio can be further determined by the third method of the present invention.

[0093] In summary, method 1 based on sound source localization and face detection is simple to implement and has high real-time performance, but has limited accuracy in complex environments; method 2 based on sound source localization and pre-trained speech state recognition model has high accuracy and adaptability, but requires a large amount of data and computing resources; method 3 does not need to be based on sound source localization, and only performs well in complex environments through the correlation analysis method of audio and video features, but has high computational complexity and low real-time performance. Method 4 based on sound source localization and the correlation analysis method of audio and video features performs best in complex environments with multiple participants. Since the positioning range of the speaker has been greatly reduced through sound source localization, the computational complexity of method 4 is reduced compared with method 3 and has higher real-time performance. The selection of methods should be based on the requirements of specific application scenarios, such as real-time performance, accuracy, and computing power. Which method to choose depends on the specific application scenario and requirements, and the above four methods can also be used in combination.

[0094] Step S103: Based on the positional relationship between the panoramic camera and the close-up camera and the face position information of the speaker, the angle and focal length of the close-up camera are determined to take a close-up shot of the speaker.

[0095] In a meeting, when a speaker starts speaking, the close-up camera will immediately adjust its angle and focal length to take a close-up shot of the speaker so that the speaker is displayed in an appropriate position and size in the picture. In this embodiment, the face position of the speaker is locked in the panoramic picture taken by the panoramic camera, and the angle offset and focal length adjustment parameters required for the panoramic camera to focus on the speaker are determined based on the face position. Then, based on the positional relationship between the panoramic camera and the close-up camera, the parameter offset between the two cameras is calculated and added to the parameters of the panoramic camera to obtain the specific angle and focal length parameters of the close-up camera. Finally, the close-up camera adjusts the rotation and zoom of the lens based on these parameters to ensure that the speaker is clearly visible and in an appropriate position in the picture.

[0096] First, if Figure 5 As shown, based on the face position information of the speaker (the coordinates and size of the red face box), the coordinates (x1, y1) of the face center point p are calculated, and the coordinates (x0, y0) of the panoramic image center point q are determined. In addition, the horizontal angle AngleH and the vertical angle AngleV of the panoramic camera are determined, and the face range size Area is calculated.

[0097] Determine the coordinates of the speaker's face center point p (x1, y1) and the panoramic image center point q (x0, y0).

[0098] Get the horizontal field of view HFOV and vertical field of view VFOV of the panoramic camera, as well as the width W and height H of the panoramic image, such as Figure 5 shown.

[0099] A panoramic camera is a camera used to capture a wide range of scenes. The field of view refers to the field of view that a panoramic camera can capture, which is usually divided into the horizontal field of view HFOV and the vertical field of view VFOV. The horizontal field of view HFOV refers to the field of view that a panoramic camera can cover in the horizontal direction, which determines the width of the scene that the panoramic camera can see in the left and right directions. The vertical field of view HFOV refers to the field of view that a panoramic camera can cover in the vertical direction, which determines the height of the scene that the panoramic camera can see in the up and down directions.

[0100] Calculate the optical center distance W from the panoramic camera to the center point q of the panoramic image under the two field of view angles ′ and H ′

[0101] The corresponding optical center distance of the vertical field of view VFOV is the vertical viewing angle distance W ′ , calculated using the following formula: Vertical viewing angle distance W ′ =(H / 2) / tan(VFOV / 2)

[0102] The optical center distance corresponding to the horizontal field of view HFOV is the horizontal viewing angle distance H ′ , calculated using the following formula: Horizontal viewing distance H ′ =(W / 2) / tan(HFOV / 2)

[0103] The horizontal angle AngleH, the vertical angle AngleV and the face range size Area are calculated using the following formula.

[0104] The horizontal angle AngleH is calculated using the following formula: AngleH=arctan(|x1-x0| / H ′ )

[0105] Here, |x1-x0| is the horizontal distance between the center point p of the speaker's face and the center point q of the panoramic image. The horizontal angle describes the field of view angle of the offset of the center point p of the speaker's face in the horizontal direction relative to the center point q of the panoramic image.

[0106] The vertical angle is calculated using the following formula: AngleV=arctan(|y1-y0| / W ′ )

[0107] Here |y1-y0| is the vertical distance between the center point p of the speaker's face and the center point q of the panoramic image. The vertical angle describes the field of view angle of the vertical offset of the center point p of the face relative to the center point q of the panoramic image.

[0108] The following formula is used to calculate the face range size: Area=|x1-x0|·|y1-y0|

[0109] The face range size describes the size of the rectangular image range formed by the speaker's face center point p and the panoramic image center point.

[0110] The horizontal angle (AngleH), vertical angle (AngleV) and face range size (Area) are geometric descriptions and are not directly related to the mechanical movement of the camera. Therefore, these geometric parameters need to be converted into parameters (pan_p, tilt_p, zoom_p) used to control the actual movement of the panoramic camera.

[0111] First, determine the horizontal angle conversion coefficient a1, vertical angle conversion coefficient a2 and zoom conversion coefficient a3 of the panoramic camera.

[0112] The conversion factors a1, a2 and a3 are determined based on the characteristics of the panoramic camera, the mechanical design and the required range of motion. They are usually determined by the panoramic camera manufacturer based on specific product specifications and functional requirements.

[0113] The panoramic horizontal offset angle pan_p, the panoramic vertical offset angle tilt_p and the panoramic zoom factor zoom_p are the actual control parameters of the panoramic camera. By calculating the horizontal angle and the vertical angle, combined with the scale factor, they are converted into the actual mechanical movement required for the panoramic camera to rotate and zoom.

[0114] The panoramic horizontal offset angle pan_p is a parameter used to control the rotation of the panoramic camera on the horizontal plane. It is calculated using the following formula: pan_p=(AngleH)·a1

[0115] The panoramic vertical offset angle tilt_p is a parameter used to control the rotation of the panoramic camera on the vertical plane. It is calculated using the following formula: tilt_p=(AngleV)·a2

[0116] The panoramic zoom factor zoom_p is a parameter used to control the zoom in and zoom out of the panoramic camera. It is calculated using the following formula: zoom_p=(W·H / Area)·a3

[0117] like Figure 6 As shown in the figure, the optical center positions of the close-up camera and the panoramic camera in the panoramic image are different, so there is a certain offset between the two. In order to ensure that the close-up camera can accurately focus on the speaker, we need to calculate the horizontal angle offset, vertical angle offset and focal length offset between the two cameras, and determine the PTZ (Pan, Tilt, Zoom) value of the close-up camera based on this.

[0118] Based on the positional relationship between the panoramic camera and the close-up camera, the close-up horizontal movement angle pan_l, the close-up vertical movement angle tilt_l and the close-up zoom factor zoom_l of the close-up camera are determined.

[0119] Firstly, based on the positional relationship between the panoramic camera and the close-up camera, a horizontal offset b1, a vertical offset b2 and a zoom offset b3 between the two cameras are determined;

[0120] The following formula is used to calculate the close-up horizontal movement angle pan_l, the close-up vertical movement angle tilt_l and the close-up zoom multiple zoom_l: pan_l=pan_p+b1 tilt_l=tilt_p+b2 zoom_l=zoom_p+b3

[0121] The close-up camera rotates and zooms according to the horizontal movement angle pan_l, the close-up vertical movement angle tilt_l and the close-up zoom multiple zoom_l to take a close-up shot of the speaker.

[0122] Step S104: Analyze the position of the speaker in the close-up picture, and fine-tune the close-up camera to make the speaker centered in the close-up picture.

[0123] Although the close-up camera has been preliminarily aligned, fine-tuning is necessary to achieve higher accuracy and visual effects. This can compensate for errors in the preliminary adjustment, mechanical movement errors, and dynamic changes in the speaker, thereby providing a more accurate and professional close-up picture.

[0124] Detect the face position of the speaker in the close-up picture. The output of this step is usually a rectangular box that defines the position and size of the face.

[0125] The offset between the center point of the face and the center point of the close-up image is calculated, and the offset in the horizontal direction and the vertical direction is determined respectively.

[0126] Based on the offset, the close-up horizontal movement angle pan_l, the close-up vertical movement angle tilt_l and the close-up zoom factor zoom_l of the close-up camera are readjusted. In this step, the horizontal and vertical offsets are converted into the close-up horizontal movement angle pan_l and the close-up vertical movement angle tilt_l of the close-up camera through conversion coefficients. Then, the close-up zoom factor zoom_l is adjusted according to the size of the speaker's face. If the area occupied by the face is too large or too small, the lens can be reduced or enlarged accordingly.

[0127] Recapture the close-up image and perform face detection, repeat until the speaker's face is close to the center of the frame. Adjust the camera and capture a new close-up frame based on the calculated new tilt_l, pan_l, and zoom_l parameters. Re-perform face detection in the new close-up frame to determine the new position of the face. If the face center is still not close to the center of the frame, repeat the above steps (recalculate the offset and adjust the camera) until the offset between the face center and the center of the frame is within an acceptable range.

[0128] The close-up camera can continuously adjust its angle and focus to ensure that the speaker's face is always in the center of the picture. This dynamic adjustment process ensures accurate and stable shooting, thus providing high-quality close-up images.

[0129] In summary, the core of the technical solution of this embodiment is to use a binocular camera (panoramic + close-up) to track the speaker and use sound source + face detection to locate the speaker. Using a binocular camera system (including a panoramic camera and a close-up camera) for speaker tracking has the following significant advantages compared to a single camera system:

[0130] The panoramic camera provides a wide viewing angle and can capture the scene of the entire conference room and the positions of all participants. It can monitor the dynamics of all speakers in real time, automatically and quickly lock the speaker's face position, and calculate the angle rotation and zoom parameters of the corresponding close-up camera. When a speaker is detected, the close-up camera can directly and quickly aim at the target, ensuring that the close-up camera can smoothly transition to a close-up of the speaker, avoiding blurry or jittering images that may occur when a single camera rotates significantly.

[0131] In addition, the present invention combines sound source localization and face detection technology to lock the speaker, improve positioning accuracy and reduce false locks. The panoramic camera can monitor all participants in real time, use sound source and face detection to accurately locate the speaker, and the panoramic camera can continuously update the position information of the speaker in the scene, dynamically adjust the angle and focal length of the close-up camera, thereby achieving more intelligent and automatic speaker tracking and picture adjustment, ensuring that no human intervention is required throughout the meeting.

[0132] Exemplary Devices

[0133] Accordingly, an embodiment of the present invention also provides a system for tracking a speaker based on a panoramic and close-up camera, such as Figure 7 shown, including:

[0134] A panoramic camera 701 is used to capture all persons in a panoramic picture;

[0135] A positioning module 702, configured to locate the face position information of the speaker in the panoramic picture based on the speaker's sound source and / or analyzing the facial movements of all persons in the panoramic picture;

[0136] A determination module 703, configured to determine the angle and focal length of the close-up camera based on the positional relationship between the panoramic camera 701 and the close-up camera and the face position information of the speaker;

[0137] A close-up camera 704 is used to take a close-up shot of the speaker;

[0138] The fine-tuning module 705 is used to analyze the face position of the speaker in the close-up picture and fine-tune the close-up camera 704 to make the speaker centered in the close-up picture.

[0139] The positioning module 702 detects the faces of all people in the panoramic picture and determines the facial position information of each person in the panoramic picture; divides the sound source area of ​​the corresponding range in the panoramic picture according to the sound source of the speaker; searches for the faces located in the sound source area according to the facial position information of each person; and determines the facial position information located in the sound source area as the facial position information of the speaker.

[0140] Alternatively, the positioning module 702 pre-trains a model so that the model learns to distinguish the movement change features of the speaking and non-speaking states; collects the facial features of each person in the panoramic picture in time series, and inputs the facial image of each person into the trained model, and the model outputs a sign of whether each person is speaking; divides the corresponding sound source area in the panoramic picture according to the sound source of the speaker; in the panoramic picture, the person whose face position information is located in the sound source area and is marked as speaking is determined as the speaker.

[0141] The system also includes a multi-microphone array, which includes a plurality of microphones arranged in a fixed position and is used to collect the voice of the speaker and obtain the sound source of the speaker.

[0142] The operation of dividing the sound source area of ​​the corresponding range in the panoramic picture according to the sound source of the speaker specifically includes: using the multi-microphone array to capture the sound signal of the speaker; calculating the sound source direction angle through the time difference when each microphone receives the speaker's sound; converting the sound source direction angle into the x-coordinate and / or y-coordinate of the panoramic picture; and defining an area of ​​fixed width as the sound source area with the x-coordinate and / or y-coordinate as the center.

[0143] The positioning module 702 can also simultaneously start recording the audio and the video of the facial movements of each person in the panoramic picture; synchronously extract the facial movement features of each person in the video and the sound waveform features of the audio; analyze the correlation between the facial features and the sound waveform features of each person in the same time sequence, find out the face with relevant features as the speaker, and determine its face position information in the panoramic picture.

[0144] The determination module 703 calculates the horizontal angle AngleH and the vertical angle AngleV between the center point of the face and the center point q of the panoramic image, and calculates the face range size Area based on the face position information of the speaker; determines the panoramic horizontal offset angle pan_p, the panoramic vertical offset angle tilt_p and the panoramic zoom factor zoom_p of the speaker's face relative to the center point of the panoramic image taken by the panoramic camera 701 based on the horizontal angle AngleH, the vertical angle AngleV and the face range size Area if the speaker's face is enlarged to fill the screen; determines the close-up horizontal movement angle pan_l, the close-up vertical movement angle tilt_l and the close-up zoom factor zoom_l of the close-up camera 704 based on the positional relationship between the panoramic camera 701 and the close-up camera 704.

[0145] The operations of calculating the horizontal angle AngleH and the vertical angle AngleV between the face center point p and the panoramic image center point q based on the face position information of the speaker, and calculating the face range size Area also include: determining the coordinates (x1, y1) of the face center point of the speaker and the coordinates (x0, y0) of the panoramic image center point;

[0146] Obtain the horizontal field of view angle HFOV and the vertical field of view angle VFOV of the panoramic camera 701, as well as the width W and height H of the panoramic image;

[0147] The vertical viewing angle distance W between the panoramic lens and the optical center of the panoramic image is calculated using the following formula: ′ and horizontal viewing distance H ′ : Vertical viewing angle distance W ′ =(H / 2) / tan(VFOV / 2); Horizontal viewing distance H′ =(W / 2) / tan(HFOV / 2);

[0148] The horizontal angle AngleH, the vertical angle AngleV and the face range size Area are calculated using the following formula: AngleH=arctan(|x1-x0| / H ′ ) AngleV=arctan(|y1-y0| / W ′ ) Area=|x1-x0|·|y1-y0|

[0149] The operation of determining the panoramic horizontal offset angle pan_p, the panoramic vertical offset angle tilt_p and the panoramic zoom factor zoom_p based on the horizontal angle AngleH, the vertical angle AngleV and the face range size Area further includes:

[0150] Determine the horizontal angle conversion coefficient a1, the vertical angle conversion coefficient a2 and the zoom conversion coefficient a3 of the panoramic camera 701;

[0151] The panoramic horizontal offset angle pan_p, the panoramic vertical offset angle tilt_p and the panoramic view are calculated using the following formula: pan_p=(AngleH)·a1 tilt_p=(AngleV)·a2 zoom_p=(W·H / Area)·a3

[0152] The operation of determining the close-up horizontal movement angle pan_1, the close-up vertical movement angle tilt_1 and the close-up zoom factor zoom_1 of the close-up camera 704 based on the positional relationship between the panoramic camera 701 and the close-up camera 704 further includes:

[0153] Based on the positional relationship between the panoramic camera 701 and the close-up camera 704, determine the horizontal offset b1, the vertical offset b2 and the zoom offset b3 between the two cameras;

[0154] The following formula is used to calculate the close-up horizontal movement angle pan_l, the close-up vertical movement angle tilt_l and the close-up zoom multiple zoom_l: pan_l=pan_p+b1; tilt_l = tilt_p + b2; zoom_l=zoom_p+b3.

[0155] The fine-tuning module 705 detects the face position of the speaker in the close-up picture; calculates the offset between the center point of the face and the center point of the close-up picture; readjusts the close-up horizontal movement angle pan_l, the close-up vertical movement angle tilt_l and the close-up zoom factor zoom_l of the close-up picture based on the offset; recaptures the close-up picture image and performs face detection, and repeats until the speaker's face is close to the center of the picture.

[0156] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims. The present invention provides: 1. A method for tracking a speaker based on a panoramic and close-up camera, comprising: The panoramic camera captures everyone in the panoramic picture; Locating the face position information of the speaker in the panoramic picture based on the speaker's sound source and / or analyzing the facial movements of all persons in the panoramic picture; Based on the positional relationship between the panoramic camera and the close-up camera and the face position information of the speaker, the angle and focal length of the close-up camera are determined to take a close-up shot of the speaker; The face position of the speaker in the close-up picture is analyzed, and the close-up camera is fine-tuned to center the speaker in the close-up picture. 2. According to the method described in item 1, the step of locating the face position information of the speaker in the panoramic picture based on the speaker's sound source and / or analyzing the facial movements of all persons in the panoramic picture further comprises the following steps: Detecting faces of all persons in the panoramic picture, and determining face position information of each person in the panoramic picture; Dividing the sound source area of ​​corresponding range in the panoramic picture according to the sound source of the speaker; According to the facial position information of each person, searching for the face located in the sound source area; The face position information located in the sound source area is determined as the face position information of the speaker. 3. According to the method described in item 1, the step of locating the face position information of the speaker in the panoramic picture based on the speaker's sound source and / or analyzing the facial movements of all persons in the panoramic picture further comprises the following steps: Pre-train the model so that it can learn to distinguish the motion change features between speaking and non-speaking states; The facial features of each person in the panoramic picture are collected in time series, and the facial image of each person is input into a trained model, and the model outputs a sign of whether each person is speaking; Dividing corresponding sound source areas in the panoramic picture according to the sound source of the speaker; In the panoramic picture, a person whose face position information is located in the sound source area and is marked as a speaking state is determined as a speaker. 4. According to the method described in item 1, the step of locating the face position information of the speaker in the panoramic picture based on the speaker's sound source and / or analyzing the facial movements of all persons in the panoramic picture further comprises the following steps: Start recording audio and video of the facial movements of everyone in the panoramic view at the same time; Synchronously extract each person's video facial action features and audio sound waveform features; Analyze the correlation between each person's facial features and voice waveform features in the same time series; The face with relevant features is found to be the speaker, and its face position information is determined in the panoramic picture. 5. According to the method described in item 1, the step of locating the face position information of the speaker in the panoramic picture based on the speaker's sound source and / or analyzing the facial movements of all persons in the panoramic picture further comprises the following steps: Dividing the sound source area of ​​corresponding range in the panoramic picture according to the sound source of the speaker; If there are multiple faces in the sound source area, a face whose facial features and sound waveform features are related in the same time series is found among these faces as the speaker, and the face position information of the speaker is determined. 6. According to the method described in item 2, 3 or 5, the step of dividing the sound source areas of corresponding ranges in the panoramic picture according to the sound source of the speaker further comprises the following steps: Use a multi-microphone array to capture the speaker's voice signal; The angle of the sound source direction is calculated by the time difference when each microphone receives the speaker's voice; Convert the sound source direction angle into an x-coordinate and / or a y-coordinate of a panoramic picture; Taking the x coordinate and / or y coordinate as the center, define an area with a fixed width as the sound source area. 7. According to the method described in item 1, the step of determining the angle and focal length of the close-up camera based on the positional relationship between the panoramic camera and the close-up camera and the face position information of the speaker, and taking a close-up of the speaker further comprises the following steps: Based on the face position information of the speaker, the horizontal angle AngleH and the vertical angle AngleV between the center point of the face and the center point of the panoramic image are calculated, and the size of the face range Area is calculated; Based on the horizontal angle AngleH, the vertical angle AngleV and the face range size Area, determine the panoramic horizontal offset angle pan_p, the panoramic vertical offset angle tilt_p of the speaker's face relative to the center point of the panoramic image taken by the panoramic camera, and the panoramic zoom factor zoom_p if the speaker's face is enlarged to fill the screen; Based on the positional relationship between the panoramic camera and the close-up camera, the close-up horizontal movement angle pan_l, the close-up vertical movement angle tilt_l and the close-up zoom factor zoom_l of the close-up camera are determined. 8. According to the method described in item 7, the step of calculating the horizontal angle AngleH and the vertical angle AngleV between the center point of the face and the center point of the panoramic image based on the face position information of the speaker, and calculating the size of the face range Area further comprises the following steps: Determine the coordinates of the speaker's face center point (x1, y1) and the coordinates of the panoramic image center point (x0, y0); Get the horizontal field of view angle HFOV and the vertical field of view angle VFOV of the panoramic camera, as well as the width W and height H of the panoramic image; The vertical viewing angle distance W between the panoramic lens and the optical center of the panoramic image is calculated using the following formula: ′ and horizontal viewing distance H ′ : Vertical viewing angle distance W ′ =(H / 2) / tan(VFOV / 2); Horizontal viewing distance H ′ =(W / 2) / tan(HFOV / 2); The horizontal angle AngleH, the vertical angle AngleV and the face range size Area are calculated using the following formula: AngleH=arctan(|x1-x0| / H ′ ); AngleV=arctan(|y1-y0| / W ′ ); Area = |x1-x0|·|y1-y0|. 9. According to the method described in item 8, the step of determining the panoramic horizontal offset angle pan_p, the panoramic vertical offset angle tilt_p and the panoramic zoom factor zoom_p based on the horizontal angle AngleH, the vertical angle AngleV and the face range size Area further comprises the following steps: Determine the horizontal angle conversion coefficient a1, vertical angle conversion coefficient a2 and zoom conversion coefficient a3 of the panoramic camera; The panoramic horizontal offset angle pan_p, the panoramic vertical offset angle tilt_p and the panoramic zoom factor zoom_p are calculated using the following formula: pan_p=(AngleH)·a1; tilt_p = (AngleV)·a2; zoom_p=(W·H / Area)·a3. 10. According to the method described in item 9, the step of determining the close-up horizontal movement angle pan_1, the close-up vertical movement angle tilt_1 and the close-up zoom factor zoom_1 of the close-up camera based on the positional relationship between the panoramic camera and the close-up camera further comprises the following steps: Based on the positional relationship between the panoramic camera and the close-up camera, determine a horizontal offset b1, a vertical offset b2, and a zoom offset b3 between the two cameras; The following formula is used to calculate the close-up horizontal movement angle pan_l, the close-up vertical movement angle tilt_l and the close-up zoom multiple zoom_l: pan_l=pan_p+b1; tilt_l = tilt_p + b2; zoom_l=zoom_p+b3. 11. According to the method described in item 1, the step of analyzing the face position of the speaker in the close-up picture and fine-tuning the close-up camera to center the speaker in the close-up picture further comprises the following steps: Detect the face position of the speaker in the close-up picture; Calculate the offset between the center point of the face and the center point of the close-up image; Re-adjust the close-up horizontal movement angle pan_1, the close-up vertical movement angle tilt_1 and the close-up zoom multiple zoom_1 of the close-up shot based on the offset; Recapture the close-up image and perform face detection, repeating until the speaker's face is close to the center of the frame. 12. A system for tracking a speaker based on panoramic and close-up cameras, comprising: Panoramic camera, used to capture all people in the panoramic picture; A positioning module, used to locate the face position information of the speaker in the panoramic picture based on the speaker's sound source and / or analyzing the facial movements of all persons in the panoramic picture; A determination module, configured to determine an angle and a focal length of the close-up camera based on a positional relationship between the panoramic camera and the close-up camera and facial position information of the speaker; Close-up camera, used to take close-up shots of speakers; The fine-tuning module is used to analyze the face position of the speaker in the close-up picture and fine-tune the close-up camera to center the speaker in the close-up picture. 13. According to the system described in item 12, the positioning module specifically includes the following operations: Detecting faces of all persons in the panoramic picture, and determining face position information of each person in the panoramic picture; Dividing the sound source area of ​​corresponding range in the panoramic picture according to the sound source of the speaker; According to the facial position information of each person, searching for the face located in the sound source area; The face position information located in the sound source area is determined as the face position information of the speaker. 14. According to the system described in item 12, the positioning module specifically includes the following operations: Pre-train the model so that it can learn to distinguish the motion change features between speaking and non-speaking states; The facial features of each person in the panoramic picture are collected in time series, and the facial image of each person is input into a trained model, and the model outputs a sign of whether each person is speaking; Dividing corresponding sound source areas in the panoramic picture according to the sound source of the speaker; In the panoramic picture, a person whose face position information is located in the sound source area and is marked as a speaking state is determined as a speaker. 15. According to the system described in item 12, the positioning module further includes the following operations: Start recording audio and video of the facial movements of everyone in the panoramic view at the same time; Synchronously extract each person's video facial action features and audio sound waveform features; In the same time series, the correlation between each person's facial features and sound waveform features is analyzed; the face with relevant features is found to be the speaker, and its face position information is determined in the panoramic picture. 16. According to the system described in item 12, the positioning module further includes the following operations: Dividing the sound source area of ​​corresponding range in the panoramic picture according to the sound source of the speaker; If there are multiple faces in the sound source area, a face whose facial features and sound waveform features are related in the same time series is found among these faces as the speaker, and the face position information of the speaker is determined. 17. The system according to item 13, 14 or 16, further comprising a multi-microphone array, which includes a plurality of microphones arranged in a fixed position, for collecting the voice of the speaker and obtaining the sound source of the speaker; The operation of dividing the sound source area of ​​the corresponding range in the panoramic picture according to the sound source of the speaker also includes: Using the multi-microphone array to capture the speaker's voice signal; Calculate the sound source direction angle by the time difference when each microphone receives the speaker's voice; Convert the sound source direction angle into an x-coordinate and / or a y-coordinate of a panoramic picture; Taking the x coordinate and / or y coordinate as the center, define an area with a fixed width as the sound source area. 18. According to the system described in item 12, the determining module further includes the following operations: Based on the face position information of the speaker, the horizontal angle AngleH and the vertical angle AngleV between the center point of the face and the center point of the panoramic image are calculated, and the size of the face range Area is calculated; Based on the horizontal angle AngleH, the vertical angle AngleV and the face range size Area, determine the panoramic horizontal offset angle pan_p, the panoramic vertical offset angle tilt_p of the speaker's face relative to the center point of the panoramic image taken by the panoramic camera, and the panoramic zoom factor zoom_p if the speaker's face is enlarged to fill the screen; Based on the positional relationship between the panoramic camera and the close-up camera, the close-up horizontal movement angle pan_l, the close-up vertical movement angle tilt_l and the close-up zoom factor zoom_l of the close-up camera are determined. 19. According to the system described in item 18, the operation of calculating the horizontal angle AngleH and the vertical angle AngleV between the center point of the face and the center point of the panoramic image based on the face position information of the speaker, and calculating the size of the face range Area also includes: Determine the coordinates of the speaker's face center point (x1, y1) and the coordinates of the panoramic image center point (x0, y0); Get the horizontal field of view angle HFOV and the vertical field of view angle VFOV of the panoramic camera, as well as the width W and height H of the panoramic image; The vertical viewing angle distance W between the panoramic lens and the optical center of the panoramic image is calculated using the following formula: ′and horizontal viewing distance H ′ : Vertical viewing angle distance W ′ =(H / 2) / tan(VFOV / 2); Horizontal viewing distance H ′ =(W / 2) / tan(HFOV / 2); The horizontal angle AngleH, the vertical angle AngleV and the face range size Area are calculated using the following formula: AngleH=arctan(|x1-x0| / H ′ ); AngleV=arctan(|y1-y0| / W ′ ); Area = |x1-x0|·|y1-y0|. 20. According to the system of item 19, the operation of determining the panoramic horizontal offset angle pan_p, the panoramic vertical offset angle tilt_p and the panoramic zoom factor zoom_p based on the horizontal angle AngleH, the vertical angle AngleV and the face range size Area further comprises: Determine the horizontal angle conversion coefficient a1, vertical angle conversion coefficient a2 and zoom conversion coefficient a3 of the panoramic camera; The panoramic horizontal offset angle pan_p, the panoramic vertical offset angle tilt_p and the panoramic zoom factor zoom_p are calculated using the following formula: pan_p=(AngleH)·a1; tilt_p = (AngleV)·a2; zoom_p=(W·H / Area)·a3. 21. According to the system of item 20, the operation of determining the close-up horizontal movement angle pan_1, the close-up vertical movement angle tilt_1 and the close-up zoom factor zoom_1 of the close-up camera based on the positional relationship between the panoramic camera and the close-up camera further comprises: Based on the positional relationship between the panoramic camera and the close-up camera, determine a horizontal offset b1, a vertical offset b2, and a zoom offset b3 between the two cameras; The following formula is used to calculate the close-up horizontal movement angle pan_l, the close-up vertical movement angle tilt_l and the close-up zoom multiple zoom_l: pan_l=pan_p+b1; tilt_l = tilt_p + b2; zoom_l=zoom_p+b3. 22. According to the system described in item 12, the fine-tuning module further includes the following operations: Detect the face position of the speaker in the close-up picture; Calculate the offset between the center point of the face and the center point of the close-up image; Re-adjust the close-up horizontal movement angle pan_1, the close-up vertical movement angle tilt_1 and the close-up zoom multiple zoom_1 of the close-up shot based on the offset; Recapture the close-up image and perform face detection, repeating until the speaker's face is close to the center of the frame.

Claims

1. A method for tracking a speaker based on a panoramic and close-up camera, comprising: The panoramic camera captures everyone in the panoramic picture; Locating the face position information of the speaker in the panoramic picture based on the speaker's sound source and / or analyzing the facial movements of all persons in the panoramic picture; Based on the positional relationship between the panoramic camera and the close-up camera and the face position information of the speaker, the angle and focal length of the close-up camera are determined to take a close-up shot of the speaker; The face position of the speaker in the close-up picture is analyzed, and the close-up camera is fine-tuned to center the speaker in the close-up picture.

2. The method according to claim 1, wherein the step of locating the face position information of the speaker in the panoramic picture based on the speaker's sound source and / or analyzing the facial movements of all persons in the panoramic picture further comprises the following steps: Detecting faces of all persons in the panoramic picture, and determining face position information of each person in the panoramic picture; Dividing the sound source area of ​​corresponding range in the panoramic picture according to the sound source of the speaker; According to the facial position information of each person, searching for the face located in the sound source area; The face position information located in the sound source area is determined as the face position information of the speaker.

3. The method according to claim 1, wherein the step of locating the face position information of the speaker in the panoramic picture based on the speaker's sound source and / or analyzing the facial movements of all persons in the panoramic picture further comprises the following steps: Pre-train the model so that it can learn to distinguish the motion change features between speaking and non-speaking states; The facial features of each person in the panoramic picture are collected in time series, and the facial image of each person is input into a trained model, and the model outputs a sign of whether each person is speaking; Dividing corresponding sound source areas in the panoramic picture according to the sound source of the speaker; In the panoramic picture, a person whose face position information is located in the sound source area and is marked as a speaking state is determined as a speaker.

4. The method according to claim 1, wherein the step of locating the face position information of the speaker in the panoramic picture based on the speaker's sound source and / or analyzing the facial movements of all persons in the panoramic picture further comprises the following steps: Start recording audio and video of the facial movements of everyone in the panoramic view at the same time; Synchronously extract each person's video facial action features and audio sound waveform features; Analyze the correlation between each person's facial features and voice waveform features in the same time series; The face with relevant features is found to be the speaker, and its face position information is determined in the panoramic picture.

5. The method according to claim 1, wherein the step of locating the face position information of the speaker in the panoramic picture based on the speaker's sound source and / or analyzing the facial movements of all persons in the panoramic picture further comprises the following steps: Dividing the sound source area of ​​corresponding range in the panoramic picture according to the sound source of the speaker; If there are multiple faces in the sound source area, a face whose facial features and sound waveform features are related in the same time series is found among these faces as the speaker, and the face position information of the speaker is determined.

6. The method according to claim 2, 3 or 5, wherein the step of dividing the sound source areas of corresponding ranges in the panoramic picture according to the sound source of the speaker further comprises the following steps: Use a multi-microphone array to capture the speaker's voice signal; The angle of the sound source direction is calculated by the time difference when each microphone receives the speaker's voice; Convert the sound source direction angle into an x-coordinate and / or a y-coordinate of a panoramic picture; Taking the x coordinate and / or y coordinate as the center, define an area with a fixed width as the sound source area.

7. The method according to claim 1, wherein the step of determining the angle and focal length of the close-up camera based on the positional relationship between the panoramic camera and the close-up camera and the face position information of the speaker, and taking a close-up of the speaker further comprises the following steps: Based on the face position information of the speaker, the horizontal angle AngleH and the vertical angle AngleV between the center point of the face and the center point of the panoramic image are calculated, and the size of the face range Area is calculated; Based on the horizontal angle AngleH, the vertical angle AngleV and the face range size Area, determine the panoramic horizontal offset angle pan_p, the panoramic vertical offset angle tilt_p of the speaker's face relative to the center point of the panoramic image taken by the panoramic camera, and the panoramic zoom factor zoom_p if the speaker's face is enlarged to fill the screen; Based on the positional relationship between the panoramic camera and the close-up camera, the close-up horizontal movement angle pan_l, the close-up vertical movement angle tilt_l and the close-up zoom factor zoom_l of the close-up camera are determined.

8. The method according to claim 7, wherein the step of calculating the horizontal angle AngleH and the vertical angle AngleV between the center point of the face and the center point of the panoramic image based on the face position information of the speaker, and calculating the size of the face area Area further comprises the following steps: Determine the coordinates of the speaker's face center point (x1, y1) and the coordinates of the panoramic image center point (y0, y0); Get the horizontal field of view angle HFOV and the vertical field of view angle VFOV of the panoramic camera, as well as the width W and height H of the panoramic image; The vertical viewing angle distance W between the panoramic lens and the optical center of the panoramic image is calculated using the following formula: ′ and horizontal viewing distance H ′ : Vertical viewing angle distance W ′ =(H / 2) / tan(VFOV / 2); Horizontal viewing distance H ′ =(W / 2) / tan(HFOV / 2); The horizontal angle AngleH, the vertical angle AngleV and the face range size Area are calculated using the following formula: AngleH_Arctan(|x1-x0| / H ′ )4 AngleV=arctan(|y1-y0| / W ′ ); Area = |x1-x0|·|y1-y0|.

9. The method according to claim 8, wherein the determining the panoramic horizontal offset angle pan_p, the panoramic vertical offset angle tilt_p and the panoramic zoom factor zoom_p based on the horizontal angle AngleH, the vertical angle AngleV and the face range size Area further comprises the following steps: Determine the horizontal angle conversion coefficient a1, vertical angle conversion coefficient a2 and zoom conversion coefficient a3 of the panoramic camera; The panoramic horizontal offset angle pan_p, the panoramic vertical offset angle tilt_p and the panoramic zoom factor zoom_p are calculated using the following formula: pan_p=(AngleH)·a1; tilt_p = (AngleV)·a2; zoom_p=(W·H / Area)·a3.

10. A system for tracking a speaker based on panoramic and close-up cameras, comprising: Panoramic camera, used to capture all people in the panoramic picture; A positioning module, used to locate the face position information of the speaker in the panoramic picture based on the speaker's sound source and / or analyzing the facial movements of all persons in the panoramic picture; A determination module, configured to determine an angle and a focal length of the close-up camera based on a positional relationship between the panoramic camera and the close-up camera and facial position information of the speaker; Close-up camera, used to take close-up shots of speakers; The fine-tuning module is used to analyze the face position of the speaker in the close-up picture and fine-tune the close-up camera to center the speaker in the close-up picture.

Images