Intelligent conference management method and system

Abstract

The invention discloses an intelligent conference management method and system, and relates to the technical field of conference management, and the system comprises a voice recognition and processing module, a camera control module, a face recognition and tracking module, and a user management module. The voice recognition and processing module captures sound signals at all time points through a microphone array, obtains the amplitude of the sound signals through processing, compares the time difference and the intensity difference of the sound signals received by different microphones, generates sound source position coordinates, carries out preliminary judgment, and carries out voice recognition when the positions of the microphones coincide with the position coordinates of the sound source. At the moment, the sound source position is defined as a fuzzy position, and the fuzzy position is output; and the camera control module is used for acquiring the fuzzy position, calculating the distance between the fuzzy position and the positions of the N microphones for participants at the positions of the N microphones close to the fuzzy position, capturing face images, and outputting the captured face images.

Description

Technical Field

[0001] The present invention relates to the technical field of conference management, and in particular to an intelligent conference management method and system. Background Art

[0002] With the development of computer technology and network technology, enterprises, government agencies, educational departments and other organizations have found that the traditional meeting management model is inefficient and difficult to cope with the growing number and complexity of meetings. More and more organizations have begun to use digital tools to optimize internal processes. Meeting management is no exception. People have begun to use software to assist meeting management, including electronic documents and emails. With the continuous development of technology, the existing working methods have been changed to make communication more efficient and convenient, especially the birth of remote work and virtual meetings. However, traditional conference management usually uses fixed camera angles for conference recording and remote conferences, which makes it difficult to flexibly capture all participants, especially in large conferences where people in corners or edges may not be in the best field of view. Fixed cameras are difficult to automatically adjust the angle of view to maintain the best shooting effect, and usually do not have intelligent recognition functions such as face recognition, making it difficult to automatically obtain close-up shots of important speakers; In view of the above technical defects, a solution is now proposed. Summary of the invention

[0003] The purpose of the present invention is to provide an intelligent conference management system to solve the above-mentioned technical defects. The present invention obtains the voice signal of the microphone array in real time, and makes a preliminary judgment on the sound source position based on the time difference and intensity difference of the voice signals received by different microphones. When the microphone position partially overlaps with the coordinates of the sound source position, the fuzzy position is obtained, the face data near the fuzzy position is obtained, and the speaker position is judged through lip movement recognition. The speaker information is processed and output through the obtained speaker information, and the image information is digitally zoomed and cropped; for the voice information, the spectral characteristics of the background noise are obtained, the noise component is subtracted from the mixed signal, and the processed speaker information is output.

[0004] The object of the present invention can be achieved by the following technical solutions: an intelligent conference management system, including a speech recognition and processing module, a camera control module, a face recognition and tracking module and a user management module; The speech recognition and processing module uses a microphone array to capture sound signals at each time point, obtains the amplitude of the sound signal through processing, compares the time difference and intensity difference of the sound signals received by different microphones, generates the coordinates of the sound source position, and makes a preliminary judgment. When the microphone position and the sound source position coordinates partially overlap, the sound source position is defined as a fuzzy position and the fuzzy position is output; The camera control module obtains the fuzzy position, calculates the distance between the fuzzy position and the positions of N surrounding microphones, captures the face image, and outputs the captured face image; The face recognition and tracking module is used to obtain the captured face image, pre-process the face image, perform lip movement recognition and judgment on the pre-processed face image, and output the lip movement judgment result; The user management module outputs the speaker information at the position by acquiring the lip movement judgment result, wherein the speaker information includes image information and voice information.

[0005] Preferably, the time difference acquisition process of the sound signal is as follows: According to the sound signal of the microphone at each time point, the sound signal of the x microphone and the y microphone at each time point is normalized to generate a cross-correlation index; Solve by numerical method, construct geometric equations, and calculate the preliminary position of the sound source by TDOA method .

[0006] Preferably, the process of obtaining the sound signal strength difference is as follows: By obtaining the amplitude of the sound signal received by microphone i and the sound signal at microphone time point n, an amplitude peak is generated. For microphone i, an amplitude peak A is generated. i , and calculate the intensity difference between different microphones; The relative distance between the sound source and each microphone is calculated by the intensity difference, and a set of geometric equations is constructed to calculate the position of the sound source by the intensity difference method. .

[0007] Preferably, the process of obtaining the coordinates of the sound source position is as follows: The position information provided by TDOA and intensity difference is adjusted by the proportional coefficient k value to obtain the coordinates of the sound source position. The formula is: In the formula, x s and s The estimated point sound source coordinates are The horizontal and vertical coordinates of ; k is the proportional coefficient.

[0008] Preferably, the process of preliminary judgment is as follows: According to the coordinates of the sound source Draw a virtual circle with the center and radius r, and at the same time, the position of each microphone Draw a virtual circle with center and radius r, calculate the intersection area of ​​the two circles, and use geometric methods to calculate the intersection area A of the two circles; Set the intersection threshold C1 of the intersection area A and perform threshold judgment on the intersection area A: When the intersection area A of the two circles is greater than or equal to C1, it is determined that the coordinates of the sound source position coincide with the microphone position, and the speaker at the microphone position is speaking; When the intersection area A of the two circles is less than C1, it is determined that the coordinates of the sound source position partially coincide with the microphone position. Defined as fuzzy position.

[0009] Preferably, the process of calculating the distance between the fuzzy position and the N microphone positions is as follows: Through the microphone array, the Euclidean distance between the blurred position and the microphone position is calculated, and a list is created to store the distances of N microphones and their corresponding indices. A tuple list is used, where the tuple contains the indices of N microphones and the corresponding distances. The stored distance list is sorted, and the corresponding microphone position is extracted according to the selected microphone index.

[0010] Preferably, the facial image capturing process is as follows: The face image is captured by moving the camera to focus on the blurred position area; the Haar cascade classifier in the face detection algorithm OpenCV is used to perform face detection on the captured face image, the face detection algorithm is used to process the face image, and the corresponding face image is cropped from the original image.

[0011] Preferably, the face image preprocessing process is as follows: Convert the cropped face image to a grayscale image, normalize the grayscale image, and adjust the histogram distribution of the image. Obtain the Dlib library, obtain the lip pixels through the pre-trained facial landmark detection model in facial landmark detection, and use edge detection to extract the edge and contour of the lip area. The LBP values ​​of the pixels are counted to generate an LBP histogram. The frequency difference between the two LBP histograms at each LBP value is calculated, and the chi-square distance is calculated by weighted summation. The chi-square distance is calculated element by element using the chi-square distance formula. Through statistical analysis of the training data, the threshold is set to Q to perform lip movement judgment: When the chi-square distance is greater than or equal to Q, it is considered that the lips are moving, that is, speaking at this time; When the chi-square distance is less than Q, the lips are considered to be still, that is, no speech is being made at this time.

[0012] Preferably, the image information and voice information processing process is as follows: Image information processing: By obtaining lip movement recognition, the area where the speaker is located is digitally zoomed through an interpolation algorithm, and the area where the speaker is located is cropped; Speech information processing: Through spectrum analysis technology, the spectrum characteristics of background noise are obtained, and a noise model is established based on the obtained noise spectrum characteristics to subtract the noise component from the mixed signal.

[0013] An intelligent conference management method, comprising: Step 1: By acquiring the speech signal of the multi-microphone array in real time, the time difference and intensity difference of the speech signals received by different microphones are used to make a preliminary judgment on the sound source position. When the microphone position and the sound source position coordinates partially overlap, the sound source position is defined as a fuzzy position and the fuzzy position is output; Step 2: By obtaining the blurred position, the face data near the blurred position is obtained, and the speaker position is determined by lip movement recognition: when the speaker position is identified, the speaker information is obtained; when the speaker position is not identified, the default lens is displayed; Step three: Process and output the speaker information obtained, wherein the speaker information includes image information and voice information; digitally zoom and crop the image information, and obtain the spectral characteristics of the background noise for the voice information, subtract the noise component from the mixed signal, and output the processed speaker information.

[0014] The beneficial effects of the present invention are as follows: (1) The present invention uses a microphone array to capture sound signals at each time point, obtains the amplitude of the sound signal by processing, compares the time difference and intensity difference of the sound signals received by different microphones, generates the coordinates of the sound source position, and makes a preliminary judgment. By obtaining the intersection area of ​​two unit circles, a threshold judgment is made. It is judged that the coordinates of the sound source position partially overlap with the microphone position at this time, and the speaker at the microphone position may be speaking. The speaker can be automatically locked, and the camera can be mobilized to record and save the video of the speaker, making the meeting record more efficient and convenient.

[0015] (2) The present invention extracts contours through a contour detection algorithm, counts the LBP values ​​of all pixels, generates an LBP histogram, and calculates the chi-square distance by calculating the frequency difference between the two LBP histograms at each LBP value and performing a weighted summation to measure the similarity between the two histograms. This effectively compares and distinguishes the lip movements between different frames, and combines the fuzzy position information to accurately lock the speaker, thereby improving the accuracy of recognition and avoiding misjudgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present invention will be further described below in conjunction with the accompanying drawings; Figure 1 It is a flowchart of the system of the present invention; Figure 2 is a reference diagram of the method of the present invention; Figure 3 It is a schematic diagram of the overall steps of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Example 1: Please refer to Figure 1 and Figure 3 As shown, this embodiment provides an intelligent conference management system, including the following modules: The speech recognition and processing module uses a microphone array to capture sound signals at each time point, obtains the amplitude of the sound signal through processing, compares the time difference and intensity difference of the sound signals received by different microphones, generates the coordinates of the sound source position, and makes a preliminary judgment: when the sound source position is identified, it is matched with the speaker position; when the sound source position is not identified, the sound source position is defined as a fuzzy position and the fuzzy position is output; When the scheduled participants arrive at the venue, the facial recognition system will automatically complete the check-in process without any manual operation. Once the participant is successfully identified by the system, his / her attendance status will be updated immediately and automatically saved in the system. At the same time, relevant managers will receive notifications immediately to ensure that they can grasp the attendance of participants in real time. This not only improves the check-in efficiency, but also ensures the accuracy and timeliness of the data, providing strong support for conference management. By placing a circle of microphone arrays in the center of the conference table, the spacing between the microphones should be determined according to the expected speaker position, generally ranging from several tens of centimeters, effectively covering the entire conference table speaker, ensuring that the voice of each speaker can be clearly captured. By properly arranging the microphone array, the quality of the audio in the meeting and the accuracy of the sound source positioning are significantly improved, thereby enhancing the overall meeting experience; Through a unified data acquisition system, all microphones are synchronously collected for sound signals, and the sound signals of each microphone are recorded at the same time, and the timestamps of all sound signals are ensured to be consistent. The analog sound signals are converted into digital signals for subsequent digital signal processing, and the microphones are calibrated to increase or decrease the level of the audio signal to make the sound clearer, ensuring that the voice of each speaker can be clearly captured; The process of obtaining the time difference of the sound signal is as follows: According to the sound signal of the microphone at each time point, the sound signal of microphone i and microphone j at each time point is normalized to generate a cross-correlation index, and the formula based on it is: In the formula, i n and j n is the sound signal received by the two microphones, i n is the sound signal received by the first microphone, n is the time point; j n is the sound signal received by the second microphone, and n is the time point; is the cross-correlation index, which indicates the time offset between signal i and signal j. the relevance of the place; is the time offset, which indicates the relative time difference between two signals. When it is positive, it means that the j signal is delayed relative to the i signal. time unit; When it is negative, it means that the j signal is advanced time unit; is the summation symbol, indicating the summation of all time points n; n is the time point, from 1 to positive infinity; Calculated by the highest point In different The value under value, find The maximum value corresponding to , that is, the time difference between the two signals. The preliminary position of the sound source is calculated through the time difference information of multiple microphone pairs, and a set of geometric equations is constructed to calculate the preliminary position of the sound source through the TDOA method. ; Example: There are three microphones M1, M2 and M3, whose positions on the two-dimensional plane are (x1, y1), (x2, y2), and (x3, y3) respectively; The sound source position is (x s ,y s ), the speed of sound is c, and the time difference between the sound source and each microphone is calculated by the cross-correlation index. , where i and j are any two microphones among the three microphones M1, M2 and M3; is the time difference between the sound source arriving at M1 and M2; is the time difference between the sound source arriving at M1 and M3; is the time difference between the sound source arriving at M1 and M3; assumed , and , and according to the time difference and the speed of sound, establish the geometric equation: Wherein, x1, x2 and x3 represent the horizontal coordinates of the microphones M1, M2 and M3 respectively, and y1, y2 and y3 represent the vertical coordinates of the microphones M1, M2 and M3 respectively; s and s Represent the horizontal and vertical coordinates of the sound source position respectively, and solve them by numerical methods, construct a geometric equation system, and calculate the preliminary position of the sound source by TDOA method ; The process of obtaining the sound signal strength difference is as follows: By obtaining the amplitude of the sound signal received by microphone i and the sound signal at microphone time point n, the amplitude peak is generated. For microphone i, the amplitude peak is generated according to the following formula: In the formula, A i is the peak amplitude; is the sound signal received by microphone i, N is the number of sampling points of the sound signal, and n is the index of the time point; Similarly, obtain the peak amplitude A of microphone j j ; And calculate the intensity difference between different microphones: For microphones i and j, the intensity difference It is expressed as: In the formula, A i is the peak amplitude of microphone i, A j is the peak amplitude of microphone j; The relative distance between the sound source and each microphone is calculated by the intensity difference, and a set of geometric equations is constructed to calculate the position of the sound source by the intensity difference method. ; Adjust the scale factor k value of the position information provided by TDOA and intensity difference to obtain the precise coordinates of the sound source position , the formula is: In the formula, x s and s The estimated point sound source coordinates are The horizontal and vertical coordinates of ; k is the proportional coefficient; The preliminary judgment process is as follows: The sound source position coordinates Draw a virtual circle with the center and radius r, with the position of each microphone Draw a virtual circle with a center and radius r, and calculate the intersection area of ​​the two circles. Use geometric methods to calculate the intersection area A of the two circles. The formula is: In the formula, r is the radius of the two circles, and d is the distance between the centers of the two circles; Set the intersection threshold C1 of the intersection area A and perform threshold judgment on the intersection area A: When the intersection area A of the two circles is greater than or equal to C1, it is determined that the coordinates of the sound source position coincide with the microphone position, and the speaker at the microphone position is speaking; When the intersection area A of the two circles is less than C1, it is judged that the coordinates of the sound source position partially overlap with the microphone position. The speaker at this microphone position may be speaking. , i.e. fuzzy position; The camera control module obtains the blurred position and the positions of N microphones near the blurred position. For the participants at the meeting, the distance between the fuzzy position and the positions of N microphones is calculated, the face image is captured, and the captured face image is output; Calculate the distance between the blur position and the N microphone positions: Through the microphone array, calculate the Euclidean distance between the fuzzy position and the microphone position, create a list to store the distances of N microphones and their corresponding indices, use a tuple list, where N tuples contain the index of the microphone and the corresponding distance, sort the stored distance list to find the N microphones with the smallest distance, and extract the corresponding microphone position according to the selected microphone index; The python code example is: ```Python import numpy as np # Sound source position x_s, y_s = 0.5, 0.5 # List of microphone positions microphone_positions = [ (0.1, 0.2), (0.4, 0.5), (0.7, 0.8), (0.2, 0.3), (0.5, 0.6) ] # The number of microphones to select e = 2 # Calculate the distance between the sound source position and each microphone position distances = [] for i, (x_i, y_i, z_i) in enumerate(microphone_positions): distance = np.sqrt((x_s - x_i)**2 + (y_s - y_i)**2) distances.append((i, distance)) # Sort the distances sorted_distances = sorted(distances, key=lambda x: x[1]) # Select the microphone with the smallest distance nearest_mics = sorted_distances[:e] # Extract microphone position nearest_mic_positions = [microphone_positions[i] for (i, _) innearest_mics] # Output results print("Nearest microphones and their positions:") for i, (x_i, y_i) in enumerate(nearest_mic_positions): print(f"Microphone {i+1}: ({x_i}, {y_i}") ``` Face image capture: By moving the camera to focus on the blurred position area, the face image is captured; the Haar cascade classifier in the face detection algorithm OpenCV is used to perform face detection on the captured face image, and the face detection algorithm is used to process the face image to find the face position in the face image. Usually, the bounding box is used to extract the bounding box coordinates and crop the corresponding face image from the original image. The specific steps are as follows: Determine the cropping area: use the coordinates of the bounding box as the starting point of the cropping area, and the width and height as the size of the cropping area; Cropping image: extract pixels in the cropping area from the original image to form a new cropped face image; Save the cropped face image to a file in PNG image format; The face recognition and tracking module is used to obtain the captured face image, pre-process the face image, perform lip movement recognition and judgment on the pre-processed face image, and output the lip movement judgment result; The face image preprocessing process is as follows: Convert the cropped face image to a grayscale image to reduce computational complexity. Normalize the grayscale image and adjust the image histogram distribution to make the image brightness more evenly distributed and enhance the image contrast. This can reduce the impact of illumination changes on the image, especially in the case of inconsistent illumination conditions, making subsequent feature extraction and model training more robust and effective. The lip movement recognition process is as follows: Install Dlib using Python's package management tool pip. Open a terminal or command prompt and run the pipinstalldlib command. Get the Dlib library, use the pre-trained 68-point facial landmark detection model in facial landmark detection. Among the 68 key points, the key points with specific numbers correspond to the lips. In Dlib's 68-point model, the number range of the lip key points is as follows: upper lip: 48-60; lower lip: 61-67. Use OpenCV to read the face image to be processed, use Dlib's HOG+SVM face detector or OpenCV's Haar cascade classifier to detect the face in the image, and use Dlib's facial landmark detector to detect the 68 key points on the face. Use edge detection algorithm to extract the edge of the lip area. First, perform Gaussian filtering on the image to remove noise, calculate the gradient magnitude and direction of the image, retain the pixel with the largest gradient magnitude, suppress other pixels, use high and low thresholds to determine the edge pixels, connect the edge pixels to form a complete edge, and extract the contour from the edge image; the contour is a collection of edge pixels that form the boundary of the object in the image; The contour detection algorithm is used to extract the contour, and the extracted contour is used to describe the external shape of the lips; the shape characteristics of the image are described by the Hu moment descriptor; the Hu moment is invariant to rotation, translation and scaling. The specific steps include: Calculate moment: calculate the central moment of the image; Normalization: Normalize the central moment to eliminate the effects of translation, rotation, and scaling; Calculate Hu moments: Calculate seven Hu moments from the normalized central moment to describe the characteristics of the shape; Generate a binary pattern based on the comparison results, convert the binary pattern into a decimal number, and count the LBP values ​​of all pixels to generate an LBP histogram as a texture feature; Select the radius and number of neighborhood points of LBP, perform LBP transformation on the grayscale image to obtain LBP image, count the frequency of occurrence of each LBP value in the LBP image to form a histogram, and normalize the LBP histogram to make it a probability distribution; LBP histogram: By capturing the texture information of the local area in the image, a histogram describing the texture characteristics of the image is generated. The LBP histogram can effectively describe and compare the texture characteristics of the image; The chi-square distance is calculated by calculating the frequency difference of the two LBP histograms at each LBP value and performing weighted summation to measure the similarity between the two histograms, effectively comparing and distinguishing the lip movements between different frames, and obtaining two LBP histograms. The LBP histograms of two adjacent frames are P and R respectively, and the chi-square distance G is calculated. 2 (P, R), the formula is: Where P and R are two LBP histograms; f is the length of the histogram, that is, the number of LBP values; P h is the frequency of the hth LBP value in the first LBP histogram; R h is the frequency of the hth LBP value in the second LBP histogram; To represent the sum of all h from 1 to f; It should be noted that in a multi-person conference scenario, especially when participants are sitting close to each other, multiple participants may move their lips at the same time. In order to improve the accuracy of recognition and avoid misjudgment, the system needs to set a minimum acquisition time threshold to ensure that the lip movement of a certain speaker can be stably detected, rather than short, uncertain movements. The chi-square distance formula is used to calculate the chi-square distance element by element. Through statistical analysis of the training data, the threshold is set to Q to perform lip movement judgment: When the chi-square distance is greater than or equal to Q, it is considered that the lips are moving, that is, the speaker is speaking at this time, and the speaker information at the secondary microphone position is output; When the chi-square distance is less than Q, the lips are considered to be stationary, that is, no speech is being made at this time, and the default lens is used to obtain a panoramic image and continuously collect voice information; The user management module outputs the speaker information at the position by acquiring the lip movement judgment result, wherein the speaker information includes image information and voice information; By obtaining the lip movement judgment results, the speaker's image information and voice information are identified and output. When the system detects that a participant at a certain position is making lip movements (i.e. speaking), the speaker's image is acquired and the speaker's voice information is processed to increase the speaker's voice clarity. This module will perform the following operations: The image information processing process is as follows: By acquiring lip movement recognition, the speaker's area is digitally zoomed, and the image is enlarged to highlight the speaker. Digital zoom increases the image resolution through an interpolation algorithm, thereby visually enlarging the image. Through digital zoom, the details of the speaker's face are clearly visible, so that participants can clearly see the speaker's expression and lip movements. Crop the area where the speaker is located from the video stream to generate a close-up image focused on the speaker; the cropped image should maintain good composition, ensuring that the speaker is in the center of the picture and the face occupies most of the picture; The voice information processing process is as follows: Through spectrum analysis technology, the spectrum characteristics of background noise are obtained. Based on the obtained noise spectrum characteristics, a noise model is established to prepare for subsequent noise reduction. The noise component is subtracted from the mixed signal to improve the quality of the speech signal. Through noise reduction, the speaker's speech clarity is enhanced to ensure that other participants can clearly hear the speaker's voice; It should be noted that the camera should try to capture the speaker's frontal perspective, that is, the camera should be at a 90-degree angle to the speaker's face; this can ensure that the speaker's facial features are fully presented, so that the audience can clearly see the speaker's expression and lip movements; the camera's height should be adjusted to be roughly level with the speaker's eyes; this can avoid distortion caused by looking down or up, making the image more natural; use soft lighting to evenly illuminate the speaker's face to avoid one side of the face being too bright and the other side too dark, resulting in excessive image contrast; at the same time, the meeting host can manually intervene to manually lock the camera and switch the perspective to focus on different participants.

[0019] Example 2: Please refer to Figure 2 Based on Example 1, this embodiment also provides an intelligent conference management method, including the following specific steps: Step 1: By acquiring the speech signal of the multi-microphone array in real time, the time difference and intensity difference of the speech signals received by different microphones are used to make a preliminary judgment on the sound source position. When the microphone position and the sound source position coordinates partially overlap, the sound source position is defined as a fuzzy position and the fuzzy position is output; Step 2: By obtaining the blurred position, the face data near the blurred position is obtained, and the speaker position is determined by lip movement recognition: when the speaker position is identified, the speaker information is obtained; when the speaker position is not identified, the default lens is displayed; Step three: Process and output the speaker information obtained, wherein the speaker information includes image information and voice information; digitally zoom and crop the image information, and obtain the spectral characteristics of the background noise for the voice information, subtract the noise component from the mixed signal, and output the processed speaker information.

[0020] In the application, the several formulas involved are all calculated by removing the dimensions and taking their numerical values. The formula is a formula obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The formula is set by technical personnel in this field according to actual conditions.

[0021] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. A person of ordinary skill in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution.

[0022] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0023] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. An intelligent conference management system, characterized in that: It includes voice recognition and processing module, camera control module, face recognition and tracking module and user management module; The speech recognition and processing module uses a microphone array to capture sound signals at each time point, obtains the amplitude of the sound signal through processing, compares the time difference and intensity difference of the sound signals received by different microphones, generates the coordinates of the sound source position, and makes a preliminary judgment. When the microphone position and the sound source position coordinates partially overlap, the sound source position is defined as a fuzzy position and the fuzzy position is output; The camera control module obtains the blur position, calculates the distance between the blur position and the surrounding N microphone positions, captures the face image, and outputs the captured face image; The face recognition and tracking module is used to obtain the captured face image, pre-process the face image, perform lip movement recognition and judgment on the pre-processed face image, and output the lip movement judgment result; The user management module outputs the speaker information at the position by acquiring the lip movement judgment result, wherein the speaker information includes image information and voice information.

2. An intelligent conference management system according to claim 1, characterized in that: The process of obtaining the time difference of the sound signal is as follows: According to the sound signal of the microphone at each time point, the sound signal of the x microphone and the y microphone at each time point is normalized to generate a cross-correlation index; Solve by numerical method, construct geometric equations, and obtain the preliminary position of the sound source by TDOA method .

3. The intelligent conference management system according to claim 1, characterized in that: The process of obtaining the sound signal strength difference is as follows: By obtaining the amplitude of the sound signal received by microphone i and the sound signal at microphone time point n, an amplitude peak is generated. For microphone i, an amplitude peak A is generated. i , and calculate the intensity difference between different microphones; The relative distance between the sound source and each microphone is calculated by the intensity difference, and a set of geometric equations is constructed to calculate the position of the sound source by the intensity difference method. .

4. An intelligent conference management system according to claim 3, characterized in that: The process of obtaining the coordinates of the sound source position is as follows: The position information provided by TDOA and intensity difference is adjusted by the proportional coefficient k value to obtain the coordinates of the sound source position. The formula is: In the formula, x s and s The estimated point sound source coordinates are The horizontal and vertical coordinates of ; k is the proportional coefficient.

5. An intelligent conference management system according to claim 4, characterized in that: The initial judgment process is as follows: According to the coordinates of the sound source Draw a virtual circle with the center and radius r, and at the same time, the position of each microphone Draw a virtual circle with center and radius r, calculate the intersection area of ​​the two circles, and use geometric methods to calculate the intersection area A of the two circles; Set the intersection threshold C1 of the intersection area A and perform threshold judgment on the intersection area A: When the intersection area A of the two circles is greater than or equal to C1, it is determined that the coordinates of the sound source position coincide with the microphone position, and the speaker at the microphone position is speaking; When the intersection area A of the two circles is less than C1, it is determined that the coordinates of the sound source position partially coincide with the microphone position. Defined as fuzzy position.

6. The intelligent conference management system according to claim 1, characterized in that: The process of calculating the distance between the blurred position and the N microphone positions is as follows: Through the microphone array, the Euclidean distance between the blurred position and the microphone position is calculated, and a list is created to store the distances of N microphones and their corresponding indices. A tuple list is used, where the tuple contains the indices of N microphones and the corresponding distances. The stored distance list is sorted, and the corresponding microphone position is extracted according to the selected microphone index.

7. The intelligent conference management system according to claim 1, characterized in that: The face image capture process is as follows: The face image is captured by moving the camera to focus on the blurred position area; the Haar cascade classifier in the face detection algorithm OpenCV is used to perform face detection on the captured face image, the face detection algorithm is used to process the face image, and the corresponding face image is cropped from the original image.

8. The intelligent conference management system according to claim 1, characterized in that: The face image preprocessing process is as follows: Convert the cropped face image to a grayscale image, normalize the grayscale image, and adjust the histogram distribution of the image. Obtain the Dlib library, obtain the lip pixels through the pre-trained facial landmark detection model in facial landmark detection, and use edge detection to extract the edge and contour of the lip area. The LBP values ​​of the pixels are counted to generate an LBP histogram. The frequency difference between the two LBP histograms at each LBP value is calculated, and the chi-square distance is calculated by weighted summation. The chi-square distance is calculated element by element using the chi-square distance formula. Through statistical analysis of the training data, the threshold is set to Q to perform lip movement judgment: When the chi-square distance is greater than or equal to Q, it is considered that the lips are moving, that is, speaking at this time; When the chi-square distance is less than Q, the lips are considered to be still, that is, no speech is being made at this time.

9. An intelligent conference management system according to claim 8, characterized in that: The image information and voice information processing process is as follows: Image information processing: By obtaining lip movement recognition, the area where the speaker is located is digitally zoomed through an interpolation algorithm, and the area where the speaker is located is cropped; Speech information processing: Through spectrum analysis technology, the spectrum characteristics of background noise are obtained, and a noise model is established based on the obtained noise spectrum characteristics to subtract the noise component from the mixed signal.

10. An intelligent conference management method, the system is applied to an intelligent conference management system according to any one of claims 1 to 9, characterized in that: The following steps are involved: Step 1: By acquiring the speech signal of the multi-microphone array in real time, the time difference and intensity difference of the speech signals received by different microphones are used to make a preliminary judgment on the sound source position. When the microphone position and the sound source position coordinates partially overlap, the sound source position is defined as a fuzzy position and the fuzzy position is output; Step 2: By obtaining the blurred position, the face data near the blurred position is obtained, and the speaker position is determined by lip movement recognition: when the speaker position is identified, the speaker information is obtained; When the speaker location is not identified, the default lens is displayed; Step three: Process and output the speaker information obtained, wherein the speaker information includes image information and voice information; digitally zoom and crop the image information, and obtain the spectral characteristics of the background noise for the voice information, subtract the noise component from the mixed signal, and output the processed speaker information.

Images