Sound source direction positioning and directional sound enhancement method based on microphone array and camera

Abstract

The invention discloses a sound source direction positioning and directional sound enhancement method based on a microphone array and a camera, and the method comprises the steps: detecting all persons in a scene through a target detection model (such as YOLO), and generating a corresponding person frame; then, the depth information of each pixel is calculated by utilizing a depth Anything model, and the depth information is matched with the detected human frame; a sounder closest to the camera is screened out through average depth calculation, and only a close-range effective target is reserved. According to the invention, the problems of inaccurate sound source positioning and poor sound enhancement effect in a complex environment can be solved. According to the invention, images are captured in real time through the camera, and sound data of the microphone array are combined, so that the system can accurately position and enhance a target sound source, and especially shows excellent performance in a multi-sound-source and high-noise environment.

Description

Technical Field

[0001] The present invention relates to the field of sound positioning and enhancement, and in particular to a method for sound source direction positioning and directional sound enhancement based on a microphone array and a camera. Background Art

[0002] With the rapid development of artificial intelligence (AI), Internet of Things (IoT) and multimedia technology, fields such as speech recognition, smart home, and video conferencing have put forward higher requirements for accurate sound capture and processing. Especially in complex acoustic environments, such as scenes with large background noise or multiple people speaking at the same time, traditional sound localization and enhancement technologies are difficult to meet actual needs. In this context, multimodal fusion technology that combines visual information (camera data) and audio information (microphone array data) has become a research hotspot.

[0003] Traditional methods mainly rely on microphone arrays to locate the sound source, and determine the direction of the sound source by analyzing the phase difference and time delay of the sound signal. However, this method has obvious limitations when facing complex environments. Therefore, the research trend in recent years has begun to try to combine cameras and microphone arrays to improve the accuracy of sound positioning and enhancement through visual assistance technology.

[0004] "CN112015364B- Method and device for adjusting sound pickup sensitivity" provides a technology for achieving sound enhancement by combining a microphone array with a camera, relying on the camera to track the movement of the lips to assist the microphone array in adjusting the sound pickup sensitivity. However, this technology has several obvious limitations:

[0005] Rough positioning: This technology relies on the camera to identify the speaker's lip movements and uses them as a basis for adjusting the microphone sensitivity. However, this recognition method is difficult to accurately identify the speaker's position in a complex multi-person environment, especially when the background is complex, and may mistakenly enhance irrelevant sound sources.

[0006] Lack of accurate depth information: This technology does not use depth detection to filter out irrelevant sound sources at a distance, but relies entirely on the initial positioning of camera images and microphone arrays. This method is prone to errors in multi-person scenarios and lacks effective distinction between distant speakers.

[0007] Unable to dynamically adjust the filter: For sound enhancement, the patent only uses a fixed method to adjust the pickup sensitivity, lacking real-time analysis and adaptive adjustment of sound characteristics.

[0008] "CN115862682B-Sound detection method and related equipment" adopts multimodal fusion of audio and video features, comprehensively judges the location of the sound source through visual and audio data, and then enhances the target sound. However, this method also has limitations:

[0009] Inaccurate positioning in multi-sound source environments: This technology relies solely on the fusion of visual and audio features to determine the sound source. In a scene with multiple people speaking, it is difficult to accurately distinguish who are the main speakers, especially when confusion is likely to occur between multiple targets at relatively close distances.

[0010] Poor real-time performance: The process of multimodal data fusion is complex and requires a lot of computing resources, resulting in poor performance in application scenarios with high real-time requirements. Summary of the invention

[0011] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a method for sound source direction localization and directional sound enhancement based on a microphone array and a camera.

[0012] The purpose of the present invention is achieved through the following technical solutions:

[0013] The method for sound source direction localization and directional sound enhancement based on microphone array and camera includes the following steps:

[0014] S1. Install the camera at the center of the microphone array and pre-process the image captured by the camera; detect all people in the image through the target detection model and generate corresponding person frames;

[0015] S2. Calculate the depth information of each pixel in the image using the depth estimation module, and match the depth information of the pixel with the detected human frame;

[0016] By calculating the average depth of the pixels inside each person's frame, the distance between the speaker and the camera is determined. By setting a depth threshold, targets that are far away are filtered out, and valid frames of those that are close and may be the speaker are retained.

[0017] S3, analyzing the key points of the human body on the valid human frame, and extracting multiple key points of the human body including the mouth, eyes, nose, etc.;

[0018] S4, analyzing the mouth shape movement state of the key points of the mouth: detecting the degree of mouth opening and closing representing the intensity of the sound of the speaker, and detecting the lip movement speed representing the frequency and speed of the sound of the speaker;

[0019] The gain of the sound filter increases as the speaker's mouth opens and closes more, and the strength of the sound signal increases accordingly;

[0020] The frequency response of the sound filter increases as the speed of the speaker's mouth movement becomes faster;

[0021] S5. Calculate the angle of the speaker's mouth relative to the center point of the camera, thereby identifying the speaker's position, and adjust the gain of the microphone in the corresponding direction according to the layout of the microphone array.

[0022] The depth estimation module generates a depth map for each image. The depth map D(x, y) represents the distance between the pixel at each position (x, y) in the image and the camera. The depth map has the same coordinate system and resolution as the input image, and the depth information of the pixel is matched with the detected human frame through coordinate matching.

[0023] The depth information of the pixel is matched with the detected human frame, and the specific steps are as follows:

[0024] (1) Traverse each person box: The person box set B detected by the target detection model = {B1, B2, ..., B n}, process each person's box B one by one i , each person box B i The boundary coordinates are (x i1 ,x i2 ,y i1 ,y i2 ), represents the position range of the target person in the image; where x i1 ,y i1 Human frame B i The horizontal and vertical coordinates of the upper left corner, x i2 ,y i2 Human frame B i The horizontal and vertical coordinates of the lower right corner;

[0025] (2) Find the corresponding coordinate area from the depth map of the image: For each person box B i , find the pixel range corresponding to the human frame from the depth map D(x,y) generated by the depth estimation model. The coordinates of this range are consistent with the boundary coordinates of the human frame, that is, x i1 ≤x≤x i2 and i1 ≤y≤y i2 , where x and y are the horizontal and vertical coordinates of the pixel respectively;

[0026] (3) Crop the corresponding depth map area: Crop the area corresponding to the human frame B in the depth map D(x,y). i The cropped area contains the depth values ​​of all pixels in the person frame, that is, the depth information sub-image of the target person.

[0027] The clipping formula is:

[0028] in, Is with people frame Bi The matched depth map sub-region indicates the corresponding region of the person in the depth map;

[0029] (4) Collecting depth information within the human frame: Through cropping operations, a set of depth values ​​of all pixels corresponding to the human frame area of ​​the person is extracted from the depth map. This set represents the depth information of the person in three-dimensional space and is used for subsequent average depth calculations.

[0030] The average depth of the pixels inside the human frame is calculated as follows:

[0031]

[0032] Among them, D avg (B i ) is human frame B i where N is the total number of pixels in the human frame; D(x,y) is the depth value of each pixel in the human frame area, taken from the cropped depth map sub-area D Bi (x,y).

[0033] The degree of opening and closing of the mouth of the speaker is calculated as follows:

[0034] Assume that the set of upper lip key points is {(x u1 ,y u1 ),(x u2 ,y u2 ),...(x un ,y un )}, the set of lower lip key points is {(x l1 ,y l1 ),(x l2 ,y l2 ),...(x ln ,y ln )}, each pair of upper lip and lower lip key points is one-to-one corresponding;

[0035] Calculate the vertical distance between each pair of upper and lower lip key points, and then find the average of these distances;

[0036]

[0037] Where n is the number of key points of the upper and lower lips; (x ui ,y ui ) is the coordinate of the i-th upper lip key point;

[0038] (x li ,y li ) is the coordinate of the i-th lower lip key point; d open It is the average degree of opening and closing of the mouth.

[0039] The lip movement speed of the speaker is calculated as follows:

[0040]

[0041] Where v(t) is the velocity of the lips at time t, d open (t), d open (t-1) are the degree of mouth opening and closing at time points t and t-1 respectively, and Δt is the time interval.

[0042] The gain G(t) of the sound filter is adjusted according to the degree of opening and closing of the speaker's mouth:

[0043] G(t)=G0+k·d open ;

[0044] Where G0 is the basic gain, k is the gain adjustment factor, and d open It is the degree of opening and closing of the mouth.

[0045] The frequency response f(t) of the filter is adjusted according to the lip movement speed of the speaker:

[0046] f(t)=f0+k f v(t);

[0047] Where f0 is the fundamental frequency, k f is the frequency adjustment coefficient, and v(t) is the lip movement speed.

[0048] The specific implementation steps of step S5 are as follows:

[0049] (1) Accurate azimuth calculation: Obtain the coordinates of the speaker’s lip key points P through the camera lips =(x lips ,y lips ) and the image center point (x center ,y center ), calculate the angle of the lips relative to the center point of the camera, that is, the azimuth angle θ of the lips:

[0050]

[0051] (2) Adjust the gain of the microphone array based on the azimuth angle: For each microphone i, we set its center angle to θ, then the gain adjustment can be calculated by the following formula:

[0052] G i (t) = G0 + k·max(0, cos(θ-θ i ));

[0053] Among them, G i(t) is the gain value of the i-th microphone at time t, G0 is the basic gain, θ i is the center angle of the i-th microphone, k is the gain adjustment coefficient;

[0054] By using the cosine function cos(θ-θ i ) to represent the relationship between the lip position and the center angle of each microphone, and the system can dynamically adjust the gain of each microphone;

[0055] When the lip azimuth angle θ is close to the center angle θ of a certain microphone i When the gain G i will be maximized; other microphones further away will receive less gain.

[0056] The microphone array also performs distributed gain adjustment: the gain distribution based on the lip azimuth angle makes a smooth transition between multiple microphones; when the lip position is in the middle of the coverage area of ​​two microphones, the gains of the two adjacent microphones will be adjusted accordingly to enhance the sound signal in this direction; for microphones that are far away from the lip position, the gain will be reduced or even suppressed.

[0057] At the same time, the present invention provides:

[0058] A server comprises a processor and a memory, wherein the memory stores at least one program, and the program is loaded and executed by the processor to implement the above-mentioned sound source direction positioning and directional sound enhancement method based on a microphone array and a camera.

[0059] A computer-readable storage medium stores at least one program, which is loaded and executed by a processor to implement the above-mentioned sound source direction positioning and directional sound enhancement method based on a microphone array and a camera.

[0060] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0061] 1. The present invention can accurately screen the speaker: by combining depth detection, it can effectively filter out distant or irrelevant speakers. This process enables the system to focus on important sound sources at close range, greatly reducing the interference of distant noise.

[0062] 2. The present invention can improve positioning accuracy: Different from the prior art, the present invention uses depth information to accurately determine the distance between each human frame and the camera, thereby accurately locating each speaker and avoiding the errors that may be caused by relying solely on images or microphone signals.

[0063] 3. The present invention can reduce background noise interference: through depth detection, the system can accurately lock the speaker at a close distance in a multi-person scene, and can effectively shield irrelevant noise even in a noisy environment.

[0064] 4. The mouth key point detection and dynamic filtering adjustment of the present invention can improve the intelligence of sound processing. After completing the effective target screening, the present invention uses the human key point detection model (DWPose) to carefully analyze the mouth shape of the speaker to obtain key information such as the degree of mouth opening and closing and the lip movement speed. Through this information, the gain and frequency response of the sound filter are adjusted in real time, thereby optimizing the enhancement effect of the sound signal.

[0065] 5. The present invention can realize adaptive filter adjustment: the system can dynamically adjust the sound filter by the degree of mouth opening and closing and the speed of lip movement. When the degree of opening and closing is large, the gain is increased to ensure a higher sound signal strength; when the degree of opening and closing is small, the gain is reduced to suppress irrelevant noise. This adaptive adjustment method significantly improves the clarity of the sound signal.

[0066] 6. The present invention can accurately capture sound features: the dynamic analysis of key points of the mouth is not only used to determine whether a sound is being made, but also to accurately adjust the frequency response of the filter according to subtle changes in the mouth shape, so that the system can respond to the speaker's speech speed and pitch changes in real time, thereby improving the intelligence level of sound enhancement.

[0067] 7. The filter optimization of the present invention is consistent with the direction of the sound source: the position change of the mouth corners can help the system dynamically adjust the directional gain of the microphone array, so that the enhanced direction is consistent with the actual position of the speaker, further improving the accuracy of sound source positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a structural schematic diagram of the sound source direction localization and directional sound enhancement system based on microphone array and camera.

[0069] Figure 2 The present invention is a flowchart of a method for sound source direction localization and directional sound enhancement based on a microphone array and a camera. DETAILED DESCRIPTION

[0070] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0071] The present invention proposes a sound source direction positioning and directional sound enhancement system based on a microphone array and a camera, aiming to solve the problems of inaccurate sound source positioning and poor sound enhancement effect in complex environments. By capturing images in real time with a camera and combining the sound data of a microphone array, the system can accurately locate and enhance the target sound source, especially in environments with multiple sound sources and high noise, showing excellent performance. In order to make the technical solution clearer, the various modules of the present invention and their implementation principles will be described in detail below in conjunction with the system architecture diagram.

[0072] 3.1 Overall system architecture

[0073] The overall system architecture is as follows Figure 1 As shown in Figure 1, the system is mainly divided into three layers: basic layer, processing layer and output layer. Each layer contains multiple key modules, and the modules work together to ensure that the system can achieve efficient and accurate sound source localization and sound enhancement in complex environments.

[0074] 3.1.1 Basic Layer

[0075] The basic layer is the underlying support part of the system, including basic equipment and infrastructure:

[0076] 3.1.1.1 Basic equipment

[0077] Camera: Used to capture real-time images of the scene. The position of the camera is precisely calculated and installed in the center of the microphone array to ensure spatial consistency of image and sound data.

[0078] Microphone array: The system uses an array of four directional microphones arranged in a specific geometric layout (ring) to optimize the accuracy and coverage of sound source localization. The microphone array is not only used to capture sound signals in the environment, but also to determine the approximate direction of the sound source through a preliminary sound source localization algorithm.

[0079] Edge computing devices: As the core processing platform of the system, edge computing devices are responsible for tasks such as image processing, sound signal processing, data synchronization and fusion. The introduction of edge computing not only improves the real-time performance of the system, but also reduces the dependence on cloud computing resources and adapts to the deployment requirements in resource-constrained environments.

[0080] 3.1.1.2 Infrastructure

[0081] High-performance computing hardware, such as NVIDIA graphics cards, is used to process image and sound data. The efficient operation of the computing part is the basis for ensuring the real-time and accuracy of the system.

[0082] The system accelerates the reasoning process of AI models through inference engines such as ONNX and TensorRT. Especially in image processing and speech recognition, inference acceleration significantly reduces latency and improves the response speed of the system. In order to ensure the stability and scalability of the system, containerization technologies such as Docker are used in the deployment part. Containerization technology allows each module of the system to be independently developed, tested and deployed, making it easy to maintain and upgrade.

[0083] 3.1.2 Processing Layer

[0084] The processing layer is the core functional part of the system, responsible for processing the input image and sound data to achieve sound source localization and sound enhancement. The processing layer is divided into two main modules: image processing and audio processing:

[0085] 3.1.2.1 Image processing

[0086] Target detection: The image processing module uses the images captured by the camera for target detection. The system uses a deep learning model (YOLO) to quickly identify and mark people or objects in the image. The target detection module can accurately locate the possible sound source in the scene and generate the corresponding human frame data for subsequent processing. Depth detection: The system uses the depth detection model of DepthAnything to calculate the distance information between the detection target and the camera. The depth detection module can filter out distant background targets and only retain targets that are closer and may be sound sources to improve the processing efficiency and accuracy of the system.

[0087] Human key point detection: Based on target detection and depth detection, the system further identifies the human key points of the detected target, especially the important feature points such as the mouth, eyes, and nose.

[0088] By detecting the mouth shape, the system can determine whether the detected target is speaking, and assist the sound source localization module to improve the positioning accuracy.

[0089] 3.1.2.2 Audio Processing

[0090] Directional sound pickup: The microphone array performs preliminary sound source positioning by analyzing the phase difference and time delay of the sound signal. The directional sound pickup module will enhance the sound in a specific direction based on the positioning result, while suppressing background noise in other directions. This process can effectively improve the signal-to-noise ratio of the sound, making the target sound source clearer and more identifiable.

[0091] Speech recognition: Based on directional sound pickup, the system further processes the enhanced sound signal through speech recognition technology to identify the speech content. This function is not only used to provide real-time feedback, but also to provide data support for subsequent advanced applications (such as voice commands and dialogue systems).

[0092] 3.1.3 Output Layer

[0093] The output layer is responsible for the final output of the processed data and provides support for downstream applications or devices.

[0094] The main function of this level is to ensure the quality and real-time nature of the processing results, including:

[0095] Processing result output: The edge computing device outputs the processed sound signal and image recognition results to the system's output module in real time. The output module is responsible for transmitting the enhanced sound signal to the user end to ensure that the final output sound has high quality and low noise characteristics. This module can also feed back the processing results to other application systems to meet the needs of different scenarios.

[0096] 3.2 Image Processing Flow

[0097] Image processing flow Figure 2 As shown in FIG. 1 , it is a key part of the present invention for analyzing and processing image data captured by the camera. Its main functions include target detection, human key point detection, depth detection, and mouth shape detection and motion analysis. The implementation method and technical details of each technical link are described in detail below.

[0098] 3.2.1 Image Capture and Preprocessing Image Capture: The camera module is installed at the center of the microphone array to ensure the spatial consistency of image and sound data. The camera has high resolution and good low-light imaging capabilities, can capture images in the environment in real time, and transmit data to the image processing module at a high frame rate.

[0099] Image preprocessing: Before the image enters the processing stage, it is first preprocessed, such as denoising, resizing, etc.

[0100] 3.2.2 YOLO object detection

[0101] The YOLO (You Only Look Once) model is used to detect people in an image and generate corresponding human frames. In the system of the present invention, the main task of YOLO is to identify people in an image through fast target detection and to intercept each detected target person. The detection results will be combined with the results of depth estimation to filter out human frames close to the camera.

[0102] YOLO detection process: The system feeds the input image into the YOLO model for target detection, and the output is the box of each person detected. These boxes define the area of ​​each person in the image for subsequent processing steps. The results of YOLO target detection are as follows:

[0103] Β={B1,B2,...,Bn};

[0104] Among them, Β is the set of all human boxes detected by YOLO, B i is the i-th detection box, each of which contains a person.

[0105] 3.2.3 Depth Detection

[0106] The depth estimation model (using DepthAnything) is used to calculate the depth information of each pixel in the image, that is, the distance between each pixel and the camera. The result of depth estimation is a depth map in which each pixel is assigned a depth value.

[0107] Depth map generation: The depth estimation model calculates the depth map D(x,y) from the input image, where D(x,y) represents the distance between the pixel at position (x,y) in the image and the camera. The depth map provides three-dimensional information in the scene, which can help the system identify the distance of each pixel from the camera.

[0108] Depth estimation formula:

[0109] D(x,y)=Depth(x,y);

[0110] Among them, D(x,y) is the depth value of each pixel, and Depth(x,y) is the output result of the depth estimation model, which indicates the distance between each pixel and the camera.

[0111] 3.2.4 Depth Estimation Combined with YOLO to Implement Human Frame Filtering

[0112] To determine the detected box B for each person i To determine the distance from the camera, the system combines the human frame output by YOLO with the pixel depth information generated by the depth estimation model. By calculating the average pixel depth inside each human frame, the system can determine the distance of the target and filter out targets that are farther away by setting a depth threshold.

[0113] A. Matching of human frame and depth map

[0114] The human frame detected by YOLO: The YOLO model outputs the human frame of each target person, defined as B i , each person box has its rectangular boundary, defined by the coordinates of the upper left corner (x1, y1) and the lower right corner (x2, y2). This means that the person box contains pixels ranging from x1 to x2 and y1 to y2.

[0115] Depth map generated by the depth estimation model: The depth estimation model generates depth information for each pixel in the entire image. This depth map D(x,y) represents the distance between the pixel at each position (x,y) in the image and the camera. This depth map has the same resolution and coordinate system as the input image of YOLO, so it can be matched directly by coordinates.

[0116] The matching steps are as follows:

[0117] (1) Traverse each person box: The person box set B detected by the target detection model = {B1, B2, ..., B n}, process each person's box B one by one i , each person box B i The boundary coordinates are (x i1 ,x i2 ,y i1 ,y i2 ), represents the position range of the target person in the image; where x i1 ,y i1 Human frame B i The horizontal and vertical coordinates of the upper left corner, x i2 ,y i2 Human frame B i The horizontal and vertical coordinates of the lower right corner;

[0118] (2) Find the corresponding coordinate area from the depth map of the image: For each person box B i , find the pixel range corresponding to the human frame from the depth map D(x,y) generated by the depth estimation model. The coordinates of this range are consistent with the boundary coordinates of the human frame, that is, x i1 ≤x≤x i2 and i1 ≤y≤y i2 , where x and y are the horizontal and vertical coordinates of the pixel respectively;

[0119] (3) Crop the corresponding depth map area: Crop the area corresponding to the human frame B in the depth map D(x,y). i The cropped area contains the depth values ​​of all pixels in the person frame, that is, the depth information sub-image of the target person.

[0120] The clipping formula is: for x i1 ≤x≤x i2 ,y i1 ≤y≤y i2 ;

[0121] in, Is with people frame B iThe matched depth map sub-region indicates the corresponding region of the person in the depth map;

[0122] (4) Collecting depth information within the human frame: Through cropping operations, a set of depth values ​​of all pixels corresponding to the human frame area of ​​the person is extracted from the depth map. This set represents the depth information of the person in three-dimensional space and is used for subsequent average depth calculations.

[0123] B. Average depth calculation within the human frame

[0124] After getting the person frame B i Corresponding depth map sub-region The system can then calculate the average depth value of the area to determine how far the target person is from the camera.

[0125] Average depth calculation formula:

[0126]

[0127] in:

[0128] D avg (B i ) is human frame B i The average depth value of .

[0129] N is the total number of pixels in the human frame, calculated as N = (x2-x1+1)*(y2-y1+1). D(x,y) is the depth value of each pixel in the human frame area, taken from the cropped depth map sub-area

[0130] C. Distance Filtering

[0131] In Get Everyone Box B i The average depth value D avg (B i ), the system uses the preset depth threshold D thresh Perform distance filtering. If the average depth of the target person exceeds the threshold, the target is considered to be too far away from the camera and the system filters it out.

[0132] Distance filtering formula:

[0133]

[0134] D avg (B i ) is the average depth value within the target person’s frame.

[0135] D thresh It is the depth threshold set by the system, indicating the maximum acceptable distance.

[0136] Indicates a valid human frame after depth filtering; if the person is too far away, the system will no longer process the target (i.e. ).

[0137] 3.2.4 Human key point detection

[0138] After completing depth estimation and distance filtering, the system only retains valid human frames that are close and may be the speaker. Next, these valid human frames will be input into the human key point detection model one by one to further extract the key point information of the speaker, especially the key points of the mouth area. This key point information will provide important basis for tasks such as sound enhancement and sound source localization.

[0139] The specific steps are as follows:

[0140] (1) The system traverses each filtered person frame It is cropped into separate image regions and fed into the DWPose keypoint detection model.

[0141] (2) The DWPose model will analyze the human body key points in the area and extract multiple key points including the mouth, eyes, nose, etc.

[0142] (3) The detected key point set K i Including the coordinates of the upper and lower lips and corners of the mouth, providing basic data for subsequent analysis of mouth shape changes.

[0143] Key point set formula:

[0144] K i = {P lips ,P eyes ,P nose ...};

[0145] Among them, P lips The key points of the mouth area include the position coordinates of the upper and lower lips and the corners of the mouth. The key points of the mouth area are an important basis for the system to analyze the speaker's mouth shape movement.

[0146] 3.2.5 Mouth Movement Analysis

[0147] After detecting the key points of the speaker's mouth, the system begins a deeper analysis. Through the changes in mouth shape, the system can monitor the speaker's vocal state in real time and generate dynamic parameters for sound enhancement and filtering. The following are several key analysis steps:

[0148] (1) Mouth opening degree detection: In order to determine the speaker's vocalization state, the system estimates the mouth opening degree by calculating the vertical distance between the key points of the upper and lower lips. The mouth opening degree is directly related to the intensity of the sound. When the mouth is opened wider, it usually indicates that the sound intensity is stronger, while a closed mouth indicates that the sound is weak or has stopped.

[0149] Assume that the set of upper lip key points is {(x u1 ,y u1 ),(x u2 ,y u2 ),...(x un ,y un )}, the set of lower lip key points is {(x l1 ,y l1 ),(x l2 ,y l2 ),...(x ln ,y ln )}, each pair of upper lip and lower lip key points is one-to-one corresponding.

[0150] To calculate the overall degree of mouth opening, the vertical distance between each pair of upper and lower lip key points can be calculated, and then the average of these distances can be found.

[0151]

[0152] in:

[0153] n is the number of key points of the upper and lower lips.

[0154] (x ui ,y ui ) is the coordinate of the i-th upper lip key point.

[0155] (x li ,y li ) is the coordinate of the i-th lower lip key point.

[0156] d open It is the average degree of opening and closing of the mouth.

[0157] (2) Lip movement speed detection: The system not only cares about the degree of mouth opening and closing, but also tracks the speed of lip movement. By calculating the rate of change of mouth opening and closing between consecutive time points, the system can infer the speaker's frequency and speed of speech. Fast lip movement usually means a higher frequency of speech, and vice versa.

[0158] Lip movement speed formula:

[0159]

[0160] Where v(t) is the velocity of the lips at time t, dopen (t) is the degree of mouth opening and closing at that point in time, and Δt is the time interval. The system infers the speaker's current vocal rhythm based on the speed of lip movement.

[0161] 3.3 Interaction with Microphone Array

[0162] Interaction with the microphone array is the core link in the sound source localization and directional sound enhancement system of the present invention. Through precise interaction, the directivity and gain setting of the microphone array can be controlled in real time to ensure accurate capture of the target sound source and effective suppression of background noise. The following describes in detail the various steps and technical details of interacting with the microphone array.

[0163] 3.3.1 Receiving the position information of the image processing module

[0164] Input of position information: First, the position information of the sound source target is received from the image processing module. This information includes:

[0165] Spatial position of the target: the position of the human target detected from the image.

[0166] Position information of mouth shape key points: Position information of mouth shape key points based on human body key point detection.

[0167] Mouth shape movement status: Based on the results of mouth shape detection and motion analysis, determine whether the target is speaking.

[0168] This information provides the precise location of the sound source and the sound status of the target, and is the key basis for the subsequent adjustment of the directionality of the microphone array.

[0169] 3.3.2 Multimodal Fusion of Image and Audio Data

[0170] After the system captures the image data, the mouth shape information is transmitted to the microphone array in real time to further optimize the sound filtering and enhancement process. The microphone array initially determines the approximate direction of the sound source through acoustic algorithms (such as phase difference method or time delay method), while the image data provides more accurate information on the speaker's position and mouth shape movement. Through the fusion of multimodal data, the system can improve positioning accuracy and dynamically adjust sound processing parameters.

[0171] (1) Mouth shape information generates dynamic filter parameters

[0172] The system uses dynamic parameters generated by the mouth key points to adjust the gain and frequency response of the sound filter, ensuring that the sound enhancement process can respond to the dynamic changes of the speaker in real time. Specifically:

[0173] When it is detected that the mouth is open or closed to a large extent, the system will increase the gain of the filter accordingly to enhance the strength of the sound signal; when the mouth is closed, the gain will be reduced to suppress environmental noise.

[0174] The speed of the lip movement is used to adjust the frequency response of the filter, with faster lip movements corresponding to higher frequency components and slower movements corresponding to lower frequency components. Through this adaptive adjustment, the system can dynamically optimize the effect of sound enhancement.

[0175] (2) Specific implementation of filter dynamic adjustment

[0176] Based on the above mouth key point information, the system constructs an adaptive dynamic filter function. The filter adjustment is divided into the following aspects:

[0177] Gain adjustment: adjust the filter gain according to the degree of mouth opening and closing. The gain formula is:

[0178] G(t)=G0+k·d open ;

[0179] Where G0 is the basic gain, k is the gain adjustment factor, and d open It is the degree of opening and closing of the mouth.

[0180] Frequency response adjustment: Adjust the frequency response of the filter according to the lip movement speed. The frequency adjustment formula is:

[0181] f(t)=f0+k f v(t);

[0182] Where f0 is the fundamental frequency, k f is the frequency adjustment coefficient, and v(t) is the lip movement speed.

[0183] 3.3.3 Dynamically adjust the directivity of the microphone array

[0184] The microphone array consists of 6 evenly distributed microphones. The system can calculate the corresponding angle based on the position of the lips relative to the center of the camera, and dynamically adjust the gain of each microphone based on this angle.

[0185] (1) Accurate azimuth calculation

[0186] Get the key point coordinates P of the speaker's lips through the camera lips =(x lips ,y lips ) and the image center point (x center ,y center ), we can calculate the angle of the lips relative to the center. We can use the following formula to calculate the azimuth angle θ of the lips:

[0187]

[0188] This azimuth angle θ represents the angle of the lips relative to the center point of the camera, ranging from -180° to 180°. With this angle information, the system can accurately identify the direction of the speaker and adjust the gain in the corresponding direction according to the layout of the microphone array.

[0189] (2) Adjust the gain of the 6-array microphone based on the azimuth angle

[0190] The 6-microphone array is usually distributed at 60 degrees. Assuming that each microphone covers a 60-degree area, the system can dynamically adjust the corresponding microphone gain according to the calculated azimuth angle θ.

[0191] A. Azimuth angle determines microphone gain distribution

[0192] If the azimuth angle θ of the lip keypoint falls within the area covered by a certain microphone (for example, θ is between -30° and 30°), the microphone in this direction will receive more gain; the gains of adjacent microphones can be interpolated based on the distance from the azimuth. For example, the microphone closer to the angle has a larger gain, while the microphone farther from the angle has a smaller gain.

[0193] For each microphone i, we set its center angle to θ, then the gain adjustment can be calculated by the following formula:

[0194] G i (t) = G0 + k·max(0, cos(θ-θ i ));

[0195] in:

[0196] G i (t) is the gain value of the i-th microphone at time t.

[0197] G0 is the base gain.

[0198] θ is the azimuth angle of the current speaker calculated from the lip position.

[0199] θ i is the center angle of the ith microphone (such as 0 degrees, 60 degrees, 120 degrees, etc.).

[0200] k is the gain adjustment factor.

[0201] By using the cosine function cos(θ-θ i ) to represent the relationship between the lip position and the center angle of each microphone, and the system can dynamically adjust the gain of each microphone.

[0202] When the lip azimuth angle θ is close to the center angle θ of a certain microphone i When the gain G i Other microphones further away will receive less gain.

[0203] B. Distributed gain adjustment

[0204] In addition, we also perform distributed gain adjustment. Gain distribution based on the lip azimuth can make a smooth transition between multiple microphones. This means that when the lip position is in the middle of the coverage area of ​​​​two microphones, the gains of the two adjacent microphones will be adjusted accordingly, thereby enhancing the sound signal in this direction. On the contrary, for microphones farther away from the lip position, the gain will be reduced or even suppressed.

[0205] Compared with the existing technology, the present invention has significant advantages in sound source localization and directional sound enhancement. First, the present invention combines the multimodal data of the microphone array and the camera, and realizes the precise positioning of the sound source target through target detection, depth detection and mouth movement analysis of the image processing module. This multimodal fusion effectively overcomes the problem of insufficient accuracy of the traditional single acoustic localization method in complex environments, especially in multi-person scenes and high-noise environments.

[0206] Secondly, the deep interaction between the system and the microphone array enables the present invention to not only accurately adjust the directivity of the microphone, but also suppress environmental noise through a dynamic adaptive algorithm, thereby significantly improving the clarity of the target sound source. This real-time dynamic adjustment capability enables the present invention to maintain stable performance in different application scenarios, meeting the actual needs for higher sound processing accuracy and noise suppression. In general, the present invention surpasses the existing technology in terms of the accuracy of sound source positioning, the effect of sound enhancement, and the ability to adapt to complex environments, and has broad application prospects.

[0207] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for sound source direction localization and directional sound enhancement based on a microphone array and a camera, characterized in that: The following steps are involved: S1, installing the camera at the center of the microphone array and preprocessing the image collected by the camera; Detect all people in the image through the object detection model and generate corresponding person frames; S2. Calculate the depth information of each pixel in the image using the depth estimation module, and match the depth information of the pixel with the detected human frame; By calculating the average depth of the pixels inside each person's frame, the distance between the speaker and the camera is determined. By setting a depth threshold, targets that are far away are filtered out, and valid frames of those that are close and may be the speaker are retained. S3, analyzing the key points of the human body on the valid human frame, and extracting multiple key points of the human body including the mouth, eyes, nose, etc.; S4, analyzing the mouth shape movement state of the key points of the mouth: detecting the degree of mouth opening and closing representing the intensity of the sound of the speaker, and detecting the lip movement speed representing the frequency and speed of the sound of the speaker; The gain of the sound filter increases as the speaker's mouth opens and closes more, and the strength of the sound signal increases accordingly; The frequency response of the sound filter increases as the speed of the speaker's mouth movement becomes faster; S5. Calculate the angle of the speaker's mouth relative to the center point of the camera, thereby identifying the speaker's position, and adjust the gain of the microphone in the corresponding direction according to the layout of the microphone array.

2. The method for sound source direction localization and directional sound enhancement based on microphone array and camera according to claim 1, characterized in that: The depth information of the pixel is matched with the detected human frame, and the specific steps are as follows: (1) Traverse each person box: The person box set B detected by the target detection model = {B1, B2, ..., B n }, process each person's box B one by one i , each person box B i The boundary coordinates are (x i1 ,x i2 ,y i1 ,y i2 ), represents the position range of the target person in the image; where x i1 ,y i1 Human frame B i The horizontal and vertical coordinates of the upper left corner, x i2 ,y i2 Human frame B i The horizontal and vertical coordinates of the lower right corner; (2) Find the corresponding coordinate area from the depth map of the image: For each person box B i , find the pixel range corresponding to the human frame from the depth map D(x,y) generated by the depth estimation model. The coordinates of this range are consistent with the boundary coordinates of the human frame, that is, x i1 ≤x≤x i2 and i1 ≤y≤y i2 , where x and y are the horizontal and vertical coordinates of the pixel respectively; (3) Crop the corresponding depth map area: Crop the area corresponding to the human frame B in the depth map D(x,y). i The cropped area contains the depth values ​​of all pixels in the person frame, that is, the depth information sub-image D of the target person. Bi (x,y); The cutting formula is: D Bi (x,y)=D(x,y)for x i1 ≤x≤x i2 ,y i1 ≤y≤y i2 ; Among them, D Bi (x,y) is the same as the human frame B i The matched depth map sub-region indicates the corresponding region of the person in the depth map; (4) Collecting depth information within the human frame: Through cropping operations, a set of depth values ​​of all pixels corresponding to the human frame area of ​​the person is extracted from the depth map. This set represents the depth information of the person in three-dimensional space and is used for subsequent average depth calculations.

3. The method for sound source direction localization and directional sound enhancement based on microphone array and camera according to claim 1, characterized in that: The average depth of the pixels inside the human frame is calculated as follows: Among them, D avg (B i ) is human frame B i where N is the total number of pixels in the human frame; D(x,y) is the depth value of each pixel in the human frame area, taken from the cropped depth map sub-area D Bi (x,y).

4. The method for sound source direction localization and directional sound enhancement based on microphone array and camera according to claim 1, characterized in that: The degree of opening and closing of the mouth of the speaker is calculated as follows: Assume that the set of upper lip key points is {(x u1 ,y u1 ),(x u2 ,y u2 ),...(x un ,y un )}, the set of lower lip key points is {(x l1 ,y l1 ),(x l2 ,y l2 ),...(x ln ,y ln )}, each pair of upper lip and lower lip key points is one-to-one corresponding; Calculate the vertical distance between each pair of upper and lower lip key points, and then find the average of these distances; Where n is the number of key points of the upper and lower lips; (x ui ,y ui ) is the coordinate of the i-th upper lip key point; (x li ,y li ) is the coordinate of the i-th lower lip key point; d open It is the average degree of opening and closing of the mouth.

5. The method for sound source direction localization and directional sound enhancement based on microphone array and camera according to claim 1, characterized in that: The lip movement speed of the speaker is calculated as follows: Where v(t) is the velocity of the lips at time t, d open (t), d open (t-1) are the degree of mouth opening and closing at time points t and t-1 respectively, and Δt is the time interval.

6. The method for sound source direction localization and directional sound enhancement based on microphone array and camera according to claim 1, characterized in that: The gain G(t) of the sound filter is adjusted according to the degree of opening and closing of the speaker's mouth: G(t)=G0+k·d open ; Where G0 is the basic gain, k is the gain adjustment factor, and d open is the degree of mouth opening and closing; The frequency response f(t) of the filter is adjusted according to the lip movement speed of the speaker: f(t)=f0+k f ·v(t); Where f0 is the fundamental frequency, k f is the frequency adjustment coefficient, and v(t) is the lip movement speed.

7. The method for sound source direction localization and directional sound enhancement based on microphone array and camera according to claim 1, characterized in that: The specific implementation steps of step S5 are as follows: (1) Accurate azimuth calculation: Obtain the coordinates of the speaker’s lip key points P through the camera lips =(x lips ,y lips ) and the image center point (x center ,y center ), calculate the angle of the lips relative to the center point of the camera, that is, the azimuth angle θ of the lips: (2) Adjust the gain of the microphone array based on the azimuth angle: For each microphone i, we set its center angle to θ, then the gain adjustment can be calculated by the following formula: G i (t)=G0+k·max(0,cos(θ-θ) i )); Among them, G i (t) is the gain value of the i-th microphone at time t, G0 is the basic gain, θ i is the center angle of the i-th microphone, k is the gain adjustment coefficient; By using the cosine function cos(θ-θ i ) to represent the relationship between the lip position and the center angle of each microphone, and the system can dynamically adjust the gain of each microphone; When the lip azimuth angle θ is close to the center angle θ of a certain microphone i When the gain G i will be maximized; other microphones further away will receive less gain.

8. The method for sound source direction localization and directional sound enhancement based on microphone array and camera according to claim 1, characterized in that: The microphone array also performs distributed gain adjustment: the gain distribution based on the lip azimuth angle makes a smooth transition between multiple microphones; when the lip position is in the middle of the coverage area of ​​two microphones, the gains of the two adjacent microphones will be adjusted accordingly to enhance the sound signal in this direction; for microphones that are far away from the lip position, the gain will be reduced or even suppressed.

9. A server, comprising a processor and a memory, characterized in that: At least one program is stored in the memory, and the program is loaded and executed by the processor to implement the method for sound source direction positioning and directional sound enhancement based on a microphone array and a camera as described in any one of claims 1 to 8.

10. A computer-readable storage medium, wherein at least one program is stored in the storage medium, characterized in that: The program is loaded and executed by a processor to implement the method for sound source direction positioning and directional sound enhancement based on a microphone array and a camera as described in any one of claims 1 to 8.

Images