Voiceprint face dynamic binding method and system based on sight distance and listening distance fusion

Abstract

The invention discloses a voiceprint face dynamic binding method and system based on sight distance and listening distance fusion, and relates to the technical field of computer vision and voice signal processing. The method comprises the following steps: synchronously acquiring a video stream and an audio stream, detecting a plurality of faces from the video stream and acquiring visual distances of the faces, and determining a candidate face region for pronunciation and a lip movement starting timestamp through lip movement detection; performing sound source separation on the audio stream to obtain candidate voiceprint fragments and voiceprint starting timestamps thereof; the candidate face areas and the candidate voiceprint segments are combined pairwise to construct candidate pairs, and for each candidate pair, the auditory distance is calculated according to the lip movement and voiceprint starting time difference in combination with the sound velocity; and determining a binding pair from all candidate pairs by comparing a difference value between the visual distance and the auditory distance in each candidate pair, thereby realizing dynamic association between the voiceprint and the human face. According to the method and the device, the voiceprint of the speaker and the corresponding face can be dynamically bound accurately in real time under the condition that a binding model does not need to be trained in advance.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and speech signal processing technology, and in particular to a method and system for dynamic binding of voiceprints to faces based on the fusion of visual distance and auditory distance. Background Technology

[0002] In practical applications such as smart education, video conferencing, and multi-person collaborative work, multiple people often speak alternately or even simultaneously. While independent facial recognition and voiceprint recognition technologies are relatively mature in these applications, the key bottleneck hindering the development of accurate speech recording, personalized interaction, and advanced human-computer interaction applications remains how to automatically, in real-time, and accurately associate specific voiceprints with corresponding faces—that is, how to solve the "who is speaking" problem—in the absence of prior knowledge.

[0003] In existing technologies, most multimodal fusion methods focus on using the intrinsic modal features of faces and voiceprints for identity recognition or verification. While they perform well under conditions of high data quality and good modal alignment, in complex scenarios with multiple concurrent users, multiple voice sources, and multiple faces intermingled in time and space, direct feature matching often leads to binding errors or failures due to modal misalignment and feature cross-interference. There is a lack of an effective mechanism for fast, coarse-grained association at the signal level. Furthermore, existing deep learning-based cross-modal association research typically focuses on mapping two modalities to a shared embedding space to learn semantic associations. These methods not only require training with large amounts of pairwise data but also face high computational complexity and poor real-time performance when dealing with real-time, streaming, and many-to-many dynamic binding scenarios. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a method and system for dynamic voiceprint-face binding based on the fusion of visual and auditory distances. This method can dynamically bind a speaker's voiceprint to their corresponding face in real time and accurately without the need for pre-training a binding model.

[0005] To achieve the above objectives, the present invention provides a method for dynamic voiceprint-face binding based on the fusion of visual and auditory distances, comprising: Acquire video streams captured by camera devices and audio streams captured by microphone arrays; Face detection is performed on the video stream to determine multiple face regions and their corresponding visual distances; Lip movement detection is performed on multiple face regions to determine the candidate face regions where lip movements occur and the corresponding lip movement start timestamps; The audio stream is subjected to sound source separation to determine at least one candidate voiceprint segment and its corresponding voiceprint start timestamp; The candidate face region and the candidate voiceprint segment are combined in pairs to construct at least one face-voiceprint candidate pair; For each candidate pair, the auditory distance is calculated based on the time difference between the corresponding lip movement start time stamp and the voiceprint start time stamp, combined with the speed of sound. Based on the distance difference between the visual distance and the corresponding auditory distance of the candidate face region in each candidate pair, a binding pair is determined from all candidate pairs to complete the voiceprint face binding.

[0006] Optionally, lip movement detection is performed on multiple facial regions to determine the candidate facial regions where lip movements occur and the corresponding lip movement start timestamps, including: For each facial region, extract the corresponding mouth region image sequence; Analyze the image sequence of the mouth region to detect the start time of the mouth opening action; The detected face region at the start time is identified as the candidate face region, and the timestamp of the video frame corresponding to the start time is identified as the lip movement start timestamp of the corresponding candidate face region.

[0007] Optionally, the image sequence of the mouth region is analyzed to detect the start time of the mouth opening action, including: Optical flow calculations are performed on the image sequence of the mouth region to obtain the changes in optical flow features between consecutive frames; When the change in optical flow characteristics exceeds a preset threshold, the corresponding video frame time is determined as the start time of the mouth opening action.

[0008] Optionally, the audio stream is subjected to source separation to determine at least one candidate voiceprint segment and its corresponding voiceprint start timestamp, including: The audio stream is subjected to source separation to obtain at least one separated audio signal; Speech activity detection is performed on each of the audio signals to locate the starting point of the speech signal; The speech signal segment starting from the aforementioned starting point is identified as a candidate voiceprint segment, and the timestamp corresponding to the starting point is identified as the voiceprint start timestamp of the corresponding candidate voiceprint segment.

[0009] Optionally, bind pairs are determined from all candidate pairs, including: The distance difference is used as the matching cost. Based on the matching cost, the Hungarian algorithm is used to perform optimal allocation between the candidate face region and the candidate voiceprint segment; Based on the optimal allocation, candidate pairs consisting of corresponding candidate face regions and candidate voiceprint segments are determined as binding pairs.

[0010] Optionally, the calculation of auditory distance by combining sound speed includes: Obtain current ambient temperature and / or humidity information; The current speed of sound is determined based on the temperature and / or humidity information. The perceived hearing distance is calculated based on the current speed of sound.

[0011] Optionally, the method further includes: When no binding pair is successfully determined from all candidate pairs, an indication signal of binding failure is output, and the identification information of the corresponding candidate face region is associated with the identification information of the candidate voiceprint segment and recorded.

[0012] This invention also provides a voiceprint-based dynamic face binding system based on the fusion of visual and auditory distances, comprising: The data acquisition unit is used to acquire video streams captured by camera equipment and audio streams captured by microphone array; The vision processing unit is used for: Face detection is performed on the video stream to determine multiple face regions and their corresponding visual distances; Lip movement detection is performed on multiple face regions to determine the candidate face regions where lip movements occur and the corresponding lip movement start timestamps; An audio processing unit is used to perform sound source separation on the audio stream and determine at least one candidate voiceprint segment and the corresponding voiceprint start timestamp. The candidate pair generation unit is used to combine the candidate face region with the candidate voiceprint segment in pairs to construct at least one face-voiceprint candidate pair; The auditory distance calculation unit is used to calculate the auditory distance for each candidate pair based on the time difference between the corresponding lip movement start time stamp and the voiceprint start time stamp, combined with the speed of sound. The binding decision unit is used to determine the binding pair from all candidate pairs based on the distance difference between the visual distance and the corresponding auditory distance of the candidate face region in each candidate pair, so as to complete the voiceprint face binding.

[0013] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: The present invention provides a dynamic voiceprint-face binding method based on the fusion of visual and auditory distances. By acquiring and simultaneously processing video and audio streams, it first detects multiple faces in the video and directly obtains their physical visual distance, while simultaneously separating multiple voiceprint segments from the audio. Then, it constructs candidate pairs for each face and each voiceprint segment, and calculates the auditory distance for each candidate pair based on the time difference between lip movement and voiceprint initiation and the speed of sound. Finally, it determines and completes the binding by directly comparing the consistency of visual and auditory distances in each candidate pair. This method creatively transforms the multimodal binding problem of "who is speaking" into a verification problem of whether the two physical quantities, visual and auditory distance, are consistent. It utilizes the physical laws and geometric spatial relationships of sound wave propagation, rather than relying on deep content features of faces and voiceprints for matching.

[0014] Since the core criterion of this invention is based on the consistency of physical spatiotemporal relationships (distance) rather than the similarity of intrinsic modal features, it effectively overcomes the binding errors or failures caused by modal misalignment and feature cross-interference in scenarios with multiple concurrent users, multiple sound sources, and spatiotemporally mixed faces. This method establishes a fast, coarse-grained association mechanism at the signal level, providing an effective way to solve initial binding in complex dynamic scenarios. Furthermore, this invention does not rely on a large amount of already bound pairwise data for model training and avoids complex computational processes such as mapping two modalities to a shared embedding space. Its computational process mainly involves basic operations and logical judgments, without relying on complex deep neural network inference, thus significantly reducing computational complexity and improving processing efficiency. This better meets the urgent needs for real-time performance, streaming processing, and lightweight deployment in practical application scenarios such as smart education and video conferencing. Attached Figure Description

[0015] The above and other objects, features and advantages of the present invention will become more apparent from the more detailed description of exemplary embodiments of the invention in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same parts.

[0016] Figure 1 This is a schematic diagram of the method flow of the voiceprint-face dynamic binding method based on the fusion of visual distance and auditory distance, as shown in an embodiment of the present invention. Figure 2 This is a schematic diagram of distance matching shown in an embodiment of the present invention; Figure 3 This is a schematic diagram of the module structure of a voiceprint-based dynamic face binding system based on the fusion of visual and auditory distances, as shown in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Please see Figure 1 and Figure 2 , Figure 1 This is a flowchart illustrating the method of dynamic face binding based on the fusion of visual and auditory distances.

[0019] A method for dynamic voiceprint-face binding based on the fusion of visual and auditory distances includes: S101: Acquire the video stream captured by the camera device and the audio stream captured by the microphone array.

[0020] To achieve accurate dynamic binding of voiceprints and faces, it is first necessary to simultaneously acquire high-quality visual and auditory raw data. Specifically, a dedicated camera device can be used to capture video streams of the target scene. This camera device is preferably an RGB-D camera with a built-in infrared or laser ranging module, enabling it to directly acquire depth information of the target in the image while capturing color video information, laying the hardware foundation for subsequently directly acquiring the physical distance of the face. Simultaneously, an audio stream of the scene is acquired through a microphone array. The microphone array has multi-channel acquisition capabilities, enabling the acquisition of spatial information of sound, providing the possibility for subsequent sound source separation and high-precision analysis. To ensure strict alignment of visual and auditory information in the temporal dimension, so that the start time points of lip movements and speech signals can be accurately compared, hardware-level time synchronization is required between the camera device and the microphone array. For example, a Precise Time Protocol (PTP) synchronization mechanism is needed to ensure that the timestamps of each frame of the video stream and the corresponding samples of the audio stream have a unified reference coordinate system.

[0021] S102: Perform face detection on the video stream to determine multiple face regions and their corresponding visual distances.

[0022] After acquiring the video stream, the video sequence needs to be processed in real time to locate people in the scene and obtain their spatial location information. Specifically, a lightweight face detection model can be applied to analyze each frame of the video stream. This model can quickly and accurately locate all faces appearing in the frame and return their corresponding pixel coordinate regions, i.e., the face regions.

[0023] After successfully detecting multiple face regions, the next step is to obtain the precise spatial distance, or visual distance, of each face relative to the camera device. This process can be achieved through a ranging module (such as an infrared or laser ranging module) built into the aforementioned camera device. This ranging module can directly measure the physical distance from the device to a specified pixel area in the image (e.g., the center point of the detected face region) based on principles such as time-of-flight (ToF) or triangulation. Therefore, without performing complex stereo vision calculations, the visual distance value A of each face region can be obtained simultaneously. This distance information is crucial for mapping faces from two-dimensional images to three-dimensional physical space, providing a geometric reference for subsequent consistency verification with auditory distance.

[0024] S103: Perform lip movement detection on multiple face regions to determine the candidate face regions where lip movements occur and the corresponding lip movement start timestamp.

[0025] To identify the actual speaker from multiple detected faces and accurately capture the moment they begin speaking, continuous lip movement analysis is required for each facial region. This process focuses on the mouth region of each detected face, analyzing image sequences of this region to determine if valid, articulation-related lip movements have occurred. Continuous monitoring of the mouth region image sequences allows for the identification of specific movement patterns that signal the start of speech, such as a significant change in mouth position from closed to open. Once such an initiation is detected, the facial region is identified as a candidate for speech production. Crucially, the precise moment this initiation occurs—the lip movement initiation timestamp—must be recorded. This timestamp is directly derived from the video stream's timeline, corresponding to the time information of the video frame where the initiation was detected.

[0026] Specifically, S103 above includes: For each facial region, extract the corresponding mouth region image sequence; Analyze the image sequence of the mouth region to detect the start time of the mouth opening action; The detected face region at the start time is identified as the candidate face region, and the timestamp of the video frame corresponding to the start time is identified as the lip movement start timestamp of the corresponding candidate face region.

[0027] For example, analyzing a sequence of images of the mouth region to detect the start time of the mouth opening action includes: Optical flow calculations were performed on the image sequence of the mouth region to obtain the changes in optical flow features between consecutive frames; When the change in optical flow characteristics exceeds a preset threshold, the corresponding video frame time is determined as the start time of the mouth opening action.

[0028] In this application, to achieve lip movement detection, a series of meticulous processes need to be performed on each located facial region. First, based on keypoint or region localization techniques, the sub-region containing the mouth is further precisely located and extracted from the facial region. Then, in consecutive video frames, images of this mouth region are continuously tracked and captured, forming a temporally continuous sequence of mouth region images. This sequence dynamically records the changes in lip opening and closing. Next, this mouth region image sequence is analyzed in depth, with the goal of detecting the visual event representing the start of speech, i.e., the initial moment when the lips move from closing to opening. Specifically, this can be achieved by analyzing pixel changes between consecutive frames, such as performing optical flow calculations on the image sequence and capturing the starting point of the movement by analyzing the changes in optical flow features of the mouth region; alternatively, a lightweight lip movement classification network model can be used to perform pattern recognition on the sequence to determine whether there are lip opening movements and their starting points that match speech characteristics.

[0029] Once a valid starting point is successfully detected in a mouth region image sequence through the above analysis, two key outputs are generated: First, the face region corresponding to the starting point is marked as a visual target in the process of speaking, i.e., a candidate face region, thus completing the screening from a static face to an active speaker; Second, the precise timestamp of the specific video frame corresponding to the starting point (obtained from the synchronized timeline) is recorded as the lip movement start timestamp of the candidate face region, denoted as T_face.

[0030] S104: Perform sound source separation on the audio stream to determine at least one candidate voiceprint segment and its corresponding voiceprint start timestamp.

[0031] Specifically, S104 above includes: Perform sound source separation on the audio stream to obtain at least one separated audio signal; Speech activity detection is performed on each audio signal to locate the starting point of the speech signal; The speech signal segment starting from the starting point is identified as a candidate voiceprint segment, and the timestamp corresponding to the starting point is identified as the voiceprint start timestamp of the corresponding candidate voiceprint segment.

[0032] Simultaneously with video stream processing, the synchronously acquired audio stream also needs to be processed to separate and identify individual speaker units from environments that may contain mixed speech from multiple speakers. The audio stream is acquired by the aforementioned high-precision multi-microphone array, whose multi-channel characteristics provide a spatial information basis for subsequent processing. First, based on the multi-channel mixed audio signal acquired by the microphone array, source separation technology is used for processing. This technology can utilize information such as the time difference and phase difference of sound arriving at different microphones to initially separate speech signals from different spatial directions, obtaining relatively independent audio signal segments corresponding to different speakers or different sound sources.

[0033] After obtaining the separated audio signal segments, it is necessary to further extract the effective articulation units and their timing information that can be used for binding. For each separated audio signal segment, speech activity detection technology is used for analysis. Specifically, by analyzing the short-time energy, zero-crossing rate, or more complex frequency domain characteristics of the signal, the start, duration, and end points of the speech signal in the audio stream are accurately determined. The core objective is to locate the starting point in the audio signal where the speech energy significantly exceeds the background noise threshold; this starting point marks the beginning of a valid articulation.

[0034] Based on the detected speech start point, a valid candidate voiceprint segment can be defined. Specifically, a continuous speech signal starting from this start point (e.g., from the start point to the end of the speech activity or to a reasonable time window) is identified as an independent unit representing that pronunciation, i.e., a candidate voiceprint segment. Simultaneously, the precise audio timestamp corresponding to this speech start point in a unified time coordinate system is extracted and recorded as the voiceprint start timestamp of the candidate voiceprint segment, denoted as T_audio. This timestamp is comparable to the lip movement start timestamp T_face from the video source. Under a unified time reference, the difference between the two directly reflects the flight time required for sound to travel from the sound source (mouth) to the receiver (microphone), and is a key input for calculating auditory distance.

[0035] S105: Combine candidate face regions with candidate voiceprint segments in pairs to construct at least one face-voiceprint candidate pair.

[0036] After extracting independent information from both visual and auditory perspectives, multiple candidate face regions and their lip movement start timestamps, as well as multiple candidate voiceprint segments and their voiceprint start timestamps, are obtained. The next step is to establish and validate the association hypothesis between these two groups. Especially in multi-person scenarios, it's impossible to know in advance which face corresponds to which voiceprint segment; therefore, it's necessary to construct all possible association combinations as candidate solutions to be evaluated. Specifically, each detected candidate face region is paired with each separated candidate voiceprint segment to form a combination consisting of a single face and a single voiceprint segment—a face-voiceprint candidate pair.

[0037] S106: For each candidate pair, calculate the auditory distance based on the time difference between the corresponding lip movement start time stamp and the voiceprint start time stamp, combined with the sound speed.

[0038] For each constructed face-voiceprint candidate pair, the core processing involves using time synchronization information to calculate the possible source distance of the voiceprint fragment in physical space. Specifically, for a candidate pair, its corresponding visual event time (lip movement start timestamp T_face) and auditory event time (voiceprint start timestamp T_audio) have been obtained. Since the video and audio acquisition devices have performed hardware-level time synchronization, these two timestamps are in a unified time coordinate system, so their absolute difference can be directly calculated, i.e., time difference ΔT = |T_face - T_audio|. This time difference represents the time required for sound to be generated from the mouth of the face (assuming the sound source location) and propagate to the microphone array, i.e., the time of flight of the sound wave.

[0039] After obtaining the time difference ΔT, the estimated distance to the sound source can be calculated based on the physical laws of sound propagation in air. According to the speed of sound formula, the audible distance B from the sound source to the microphone can be calculated using the formula B = vΔT + c. Here, v is the speed of sound in the current medium (usually air), which is approximately 343 m / s under standard room temperature conditions; c is an optional system calibration constant used to compensate for the fixed delays introduced by image and audio processing, ensuring that the calculated distance is closer to the true geometric distance. Through this calculation, each candidate voiceprint segment is assigned an estimated physical spatial distance value B based on the sound propagation time. This audible distance B and the visual distance A corresponding to the candidate's face region, measured directly by the camera, essentially refer to the same spatial length (i.e., the distance from the face to the acquisition device). Therefore, as... Figure 2 As shown, for a correct pair of bindings (i.e., the voiceprint does indeed originate from the face), the auditory distance B should be highly similar to the visual distance A.

[0040] S107: Based on the distance difference between the visual distance and the corresponding auditory distance of the candidate face region in each candidate pair, determine the binding pair from all candidate pairs to complete the voiceprint face binding.

[0041] For each candidate pair obtained from the aforementioned calculations, its corresponding visual distance A (from the candidate's face region) and auditory distance B (calculated based on the time difference of the pair) are extracted, and the absolute difference between the two is calculated, i.e., the distance difference D_diff = |A - B|. This difference quantifies the degree of deviation of the association hypothesis represented by the candidate pair from physical spatial consistency; the smaller the D_diff value, the more closely the auditory inferred distance matches the visually measured distance, and the higher the probability that the candidate pair represents a correct binding.

[0042] In one embodiment, determining the binding pair from all candidate pairs includes: The distance difference is used as the matching cost. Based on the matching value, the Hungarian algorithm is used to optimally allocate candidate face regions and candidate voiceprint segments; Based on the optimal allocation, candidate pairs consisting of corresponding candidate face regions and candidate voiceprint segments are determined as binding pairs.

[0043] In complex scenarios where multiple people speak simultaneously or alternately, simple threshold judgments may lead to binding ambiguity because multiple candidate pairs simultaneously meet the conditions. For example, a voiceprint may have a distance difference less than the threshold with multiple faces, or a face may have a similar distance difference with multiple voiceprints. To solve this type of many-to-many global assignment problem, the binding decision can be modeled and solved as an optimal assignment problem. This approach uses the distance difference D_diff calculated for each candidate pair as the basis, regarding it as the matching cost of the pairing hypothesis; the smaller the distance difference, the greater the probability that the voiceprint originates from that face, and the lower the matching cost, and vice versa.

[0044] Based on this, the matching costs between all candidate face regions and all candidate voiceprint segments constitute a cost matrix. Subsequently, the classic Hungarian algorithm is used to solve this matrix. The Hungarian algorithm can find an optimal matching scheme for a bipartite graph (one side being candidate face regions, and the other side being candidate voiceprint segments) in polynomial time, minimizing the total cost of the entire matching scheme. This means that the algorithm takes a global perspective, automatically deciding which voiceprint is assigned to which face, thus systematically avoiding conflicts and ensuring that each pairing is the globally optimal solution.

[0045] After the algorithm completes its operation, its optimal allocation result clearly indicates which candidate voiceprint segment should be paired with each candidate face region, and which face region each candidate voiceprint segment should belong to. Based on this defined correspondence, specific candidate pairs whose pairing relationship matches the algorithm's allocation result can be selected from all the initially constructed candidate pairs. These selected candidate pairs are the final determined binding pairs.

[0046] In one embodiment, determining the binding pair from all candidate pairs includes: Candidate pairs with a difference less than or equal to a preset threshold are selected from all candidate pairs to form a set of pairs to be bound. For each pair to be bound in the set of pairs to be bound, determine whether the pair to be bound is associated with the same candidate face region or the same candidate voiceprint segment as other pairs to be bound in the set of pairs to be bound; For pairs of objects to be bound that are determined to have no association, they are directly identified as bound pairs. For pairs of objects to be bound that are determined to be related, all related pairs of objects to be bound will be grouped into the same conflict group. For each conflict group, perform the following operations: From the conflict group, select one or more pairs of candidates with the smallest difference as the optimal candidate pairs; if there is only one optimal candidate pair, then determine the optimal candidate pair as the binding pair; if there are multiple optimal candidate pairs, then for each of the multiple optimal candidate pairs, perform multiple continuous matching verification operations, which include: continuously detecting the lip movement start timestamp of the candidate face region associated with the optimal candidate pair and the voiceprint start timestamp of the candidate voiceprint segment associated with the optimal candidate pair within a preset continuous time period; calculating and comparing multiple sets of auditory distance and visual distance based on the continuously detected multiple sets of timestamps to obtain multiple continuous differences; counting the number of matches that are less than or equal to a preset threshold among the continuous differences; and determining the optimal candidate pair that first reaches the preset number of matches as the binding pair.

[0047] In the application, a reasonable preset threshold ε needs to be set for filtering. This threshold is a preset fault tolerance parameter used to tolerate minor distance inconsistencies caused by factors such as visual ranging errors, auditory time difference calculation errors, and fluctuations in ambient sound speed. For each candidate pair, it is determined whether its distance difference D_diff is less than or equal to the threshold ε. If this condition is met, the visual distance and auditory distance in the candidate pair are considered to be physically consistent, and the binding condition is initially met.

[0048] Furthermore, to handle potential conflicts during the binding process more precisely and improve robustness, a decision-making process incorporating conflict detection and dynamic verification can be adopted. First, based on a preset threshold, candidate pairs are initially selected from all calculated candidate pairs whose difference between visual and auditory distances is less than or equal to the threshold. These pairs are considered to preliminarily meet the physical consistency condition and together constitute a set of pairs to be bound. Subsequently, it is necessary to check whether there is competition for binding resources within this set. For each pair to be bound in the set, it is determined whether its associated candidate face region or candidate voiceprint segment also appears in other pairs to be bound within the set. Through this determination, the pairs to be bound can be divided into two categories.

[0049] For pairs of faces and voiceprints that have been determined to be unique in the set, there is no binding conflict, so they can be directly identified as the final binding pairs.

[0050] For pairs of unbound devices that are found to be conflicting—that is, pairs of the same face with multiple voiceprint segments, or the same voiceprint segment with multiple faces—that simultaneously meet the initial threshold conditions, further arbitration is required. All interrelated pairs of unbound devices (i.e., sharing the same face or the same voiceprint) are grouped together to form a conflict group.

[0051] For each such conflict group, a conflict resolution operation is performed. Specifically, first, within the conflict group, the differences between all pairs to be bound are compared, and one or more pairs with the smallest differences are selected as the optimal candidate pairs for that conflict group. If there is only one optimal candidate pair, it means that this pair is significantly better than other choices in the group in terms of distance consistency, so it can be determined as the binding pair.

[0052] If there are multiple optimal candidate pairs, meaning multiple pairs are tied for the best distance difference, a single distance comparison cannot make a decision. To resolve this subtle ambiguity, a time-based verification mechanism is introduced. For each tied optimal candidate pair, multiple consecutive matching verification operations are initiated. Specifically, within a preset continuous time period, the lip movements of the specific candidate face region associated with the candidate pair, as well as the subsequent speech activities of the specific candidate voiceprint segment associated with it, are continuously tracked and monitored, thereby obtaining multiple sets of consecutive lip movement start timestamps and voiceprint start timestamps. Based on each set of timestamps, the auditory distance is recalculated and compared with the relatively stable visual distance to obtain a series of consecutive distance differences. Subsequently, the number of times these consecutive differences are still less than or equal to a preset threshold is counted. The core idea of ​​this verification process is that the distance consistency of a truly matched face-voiceprint pair should remain stable in consecutive speech events. Ultimately, during this continuous verification period, the optimal candidate pair whose distance difference meets the threshold requirement and whose number of matches first reaches the preset number requirement will be determined as the final binding pair for the conflict group.

[0053] In one embodiment, calculating the audible distance by incorporating the speed of sound includes: Obtain current ambient temperature and / or humidity information; Determine the current speed of sound based on temperature and / or humidity information; Calculate the hearing distance based on the current speed of sound.

[0054] When calculating audible distance, the speed of sound, *v*, is a key physical parameter. Its value is not fixed but varies with the state of the propagation medium (air), primarily influenced by ambient temperature and humidity. To improve the accuracy of distance calculations, a dynamic adjustment mechanism for the speed of sound can be introduced. Specifically, temperature and humidity sensors deployed near or integrated into the acquisition device can be used to acquire real-time information on the current environment's temperature and / or humidity. These environmental parameters reflect the actual physical conditions under which sound propagates through the air.

[0055] Subsequently, based on the physical laws of sound propagation in air, the current speed of sound is determined using the acquired temperature and humidity parameters. The speed of sound has a clear functional relationship with temperature (for example, in dry air, the speed of sound increases with increasing temperature); humidity also has a corrective effect on the speed of sound. The environmental parameters can be transformed into a more accurate estimate of the speed of sound, v_current, under the current environment by consulting a preset table of physical parameters or by substituting empirical formulas.

[0056] When calculating the auditory distance B, a fixed standard sound speed value (such as 343 m / s) is no longer used. Instead, the dynamically determined current sound speed v_current is used. Based on the formula B = v_currentΔT + c, a more accurate auditory distance is calculated. This dynamic adjustment mechanism directly addresses the potential source of error—sound speed fluctuations caused by environmental changes. By matching the sound speed parameter to environmental conditions, the distance calculation deviation caused by inaccurate sound speed is effectively reduced, thereby improving the consistency between visual and auditory distance comparisons.

[0057] In one embodiment, the method further includes: When no binding pair is successfully determined from all candidate pairs, an indication signal of binding failure is output, and the identification information of the corresponding candidate face region is associated with the identification information of the candidate voiceprint segment and recorded.

[0058] In practical applications, due to factors such as environmental noise, people obstructing the view, or sound wave aliasing, there may be situations where a reliable binding pair cannot be found through the aforementioned distance consistency verification. When the binding decision process ends, if the binding pair set is empty, a clear binding failure indication signal is generated and output. This signal can be a specific status code, a message, or a triggering event, used to report to the upper-layer application or user interface that no voiceprint-face association could be successfully established within the current processing cycle. This avoids silent system failures or outputting incorrect bindings, ensuring the clarity and reliability of the interaction.

[0059] Furthermore, to facilitate subsequent analysis, debugging, or potential offline processing, an information recording step can be included. Specifically, within the current processing frame or time period, relevant information about candidate face regions and candidate voiceprint segments that participated in candidate pair construction but ultimately failed to bind is extracted, associated, and stored. The recorded identification information may include, but is not limited to: the position coordinates of the candidate face region in the image, visual distance values, and facial feature summaries; the start and end times of the candidate voiceprint segments, voiceprint feature vectors, or segment indexes, etc. Crucially, associating and recording the identification information of these visual and auditory targets means that even if the real-time binding fails, a record is retained of a specific face and a specific voice segment coexisting in the same spatiotemporal context. These associated records provide a data foundation for subsequent possible model optimization or offline verification in specific scenarios, facilitating subsequent failure case analysis.

[0060] Corresponding to the aforementioned application function implementation method embodiments, the present invention also provides a voiceprint-based dynamic face binding system based on the fusion of visual distance and auditory distance, and corresponding embodiments.

[0061] Please see Figure 3 , Figure 3This is a schematic diagram of the module structure of a voiceprint-based dynamic face binding system that integrates visual and auditory distances.

[0062] A voiceprint-based dynamic face binding system based on the fusion of visual and auditory distances includes: The data acquisition unit 31 is used to acquire the video stream captured by the camera device and the audio stream captured by the microphone array; The visual processing unit 32 is used for: Perform face detection on the video stream to identify multiple face regions and their corresponding visual distances; Lip movement detection is performed on multiple face regions to identify candidate face regions where lip movements occur and the corresponding lip movement start timestamps; Audio processing unit 33 is used to perform sound source separation on the audio stream and determine at least one candidate voiceprint segment and the corresponding voiceprint start timestamp. The candidate pair generation unit 34 is used to combine candidate face regions and candidate voiceprint segments in pairs to construct at least one face-voiceprint candidate pair; The auditory distance calculation unit 35 is used to calculate the auditory distance for each candidate pair based on the time difference between the corresponding lip movement start time stamp and the voiceprint start time stamp, combined with the speed of sound. Binding decision unit 36 ​​is used to determine binding pairs from all candidate pairs based on the distance difference between the visual distance and the corresponding auditory distance of the candidate face region in each candidate pair, so as to complete the voiceprint face binding.

[0063] In one embodiment, when performing lip movement detection on multiple facial regions to determine the candidate facial region where lip movement occurred and the corresponding lip movement start timestamp, the visual processing unit 32 is specifically used for: For each facial region, extract the corresponding mouth region image sequence; Analyze the image sequence of the mouth region to detect the start time of the mouth opening action; The detected face region at the start time is identified as the candidate face region, and the timestamp of the video frame corresponding to the start time is identified as the lip movement start timestamp of the corresponding candidate face region.

[0064] In analyzing the image sequence of the mouth region and detecting the start time of the mouth opening action, the visual processing unit 32 is specifically used for: Optical flow calculations are performed on the image sequence of the mouth region to obtain the changes in optical flow features between consecutive frames; When the change in optical flow characteristics exceeds a preset threshold, the corresponding video frame time is determined as the start time of the mouth opening action.

[0065] In one embodiment, in separating the audio source of the audio stream and determining at least one candidate voiceprint segment and its corresponding voiceprint start timestamp, the audio processing unit 33 is specifically configured to: The audio stream is subjected to source separation to obtain at least one separated audio signal; Speech activity detection is performed on each of the audio signals to locate the starting point of the speech signal; The speech signal segment starting from the aforementioned starting point is identified as a candidate voiceprint segment, and the timestamp corresponding to the starting point is identified as the voiceprint start timestamp of the corresponding candidate voiceprint segment.

[0066] In one embodiment, when determining a binding pair from all candidate pairs, the binding decision unit 36 ​​is specifically used for: The distance difference is used as the matching cost. Based on the matching cost, the Hungarian algorithm is used to perform optimal allocation between the candidate face region and the candidate voiceprint segment; Based on the optimal allocation, candidate pairs consisting of corresponding candidate face regions and candidate voiceprint segments are determined as binding pairs.

[0067] In one embodiment, when calculating the audible distance by combining the speed of sound, the audible distance calculation unit 35 is specifically used for: Obtain current ambient temperature and / or humidity information; The current speed of sound is determined based on the temperature and / or humidity information. The perceived hearing distance is calculated based on the current speed of sound.

[0068] In one embodiment, the system further includes: The failure handling and recording unit is used to output a binding failure indication signal when a binding pair is not successfully determined from all candidate pairs, and to associate and record the identification information of the corresponding candidate face region with the identification information of the candidate voiceprint segment.

[0069] Regarding the system in the above embodiments, the specific manner in which each unit module performs operations has been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0070] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for dynamic voiceprint-face binding based on the fusion of visual and auditory distances, characterized in that, include: Acquire video streams captured by camera devices and audio streams captured by microphone arrays; Face detection is performed on the video stream to determine multiple face regions and their corresponding visual distances; Lip movement detection is performed on multiple face regions to determine the candidate face regions where lip movements occur and the corresponding lip movement start timestamps; The audio stream is subjected to sound source separation to determine at least one candidate voiceprint segment and its corresponding voiceprint start timestamp; The candidate face region and the candidate voiceprint segment are combined in pairs to construct at least one face-voiceprint candidate pair; For each candidate pair, the auditory distance is calculated based on the time difference between the corresponding lip movement start time stamp and the voiceprint start time stamp, combined with the speed of sound. Based on the distance difference between the visual distance and the corresponding auditory distance of the candidate face region in each candidate pair, a binding pair is determined from all candidate pairs to complete the voiceprint face binding.

2. The method for dynamic voiceprint-face binding based on the fusion of visual and auditory distances according to claim 1, characterized in that, Lip movement detection is performed on multiple facial regions to determine candidate facial regions where lip movements occur and their corresponding lip movement start timestamps, including: For each facial region, extract the corresponding mouth region image sequence; Analyze the image sequence of the mouth region to detect the start time of the mouth opening action; The detected face region at the start time is identified as the candidate face region, and the timestamp of the video frame corresponding to the start time is identified as the lip movement start timestamp of the corresponding candidate face region.

3. The method for dynamic voiceprint-face binding based on the fusion of visual and auditory distances according to claim 2, characterized in that, Analyze the image sequence of the mouth region to detect the start time of the mouth opening action, including: Optical flow calculations are performed on the image sequence of the mouth region to obtain the changes in optical flow features between consecutive frames; When the change in optical flow characteristics exceeds a preset threshold, the corresponding video frame time is determined as the start time of the mouth opening action.

4. The method for dynamic voiceprint-face binding based on the fusion of visual and auditory distances according to claim 1, characterized in that, The audio stream is subjected to source separation to determine at least one candidate voiceprint segment and its corresponding voiceprint start timestamp, including: The audio stream is subjected to source separation to obtain at least one separated audio signal; Speech activity detection is performed on each of the audio signals to locate the starting point of the speech signal; The speech signal segment starting from the aforementioned starting point is identified as a candidate voiceprint segment, and the timestamp corresponding to the starting point is identified as the voiceprint start timestamp of the corresponding candidate voiceprint segment.

5. The method for dynamic voiceprint-face binding based on the fusion of visual and auditory distances according to claim 1, characterized in that, The binding pairs are determined from all candidate pairs, including: The distance difference is used as the matching cost. Based on the matching cost, the Hungarian algorithm is used to perform optimal allocation between the candidate face region and the candidate voiceprint segment; Based on the optimal allocation, candidate pairs consisting of corresponding candidate face regions and candidate voiceprint segments are determined as binding pairs.

6. The method for dynamic voiceprint-face binding based on the fusion of visual and auditory distances according to claim 1, characterized in that, The calculation of auditory distance by combining sound speed includes: Obtain current ambient temperature and / or humidity information; The current speed of sound is determined based on the temperature and / or humidity information. The perceived hearing distance is calculated based on the current speed of sound.

7. The method for dynamic voiceprint-face binding based on the fusion of visual and auditory distances according to claim 1, characterized in that, The method further includes: When no binding pair is successfully determined from all candidate pairs, an indication signal of binding failure is output, and the identification information of the corresponding candidate face region is associated with the identification information of the candidate voiceprint segment and recorded.

8. A voiceprint-based dynamic face binding system based on the fusion of visual and auditory distances, characterized in that, include: The data acquisition unit is used to acquire video streams captured by camera equipment and audio streams captured by microphone array; The vision processing unit is used for: Face detection is performed on the video stream to determine multiple face regions and their corresponding visual distances; Lip movement detection is performed on multiple face regions to determine the candidate face regions where lip movements occur and the corresponding lip movement start timestamps; An audio processing unit is used to perform sound source separation on the audio stream and determine at least one candidate voiceprint segment and the corresponding voiceprint start timestamp. The candidate pair generation unit is used to combine the candidate face region with the candidate voiceprint segment in pairs to construct at least one face-voiceprint candidate pair; The auditory distance calculation unit is used to calculate the auditory distance for each candidate pair based on the time difference between the corresponding lip movement start time stamp and the voiceprint start time stamp, combined with the speed of sound. The binding decision unit is used to determine the binding pair from all candidate pairs based on the distance difference between the visual distance and the corresponding auditory distance of the candidate face region in each candidate pair, so as to complete the voiceprint face binding.

Images