Sound source localization and voiceprint recognition fused method, chip and electronic equipment

Abstract

The invention relates to an acoustic signal processing and biological recognition technology, in particular to a method for fusing sound source localization and voiceprint recognition, a chip and electronic equipment. The method comprises the following specific steps: synchronously acquiring an environment audio signal through an audio acquisition module array and processing the environment audio signal to obtain audio data; and analyzing the audio data to obtain deep fusion voiceprint features. And judging whether a registered user feature library contains the deep fusion voiceprint feature or not, if so, outputting a sounder identity, and if not, marking and endowing an identification number of the sounder identity. And positioning the sound source based on the audio data to obtain a space coordinate thereof. And establishing an identity-position association model, binding identity information output by voiceprint recognition with space coordinates obtained by sound source positioning, and generating identity-position association data. According to the method, the technology is fused and innovated, accurate linkage of identity-position is realized, high real-time performance and strong anti-interference capability are realized, and the method is adaptive to complex scenes.

Description

Technical Field

[0001] This invention relates to acoustic signal processing and biometrics, specifically to a method, chip, and electronic device that integrates sound source localization and voiceprint recognition. Background Technology

[0002] With the development of IoT and AI technologies, the demand for acoustic signal-based perception and recognition is growing, and voiceprint recognition has been widely applied in various fields. Voiceprint recognition relies on the unique physiological and behavioral characteristics of an individual when speaking (such as vocal cord structure and pronunciation habits) to achieve 1:1 or 1:N identity verification. It has the advantages of being contactless and convenient, playing an important role in scenarios such as financial authentication, access control, and attendance systems.

[0003] In existing solutions, the voiceprint recognition module focuses only on identity verification. In environments with multiple sound sources and complex conditions, problems such as "mismatch between identity and location" and "misjudgment of unknown sound sources" are likely to occur. For example, in intelligent security scenarios, if the interference is too strong, it is impossible to confirm the identity of the person or monitor their location, resulting in low efficiency in monitoring and analysis and difficulty in forming a complete security early warning loop. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method that integrates sound source localization and voiceprint recognition. Based on multi-channel audio signals acquired by an audio acquisition module array, it achieves coordinated execution of speaker identification and sound source location monitoring. The specific steps are as follows: S1. Acquire ambient audio signals synchronously through an audio acquisition module array and process them to obtain audio data.

[0005] S2. Analyze the audio data, extract acoustic features and perform feature fusion to obtain deep fused voiceprint features.

[0006] S3. Compare the deep fusion voiceprint feature with the registered user feature database to determine whether the registered user feature database contains the deep fusion voiceprint feature. If it does, output the speaker's identity. If it does not, mark it and assign an identification number to the speaker's identity.

[0007] S4. Locate the sound source and obtain its spatial coordinates based on the audio data.

[0008] S5. Establish an identity-location association model, bind the identity information output by voiceprint recognition with the spatial coordinates obtained by sound source localization, and generate "identity-location" association data.

[0009] Preferred method: The processing of environmental audio signals includes the following steps: timing calibration of each audio acquisition module to eliminate calculation deviations caused by clock skew; then calibrating the time of each audio acquisition module using the deviation values; uniformly sending a synchronous sampling trigger command so that all audio acquisition modules start audio acquisition at the same time point; calculating the time difference corresponding to the cross-correlation peak of any two channel signals, requiring the difference to be ≤1μs, otherwise re-performing clock calibration; removing background noise and power supply interference, filtering out low-frequency noise, high-frequency noise, and power supply interference in the environment, and retaining the effective frequency band of the speech signal; compensating for high-frequency attenuation of the filtered audio signal; converting continuous signals into short-time stationary frames through frame segmentation and windowing to suppress spectral leakage of FFT; identifying the start and end points of the speech signal and removing silent frames without speech; outputting the effective speech frame sequence as audio data.

[0010] Preferred method for calculating deviation includes: parsing the standard time and the 1PPS (one pulse per second) synchronization signal, where the time accuracy of the 1PPS signal is ≤1μs; then, using the 1PPS signal as a reference, calibrating the local clock of each audio acquisition module, and calculating the deviation between the local clock and the standard time. , where t' is the local sampling trigger time and t0 is the standard time.

[0011] Preferred method for compensating audio signals: Where α is the pre-emphasis coefficient, and in this example, α = 0.97; i is the number of the audio acquisition module, n is the index number of the discrete sampling point, and f s It is the sampling frequency of the audio signal; p n i P represents the pre-emphasis signal value at the nth discrete sampling point of the i-th audio acquisition module. n i ' is the pre-filtered input signal value of the nth discrete sampling point of the i-th audio acquisition module; P n-1 i The input signal value of the (n-1)th discrete sampling point of the i-th audio acquisition module after pre-filtering.

[0012] The preferred method for obtaining deep fusion voiceprint features is as follows: S21, perform acoustic feature preprocessing on the audio data; S22, perform deep feature extraction on the preprocessed audio data to obtain deep fusion voiceprint features.

[0013] The preferred method for deep fusion voiceprint feature acquisition specifically includes: 1. Adopting a symmetrical structure of "input layer - encoder layer - hidden layer - decoder layer - output layer", with an input layer dimension of 36, an encoder layer containing two fully connected layers, a hidden layer dimension of 32, and a decoder layer symmetrical to the encoder layer; 2. Aiming to minimize the reconstruction error, the loss function uses mean squared error. Where m is the training sample index, representing the m-th sample, M is the total number of samples in the training set, and x is the training sample index. m The feature vector value of the m-th original input sample, x m The reconstructed output sample feature vector value of the m-th original input sample after the above operations. 3. Take the 32-dimensional output of the hidden layer of the encoder layer as the deep fusion voiceprint feature, which includes the physiological and behavioral characteristics of the speaker.

[0014] Preferred method for constructing registered user feature database includes: training dataset, model structure, and loss function training.

[0015] Preferred: The training dataset includes: collecting a certain number of voice samples from registered users, with each user having at least 5 different sentences of voice text, and extracting deep fusion voiceprint features to construct the training set.

[0016] Preferred model structure includes: an input layer with a dimension of 32, containing 128 hidden units, and using Softmax as the activation function.

[0017] Preferred loss functions include: cross-entropy loss, Adam optimizer, and a learning rate of 0.001. Where Lc is the average cross-entropy loss value of the entire training set, S is the total number of samples in the training set; s is the index number of the training samples, s=1,2,…,S; R is the total number of categories in the classification task, r is the index number of the classification task, r=1,2,…,R, y sr It is the encoded value of the actual tag, with a value of 1 or 0, p sr The predicted output probability value is the probability value of determining that the i-th deep fusion voiceprint feature sample belongs to the j-th type of user, and the value range is [0, 1].

[0018] Preferred: The identity matching method includes the following: inputting the deep fusion voiceprint feature sequence of the speech to be recognized into the registered user feature database; outputting the final judgment probability distribution vector P=[p1, p2, ..., p] for each user in the registered user feature database. SThe user tag corresponding to the highest probability of judgment is taken as the recognition result; then the coefficient of variation of the fundamental frequency is calculated; it is determined whether the coefficient of variation CV is ≥ 0.05. If it is, it is judged as live speech; if not, it is judged as fake speech and the identity matching result is directly rejected.

[0019] Preferred formula for calculating the coefficient of variation of fundamental frequency: , where μ is the mean of the fundamental frequency sequence and σ is the standard deviation of the fundamental frequency sequence.

[0020] Preferably: the determination probability p S The calculation method includes: the deep fusion voiceprint feature sequence to be identified: F = [f1, f2, ..., f T ], where: ft is the 32-dimensional deep fusion voiceprint feature vector of frame t; the cell state ct and hidden state ht are updated through a gating unit, the deep fusion voiceprint feature sequence F is processed frame by frame, and finally the global temporal feature is output. The output 128-dimensional global feature ht is input into a fully connected layer and converted into an R-dimensional "class score vector". , where W is the weight matrix of the fully connected layer and b is the bias vector of the fully connected layer; then, they are normalized to obtain the decision probability.

[0021] Preferably, the method for calculating the determination probability ps includes: calculating the initial probability. Where d is the dimension number of the deep fused voiceprint feature, D is the total number of dimensions of the deep fused voiceprint feature, d=1,2,…,D; f d f represents the numerical value of the dimension numbered d for the deep fusion of voiceprint features. d 'This represents the value of the deep fusion voiceprint feature dimension d corresponding to registered user s in the registered user feature database, f' d 0 This is the unit correction coefficient.

[0022] Preferred method: The spatial coordinate calculation process includes: obtaining a reference audio acquisition module, numbered 0, and constructing a spatial coordinate system with the reference audio acquisition module as the origin to obtain the spatial coordinates of each audio acquisition module; then calculating the cross-correlation function between the audio acquisition modules and the reference audio acquisition module within a preset number; and using the cross-correlation function K... 0j The peak position corresponds to the maximum similarity of the signal. The cross-correlation peak position is calculated, and the time difference between the wave arriving at the target audio acquisition module and the reference audio acquisition module is used. The t-values ​​of all target audio acquisition modules are fitted using the least squares method. 0j The initial position is obtained. Spatial scanning grid is divided for beamforming. For each candidate position (θ') in the grid... , r'), calculate its controllable response power value, and find the maximum controllable response power value in a random region as the spatial coordinates of the sound source.

[0023] Preferred: Cross-correlation function Where 0 is the reference audio acquisition module number, j is the target audio acquisition module number, and K... 0j The similarity measure of the signals from the reference audio acquisition module and the target audio acquisition module at a delay τ; the peak value corresponds to τ, which is the number of delayed sampling points for the two channels, where n is the sampling point index number and N is the total number of sampling points for a single frame signal, n=1, 2, ..., N; X0 n The reference audio acquisition module acquires the effective speech signal amplitude at the nth sampling point, x. j n+τ It is the signal amplitude of the target audio acquisition module j at the (n+τ)th sampling point; τ is the time delay, with a value range of τ∈[-τ]. max , τ max ].

[0024] Preferred: τ max Determined by the maximum aperture of the array and the velocity of sound waves: Where c is the velocity of sound (c = 340 m / s), R' is the array radius, and fs is the sampling frequency. This is the floor function.

[0025] Preferred: t 0j Satisfies geometric relations: ,in Let be the position vector of the target audio acquisition module relative to the reference audio acquisition module; =[cosθcos sinθcos sin ] T is the unit vector in the direction of the sound source.

[0026] Preferred method: The spatial scanning grid division for beamforming includes: dividing the scanning grid around an azimuth angle θ' = 30°; azimuth scanning range: θ' ∈ [25°, 35°], step size Δθ = 0.5°; elevation scanning range: ∈[-5°, 5°], step size Δ =0.5°; Distance scan range: r'∈[1m, 20m], step size Δr'=0.5m.

[0027] Preferred: Controllable response power value Where: l is the index number of the target audio acquisition module, which is the index number of the target audio acquisition module paired with the outer loop variable j; J is the total number of target audio acquisition modules, l < j = 1, 2,..., J; θ' is the azimuth angle of the candidate position, the elevation angle of the candidate position, r' is the distance of the candidate position; K jl is the cross-correlation function between the target audio acquisition module j and the target audio acquisition module l; τ jl (θ', , r') is the theoretical time difference between the target audio acquisition module j and the target audio acquisition module l at the candidate position.

[0028] Preferably: the theoretical time difference , = [r'cosθ'cos , r'sinθ'cos , r'sin T is the position vector of the candidate position.

[0029] Preferably: the random region search includes: selecting the top 5 candidate points with controllable response power values, dividing smaller sub-regions centered on each point; calculating the controllable response power values for the points within the sub-regions, and taking the position corresponding to the maximum value as the sound source space coordinate.

[0030] The present invention also proposes a chip, which is an application specific integrated circuit (ASIC) for executing the above-mentioned method for fusing sound source localization and voiceprint recognition.

[0031] The present invention also proposes an electronic device, which integrates the above-mentioned chip and includes a device body, an audio acquisition module array, a display module, an alarm module and a power supply module.

[0032] The technical effects and advantages of the present invention: By deeply integrating voiceprint recognition and sound source localization technologies and combining with the heterogeneous architecture design of a dedicated ASIC chip, the present invention breaks through the bottlenecks of traditional acoustic perception technologies such as fragmentation, poor real-time performance, and weak anti-interference ability, and achieves significant improvements in identity-position association accuracy, processing efficiency, hardware adaptability and scene applicability. The specific technical effects and advantages are as follows: First, the present invention establishes a deep cooperation mechanism between voiceprint recognition and sound source localization, breaks the technical barriers of their independent operation, and solves the core pain points of traditional solutions such as "only identifying identity but not knowing the position" and "only determining the position but not identifying the identity". Through the fusion decision-making model of "voiceprint identity binding + spatial coordinate positioning", it can accurately output the associated data of "who is speaking where". In a multi-source sound scenario, with the help of a spatial filter, it can achieve a one-to-one matching of the identities and corresponding positions of multiple speakers, exclude the influence of interfering sound sources, accurately distinguish the position trajectories of registered users and unknown identity personnel, and provide complete data support for security early warning.​

[0033] Second, a deep fusion model is employed to extract 32-dimensional high-discrimination deep voiceprint features, combined with liveness detection to resist recording and synthesized speech attacks, significantly enhancing anti-interference capabilities. A coarse localization + beamforming fine localization fusion strategy is adopted, quickly locating the approximate sound source region through cross-correlation calculations, and precisely focusing the sound source location using a random region search method, achieving high-precision localization. Clock synchronization technology ensures multi-channel time deviation ≤1μs, guaranteeing the stability of subsequent algorithm input signals and further improving the reliability of recognition and localization in complex environments.

[0034] Third, we designed a heterogeneous computing ASIC chip that integrates modules such as preprocessing, voiceprint recognition engine, sound source localization engine, and fusion decision-making into a single chip. Compared with the traditional "general-purpose processor + separate algorithm" solution, it has the advantages of high integration, low power consumption and high efficiency, and controllable data security.

[0035] IV. The technical solution of this invention can be flexibly adapted to different application scenarios and hardware configurations, possessing strong practicality and scalability: it can be widely applied to scenarios such as intelligent security monitoring, intelligent conference systems, public safety early warning, and smart homes. It can meet the long-distance (5-20 meters) dynamic tracking requirements of security scenarios, and also adapt to the close-range identity check-in and sound source focusing requirements of conference scenarios. It supports various microphone array layouts such as linear, circular, and spherical arrays, and the number of microphones can be adjusted from 2 to 16. It supports switching between 1:1 identity verification and 1:N identity recognition modes, and the model parameters can be optimized through firmware updates to adapt to more complex scenario requirements. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating a method for integrating sound source localization and voiceprint recognition proposed in this invention.

[0037] Figure 2 This is a schematic diagram illustrating the process of processing environmental audio signals in a method for fusing sound source localization and voiceprint recognition proposed in this invention.

[0038] Figure 3 This is a flowchart illustrating the identity matching method in the fusion sound source localization and voiceprint recognition method proposed in this invention.

[0039] Figure 4 This is a schematic diagram of the process for obtaining the probability determination in a method that integrates sound source localization and voiceprint recognition proposed in this invention.

[0040] Figure 5 This is a flowchart illustrating the spatial coordinate calculation method in the fusion sound source localization and voiceprint recognition method proposed in this invention. Detailed Implementation

[0041] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the invention, and should not be construed as limiting the invention. Rather, embodiments of the invention include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.

[0042] Example 1 refer to Figure 1 This embodiment proposes a method that integrates sound source localization and voiceprint recognition. Based on multi-channel audio signals acquired by an audio acquisition module array, it achieves coordinated execution of speaker identification and sound source location monitoring. The specific steps are as follows: S1. Acquire ambient audio signals synchronously through an array of audio acquisition modules and process them to obtain audio data. Multiple audio acquisition modules can be arranged in an array within the management area to collect ambient audio signals. These modules can be standalone microphones or microphone units within cameras; details will not be elaborated here. The management area is the area we need to control, which could be a factory, office building, etc.; details will not be elaborated here. (Reference) Figure 2 The processing of environmental audio signals includes the following steps: timing calibration of each audio acquisition module to synchronize their times, ensuring the timing consistency of multi-channel signals and eliminating calculation deviations caused by clock skew. Specifically, this may include resolving the standard time and a 1PPS (one pulse per second) synchronization signal, where the time accuracy of the 1PPS signal is ≤1μs. Then, using the 1PPS signal as a reference, the local clock of each audio acquisition module is calibrated, and the deviation between the local clock and the standard time is calculated. Where t' is the local sampling trigger time and t0 is the standard time. The standard time can be obtained by receiving satellite timing signals through the built-in GNSS module of the chip, which will not be elaborated here. Then, the time of each audio acquisition module is calibrated by the deviation value ∆t, the specific process of which will not be elaborated here. After calibration, a unified synchronous sampling trigger command is sent, and all audio acquisition modules start audio acquisition at the same time point to ensure that the sampling period is strictly synchronized and that the audio signal frames acquired by the multi-channel are completely aligned. Then, the time difference corresponding to the cross-correlation peak of any two channel signals is calculated. The difference is required to be ≤1μs; otherwise, clock calibration is re-executed. Background noise and power interference are removed. Low-frequency noise (such as air conditioner and footsteps, <50Hz), high-frequency noise (such as electromagnetic interference, >8kHz), and 50Hz / 60Hz power interference in the environment are filtered out, while retaining the effective frequency band of the voice signal (50Hz-8kHz). The specific technology is existing and will not be elaborated here. The filtered audio signal is compensated for high-frequency attenuation of the voice signal, which enhances the high-frequency components of the voice signal, improves the recognizability of voiceprint features, and compensates for the high-frequency energy attenuation during voice propagation. The calculation formula is as follows: Where α is the pre-emphasis coefficient, and in this example, α = 0.97 (adapted to a 16kHz sampling rate speech signal); i is the number of the audio acquisition module, and n is the index number of the discrete sampling point, the conversion relationship between which and physical time is as follows: f s This is the sampling frequency of the audio signal (unit: Hz, i.e., number of sampling points per second). n i P represents the pre-emphasis signal value at the nth discrete sampling point of the i-th audio acquisition module. n i ' is the pre-filtered input signal value (discrete sampling point amplitude) of the nth discrete sampling point of the i-th audio acquisition module, P n-1 iThe input signal value of the (n-1)th discrete sampling point of the i-th audio acquisition module, after pre-filtering, can increase the amplitude of the high-frequency signal component by about 10dB, enabling subsequent feature extraction to more accurately capture the subtle differences in the speaker's pronunciation. Speech signals are non-stationary, but can be approximated as stationary signals within a short time (10ms-30ms). Frame segmentation transforms the continuous signal into short-time stationary frames, and windowing suppresses spectral leakage in the FFT. Specifically, Hamming window calculation can be used, which will not be elaborated here. The start and end points of the speech signal are identified, and silent frames without speech (such as ambient silence or breathing sounds) are removed to reduce the computational load of subsequent algorithms. A dual-threshold method (based on short-time energy and short-time zero-crossing rate) can be used; the specific steps will not be elaborated here. Invalid frames can be removed: all frames before the start frame and after the end frame are considered silent frames and removed. Each audio acquisition module ultimately outputs a sequence of valid speech frames, which serves as audio data for subsequent voiceprint recognition and sound source localization.

[0043] S2. Analyze the audio data, extract acoustic features (including spectrum, formants, prosody, etc.) and perform feature fusion to obtain deep fused voiceprint features. The specific process is as follows: S21. Preprocess the audio data for acoustic features, which may include pre-emphasis calculation and Mel-frequency cepstral coefficient (MFCC) calculation to provide underlying data support for deep features. This is existing technology and will not be elaborated here. S22. Extract deep features from the preprocessed audio data. Specifically, this includes: 1. Adopting a symmetrical structure of "input layer - encoder layer - hidden layer - decoder layer - output layer". The input layer has a dimension of 36 (corresponding to the MFCC feature vector), the encoder layer contains two fully connected layers (dimensions of 128 and 64 respectively), the hidden layer has a dimension of 32 (core deep feature dimension), and the decoder layer is symmetrical to the encoder layer. 2. With the goal of minimizing reconstruction error, the loss function adopts mean squared error. Where m is the training sample index, representing the m-th sample, M is the total number of samples in the training set, and x is the training sample index. m The feature vector value of the m-th original input sample, x m The reconstructed output sample feature vector value of the m-th original input sample after the above operations. 3. Take the 32-dimensional output of the hidden layer of the encoder layer as the deep fusion voiceprint feature. This deep fusion voiceprint feature includes the speaker's physiological characteristics (vocal cords, vocal tract structure) and behavioral characteristics (pronunciation habits, rhythm).

[0044] S3. Compare the deep fused voiceprint features with the registered user feature database to determine if the database contains the deep fused voiceprint features. If it does, output the speaker's identity; otherwise, mark the user and assign them an identification number. The registered user feature database is pre-constructed, including user identities and corresponding deep fused voiceprint features. Its construction method can include: Training dataset: Collecting speech samples from a certain number (e.g., 10,000) of registered users, with at least 5 different sentences of speech for each user, extracting deep fused voiceprint features, and constructing a training set. Model structure: Input layer dimension 32 (deep fused voiceprint feature dimension), containing 128 hidden units; fully connected layer output dimension 10,000 (corresponding to the number of registered users); Softmax activation function; Loss function: Cross-entropy loss function; Adam optimizer; learning rate set to 0.001. Where Lc is the average cross-entropy loss value of the entire training set, which measures the overall deviation between the model's predicted identity probability distribution and the real identity label; the smaller the value, the more accurate the model prediction. S is the total number of samples in the training set, the total number of deep fusion voiceprint feature samples participating in model training (e.g., 10,000 samples in this scheme, i.e., S=10,000); s is the index number of the training sample, s=1,2,…,S; the number of the s-th deep fusion voiceprint feature sample, corresponding to the 32-dimensional deep feature extracted from a segment of speech of a registered user; R is the total number of categories for the classification task, the total number of registered users for voiceprint recognition (e.g., this scheme supports R=10,000, i.e., 10,000 registered users); r is the index number of the classification task, r=1,2,…,R; y sr y is the encoded value of the real label, which is either 1 or 0. If the real identity of the s-th sample is a user of the r-th type, then y sr =1; otherwise y sr =0. Example: If sample r belongs to user 3, then y s3 =1, the rest of y s1 y s2 y s4 …y sR All are 0. p sr The predicted output probability value is specifically the probability value of determining that the i-th deep fusion voiceprint feature sample belongs to the j-th type of user, with a value range of [0, 1], and satisfying the following conditions: The core function of the cross-entropy loss function is to guide the optimization of model parameters through backpropagation, ultimately minimizing the loss value. (Reference) Figure 3 Identity matching calculation (using a 1:N recognition mode) specifically includes the following: inputting the deep fusion voiceprint feature sequence of the speech to be recognized into the registered user feature database. The final judgment probability distribution vector P=[p1, p2, ..., p...] is then calculated for each user in the registered user feature database. SThe user tag corresponding to the highest probability of judgment is taken as the recognition result. If the highest probability of judgment is less than a preset threshold (0.85 in this scheme), it is judged as an unknown identity. Then, the coefficient of variation of the fundamental frequency is calculated. The fundamental frequency is the frequency of vocal cord vibration. The fundamental frequency of real speech has continuous dynamic changes, while the fundamental frequency of recorded or synthesized speech changes relatively smoothly. The formula for calculating the coefficient of variation of the fundamental frequency is: Where μ is the mean of the fundamental frequency sequence and σ is the standard deviation of the fundamental frequency sequence. If CV ≥ 0.05, the speech is determined to be live; otherwise, it is determined to be spoofed, the identity matching result is rejected, and an alarm is triggered. (Reference) Figure 4 The determination probability p S The calculation method may include: a deep fusion voiceprint feature sequence to be identified: F = [f1, f2, ..., f T ], where: ft is the 32-dimensional deep fusion voiceprint feature vector of the t-th frame (t=1, 2, ..., T, in this embodiment T=80 frames, corresponding to 1.6s of speech); feature dimension D=32, that is, ft=[f t1 f t2 , ..., f t32 The cell state ct and hidden state ht are updated through gating units (input gate it, forget gate ft, output gate ot). The deep fusion voiceprint feature sequence F is processed frame by frame, and finally the global temporal feature is output. The output 128-dimensional global feature ht is input into a fully connected layer and converted into an R=10000-dimensional "class score vector". Where W is the weight matrix of the fully connected layer and b is the bias vector of the fully connected layer, which are specifically constructed from the input of the model, and will not be elaborated here. Then, they are normalized to obtain the decision probability. The method for calculating the determination probability ps may further include: calculating the initial probability. Where d is the dimension number of the deep fused voiceprint feature, and D is the total number of dimensions of the deep fused voiceprint feature, d=1, 2, ..., D; D can be 32. d f represents the numerical value of the dimension numbered d for the deep fusion of voiceprint features. d 'This represents the value of the deep fusion voiceprint feature dimension d corresponding to registered user s in the registered user feature database, f' d 0 For unit correction coefficients, if f d and f d 'By normalizing all values, the unit correction coefficient can be 1, although other values ​​are not excluded. Then, an initial probability distribution vector is constructed from the initial probabilities, and normalized to obtain the decision probability distribution vector. The decision probability distribution vector calculated in this way can eliminate accidental data fluctuations and distortions, while also evaluating and considering them, thus improving the accuracy of matching.'

[0045] S4. Locate the sound source and obtain its spatial coordinates based on audio data. (Reference) Figure 5 The specific calculation process may include: obtaining a reference audio acquisition module, which can be the first audio acquisition module to acquire audio and numbered 0; constructing a spatial coordinate system with the reference audio acquisition module as the origin; and obtaining the spatial coordinates of each audio acquisition module to indicate its closest relative to the sound source. Calculating the cross-correlation function between a preset number of audio acquisition modules (generally 8, depending on the actual situation) and the reference audio acquisition module. Where 0 is the reference audio acquisition module number, j is the target audio acquisition module number, and K... 0j The similarity measure of the signals from the reference audio acquisition module and the target audio acquisition module at a delay τ; the peak value corresponding to τ is the number of delayed sampling points between the two channels, the time difference between the arrival of the same sound source signal at different audio acquisition modules, n is the sampling point index number, N is the total number of sampling points in a single frame signal, n=1, 2, ..., N; X0 n The reference audio acquisition module acquires the effective speech signal amplitude at the nth sampling point, x. j n+τ It is the signal amplitude of the target audio acquisition module j at the (n+τ)th sampling point; τ is the time delay (unit: sampling point), and its value range is τ∈[-τ]. max , τ max ], τ max Determined by the maximum aperture of the array and the velocity of sound waves: Where c is the velocity of sound (c = 340 m / s), R' is the array radius, and fs is the sampling frequency. This is the floor function. Substituting the data R'=0.05m, fs=16kHz, we get τ max =2×0.05×16000 / 340≈4.7, rounded down to τ max =5 sampling points. Through the cross-correlation function K... 0j The peak position corresponds to the maximum similarity of the signal, and this position is the number of delay sampling points τ between the target audio acquisition module and the reference audio acquisition module. 0j Then convert it to physical time difference t 0j : For example, for microphone 3 (which is at an angle of 135° to the reference audio acquisition module), the cross-correlation peak position τ is calculated. 03 =2 sampling points, then the time difference t 03 =2 / 16000=125μs. The time difference t between the wave reaching the target audio acquisition module and the reference audio acquisition module is 125μs. 0j Satisfies geometric relations: , where is the position vector of the target audio acquisition module relative to the reference audio acquisition module (taking a circular array as an example, = [R’cosα j , R’sinα j , 0] T , and α j is the azimuth angle of the target audio acquisition module); = [cosθcos , sinθcos , sin T is the unit vector of the sound source direction (θ is the azimuth angle, is the elevation angle). By using the least squares method to fit the t of all target audio acquisition modules 0j . Beamforming spatial scan grid division. Specifically, centered on the azimuth angle θ’ = 30°, the scan grid is divided as follows: Azimuth angle scan range: θ’ ∈ [25°, 35°], step size Δθ = 0.5°; Elevation angle scan range: ϕ’ ∈ [-5°, 5°], step size Δ = 0.5°; Distance scan range: r’ ∈ [1m, 20m], step size Δr’ = 0.5m. For each candidate position (θ’, , r’) in the grid, calculate its controllable response power value, and the formula is: where: l is the index number of the target audio acquisition module (inner loop variable), the index number of the target audio acquisition module paired with the outer loop variable j, and the two form a channel pair; J is the total number of target audio acquisition modules, l < j = 1, 2,..., J. θ’ is the azimuth angle of the candidate position, is the elevation angle of the candidate position, r’ is the distance of the candidate position. K jl is the cross-correlation function between target audio acquisition module j and target audio acquisition module l; τ jl (θ’, , r’) is the theoretical time difference between target audio acquisition module j and target audio acquisition module l at the candidate position, which is calculated by geometric relationship: = [r’cosθ’cos , r’sinθ’cos , r’sin TThis is the position vector of the candidate location. The maximum controllable response power value is found within a random region and used as the spatial coordinates (azimuth, elevation, and distance) of the sound source. Specifically, this involves: selecting the top 5 candidate points with the highest controllable response power values; dividing each point into smaller sub-regions (with a step size reduced to 0.1°); calculating the controllable response power value for each point within the sub-region; and taking the location corresponding to the maximum value as the spatial coordinates (azimuth, elevation, and distance) of the sound source. This method can accurately obtain the spatial coordinates of the sound source.

[0046] S5. Establish an identity-location association model, bind the identity information output by voiceprint recognition with the spatial coordinates obtained by sound source localization, and generate "identity-location" association data; for multi-sound source scenarios, use spatial filters to distinguish the signal frequency bands of different sound sources, realize one-to-one matching of multiple voices with their corresponding locations, and eliminate the influence of interfering sound sources. The details are not elaborated here.

[0047] Example 2 This invention also proposes a chip, which is an application-specific integrated circuit (ASIC) that integrates the aforementioned fusion sound source localization and voiceprint recognition method modules. It adopts a heterogeneous computing architecture, including an audio acquisition module array interface module, a preprocessing module, a voiceprint recognition engine, a sound source localization engine, a fusion decision module, a storage module, and a peripheral interface module. These modules work collaboratively to achieve low-latency data processing. Audio acquisition module array interface module: Supports 2-16 channels of audio acquisition module input, integrates A / D converter and time synchronization unit, realizes synchronous acquisition and analog-to-digital conversion of multi-channel audio signals, and the sampling rate is configurable (16kHz-48kHz).

[0048] Preprocessing module: It adopts hardware acceleration circuit to implement preprocessing algorithms such as filtering and frame windowing, with a processing delay of ≤1ms, which greatly reduces the computational load of subsequent modules.

[0049] Voiceprint recognition engine: Built-in voiceprint feature library storage area, supports the storage of features of 10,000+ registered users, identity recognition accuracy ≥99.5%, recognition latency ≤50ms.

[0050] Sound source localization engine: integrates TDOA computing unit and beamforming acceleration unit, supports adaptation to various audio acquisition module array layouts such as linear, circular and spherical, with positioning accuracy ≤0.5° (azimuth angle) and positioning delay ≤30ms.

[0051] Fusion Decision Module: This is the core control unit of the chip, coordinating the working sequence of each engine, realizing identity-location data binding and dynamic tracking algorithms, and supporting multi-source priority sorting and early warning logic configuration.

[0052] Storage module: Built-in high-speed SRAM for temporary data caching, and Flash for storing algorithm firmware, voiceprint feature library and configuration parameters.

[0053] Peripheral interface module: Provides UART, I2C, SPI and Ethernet interfaces, supports data interaction with display devices, alarm devices and host computer systems, and supports edge computing mode, which can independently complete data processing and decision output without relying on cloud computing power.

[0054] Example 3 This embodiment proposes an electronic device that integrates the aforementioned chip and includes a device body, an audio acquisition module array, a display module, an alarm module, and a power supply module. The specific structure is as follows: Equipment body: It adopts a waterproof and dustproof design, which is suitable for indoor and outdoor installation scenarios, and has reserved mounting slots and interface layouts for audio acquisition module arrays.

[0055] Audio acquisition module array: Composed of multiple audio acquisition modules, using a circular or rectangular layout to ensure omnidirectional sound source acquisition and positioning coverage, supporting long-distance (5-20 meters) voice signal acquisition.

[0056] Display module: Uses an LCD or OLED screen to display "identity-location" association information, sound source movement trajectory and early warning prompts in real time, and supports multi-sound source information split-screen display.

[0057] Alarm module: Integrates an audible and visual alarm unit. When an unknown sound source, an unauthorized target identity, or an abnormal sound source location is detected, an alarm is automatically triggered, and alarm information can be pushed to the monitoring center through an external interface.

[0058] Power supply module: Supports DC power supply (12V / 24V) and POE power supply, adaptable to different installation environments, power consumption ≤5W (standby mode ≤1W).

[0059] This electronic device can be designed as a desktop, wall-mounted, or portable device according to the application scenario requirements, and is suitable for places such as conference rooms, security monitoring points, and office areas.

[0060] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0061] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method integrating sound source localization and voiceprint recognition, characterized in that, The steps of the method for fusing sound source localization and voiceprint recognition are as follows: S1. Acquire ambient audio signals synchronously through an audio acquisition module array and process them to obtain audio data; S2. Analyze the audio data, extract acoustic features and perform feature fusion to obtain deep fused voiceprint features; S3. Compare the deep fusion voiceprint feature with the registered user feature library to determine whether the registered user feature library contains the deep fusion voiceprint feature. If yes, output the speaker's identity. If no, mark it and assign an identification number to the speaker's identity. S4. Locate the sound source and obtain its spatial coordinates based on audio data; S5. Bind the identity information output by voiceprint recognition with the spatial coordinates obtained by sound source localization to generate "identity-location" associated data.

2. The method for fusing sound source localization and voiceprint recognition according to claim 1, characterized in that, The processing of the environmental audio signal includes the following steps: timing calibration of each audio acquisition module to eliminate calculation deviations caused by clock skew; then calibrating the time of each audio acquisition module using the deviation values; uniformly sending a synchronous sampling trigger command, so that all audio acquisition modules start audio acquisition at the same time point, and calculating the time difference corresponding to the cross-correlation peak of any two channel signals, requiring that the difference be ≤1μs, otherwise re-performing clock calibration; removing background noise and power supply interference, filtering out low-frequency noise, high-frequency noise, and power supply interference in the environment, and retaining the effective frequency band of the speech signal; compensating for high-frequency attenuation of the filtered audio signal; converting the continuous signal into short-time stationary frames through frame segmentation, and windowing to suppress spectral leakage of FFT; identifying the start and end points of the speech signal, and removing silent frames without speech; and outputting the effective speech frame sequence as audio data.

3. The method for fusing sound source localization and voiceprint recognition according to claim 2, characterized in that, The deviation calculation method includes: parsing the standard time and the 1PPS synchronization signal, where the time accuracy of the 1PPS signal is ≤1μs; then, using the 1PPS signal as a reference, calibrating the local clock of each audio acquisition module, and calculating the deviation between the local clock and the standard time. , where t' is the local sampling trigger time and t0 is the standard time.

4. The method for fusing sound source localization and voiceprint recognition according to claim 1, characterized in that, Audio signal compensation methods: Where α is the pre-emphasis coefficient; i is the number of the audio acquisition module; n is the index number of the discrete sampling point; f s It is the sampling frequency of the audio signal; p n i P represents the pre-emphasis signal value at the nth discrete sampling point of the i-th audio acquisition module. n i ' is the pre-filtered input signal value of the nth discrete sampling point of the i-th audio acquisition module; P n-1 i The input signal value of the (n-1)th discrete sampling point of the i-th audio acquisition module after pre-filtering.

5. The method for fusing sound source localization and voiceprint recognition according to claim 1, characterized in that, The specific process for obtaining deep fusion voiceprint features is as follows: S21, perform acoustic feature preprocessing on the audio data; S22, perform deep feature extraction on the preprocessed audio data to obtain deep fusion voiceprint features.

6. The method for fusing sound source localization and voiceprint recognition according to claim 1, characterized in that, The method for constructing a registered user feature database includes: training dataset, model structure, and loss function training.

7. The method for fusing sound source localization and voiceprint recognition according to claim 1, characterized in that, Identity matching calculation includes the following: inputting the deep fusion voiceprint feature sequence of the speech to be recognized into the registered user feature database; outputting the final judgment probability distribution vector P=[p1, p2, ..., p] for each user in the registered user feature database. S The system takes the user tag corresponding to the maximum probability as the recognition result. If the maximum probability is less than the preset threshold, it is judged as an unknown identity. Then, the coefficient of variation of the fundamental frequency is calculated. If CV ≥ 0.05, it is judged as live speech; otherwise, it is judged as fake speech, and the identity matching result is directly rejected.

8. The method for fusing sound source localization and voiceprint recognition according to claim 1, characterized in that, The process of calculating the spatial coordinates of the sound source includes: obtaining the reference audio acquisition module, numbered 0; constructing a spatial coordinate system with the reference audio acquisition module as the origin, and obtaining the spatial coordinates of each audio acquisition module; calculating the cross-correlation function between the audio acquisition modules and the reference audio acquisition module within a preset number; and using the cross-correlation function K... 0j The peak position corresponds to the maximum similarity of the signal. The cross-correlation peak position is calculated, and the time difference between the wave arriving at the target audio acquisition module and the reference audio acquisition module is used. The t-values ​​of all target audio acquisition modules are fitted using the least squares method. 0j Obtain the initial position; divide the spatial scanning grid for beamforming; for each candidate position (θ') in the grid, , r'), calculate its controllable response power value, and find the maximum controllable response power value in a random region as the spatial coordinates of the sound source.

9. A chip, characterized in that, The chip is used to execute any one of the methods for fusing sound source localization and voiceprint recognition according to claims 1-8.

10. An electronic device, characterized in that, The electronic device includes the chip of claim 9.

Images