A method and system for voice wake-up and control of a surveillance ball

Abstract

This invention relates to the field of speech signal processing technology, and discloses a method and system for voice wake-up and control of a gimbal. The method includes: acquiring speech signals and extracting extended acoustic features; using an attention-based temporal convolutional network to predict the probability value of situational abrupt changes and the target situational category label; performing dual-mode linkage control based on the probability value of situational abrupt changes, coordinating the initiation of azimuth protection and the gradual transition of speech processing parameters; performing anomaly detection and selective inclusion of the azimuth angle in the protection state; performing delay compensation forward extrapolation and generating the absolute azimuth angle of the sound source; performing speech processing and outputting the wake-up determination result; parsing directional commands and generating gimbal rotation control commands. This invention simultaneously integrates the probability value of situational abrupt changes output by the attention-based temporal convolutional network into the data admission control of the sound source azimuth motion model and the gradual transition of speech processing parameters, solving the technical problem of coupling failure between voice wake-up and directional control during the transition period of acoustic environment abrupt changes.

Description

Technical Field

[0001] This invention relates to the field of speech signal processing technology, specifically to a method and system for voice wake-up and control of a PTZ camera. Background Technology

[0002] In outdoor mobile patrol scenarios, the PTZ camera needs to continuously respond to the operator's voice wake-up words and directional control commands. Existing solutions use linear regression of the sound source azimuth time series to fit angular velocity and angular acceleration, and combine this with forward extrapolation based on processing link delay to convert relative direction words in voice commands into absolute rotation control commands for the PTZ camera; the voice processing parameters are then switched retroactively based on the sound field situation classification results.

[0003] The above scheme has the following drawbacks: The sound source azimuth motion model unconditionally includes the azimuth angle data of each frame into the regression window. When the acoustic environment undergoes a drastic change, the sudden noise causes a large jump in the angle of arrival estimation. The abnormal azimuth angle is unconditionally included, contaminating the angular velocity and angular acceleration estimates, and causing multi-cycle systematic azimuth tracking offset. At the same time, the post-event switching of speech processing parameters has a multi-frame delay from the actual occurrence of the situational change to the completion of parameter loading. During the delay, the old parameters are severely mismatched with the new sound field situation, causing the wake-up word detection to fail during the transition period. The superposition of these two types of failures makes the control ball unable to respond to the wake-up word or correctly perform directional control during the acoustic change transition period. Summary of the Invention

[0004] This invention provides a method and system for voice wake-up and control of a controllable ball, which solves the technical problems in related technologies such as insufficient robustness of voice wake-up when the acoustic environment changes abruptly, delay deviation in sound source azimuth angle estimation, and wake-up failure caused by discontinuous switching of voice processing parameters.

[0005] This invention discloses a method for voice wake-up and control of a PTZ camera, comprising at least steps A and B. Step A includes: extracting extended acoustic feature vectors from the speech signal acquired by a dual-microphone differential array and calculating the time series of the sound source azimuth angle; inputting the extended acoustic feature vectors into an attention-based temporal convolutional network to output a situational change probability value and a target situational category label. Step B includes: when the situational change probability value reaches a protection trigger threshold, storing the sound source azimuth motion model parameters and the latest reliable sound source azimuth angle into a protection snapshot cache; performing anomaly detection and selective inclusion based on residuals for subsequent azimuth angles; generating working parameters by progressively transitioning the current speech processing parameter combination to a preliminary parameter combination frame by frame based on the situational change probability value; generating the absolute azimuth angle of the sound source by performing delay compensation forward extrapolation using the currently effective angular velocity and angular acceleration estimates; processing the speech signal with the working parameters and performing a wake-up determination; after successful wake-up, converting relative direction words into PTZ rotation control commands based on the absolute azimuth angle of the sound source.

[0006] Furthermore, the generation of the extended acoustic feature vector includes: forming a spectral energy distribution feature by taking the energy values ​​of each frame of speech signal after short-time Fourier transform in each frequency sub-band; forming an inter-frame energy fluctuation feature by taking the absolute values ​​of the differences between the energy values ​​of each frequency sub-band between the current frame and the previous frame; forming a sub-band energy change rate feature by taking the logarithm of the ratio of the energy value of the current frame to the energy value of the previous frame in each frequency sub-band; and concatenating the spectral energy distribution feature, the inter-frame energy fluctuation feature, and the sub-band energy change rate feature after performing Z-score normalization on each of them.

[0007] Furthermore, the attention-enabled temporal convolutional network includes a temporal convolutional layer group, an attention-weighted layer, and a classification output layer. The temporal convolutional layer group is composed of multiple cascaded causal dilated convolutional layers with increasing dilation coefficients layer by layer. The feature maps output by each layer are concatenated along the channel dimension to generate a multi-scale temporal feature map. The attention-weighted layer performs a fully connected transformation on the feature vectors at each time window position in the multi-scale temporal feature map to obtain scalar scores. Softmax normalization is performed on all scalar scores to obtain attention weights. The attention weights are weighted and summed with the corresponding feature vectors to generate an aggregated feature vector. The classification output layer passes the aggregated feature vector through two fully connected sub-layers. One fully connected sub-layer is connected to a sigmoid activation function to output the situational change probability value, and the other fully connected sub-layer is connected to a softmax activation function to output the target situational category label.

[0008] Furthermore, the training of the attention-enabled temporal convolutional network includes: using a labeled historical acoustic event dataset as training samples, each sample containing an extended acoustic feature vector sequence for several time windows before the mutation occurs, along with corresponding mutation occurrence labels and situation category labels; the loss function is composed of a weighted sum of the binary cross-entropy loss of the mutation probability prediction branch and the multi-class cross-entropy loss of the situation category prediction branch, and the weight parameters of the attention-enabled temporal convolutional network are updated using the Adam optimization algorithm.

[0009] Further, the residual-based anomaly detection and selective inclusion includes: taking the latest reliable sound source azimuth angle in the protection snapshot cache as the starting value, superimposing the product of the angular velocity estimate and the time difference, and half of the product of the angular acceleration estimate and the square of the time difference, to obtain the model prediction value at the current moment; calculating the absolute value of the difference between the newly acquired sound source azimuth angle and the model prediction value as the residual; determining whether the residual exceeds a preset anomaly offset threshold, if it exceeds, excluding the azimuth angle data from the regression window and maintaining the predicted sound source azimuth angle by inertial extrapolation, if it does not exceed, including the azimuth angle data in the regression window and re-executing linear regression to update the angular velocity estimate and angular acceleration estimate.

[0010] Furthermore, after the protection state lasts for more than the preset maximum protection duration, the abnormal offset threshold is widened frame by frame according to a preset incremental step size, so that the regression window gradually accepts azimuth data with increased deviation but still passing the threshold test after widening.

[0011] Further, the step of progressively transitioning the current speech processing parameter combination to the preliminary parameter combination frame by frame based on the situational change probability value to generate working parameters includes: retrieving the preliminary parameter combination from the parameter configuration table based on the target situational category label; calculating the difference vector of each dimension between the current speech processing parameter combination and the preliminary parameter combination; scaling the difference vector proportionally using the situational change probability value as a scaling factor to generate a progressive transition increment; superimposing the progressive transition increment onto the current speech processing parameter combination to generate a transition parameter combination and loading it as working parameters; wherein, the parameter configuration table is a pre-stored correspondence table between multiple sets of speech processing parameter combinations and situational category labels, each set of speech processing parameter combinations includes beamforming direction gain coefficient, noise reduction threshold, and echo cancellation step size, and the parameter values ​​of each dimension have been uniformly normalized to the same numerical range before being stored.

[0012] Furthermore, the generation of the absolute azimuth angle of the sound source also includes: acquiring the three-axis acceleration and three-axis angular velocity sampling sequences output by the inertial measurement unit, and applying the Kalman filter algorithm to output device attitude estimation data; performing coordinate transformation on the predicted sound source azimuth angle based on the device attitude estimation data, converting the azimuth angle in the device body coordinate system to the azimuth angle in the geodetic coordinate system to generate the compensated absolute azimuth angle of the sound source; acquiring the current wind speed value output by the wind speed sensor, and when the current wind speed value exceeds a preset wind speed threshold, using the square of the ratio of the current wind speed value to the wind speed threshold as an amplification factor to amplify the diagonal element values ​​corresponding to the yaw angle and pitch angle in the process noise covariance matrix of the Kalman filter algorithm.

[0013] Furthermore, it also includes: after the probability value of the sudden change in the situation falls below the protection trigger threshold for several consecutive frames, removing the anomaly detection flag and clearing the protection snapshot cache, allowing the regression window to fully use the newly acquired reliable azimuth angle data to re-execute the linear regression fitting of the angular velocity estimate and the angular acceleration estimate; comparing the target situation category label output by the attention-based temporal convolutional network during the protection period with the situation category label actually confirmed after the protection is lifted, and using the comparison result and the corresponding temporal feature matrix as incremental training samples to update the weight parameters of the attention-based temporal convolutional network.

[0014] This invention discloses a voice wake-up and control system for a deployment ball, used to execute the aforementioned voice wake-up and control method for a deployment ball, comprising: a signal acquisition and feature extraction module, used to extract extended acoustic feature vectors from the voice signal acquired by a dual-microphone differential array and calculate the time series of the sound source azimuth angle; a situation prediction module, used to input the extended acoustic feature vectors into an attention-based temporal convolutional network and output a situation change probability value and a target situation category label; and a dual-mode linkage control module, used to, when the situation change probability value reaches a protection trigger threshold, input the sound source azimuth motion model parameters and the latest reliable sound source azimuth angle. The system stores a snapshot in a protective cache, performs anomaly detection and selective inclusion of subsequent azimuth angles based on residuals, and generates working parameters by progressively transitioning the current voice processing parameter combination to a preliminary parameter combination frame by frame based on the probability value of the situational abrupt change. An azimuth extrapolation module is used to generate the absolute azimuth angle of the sound source by performing delay compensation extrapolation using the currently valid angular velocity and angular acceleration estimates. A voice processing and wake-up module is used to process the voice signal with the working parameters and perform wake-up determination. An instruction generation module is used to convert relative direction words into gimbal rotation control instructions based on the absolute azimuth angle of the sound source after successful wake-up.

[0015] This invention integrates the probability values ​​of situational abrupt changes output by an attention-based temporal convolutional network into the data admission control of the sound source azimuth motion model and the progressive transition of speech processing parameters. This solves the technical problem of voice wake-up and direction control coupling failure during the transition period of acoustic environment abrupt changes, achieving the following technical effects: At the azimuth tracking level, large jumps in azimuth angles caused by noise are blocked outside the regression window, and the estimated values ​​of angular velocity and angular acceleration are not contaminated, overcoming the multi-cycle systematic tracking offset caused by the unconditional inclusion of abnormal azimuth angles in the regression window; At the speech processing level, the speech processing parameters are transitioned to the target parameter combination frame by frame before the actual acoustic environment abrupt change occurs, overcoming the beamforming direction jumps and noise reduction threshold mismatch during the multi-frame delay caused by post-parameter switching, ensuring the continuous availability of wake-up word detection during the transition period of abrupt changes; At the device attitude estimation level, the process noise covariance matrix of the Kalman filter is dynamically adjusted according to real-time wind speed data, overcoming the azimuth coordinate transformation deviation caused by the lag in device attitude estimation under outdoor gust conditions. Attached Figure Description

[0016] Figure 1 This is a flowchart of the voice wake-up and control method for the deployment ball provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the trend of the extended acoustic features of each frame changing over time, as provided in the embodiments of the present invention. Figure 3 This is a schematic diagram showing a frame-by-frame comparison of attention weights and situational change probability values ​​provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the gradual transition parameter changes from frame 445 to frame 446 provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the azimuth anomaly detection results of each frame under the protection state provided in the embodiments of the present invention; Figure 6 This is a schematic diagram comparing the azimuth residuals of each frame with the abnormal offset threshold provided in the embodiments of the present invention; Figure 7 This is a schematic diagram of the voice wake-up and direction control processing flowchart provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the attention-enabled temporal convolutional network structure provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the signal-to-noise ratio and wake-up confidence before and after speech processing provided in an embodiment of the present invention. Detailed Implementation

[0017] The steps of this implementation method are as follows: Step S100: Acquire speech signals and extract extended acoustic features. Speech signals are continuously acquired using a dual-microphone differential array. Spectral energy distribution features, inter-frame energy fluctuation features, and sub-band energy change rate features are extracted from each frame of speech signal. These three types of features are then concatenated to generate an extended acoustic feature vector. Simultaneously, the azimuth angle of the sound source is calculated from the signals received by the dual-microphone differential array at fixed time intervals. The azimuth angle of the sound source is then paired with the corresponding timestamp to form a data pair and stored in chronological order as a sound source azimuth angle time series.

[0018] It should be noted that the above-mentioned spectral energy distribution characteristics are vectors composed of the energy values ​​of each frequency sub-band after the short-time Fourier transform of each frame of speech signal; the inter-frame energy fluctuation characteristics are vectors composed of the absolute values ​​of the differences between the energy values ​​of each frequency sub-band between the current frame and the previous frame; and the sub-band energy change rate characteristics are vectors composed of the logarithms of the ratios of the energy values ​​of the current frame and the previous frame in each frequency sub-band. These three types of characteristics respectively characterize the static spectral structure of the current frame, the amplitude of energy jumps between adjacent frames, and the rate of change of energy in each sub-band.

[0019] Before concatenating the three types of features to generate the extended acoustic feature vector, Z-score normalization is performed on the spectral energy distribution feature, inter-frame energy fluctuation feature, and sub-band energy change rate feature to eliminate the influence of the differences in physical dimensions and numerical ranges of the three types of features on the input of the subsequent attention-based temporal convolutional network.

[0020] In a security monitoring project, operators used handheld surveillance cameras to conduct nighttime patrols along the perimeter of a factory area. The cameras continuously collected audio signals at a 16 kHz sampling rate, with a frame length of 25 ms and a frame shift of 10 ms, dividing the spectrum into 8 sub-bands. At 22:47 on March 12, 20XX, when the operators reached the northeast corner of the perimeter, a large truck suddenly started moving in the direction of the parking lot in the distance, with the engine noise rapidly increasing within approximately 1.2 seconds. The surveillance cameras continuously collected audio frames before and after this moment, extracted extended acoustic feature vectors from each frame, and calculated the sound source azimuth angle at fixed intervals of 100 ms, forming a sound source azimuth angle time series. Taking six frames before and after the sudden change as an example, the three types of features (standardized and using the average of representative sub-bands) and the sound source azimuth angle for each frame are shown in the table below.

[0021] Table 1. Extended acoustic features and sound source azimuth angles for each frame From frame 443 to frame 446, the average inter-frame energy fluctuation rapidly increased from 0.19 to 1.38, and the average sub-band energy change rate increased from 0.13 to 1.07. Simultaneously, the source azimuth angle showed a significant jump in frames 445 and 446, reflecting that the interference of truck noise on the angle of arrival estimation had begun to appear. The extended acoustic feature vectors of these six frames will be fed into subsequent steps in chronological order.

[0022] Step S200: Predict the probability of sudden situational changes using a temporal convolutional network with attention. Recently The extended acoustic eigenvectors of each time window are arranged in chronological order to form a temporal feature matrix. ,in Number of time windows To expand the dimension of the acoustic feature vectors, the temporal feature matrix is ​​input into an attention-enabled temporal convolutional network, which outputs the probability of a sudden change in situation occurring within a future time window. and predicted target situation category labels .

[0023] It should be noted that the aforementioned attention-enabled temporal convolutional network includes a group of temporal convolutional layers, attention-weighted layers, and a classification output layer. The temporal convolutional layer group consists of multiple cascaded causal dilation convolutional layers. Each causal dilation convolutional layer performs a one-dimensional causal convolution operation on the temporal feature matrix along the time axis, with the dilation coefficient increasing layer by layer to cover local temporal patterns across different time spans. The feature maps output by each causal dilation convolutional layer are concatenated along the channel dimension to generate a multi-scale temporal feature map. The attention-weighted layer calculates attention weights for each time window position in the multi-scale temporal feature map, and then sums the attention weights with the feature vectors at the corresponding time window positions to generate an aggregated feature vector. Specifically, the attention weights are calculated as follows: for the first time window position in the multi-scale temporal feature map... Feature vectors at each time window position Scalar fractions are obtained after fully connected transformation. The attention weights are obtained by performing softmax normalization on the scalar scores for all time window positions. The aggregated feature vector is ,in For aggregated feature vectors, For the first Attention weights for each time window position, For the first Feature vectors at each time window position Number of time windows The index of the time window position and from Take to The classification output layer passes the aggregated feature vectors through two fully connected sub-layers, one of which is connected to a sigmoid activation function to output the probability value of a sudden change in situation. Another fully connected sublayer is connected to a softmax activation function to output the target situation category label. Target situation category label These are discrete category numbers, corresponding to the predefined situation categories in the parameter configuration table.

[0024] Furthermore, the aforementioned attention-enabled temporal convolutional network is trained using a labeled historical acoustic event dataset as training samples. Each sample contains an extended acoustic feature vector sequence for several time windows prior to a mutation, along with corresponding mutation occurrence labels and situation category labels. The loss function consists of two parts: a binary cross-entropy loss for the mutation probability prediction branch and a multi-class cross-entropy loss for the situation category prediction branch. The weighted sum of the two losses is used as the total loss, and the Adam optimization algorithm is employed to update the weight parameters of the attention-enabled temporal convolutional network.

[0025] In this patrol scenario, the number of time windows for the attention-based temporal convolutional network is set to... The dimension of the extended acoustic feature vector is (Each of the 8 sub-bands has 3 types of features). The temporal feature matrix is ​​constructed using 10 consecutive time windows containing frames 441 to 446. The data is fed into the network for inference. The attention-weighted layer assigns higher weights to frames 443 to 446, which are closer to the current time step, because these frames have more significant energy jump characteristics. The network outputs a corresponding situational change probability value for each frame. and predicted target situation category labels The results are shown in the table below, where the preset protection trigger threshold is... .

[0026] Table 2. Frame-by-frame output results of the attention-based temporal convolutional network. Starting from frame 444, the network has predicted that the situational change category is "sudden strong noise" (category 3), but at this time... Protection has not yet been triggered. (Up to frame 445) Once the protection trigger condition is met for the first time, the linkage control will start immediately.

[0027] Step S300: Execute dual-mode linkage control based on the probability value of sudden situation change. Determine the probability value of sudden change in situation Has the preset protection trigger threshold been reached? ,in .

[0028] when At that time, linear regression was performed on the azimuth data within the sliding window range of the sound source azimuth time series to obtain the estimated angular velocity value. and angular acceleration estimates ,Will and The parameters are used as the sound source orientation motion model parameters; the combination of speech processing parameters corresponding to the current situation category label is used as the working parameters.

[0029] when When this happens, the following linked operations will be executed: Step S301: Store the current sound source azimuth motion model parameters and the latest reliable sound source azimuth angle into the protection snapshot cache, and activate the anomaly detection flag for the subsequently acquired sound source azimuth angle.

[0030] Step S302: Based on target situation category label Retrieve the pre-defined parameter combination from the parameter configuration table. Calculate the current combination of speech processing parameters Combined with pre-set parameters Difference vectors in each dimension ,in For the current combination of speech processing parameters, For the preliminary parameter combination, It is a difference vector; based on the probability value of sudden change in situation. Scaling the difference vector proportionally generates a gradual transition increment. ,in For gradual transition increments, This represents the probability value of a sudden change in situation; the gradual transition increment is superimposed on the current combination of speech processing parameters to generate the transition parameter combination. ,in For the transition parameter combination, load the transition parameter combination as the working parameters.

[0031] It should be noted that the above parameter configuration table is a pre-stored table mapping multiple sets of speech processing parameter combinations to situation category labels. Each set of speech processing parameter combinations includes parameters such as beamforming direction gain coefficient, noise reduction threshold, and echo cancellation step size. Each set of speech processing parameter combinations corresponds to different sound field situation categories, such as stable quiet situation, stable noisy situation, and sudden strong noise situation. The parameter values ​​of each dimension in the parameter configuration table have been uniformly normalized to the same numerical range before being stored to ensure the difference vector... The scaling across all dimensions has a consistent physical meaning.

[0032] It should be noted that the scaling factor for the aforementioned gradual transition increment is directly adopted from the probability value of the sudden change in situation. Its value range is .when Just reached the protection trigger threshold At that time, the scaling factor was small, and the parameters only shifted slightly; as the scaling factor increased in subsequent frames... As the value continues to increase, the scaling factor gradually increases, and the parameters gradually approach the pre-set parameter combination. This process recalculates each frame, causing the parameters to transition frame by frame from the current combination of speech processing parameters to the preparatory combination of parameters.

[0033] When protection is triggered in frame 445, step S301 first freezes the current sound source azimuth motion model parameters to the protection snapshot cache. Previously, linear regression was performed based on the reliable azimuth angle data (47.3°, 47.6°, 47.8°, 48.1°) from frames 441 to 444 to obtain the estimated angular velocity values. degrees / second, estimated angular acceleration degrees / second², latest reliable sound source azimuth angle These three values, along with the timestamp, are stored in the protection snapshot cache.

[0034] In step S302, the current situation is category 1 (stable and quiet situation), and the target situation is category 3 (sudden strong noise situation). The corresponding parameters are retrieved from the parameter configuration table. The current and target values ​​of the three-dimensional normalized parameters are as follows, at frame 445. The incremental transition is calculated as follows: Taking the beamforming direction gain coefficient dimension as an example:

[0035] Table 3 Calculation of progressive transition parameters for frame 445 Up to frame 446, As the value increases to 0.89, the transition working value will further approach the pre-set parameter combination. Taking the beamforming direction gain coefficient as an example: It is very close to the target value of 0.85, and the parameter pre-switching has been basically completed before the actual complete change in the acoustic environment.

[0036] Step S400: In the protected state, perform anomaly detection and selective inclusion of the azimuth angle. With the anomaly detection flag activated, the azimuth angle of each newly acquired sound source is... Perform the following processing: Step S401: Utilize the angular velocity estimate in the protected snapshot cache Angular acceleration estimates And the latest reliable sound source azimuth angle Calculate the model prediction value at the current moment using kinematic formulas. ,in This is the model prediction value at the current moment. For the latest reliable sound source azimuth angle, To protect the angular velocity estimates in the snapshot cache, To protect the angular acceleration estimates in the snapshot cache, This represents the time difference between the current moment and the moment the snapshot was stored. , , The units are degrees, degrees per second, and degrees per second², respectively. The unit is seconds, and all dimensions are consistent.

[0037] Step S402: Calculate the azimuth angle of the sound source Compared with model predictions The residuals between ,in For residuals, This is the azimuth angle of the newly acquired sound source. These are the model's predicted values. The unit is degrees.

[0038] Step S403: Determine the residual Does it exceed the preset abnormal offset threshold? ,in The unit is degrees, and The dimensions are consistent. If The azimuth data is labeled as a noise pollution sample and excluded from the regression window. The predicted azimuth angle is maintained by inertial extrapolation using the sound source azimuth motion model parameters in the protected snapshot cache. If The azimuth data is then included in the regression window, and linear regression is re-executed to update the angular velocity and angular acceleration estimates.

[0039] In this embodiment of the application, to avoid the cumulative deviation from increasing due to excessively long inertial extrapolation time, after the duration of the protection state exceeds the preset maximum protection duration, the abnormal offset threshold is set. The regression window gradually expands frame by frame by increasing the step size, allowing it to gradually incorporate azimuth data that has a large deviation but still has reference value, thereby achieving a balance between protection accuracy and model update capability.

[0040] The parameters stored in the protection snapshot cache are: , degrees / second Degrees per second², snapshot storage time corresponds to the end time of frame 444 (22:47:44.100). Preset abnormal offset threshold. After the protection state is activated, the azimuth angles of the sound source in frames 445 and 446 are 61.7° and 73.2° respectively, both of which are jump values ​​under truck noise interference. Taking frame 446 as an example, The model prediction is calculated as follows:

[0041] The residual is: 24.552° far exceeds The azimuth angle was marked as a noise pollution sample and excluded. From frame 447 to frame 449, the truck noise gradually stabilized, and the azimuth angle gradually returned to the normal range. The abnormal detection results for each frame are shown in the table below.

[0042] Table 4. Detection results of azimuth anomalies in each frame under protected conditions. The azimuth residual of frame 447 is 54.3°, which is 5.378° and does not exceed [the specified value]. The data was included in the regression window, and linear regression was re-performed along with frame 447 based on historical reliable data from frames 441 to 444, updating the angular velocity and angular acceleration estimates. The jump azimuth angles of frames 445 and 446 were completely blocked from the regression window, and the sound source azimuth motion model parameters were not contaminated.

[0043] Step S500: Perform delay compensation forward extrapolation and generate the absolute azimuth angle of the sound source. Obtain the cumulative latency of the voice processing link Using the currently available angular velocity estimates and angular acceleration estimates The latest azimuth angle of the reliable sound source Perform forward extrapolation to calculate the predicted azimuth angle of the sound source after delay compensation. ,in The predicted azimuth angle of the sound source after delay compensation. For the latest reliable sound source azimuth angle, This is the currently valid estimate of angular velocity. The current effective estimate of angular acceleration. This represents the cumulative latency of the voice processing link. The unit is degrees. The unit is degrees per second. The unit is degrees per second². The unit is seconds, and all dimensions are consistent.

[0044] The system acquires continuous triaxial acceleration and triaxial angular velocity sampling sequences from the inertial measurement unit (IMU), and applies a Kalman filter algorithm. The input consists of the triaxial acceleration and angular velocity sampling sequences, and the output is the equipment attitude estimation data (including the equipment's roll, pitch, and yaw angles). Based on the equipment attitude estimation data, the azimuth angle of the sound source is predicted. Perform a coordinate transformation to convert the azimuth angle in the device's body coordinate system to the azimuth angle in the geodetic coordinate system, generating the compensated absolute azimuth angle of the sound source. .

[0045] It should be noted that the currently valid angular velocity and angular acceleration estimates mentioned above, in the unprotected state, are the sound source azimuth motion model parameters obtained by linear regression fitting in step S300; in the protected state, they are the sound source azimuth motion model parameters in the protected snapshot cache or the sound source azimuth motion model parameters updated with reliable azimuth data in step S403. The latest reliable sound source azimuth angle mentioned above, in the unprotected state, is the azimuth angle with the latest timestamp in the sound source azimuth angle time series; in the protected state, it is the azimuth angle that most recently passed the abnormal offset threshold test and was included in the regression window in step S403.

[0046] In this embodiment, to overcome the lag in equipment attitude estimation caused by outdoor gusts, the process noise covariance matrix of the Kalman filter algorithm is dynamically adjusted based on the real-time wind speed data output by the wind speed sensor. Specifically, the current wind speed value output by the wind speed sensor is obtained. ,when Exceeding the preset wind speed threshold At that time, according to the proportional factor The diagonal elements of the noise covariance matrix during amplification, corresponding to the yaw and pitch angles, are: To amplify the scaling factor, This is the current wind speed value. The preset wind speed threshold; This is the dimensionless ratio obtained by dividing units of the same dimension. This is a dimensionless scaling factor that can be directly used for scaling operations on diagonal elements. This adjustment makes the Kalman filter more inclined to trust the angular velocity observations of the inertial measurement unit rather than the predictive model under high wind speed conditions, thereby accelerating the tracking speed of attitude changes caused by wind disturbances.

[0047] After frame 447 is included in the regression window, the angular velocity estimate is updated. degrees / second, estimated angular acceleration degrees / second², latest reliable sound source azimuth angle The cumulative latency of the voice processing link was measured to be... Seconds. The delay compensation extrapolation calculation is as follows:

[0048] At the same time, the wind speed sensor outputs the current wind speed. m / s, preset wind speed threshold meters per second, because Calculate the magnification factor: In the Kalman filtering process, the diagonal elements corresponding to the yaw and pitch angles in the noise covariance matrix are amplified to 2.074 times their original values. The inertial measurement unit output, after Kalman filtering, yields the equipment attitude estimation data: roll angle. Pitch angle Yaw angle .right After performing coordinate transformation, the compensated absolute azimuth angle of the sound source is generated. (North-east direction in the geodetic coordinate system).

[0049] Step S600: Perform voice processing and output wake-up determination result. Adaptive beamforming is applied to the speech signal using the beamforming direction gain coefficient in the current operating parameters to generate a spatially filtered speech signal. Deep learning denoising is then applied to the spatially filtered speech signal using the denoising threshold in the current operating parameters to generate a denoised speech signal. Echo cancellation is then applied to the denoised speech signal using the echo cancellation step size in the current operating parameters, and the processed speech signal is output. The processed speech signal is then sent to the wake-up word detection module for wake-up determination, and the wake-up determination result is output.

[0050] At the corresponding time frame 446, the operating parameters have been gradually switched to a preparatory parameter combination approaching category 3 (sudden strong noise situation) through step S302. Taking the time frame 446 ( Taking [example] as an example, the three-dimensional normalized operating parameters are: beamforming direction gain coefficient 0.80, noise reduction threshold 0.75, and echo cancellation step size 0.69, which are significantly biased towards the parameter configuration for sudden strong noise situations. By frame 448, the truck noise tends to stabilize, and the operator issues the wake-up word "sentinel attention" at 22:47:44.500. At this time, the noise reduction threshold in the operating parameters is 0.75 (normalized), and the beamforming direction gain coefficient is 0.80, both adapted to the strong noise environment. The speech processing link sequentially performs adaptive beamforming, deep learning noise reduction, and echo cancellation. The signal-to-noise ratio of the processed speech signal increases from approximately 11.3 dB to approximately 26.8 dB. The wake-up word detection module's detection confidence for "sentinel attention" is 0.91, exceeding the wake-up judgment threshold of 0.75, and the output wake-up judgment result is passed.

[0051] Step S700: Parse the direction command and generate the gimbal rotation control command. After the wake-up determination is passed, semantic parsing is performed on the voice commands in the processed speech signal to extract the relative direction words and map them to relative angle offsets. ; to compensate for the absolute azimuth angle of the sound source The absolute azimuth of the target is calculated by superimposing the relative angular offset as a reference. ,in The absolute azimuth of the target. To compensate for the absolute azimuth angle of the sound source, This represents the relative angle offset; it also retrieves the current orientation angle of the gimbal. Calculate the angle difference between the target's absolute azimuth and the gimbal's current orientation angle. ,in The difference in angles. The absolute azimuth of the target. The current orientation angle of the gimbal is... The gimbal rotation is used to generate and output gimbal rotation control commands, which include the rotation direction and rotation angle.

[0052] It should be noted that the aforementioned relative direction words are words indicating directional relationships in voice commands, such as "turn left," "turn right," and "straight ahead." The mapping relationship between relative direction words and relative angle offsets is pre-stored in a direction word mapping table; for example, "turn left" corresponds to... "Turn right" corresponds to "Directly in front" corresponds to .

[0053] After successful wake-up, the operator continues to issue the voice command "right front". The semantic parsing module extracts the relative direction word "right front" from the processed voice signal, queries the direction word mapping table, and maps it to a relative angle offset. The absolute azimuth angle of the sound source after compensation. Using this as a reference, calculate the target's absolute azimuth:

[0054] Get the current orientation angle of the gimbal (North-east direction in geodetic coordinate system), calculate the rotation: Generate gimbal rotation control command: rotation direction is clockwise (to the right), rotation angle is 29.2°. After the gimbal rotates according to the command, the lens faces the direction of 112.6° north of east in the geodetic coordinate system, that is, 45° to the right front of the operator, completing the image capture of the corresponding area.

[0055] Step S800: Deactivate protection and perform online model update. In the probability value of sudden change in situation Multiple consecutive frames falling back to the protection trigger threshold Following this, based on the newly acquired speech signal, the situational change probability prediction in step S200 is re-executed to confirm that the situational change probability value remains below [a certain value]. The environment is determined to have returned to a stable state. The anomaly detection flag is removed and the protection snapshot cache is cleared, allowing the regression window to re-execute linear regression to fit the angular velocity and angular acceleration estimates using the newly acquired reliable azimuth data.

[0056] Based on step S800, the predicted target situation category label output by the attention-enabled temporal convolutional network during the protection period is compared with the actual confirmed situation category label after the protection is lifted. The comparison result and the corresponding temporal feature matrix are used as incremental training samples. The weight parameters of the attention-enabled temporal convolutional network are updated according to the online incremental learning rule, so that the prediction accuracy of the attention-enabled temporal convolutional network for this type of acoustic mutation mode is gradually improved.

[0057] Based on step S800, the intelligibility score of the processed speech signal during the protection period is obtained, the intelligibility score is associated with the corresponding situation category label, and the parameter values ​​of each dimension of the corresponding entry in the parameter configuration table are fine-tuned online so that the speech processing parameter combination corresponding to each situation category in the parameter configuration table gradually adapts to the actual sound field conditions.

[0058] The truck noise gradually subsided after 22:47:45.800, and the probability value of the sudden situation change from frame 453 to frame 458 was... Six consecutive frames below The system confirmed that the environment had returned to a stable state, and at 22:47:46.400, the anomaly detection flag was removed and the protection snapshot cache was cleared. During the protection period (frames 445 to 452), the target situation category label predicted by the network was all category 3 (sudden strong noise), which was consistent with the actual confirmed situation category label after the protection was lifted, and the comparison result showed that the prediction was correct. The temporal feature matrix of this segment and the correct label were used as incremental training samples to update the network weight parameters according to the online incremental learning rules. At the same time, the mean intelligibility score of the processed speech signal during the protection period (frames 448 to 452) was extracted, which was 0.83. After associating it with the category 3 label, the noise reduction threshold dimension of the corresponding entry for category 3 in the parameter configuration table was fine-tuned online, and the normalization value was adjusted from 0.79 to 0.81, so that this dimension parameter is more in line with the actual large vehicle noise spectrum characteristics of the factory area.

[0059] In the entire processing flow, the data starts from the raw speech frames and azimuth time-series data collected in step S100. Step S200 uses an attention-based temporal convolutional network to extract the probability value of situational abrupt changes and the target situational category label. In step S300, azimuth protection snapshot storage and progressive transition of speech processing parameters are triggered simultaneously. In step S400, kinematic parameters in the snapshot cache are used to perform residual checks on subsequent azimuth frames to block contaminated samples from the regression window. In step S500, delay compensation is performed based on reliable azimuth data, and forward extrapolation is combined with wind speed adaptive Kalman filtering to generate the compensated absolute azimuth of the sound source. In step S600, speech processing is performed using the working parameters that have completed the progressive transition, and wake-up determination is completed. In step S700, the absolute azimuth and semantic parsing results are combined into a gimbal rotation control command. In step S800, after environmental recovery, the network and parameter configuration table are updated online using the event data. Data is transferred sequentially between each step, maintaining complete logical coherence and jointly supporting the voice wake-up and direction control capabilities of the control sphere during the acoustic abrupt change transition period.

[0060] Technical effects of this embodiment This implementation integrates the situational change probability value output by the attention-based temporal convolutional network into the data admission control and gradual transition of speech processing parameters of the sound source orientation motion model, thereby coordinating orientation protection and parameter pre-switching before actual changes in the acoustic environment.

[0061] At the azimuth tracking level, because a protective snapshot cache and a residual-based selective data inclusion strategy are adopted, when the probability value of a sudden change in the situation reaches the protection trigger threshold, the current parameters of the sound source azimuth motion model are immediately frozen in the protective snapshot cache. Subsequent newly acquired azimuth angle data must pass the residual test before being included in the regression window. Therefore, large jumps in azimuth angles caused by noise are blocked outside the regression window, and the estimated values ​​of angular velocity and angular acceleration are not contaminated. Inertial extrapolation maintains the prediction output based on the reliable sound source azimuth motion model parameters before the change, thereby overcoming the multi-cycle systematic tracking offset caused by the unconditional inclusion of abnormal azimuth angles in the regression window.

[0062] At the speech processing level, because a progressive transition increment method based on the probability value of situational change is adopted to update the working parameters, when the attention-based temporal convolutional network predicts the trend of situational change, the speech processing parameters begin to transition from the current speech processing parameter combination to the preparatory parameter combination frame by frame. Therefore, when the actual acoustic environment changes, the speech processing parameter combination has been partially or completely switched to a state that matches the new situation. This overcomes the abrupt jump in beamforming direction and the mismatch of noise reduction threshold during the multi-frame delay caused by post-parameter switching, and ensures the continuous availability of wake word detection during the transition period of the change.

[0063] At the equipment attitude estimation level, because the process noise covariance matrix of the Kalman filter algorithm is dynamically adjusted according to real-time wind speed data, the process noise in the yaw and pitch directions is increased under high wind speed conditions. Therefore, the Kalman filter algorithm can follow the attitude changes caused by wind disturbances faster, thereby overcoming the deviation in azimuth coordinate transformation caused by the lag in equipment attitude estimation under outdoor gust conditions.

[0064] By combining the synergistic effects of the above three levels, this implementation method enables the control ball to maintain both voice wake-up response capability and directional control accuracy during the transition period of abrupt changes in the acoustic environment, thus solving the problem of the coupling failure of voice wake-up and directional control during the transition period.

[0065] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A method for voice wake-up and control of a controlled ball, characterized in that, It includes at least steps A and B, wherein step A includes: Extended acoustic feature vectors are extracted from the speech signals acquired by the dual-microphone differential array, and the time series of the sound source azimuth angle is calculated. The extended acoustic feature vectors are input into a temporal convolutional network with attention, and the output is the probability value of the situation change and the target situation category label. Step B includes: When the probability value of the situational change reaches the protection trigger threshold, the sound source azimuth motion model parameters and the latest reliable sound source azimuth angle are stored in the protection snapshot cache. Anomaly detection and selective inclusion are performed on subsequent azimuth angles based on residuals. Based on the probability value of the situational change, the current speech processing parameter combination is progressively transitioned to the preparatory parameter combination frame by frame to generate working parameters. The sound source absolute azimuth angle is generated by using the currently effective angular velocity estimate and angular acceleration estimate to perform delay compensation and forward extrapolation. The speech signal is processed with the working parameters and a wake-up determination is performed. After the wake-up is successful, the relative direction words are converted into gimbal rotation control commands based on the sound source absolute azimuth angle.

2. The method for voice wake-up and control of a control ball as described in claim 1, wherein, The generation of the extended acoustic feature vector includes: The energy values ​​of each frequency sub-band after short-time Fourier transform of each frame of speech signal constitute the spectral energy distribution feature; the absolute values ​​of the differences between the energy values ​​of each frequency sub-band between the current frame and the previous frame constitute the inter-frame energy fluctuation feature; the logarithm of the ratio of the energy value of the current frame to the energy value of the previous frame in each frequency sub-band constitutes the sub-band energy change rate feature; the spectral energy distribution feature, the inter-frame energy fluctuation feature, and the sub-band energy change rate feature are respectively Z-score normalized and then concatenated to generate the extended acoustic feature vector.

3. The method for voice wake-up and control of a control ball as described in claim 1, wherein, The attention-enabled temporal convolutional network includes a group of temporal convolutional layers, an attention-weighted layer, and a classification output layer; The temporal convolutional layer group is composed of multiple cascaded causal dilated convolutional layers with the dilation coefficient increasing layer by layer. The feature maps output by each layer are spliced ​​along the channel dimension to generate multi-scale temporal feature maps. The attention weighting layer performs a fully connected transformation on the feature vectors at each time window position in the multi-scale temporal feature map to obtain scalar scores, performs softmax normalization on all scalar scores to obtain attention weights, and weights the attention weights and corresponding feature vectors to generate aggregated feature vectors. The classification output layer passes the aggregated feature vector through two fully connected sub-layers. One fully connected sub-layer is connected to a sigmoid activation function to output the probability value of the situational change, and the other fully connected sub-layer is connected to a softmax activation function to output the target situational category label.

4. The method for voice wake-up and control of a control ball as described in claim 3, wherein, The training of the attention-enabled temporal convolutional network includes: The labeled historical acoustic event dataset is used as training samples. Each sample contains an extended acoustic feature vector sequence of several time windows before the mutation occurs, as well as the corresponding mutation occurrence label and situation category label. The loss function is composed of the weighted sum of the binary cross-entropy loss of the mutation probability prediction branch and the multi-class cross-entropy loss of the situation category prediction branch. The Adam optimization algorithm is used to update the weight parameters of the attention-enabled temporal convolutional network.

5. The method for voice wake-up and control of a control ball as described in claim 1, wherein, The residual execution anomaly detection and selective inclusion includes: Starting with the latest azimuth angle of the trusted sound source in the protected snapshot cache, the model prediction value at the current moment is obtained by superimposing the product of the estimated angular velocity and the time difference, and half of the product of the estimated angular acceleration and the square of the time difference. The absolute value of the difference between the newly acquired sound source azimuth angle and the model prediction value is calculated as the residual; Determine whether the residual exceeds the preset abnormal offset threshold. If it does, exclude the azimuth data from the regression window and maintain the predicted sound source azimuth by inertial extrapolation. If it does not exceed the threshold, include the azimuth data in the regression window and re-execute linear regression to update the angular velocity estimate and angular acceleration estimate.

6. The method for voice wake-up and control of a control ball as described in claim 5, wherein, After the protection state lasts for more than the preset maximum protection time, the abnormal offset threshold is widened frame by frame according to the preset incremental step size, so that the regression window gradually accepts azimuth data with increased deviation but still passing the threshold test after widening.

7. The method for voice wake-up and control of a control ball as described in claim 1, wherein, The step of generating working parameters frame-by-frame progressively from the current speech processing parameter combination to the preliminary parameter combination based on the probability value of the sudden change in the situation includes: Based on the target situation category label, retrieve the preliminary parameter combination from the parameter configuration table, calculate the difference vector of each dimension between the current speech processing parameter combination and the preliminary parameter combination; use the situation change probability value as a scaling factor to scale the difference vector proportionally to generate a gradual transition increment, and superimpose the gradual transition increment onto the current speech processing parameter combination to generate a transition parameter combination and load it as working parameters; The parameter configuration table is a pre-stored table of correspondence between multiple sets of speech processing parameter combinations and situation category labels. Each set of speech processing parameter combinations includes beamforming direction gain coefficient, noise reduction threshold and echo cancellation step size. The parameter values ​​of each dimension have been uniformly normalized to the same numerical range before being stored.

8. The method for voice wake-up and control of a control ball as described in claim 1, wherein, The generation of the absolute azimuth angle of the sound source also includes: The three-axis acceleration and three-axis angular velocity sampling sequences output by the inertial measurement unit are acquired, and the Kalman filter algorithm is applied to output the device attitude estimation data. Based on the device attitude estimation data, the coordinate transformation is performed on the predicted sound source azimuth angle to convert the azimuth angle in the device body coordinate system to the azimuth angle in the geodetic coordinate system to generate the compensated absolute azimuth angle of the sound source. The current wind speed value output by the wind speed sensor is obtained. When the current wind speed value exceeds a preset wind speed threshold, the square of the ratio of the current wind speed value to the wind speed threshold is used as an amplification factor to amplify the diagonal elements in the process noise covariance matrix of the Kalman filter algorithm that correspond to the yaw angle and pitch angle.

9. The method for voice wake-up and control of a control ball as described in claim 1, wherein, Also includes: After the probability value of the sudden change in the situation falls below the protection trigger threshold for several consecutive frames, the anomaly detection flag is removed and the protection snapshot cache is cleared, allowing the regression window to fully use the newly acquired reliable azimuth angle data to re-execute the linear regression fitting of the angular velocity estimate and the angular acceleration estimate. The target situation category label output by the attention-enabled temporal convolutional network during the protection period is compared with the actual confirmed situation category label after the protection is lifted. The comparison result and the corresponding temporal feature matrix are used as incremental training samples to update the weight parameters of the attention-enabled temporal convolutional network.

10. A control system for a telescopic ball system, used to execute the telescopic ball system voice wake-up and control method according to any one of claims 1 to 9, characterized in that, include: The signal acquisition and feature extraction module is used to extract extended acoustic feature vectors from the speech signals acquired by the dual-microphone differential array and calculate the time series of the sound source azimuth angle. The situation prediction module is used to input the extended acoustic feature vector into an attention-enabled temporal convolutional network and output the situation change probability value and the target situation category label. The dual-mode linkage control module is used to store the sound source azimuth motion model parameters and the latest reliable sound source azimuth angle into the protection snapshot cache when the probability value of the situation change reaches the protection trigger threshold. It performs anomaly detection and selective inclusion on subsequent azimuth angles based on residuals, and generates working parameters by gradually transitioning the current voice processing parameter combination to the preparatory parameter combination frame by frame based on the probability value of the situation change. The azimuth extrapolation module is used to perform delay-compensated forward extrapolation using the currently valid angular velocity and angular acceleration estimates to generate the absolute azimuth of the sound source. The voice processing and wake-up module is used to process voice signals according to the aforementioned operating parameters and perform wake-up determination; The instruction generation module is used to convert relative direction words into gimbal rotation control instructions based on the absolute azimuth angle of the sound source after the wake-up is successful.

Images