A robot steering control method and system based on multi-dimensional sound source confidence evaluation

Abstract

The application relates to the technical field of intelligent control, in particular to a robot steering control method and system based on multi-dimensional sound source confidence evaluation, the method extracts feature confidence of multiple dimensions including a voiceprint matching dimension, obtains comprehensive confidence through weighted fusion, and only executes steering when the comprehensive confidence meets a threshold condition, the voiceprint matching is introduced into the confidence evaluation system as a necessary dimension, sound source identity information directly participates in the steering decision, the existing passive mode of triggering steering as energy triggers is upgraded to an active mode of triggering steering after identity identification, the problem of missteering caused by lack of identity recognition ability is eliminated, and the accuracy of locking a target user and the interaction continuity of the robot in a noisy environment are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology, and in particular to a robot steering control method and system based on multidimensional sound source confidence assessment. Background Technology

[0002] In the field of human-computer interaction for intelligent robots, auditory perception is the key entry point for realizing natural human-computer dialogue. Enabling robots to have an auditory attention mechanism similar to the human "cocktail party effect"—that is, to effectively capture the voice of the target speaker and respond in a complex acoustic environment where multiple people are talking and background noise coexists—is one of the core technologies for improving the interactive experience of service robots.

[0003] Currently, existing technologies for robot sound source localization and steering control primarily employ microphone array signal processing techniques. These algorithms calculate the azimuth angle of the sound source relative to the robot using algorithms such as Generalized Cross-Correlation (GCC), Generalized Cross-Correlation-Phase Transform (GCC-PHAT), or Beamforming. The basic decision-making logic is as follows: First, Voice Activity Detection (VAD) is used to determine if there is human voice activity exceeding a preset energy threshold. When the threshold is triggered, the system calculates the sound source azimuth angle and drives the actuator to turn the robot towards the sound source. This technical solution has been validated and applied in laboratory or quiet indoor environments and is currently the mainstream method for achieving auditory steering in the service robot field.

[0004] However, the aforementioned existing technical solutions have serious drawbacks when applied to a home kitchen environment. First, kitchens typically contain a range hood, a continuous source of strong noise, with operating noise levels reaching 65dB to 70dB or even higher. This noise has a wide spectral coverage and its energy is concentrated in the fundamental frequency and formant band of human voice, severely masking the speech signals collected by the microphone array. This makes it difficult for existing VAD algorithms to accurately distinguish between the range hood's noise and user voice commands. The robot may become "deaf" due to the masked speech, or it may misinterpret noise as valid speech, triggering meaningless turns. Second, kitchens also contain numerous transient strong interference events, such as ignition sounds, exhaust sounds, oil popping sounds, and the clattering of pots and pans. These have extremely high peak energy and are easily misinterpreted as valid commands by energy threshold mechanisms, causing the robot to frequently turn randomly and failing to establish a stable interactive posture. More importantly, the decision-making logic of existing technologies is entirely based on the coordination of acoustic energy threshold triggering and geometric orientation calculation. It lacks the ability to distinguish the speaker's identity. When multiple people are active at the same time or media devices are playing, once a non-target person speaks or the background sound pressure is loud enough, the robot will interrupt the current task and turn to the interference sound source. This lack of identity recognition makes it impossible for the robot to focus on the target user and makes it difficult to meet the needs of refined human-computer interaction in complex family scenarios. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a robot steering control method and system based on multidimensional sound source confidence assessment.

[0006] In a first aspect, embodiments of the present invention provide a robot steering control method based on multidimensional sound source confidence assessment, the method comprising: The system acquires ambient audio signals collected by the microphone array and preprocesses these signals to determine valid speech segments. Sound source localization is performed based on effective speech segments to obtain the location information of the sound source relative to the robot; Extract feature confidence scores from multiple dimensions of valid speech segments, with at least one dimension including speaker matching. The confidence scores of features from multiple dimensions are fused and calculated to obtain the comprehensive sound source confidence score; If the overall sound source confidence level meets the preset turning conditions, the robot is controlled to perform a turning action based on the orientation information.

[0007] In conjunction with the first aspect, the steps of preprocessing the environmental audio signal to determine the valid speech segments include: The ambient audio signal is subjected to dereverberation and noise reduction processing to obtain a noise-conditioned audio signal; Speech activity detection is performed on the noise-reduced audio signal, the signal segment corresponding to human voice activity is extracted from the noise-reduced audio signal, and the extracted signal segment is determined as the valid speech segment.

[0008] In conjunction with the first aspect, the steps of localizing the sound source based on effective speech segments and obtaining the sound source's orientation information relative to the robot include: A localization algorithm based on time delay estimation is used to calculate the time difference of arrival between each microphone in the microphone array for effective speech segments; Based on the time difference of arrival and the geometry of the microphone array, the azimuth angle of the sound source relative to the robot is calculated.

[0009] In addition to the first aspect, the multiple dimensions also include: speech effectiveness dimension, spatial stability dimension, and signal-to-noise ratio quality dimension; The steps for extracting feature confidence scores from multiple dimensions of valid speech segments, including at least the speaker matching dimension, include: The Mel frequency cepstral coefficients of the effective speech segments are input into the classifier for processing to obtain the feature confidence of the speech validity dimension. Based on the degree of fluctuation of location information over time, the feature confidence of the spatial stability dimension is obtained; The signal-to-noise ratio of the current speech frame is mapped to a quality score, and the feature confidence corresponding to the signal-to-noise ratio quality dimension is obtained. Extract the voiceprint feature vector of the current speech segment, and compare the similarity between the voiceprint feature vector and the pre-stored target user voiceprint template to obtain the feature confidence corresponding to the voiceprint matching dimension.

[0010] Combining the first aspect, the steps for fusing and calculating the feature confidence scores from multiple dimensions to obtain the comprehensive sound source confidence score include: The feature confidence scores for the speech validity dimension, spatial stability dimension, signal-to-noise ratio quality dimension, and speaker matching dimension are normalized to obtain the normalized confidence scores for each dimension. The normalized confidence scores corresponding to each dimension are fused using a weighted summation method to obtain the comprehensive sound source confidence score.

[0011] In conjunction with the first aspect, the steps for controlling the robot to perform turning actions based on orientation information include: The steering control quantity is generated based on the orientation information, and a smooth control algorithm is used to process the steering control quantity to generate a smooth steering command. Drive the motors according to the smooth steering command to make the robot rotate to the target direction.

[0012] In conjunction with the first aspect, after controlling the robot to perform a turning action based on the orientation information, it also includes: The camera collects the actual orientation information after turning, and calculates the angle error between the actual orientation information and the orientation information. Record and statistically analyze the angular errors generated by multiple turns, and adaptively compensate and calibrate the azimuth information obtained from subsequent sound source localization based on the statistical results.

[0013] Secondly, this application also provides a robot steering control system based on multi-dimensional sound source confidence assessment, applied to desktop robots, including: The audio acquisition module is used to acquire ambient audio signals. The audio processing module is used to locate the sound source of the environmental audio signal to obtain the location information of the sound source relative to the robot, and extract feature confidence scores in multiple dimensions, including voiceprint features. The main control module, connected to the audio processing module, is used to execute the method as described in any one of claims 1-7, to perform fusion calculation on feature confidence in multiple dimensions to obtain comprehensive sound source confidence, and to generate a steering control command based on directional information when the comprehensive sound source confidence meets the preset steering conditions. The execution drive module, connected to the main control module, is used to respond to steering control commands and drive the robot to perform steering actions.

[0014] In conjunction with the second aspect, the audio processing module includes: A speech activity detection unit is used to separate valid speech segments from environmental audio signals; The sound source localization unit is used to calculate the azimuth angle of the sound source relative to the robot based on the effective speech segment; The voiceprint feature extraction unit is used to extract the voiceprint feature vector of the effective speech segment; The feature scoring unit is used to calculate the feature confidence scores for the speech validity dimension, spatial stability dimension, signal-to-noise ratio quality dimension, and voiceprint matching dimension, respectively.

[0015] In conjunction with the second aspect, it also includes: The camera module is connected to the main control module; the camera module is used to collect actual orientation information after the robot performs a turning action and feed it back to the main control module. The main control module is also used to calculate the angle error based on the actual azimuth information and the azimuth information, and to adaptively calibrate the subsequent sound source localization results based on the statistical multiple angle errors.

[0016] Thirdly, this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor runs the computer program to cause the electronic device to perform the above-described method.

[0017] Fourthly, this application provides a readable storage medium storing computer program instructions, which are read and executed by a processor to perform the above-described method.

[0018] The embodiments of the present invention bring the following beneficial effects: The present invention provides a robot steering control method and system based on multi-dimensional sound source confidence assessment. The method includes: acquiring environmental audio signals collected by a microphone array and preprocessing the environmental audio signals to determine effective speech segments; locating sound sources based on the effective speech segments to obtain the directional information of the sound sources relative to the robot; extracting feature confidence scores of multiple dimensions of the effective speech segments, wherein the multiple dimensions include at least a voiceprint matching dimension; fusing and calculating the feature confidence scores of the multiple dimensions to obtain a comprehensive sound source confidence score; and controlling the robot to perform a steering action based on the directional information if the comprehensive sound source confidence score meets preset steering conditions.

[0019] This invention extracts feature confidence scores from multiple dimensions, including voiceprint matching, and then weights and fuses them to obtain a comprehensive confidence score. Turning is only performed when the comprehensive confidence score meets a threshold condition. By introducing voiceprint matching as a necessary dimension into the confidence score evaluation system, the voice source identity information directly participates in the turning decision. This upgrades the existing passive mode of energy-triggered turning to an active mode where turning is triggered only after identity verification is passed. This eliminates the problem of mis-turning caused by a lack of identity recognition capabilities and improves the accuracy and interaction consistency of the robot in locking onto the target user in noisy environments.

[0020] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0021] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0022] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 A flowchart of a robot steering control method based on multidimensional sound source confidence assessment provided in an embodiment of the present invention; Figure 2 A schematic diagram of a robot steering control system based on multidimensional sound source confidence assessment is provided for an embodiment of the present invention; Figure 3 This is a schematic diagram of the electronic device structure provided in an embodiment of the present invention.

[0024] Figure label: 10 - Audio acquisition module, 20 - Audio processing module, 30 - Main control module, 40 - Execution driver module; 130 - Processor, 131 - Memory, 132 - Bus, 133 - Communication interface. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] To facilitate understanding of this embodiment, the technical terms used in this application will be briefly introduced below.

[0027] Voice Activity Detection (VAD) is a technique that identifies and extracts signal segments corresponding to human voice activity from environmental audio signals. VAD determines whether the current signal segment contains human voice by analyzing multi-dimensional acoustic features such as short-time energy, zero-crossing rate, and spectral entropy of the audio signal.

[0028] Time Difference of Arrival (TDOA): This refers to the time difference between the arrival times of a sound emitted from the same sound source at different microphones in a microphone array. Since the speed of sound in air is constant, the time it takes for each microphone in the array to receive the signal from the same sound source varies due to their different spatial locations. This difference contains information about the location of the sound source.

[0029] Feature confidence: refers to the quantitative evaluation of the credibility of a valid speech segment in a certain dimension, expressed as a value between 0 and 1.

[0030] Voiceprint: refers to the vocal characteristics that carry the speaker's identity information. Based on the physiological differences of human vocal organs (such as the shape of the vocal tract and the characteristics of the vocal cords) and the differences in pronunciation habits formed after birth, each speaker's voiceprint is unique.

[0031] PID control algorithm: refers to the proportional-integral-derivative (PID) control algorithm, a classic closed-loop control method.

[0032] Field of View (FOV): refers to the range of view that a camera can capture, usually expressed in angles.

[0033] After introducing the technical terms used in this application, the application scenarios and design concepts of the embodiments of this application will be briefly described below.

[0034] Existing technologies lack the ability to identify the source of sound and rely solely on sound energy to trigger turning. This causes robots to frequently turn incorrectly when multiple people are talking or when there is background noise interference, making it impossible to lock onto the target user and affecting the continuity of interaction.

[0035] Based on this, this application provides a robot steering control method and system based on multi-dimensional sound source confidence assessment.

[0036] Example 1 This application provides a robot steering control method based on multi-dimensional sound source confidence assessment, combined with... Figure 1 As shown, the method includes: S110: Acquire the ambient audio signal collected by the microphone array and preprocess the ambient audio signal to determine the valid speech segment.

[0037] S120 performs sound source localization based on effective speech segments to obtain the location information of the sound source relative to the robot.

[0038] S130, extract feature confidence scores of multiple dimensions of the effective speech segment, including at least the voiceprint matching dimension.

[0039] S140 calculates the confidence scores of features from multiple dimensions to obtain the comprehensive sound source confidence score.

[0040] S150, if the confidence level of the integrated sound source meets the preset turning conditions, control the robot to perform turning actions based on the orientation information.

[0041] This invention introduces voiceprint matching as a necessary dimension into the confidence assessment system. Based on the voice source matching dimension, it confirms whether the voice is from a user, enabling the robot to directly obtain the identity information of the voice source during the turning decision-making process. In this way, the voice signals of non-target users will be rejected because the confidence of the voiceprint matching dimension is low, resulting in the overall confidence not meeting the turning conditions. This upgrades the passive response mode of turning based on energy triggering to an active decision-making mode that can only trigger turning after identity verification. It eliminates the problem of mis-turning caused by the lack of identity recognition ability from the root of the decision-making process, and improves the accuracy of the robot in locking onto the target user and the consistency of interaction in noisy environments.

[0042] In step S110, ambient audio signals are first acquired in real time using a microphone array configured on the robot. The microphone array can be any configuration of a linear array, a circular array, or a stereo array. Each microphone channel is sampled synchronously to preserve the phase difference information between the channels, providing a data basis for subsequent sound source localization.

[0043] In conjunction with the first aspect, step S110 involves preprocessing the environmental audio signal to determine valid speech segments, specifically including: S111 performs dreverberation and noise reduction processing on the ambient audio signal to obtain a noise-conditioned audio signal.

[0044] S112, performs speech activity detection on the noise-reduced audio signal, extracts the signal segment corresponding to human voice activity from the noise-reduced audio signal, and determines the extracted signal segment as a valid speech segment.

[0045] Due to the presence of broadband noise from the range hood, reverberation, and sudden interference in the kitchen environment, the raw acquired signals cannot be directly used for sound source localization and feature extraction. Therefore, step S110 further performs preprocessing. The first stage of preprocessing is acoustic channel compensation, specifically employing dereverberation. This involves using a deconvolution algorithm to suppress late-stage reverberation components caused by highly reflective surfaces such as walls, cabinets, and stainless steel appliances, reducing the interference of multipath effects on time delay estimation. Simultaneously, denoising algorithms such as spectral subtraction or Wiener filtering are used to suppress steady-state background noise, improving the overall signal purity. The order of dereverberation and denoising can be flexibly adjusted based on the actual microphone array configuration and computing resources.

[0046] The second stage of preprocessing is Voice Activity Detection (VAD). The noise-reduced and conditioned audio signal is fed into the VAD module, which extracts multi-dimensional features such as energy characteristics, zero-crossing rate, and spectral entropy. Combined with a short-time energy threshold and a deep learning classifier, it segments effective speech segments corresponding to human voice activity from the continuous audio stream. The output of VAD is a timestamped speech segment, serving as effective input for subsequent sound source localization and feature extraction. Unlike traditional solutions that rely solely on a fixed energy threshold, the VAD determination result in this invention is further verified by subsequent multi-dimensional confidence assessments. That is, low-confidence VAD triggers will be effectively suppressed in subsequent steps, forming a dual discrimination mechanism that combines pre-detection and post-verification.

[0047] The above preprocessing is used to extract clean, continuous, and effective speech segments from a noisy kitchen environment.

[0048] In conjunction with the first aspect, step S120 includes: S121 employs a positioning algorithm based on time delay estimation to calculate the arrival time difference between each microphone in the microphone array for the effective voice segment.

[0049] S122, based on the time difference of arrival and the geometry of the microphone array, calculate the azimuth angle of the sound source relative to the robot.

[0050] In step S121, the system employs a localization algorithm based on Time Delay Estimation (TDE) to calculate the Time Difference of Arrival (TDOA) between the effective speech segments arriving at each microphone in the microphone array. When microphones at different spatial locations in the microphone array receive signals from the same sound source, there is a time delay between the signals in each channel due to the different propagation distances of the sound waves. This delay contains information about the location of the sound source.

[0051] In a preferred embodiment of the present invention, the Generalized Cross-Correlation-Phase Transform (GCC-PHAT) algorithm is used for time delay estimation. This algorithm first calculates the cross-power spectrum between the two microphone signals, then whitens the cross-power spectrum using a phase transformation weight function, normalizing the amplitude information while retaining only the phase information. Finally, an inverse Fourier transform is performed to obtain the generalized cross-correlation function, and the time index corresponding to the peak position is the arrival time difference between the two microphones. The GCC-PHAT algorithm outperforms the traditional cross-correlation algorithm in time delay estimation under low signal-to-noise ratio and reverberation environments because the phase transformation weight function effectively suppresses the interference of noise and reverberation components in the amplitude spectrum on peak detection, making the peak of the cross-correlation function sharper and improving noise resistance. If the microphone array contains more than three microphones, the system needs to calculate the TDOA between each pair of microphones, forming multiple sets of time delay observations to provide redundant information for subsequent azimuth calculation.

[0052] In step S122, based on the time difference of arrival (TDOA) calculated in sub-step S121 and the geometric configuration of the microphone array, the azimuth angle of the sound source relative to the robot is calculated. The geometric configuration of the microphone array (including parameters such as the spacing and arrangement of each array element) is pre-calibrated and stored in the system, and together with the TDOA observations, it constitutes the geometric constraint for sound source localization. For circular or linear arrays, the system converts the TDOA into the distance difference between the sound source and each microphone, establishes a hyperbolic equation system, and calculates the azimuth angle θ of the sound source relative to the array center using a geometric localization algorithm (such as the least squares method or a direction of arrival estimation algorithm based on the far-field assumption). The range of this azimuth angle θ is typically 0° to 360° or -180° to 180°, representing the angular offset of the sound source relative to the positive direction of the robot on the horizontal plane.

[0053] In conjunction with the first aspect, multiple dimensions also include: speech effectiveness dimension, spatial stability dimension, and signal-to-noise ratio quality dimension. Step S130 includes: S131, input the Mel frequency cepstral coefficients of the effective speech segment into the classifier for processing to obtain the feature confidence of the speech validity dimension.

[0054] S132, based on the degree of fluctuation of orientation information in the time series, the feature confidence of the spatial stability dimension is obtained.

[0055] S133, map the signal-to-noise ratio of the current speech frame to a quality score to obtain the feature confidence corresponding to the signal-to-noise ratio quality dimension.

[0056] S134, extract the voiceprint feature vector of the current speech segment, and compare the similarity between the voiceprint feature vector and the pre-stored target user voiceprint template to obtain the feature confidence corresponding to the voiceprint matching dimension.

[0057] In step S131, the system extracts the speech validity features of the valid speech segments to obtain the feature confidence scores of the speech validity dimension. This represents the probability that the current signal segment is a valid human voice.

[0058] Specifically, the system first extracts the Mel-Frequency Cepstral Coefficients (MFCCs) of the effective speech segments. MFCC features simulate the characteristics of human hearing, compress and represent the spectral envelope of the speech signal, and can effectively distinguish speech from various types of environmental noise.

[0059] The extracted MFCC feature vectors are then fed into a pre-trained speech or non-speech classifier, which can be a Support Vector Machine (SVM) or a lightweight neural network (such as a convolutional neural network or a recurrent neural network). The classifier outputs a probability value between 0 and 1, indicating the degree of confidence that the current signal belongs to a valid human voice.

[0060] In a kitchen environment, the MFCC distribution of broadband noise from a range hood differs significantly from that of speech. A classifier uses this difference to differentiate between human voice segments and noise segments. The system can directly verify the output of Voice Activity Detection (VAD) during the confidence fusion stage, low Even if the scored signal segment passes the front-end VAD detection, it will not obtain high confidence in the subsequent fusion, thus realizing a dual discrimination mechanism that combines front-end detection and post-end verification.

[0061] In step S132, the system extracts the spatial stability features of the sound source corresponding to the effective speech segment, and obtains the feature confidence level of the spatial stability dimension. This is used to characterize whether the location of the sound source remains stable over time.

[0062] Specifically, the system constructs a sliding time window to store the azimuth angles calculated for the current frame and the previous N frames (such as the previous 10 or 20 frames). The system calculates the deviation between the azimuth angle of the current frame and the average azimuth angle of the previous N frames. The smaller the deviation, the more stable the sound source's position in space, and the higher the feature confidence. Conversely, if the sound source's position changes frequently, it is judged as unstable, and the feature confidence decreases accordingly. This dimension aims to address location jumps caused by reverberation, multipath effects, or sudden transient interference in a kitchen environment. For example, although the sizzling sound of oil while stir-frying has high peak energy, its duration is extremely short and its orientation is discontinuous in time, making it difficult to maintain a stable azimuth angle over multiple consecutive frames. This is addressed through the spatial stability dimension. According to the assessment, such transient disturbances will obtain low feature confidence, thus being effectively suppressed in the comprehensive confidence fusion.

[0063] In step S133, feature confidence scores for the signal-to-noise ratio quality dimension are extracted. This is used to characterize the clarity of the current speech signal.

[0064] Specifically, the system estimates the signal-to-noise ratio (SNR) of the current speech frame. This estimation can be achieved using algorithms based on noise power spectrum estimation (such as minimum tracking or speech presence probability estimation algorithms) to estimate the background noise power spectrum in real time and calculate the ratio of signal power to noise power. Since the original SNR values ​​are relatively large, the system maps them to a quality score between 0 and 1 using a nonlinear mapping function (such as a sigmoid function or a piecewise linear function). A higher SNR results in higher confidence of the mapped features. In the kitchen scenario, this dimension's physical significance lies in the fact that even if the range hood noise fails to completely mask human voices, low SNR conditions can still lead to a decrease in the accuracy of subsequent voiceprint feature extraction. Through quantitative evaluation of the SNR quality dimension, the system can suppress such low-quality signals during the fusion stage, avoiding high-confidence switching decisions on unclear speech.

[0065] In step S134, the system extracts the voiceprint matching features of the effective speech segments to obtain the feature confidence of the voiceprint matching dimension. This dimension indicates whether the current speaker is a target registered user. For the first time, identity information enters the robot's steering decision-making process as a continuous variable, rather than as a separate binary access control switch in traditional solutions.

[0066] In practical implementation, the system first extracts the voiceprint feature vector of the current speech segment. Voiceprint feature extraction can employ algorithms such as identity vector-based (i-vector) or cross-entropy vector-based (x-vector) from deep neural networks. i-vector compresses speaker information into a low-dimensional space through factor analysis, while x-vector extracts deep speaker embedding features through a time-delay neural network (TDNN). In short speech scenarios, x-vector typically outperforms i-vector in recognition accuracy. The extracted voiceprint feature vector is then compared with the target user's voiceprint template pre-stored locally. Similarity calculation can use cosine similarity or probabilistic linear discriminant analysis (PLDA) scoring. Cosine similarity calculates the cosine of the angle between two vectors, ranging from -1 to 1, and after normalization, it can be used as the matching score. PLDA scoring calculates the likelihood ratio of two feature vectors belonging to the same speaker within a probabilistic generation model framework, outputting a log-likelihood ratio score, which is then mapped by a sigmoid function to obtain a matching probability value between 0 and 1. The higher the value, the greater the probability that the current speaker belongs to the target user. This dimension allows the robot to directly utilize the identity information of the sound source in the turning decision, rather than relying solely on energy and location information as in existing technologies. This represents a qualitative shift in the decision-making logic from turning upon the presence of sound to turning only after confirming the target.

[0067] Through the four sub-steps described above, the system comprehensively evaluates the current speech segment from four dimensions: speech validity, spatial stability, signal-to-noise ratio quality, and voiceprint matching. The scores for each dimension reflect the credibility of the sound source from four perspectives: "Is it a human voice?", "Is the location stable?", "Is the signal clear?", and "Is it the target user?". These four dimensions are independent yet complementary: for example, when the target user has a cold, the feature confidence level of the voiceprint matching dimension may be lower. When it decreases, if and The confidence level is relatively high, and the system can still obtain a sufficiently high confidence level through comprehensive integration, thereby avoiding the interaction failure caused by hard threshold rejection in traditional voiceprint access control.

[0068] In conjunction with the first aspect, step S140 includes: S141, the feature confidence scores for the speech validity dimension, spatial stability dimension, signal-to-noise ratio quality dimension, and voiceprint matching dimension are normalized respectively to obtain the normalized confidence scores for each dimension.

[0069] S142 uses a weighted summation method to fuse the normalized scores of multiple dimensions to obtain the comprehensive sound source confidence.

[0070] Normalization is the process of mapping the confidence scores of features in each dimension to the same numerical range. Its purpose is to eliminate the scale differences caused by different algorithm principles, output ranges, and physical units, and to ensure that each dimension has comparability and balanced contribution in the subsequent weighted fusion.

[0071] Specifically, the feature confidence scores of the four dimensions come from different computational paths: feature confidence scores of the speech validity dimension. The probability values ​​output directly by the classifier, ranging from 0 to 1, naturally fall within the standard range; feature confidence in the spatial stability dimension. The azimuth deviation is obtained through mapping, and its original output range depends on the deviation calculation method and the setting of the mapping function; the feature confidence of the signal-to-noise ratio quality dimension. The signal-to-noise ratio is obtained through a nonlinear mapping. Although it has been mapped to the 0-1 interval, its distribution characteristics may differ from other dimensions; the feature confidence of the voiceprint matching dimension. Based on the similarity comparison results, the original range of cosine similarity is -1 to 1, while the original range of PLDA scoring is the entire real number domain, which needs to be mapped by the sigmoid function to fall into the 0 to 1 interval.

[0072] Because the original scores for the various dimensions differ in numerical range and distribution characteristics, directly inputting these original values ​​into the weighted fusion formula will result in the larger-scale dimensions dominating the fusion result. This would cause the weight coefficients to lose their intended balancing effect, and could even lead to the fusion result deviating significantly from actual reliability. Normalization is precisely designed to address this issue of inconsistency in dimensions across dimensions.

[0073] As a preferred implementation, the normalization process can employ a minimum-maximum normalization method.

[0074] in, For the first Confidence of original features in each dimension For the first The minimum value of each dimension within a preset statistical window. For the first The maximum value of each dimension within a preset statistical window. The normalized confidence score falls within the range of 0 to 1. The advantage of this method is its simplicity in computation, ease of real-time implementation on embedded platforms, and ability to strictly map the confidence scores of features across different dimensions to the same scale.

[0075] As another possible implementation, when the confidence scores of each dimension are known to follow a specific distribution, Z-score normalization (mean and standard deviation normalization) can be used followed by nonlinear mapping to eliminate differences in distribution characteristics between different dimensions. Furthermore, if the confidence scores of a certain dimension are already within the 0-1 range through classifier softmax output or sigmoid function mapping, and their distribution characteristics are consistent with the global scale requirements, then normalization can be simplified to preserving the values ​​or only making minor adjustments, without needing to perform a complete transformation operation.

[0076] Through the above normalization process, the feature confidence of the speech validity dimension, spatial stability dimension, signal-to-noise ratio quality dimension, and voiceprint matching dimension are uniformly mapped to the same numerical scale, eliminating the original dimensional differences. In the subsequent weighted fusion, they can contribute to the calculation of the comprehensive confidence in a balanced manner according to the set proportion of the weight coefficients.

[0077] In step S142, a weighted summation method is used to fuse the normalized scores of multiple dimensions to obtain the comprehensive sound source confidence score. Weighted summation is a linear fusion strategy. Its basic principle is to assign a weight coefficient representing the importance of each dimension, multiply the scores of each dimension by their corresponding weights, and then sum them to obtain the final fusion result. The specific formula is as follows:

[0078] in, Weighting coefficients for the speech effectiveness dimension For the weighting coefficients of the spatial stability dimension, Weighting coefficients for the signal-to-noise ratio quality dimension Let be the weight coefficients corresponding to the voiceprint matching dimension, and satisfy . .

[0079] The magnitude of the weighting coefficient reflects the contribution of the corresponding dimension to the turning decision; the larger the weighting coefficient, the more significant the impact of the score change in that dimension on the final confidence level. The specific values ​​of the weighting coefficients can be pre-defined according to the actual application scenario and system requirements, for example, by using the Analytic Hierarchy Process (AHP) or a parameter search method based on experimental data to determine the optimal weight combination. The weighted summation calculation process is essentially a weighted average of the scores of each dimension, with clear physical meaning, low computational cost, and easy real-time implementation on embedded platforms. Through this fusion formula, the system compresses the normalized scores of the four dimensions into a single comprehensive confidence value. This value is also in the range of 0 to 1. The closer it is to 1, the higher the overall credibility of the current sound source.

[0080] Steps S141-S142 fuse and transform multidimensional discrete evaluation into single-dimensional continuous confidence. This ensures that even if a certain dimension is affected by specific environmental factors (such as strong reverberation), the confidence level remains consistent. The price is too low, or the target user may have a cold. The score is lower (due to a decrease in scores), but as long as the scores in other dimensions can effectively compensate, the overall confidence level will be higher. It can still reach the turning threshold, thus avoiding system failure caused by a single hard threshold in traditional solutions.

[0081] It is particularly noteworthy that In this convergence framework, it is one of the influencing factors rather than the decisive switch. It increases the importance of identity information in the switching decision, but does not independently determine whether to switch or not. This design allows the system to strike a balance between security and availability.

[0082] In this embodiment, the weight coefficients of each dimension are dynamically adjusted according to the robot's current working mode, enabling the same hardware platform to adapt to different application scenarios and user needs. The robot's behavioral preferences in steering decisions are adjusted by changing the contribution ratio of the confidence scores of each dimension's features in the fusion formula. In this embodiment, the default... .

[0083] When the robot is in its first working mode, the system increases the weighting coefficient corresponding to the voiceprint matching dimension. This is to increase the weight of identity recognition in steering decisions. The first working mode corresponds to either a focused or private mode, suitable for scenarios where the robot needs to focus on a specific authorized user. For example, in a home kitchen, the robot is set to be a dedicated assistant serving only one family member, or in an office environment where interference from unauthorized voices needs to be avoided. In this mode, the weight coefficient δ can be preset to a relatively high value (e.g., between 0.40 and 0.55), making the feature confidence of the voiceprint matching dimension more influential on the overall confidence. The contribution increases significantly. At this point, the overall confidence level... Highly sensitive to the speaker's identity, even if the voice of a non-target user is clear and directional, it will be rejected because the overall confidence level is lower than the turning threshold due to the low confidence level of the voiceprint matching dimension features. This mechanism enables the robot to exhibit extremely high loyalty and effectively avoids the problem of being interfered with by the voices of strangers in multi-speaker environments. It is particularly suitable for scenarios where users need the robot to focus on providing services without being interrupted by others.

[0084] When the robot is in its second working mode, the system reduces the weight coefficient corresponding to the voiceprint matching dimension. And accordingly increase the weight coefficient of the speech effectiveness dimension. and / or the weighting coefficients corresponding to the signal-to-noise ratio quality dimension To reduce the decision weight of identity recognition, the second working mode corresponds to the public or open mode, suitable for scenarios where the robot needs to provide services to all users. For example, in business presentations, visitor reception, or multi-person meetings, the robot's service targets are not limited to specific registered users; anyone's voice commands should be responded to. In this mode, the weight coefficient δ is reduced (e.g., to between 0.05 and 0.15), and the released weight is allocated to the voice validity dimension and / or signal-to-noise ratio quality dimension, making the feature confidence of these two dimensions the dominant factor influencing the overall confidence. At this point, the overall confidence... The decision primarily depends on whether the current sound is a valid human voice and its clarity. As long as the speaker's voice is clear enough and recognized as human, the system can still make a turning decision even if the confidence level of the voiceprint matching dimension features is low. This mechanism enables the robot to adopt an open service posture, capable of responding to any visitor's instructions. At the same time, the increased weighting of the voice validity dimension and signal-to-noise ratio quality dimension ensures that device noise and low-quality interference signals are still effectively suppressed.

[0085] In conjunction with the first aspect, step S150, which involves controlling the robot to perform a turning action based on the orientation information, specifically includes: S151, Generate steering control quantity based on orientation information, and use smooth control algorithm to process the steering control quantity to generate smooth steering command.

[0086] S152 drives the motor to run according to the smooth steering command, so that the robot rotates to the target direction.

[0087] The comprehensive sound source confidence level is calculated in step S140. Next, the system enters the turning decision phase, which determines whether the current sound source warrants a turning response from the robot. Specifically, the system will consider the sound source confidence level. Compared with the preset confidence threshold The comparison is made based on the confidence threshold. It is a pre-calibrated constant whose value range is consistent with... Consistent (between 0 and 1), the specific value can be set according to the application scenario's preference for a trade-off between security (no false redirects) and availability (no missed valid commands). Understandable. The higher the value, the stricter the system's requirements for the credibility of the sound source, and the lower the probability of false turn, but at the same time, it may miss valid commands under low signal-to-noise ratio. A lower value results in a more sensitive system response, but may introduce more false turns. In practical applications, The default value can be set to the range of 0.65 to 0.75, which can achieve a good balance in a typical kitchen environment.

[0088] like ≥ If the sound source is determined to be a high-confidence target and meets the preset turning conditions, the system confirms that the current speaker is the target user and the voice quality meets the requirements. Therefore, it generates a turning decision instruction, and the process proceeds to step S150 to execute the turning action.

[0089] like < If the sound source is determined to be interference or an unreliable signal, and the preset switching conditions are not met, the system refuses to respond to the current sound source, discards the current data, and returns to step S110 to re-acquire and preprocess the audio, entering the next round of monitoring cycle.

[0090] In step S151, the system generates a steering control quantity based on the azimuth information and uses a smooth control algorithm to process the steering control quantity, generating a smooth steering command. Specifically, the system first obtains the calculated sound source azimuth angle θ from step S122. This azimuth angle represents the angular deviation of the sound source relative to the robot's current orientation, typically ranging from -180° to 180° or 0° to 360°, with its positive or negative sign indicating the steering direction (e.g., a positive value indicates clockwise steering, and a negative value indicates counterclockwise steering). The system uses this azimuth angle θ as the target steering angle, combined with the robot's current orientation angle (provided by the motor encoder or inertial measurement unit), to calculate the required steering control quantity. This control quantity typically includes parameters such as the target angle value, desired angular velocity, and acceleration.

[0091] However, directly inputting the target angle as a step command to the motor drive unit would cause the motor to start and stop abruptly at maximum acceleration, resulting in severe mechanical shock and fuselage vibration. Therefore, the system further employs a smooth control algorithm to process the steering control input during transitions.

[0092] In this embodiment, the smooth control algorithm is a PID (Proportional-Integral-Derivative) control algorithm. The PID controller calculates the control output in real time based on the current angle error (the difference between the target angle and the actual angle). Its proportional term determines the response speed, the integral term eliminates steady-state error, and the derivative term suppresses overshoot and oscillation. The synergistic effect of these three terms enables the robot to smoothly and quickly approach the target direction. Specifically, in the turning start phase, the robot accelerates uniformly from rest to the set maximum turning angular velocity with a preset acceleration. When approaching the target direction, the system decelerates uniformly to a stop with a preset deceleration, realizing "S-shaped" speed planning. The acceleration and deceleration during the entire turning process can be pre-calibrated according to the actual load and motor characteristics to adapt to the motion characteristics of different models.

[0093] In step S152, the system drives the motor according to the smooth steering command, causing the robot to rotate to the target direction. Specifically, the smooth steering command generated in sub-step S151 is sent to the actuator drive module, which includes a motor driver, a steering motor, and an angle feedback sensor (such as an encoder or potentiometer). The motor driver analyzes parameters such as the target angle, desired speed, and acceleration / deceleration time based on the received command, and generates a corresponding pulse width modulation (PWM) signal to drive the motor. During rotation, the angle feedback sensor monitors the actual rotation angle of the motor in real time and feeds it back to the PID controller, forming a closed-loop control to ensure that the steering angle accurately converges to the target value until the error between the robot's orientation and the sound source direction is kept within a preset allowable deviation range (such as ±3°), at which point the motor stops rotating and remains aligned.

[0094] Through steps S151-S152, the system realizes a complete execution chain from orientation information to mechanical rotation. The smooth transition processing in S151 effectively suppresses sudden speed changes during the turning process, making the robot's rotation posture more natural and smooth, improving the user experience, and reducing transient impact loads on the motors and transmission mechanisms, extending the service life of the actuators. The closed-loop drive in S152 ensures high-precision convergence of the turning angle, enabling the robot to stably align with the target sound source direction. Thus, a complete turning execution process is completed at the physical level.

[0095] In this embodiment, the robot refers to a service robot placed on a desktop platform. Unlike ground-based mobile robots, desktop robots are smaller and have a limited range of movement. They are typically fixed on a desktop, and their interaction targets are mainly users in a sitting or standing position. The distance between the microphone array and the user's mouth is relatively fixed, resulting in a relatively stable acoustic environment. However, they are also more susceptible to interference from desktop reflections and nearby noise sources. This application addresses the voice interaction and turning control requirements of desktop robots in complex acoustic environments with high noise levels, such as kitchens, and multiple users. Through multi-dimensional sound source confidence assessment, the desktop robot can still achieve accurate and reliable target user locking and turning response under limited computing resources and fixed acoustic conditions.

[0096] In conjunction with the first aspect, after step S150, the following also includes: The S160 uses a camera to collect the actual orientation information after turning and calculates the angle error between the actual orientation information and the orientation information.

[0097] S170 records and statistically analyzes the angular errors generated by multiple turns, and adaptively compensates and calibrates the azimuth information obtained from subsequent sound source localization based on the statistical results.

[0098] In step S160, the system collects the actual orientation information after turning using a camera, and calculates the angle error between the actual orientation information and the orientation information obtained by audio positioning in step S120.

[0099] Specifically, after the robot completes its turn and stabilizes, the camera module is triggered to acquire the current field of view image. The camera module's field of view (FOV) should cover the area in front of the robot's turning direction to ensure that the target speaker is within the frame. The system performs target detection on the acquired image, identifies and locates the speaker's face or body position in the image, and calculates the actual azimuth angle of the target relative to the robot's current orientation by using perspective projection geometry, based on the positional deviation of the face / body in the image pixel coordinate system (i.e., the pixel distance of the target from the image center point), combined with the camera's intrinsic parameter matrix (focal length, principal point coordinates, etc.) and the robot's current orientation angle. .Should This represents the target's true azimuth in physical space. It does not depend on the time delay estimation of the acoustic signal, and therefore is unaffected by reverberation or multipath effects, making it a reliable audio localization result. An objective reference benchmark. The system then calculates the angle error. The sign and magnitude of this angular error directly characterize the direction and amplitude of the deviation in the current audio positioning system. During image acquisition, if the target speaker changes position within the camera's field of view due to their own movement, the system needs to continuously lock onto the target using face tracking or human body tracking algorithms, and take the average position within a preset time window (e.g., 0.5 to 1 second) after the turn is completed as the basis for calculating the actual azimuth angle, in order to eliminate random errors caused by the instantaneous movement of the target.

[0100] In step S170, the system records and statistically analyzes the angular errors generated by multiple turns, and adaptively compensates and calibrates the azimuth information obtained from subsequent sound source localization based on the statistical results. Because the acoustic conditions in a kitchen environment are relatively stable—for example, the geometric relationship between the microphone array and the reflector is fixed, and the noise spectrum characteristics of the range hood remain essentially unchanged—the localization errors caused by reverberation and multipath effects often exhibit systematic and repetitive characteristics, manifesting as… Compared to It exhibits a fixed tendency to deviate. Utilizing this characteristic, the system calculates the angular error after each turn. The error records are stored in local non-volatile memory in time-series format, forming an error record set. The system maintains a sliding statistical window (e.g., storing the most recent 50 or 100 valid error records), and performs statistical analysis on the error data within the window. If the absolute value of the average error within the statistical window exceeds a preset calibration trigger threshold (e.g., 2°), it is determined that there is a significant systematic deviation in the current audio positioning, requiring compensation calibration. The compensation method is as follows: calculate the weighted moving average of the error within the window, and superimpose this average value as a compensation bias onto the azimuth angle output in the subsequent step S120, i.e. ,in, This represents the statistically obtained average error. This compensation bias remains in effect until it is updated when the average error changes in the subsequent statistical window. If the variance of the error within the statistical window is large, it indicates that the error mainly originates from random factors rather than systematic bias. In this case, the system pauses compensation or reduces the compensation intensity to avoid overcompensation.

[0101] Through steps S160 and S170, the system utilizes visual information to establish a long-term monitoring and adaptive calibration mechanism for audio positioning accuracy, enabling the robot to maintain high steering alignment accuracy continuously in a kitchen reverberant environment.

[0102] Because error statistics are accumulated incrementally, the system does not need to perform visual calibration with each turn. The compensation bias is only updated when a new error record causes a significant change in the average error of the statistical window, thus controlling computational overhead while ensuring calibration effectiveness. This mechanism fully considers the stability of the kitchen acoustic environment and the systematic characteristics of errors, achieving continuous optimization of positioning accuracy with extremely low computational overhead. When the target speaker is outside the camera's field of view or insufficient ambient light causes visual positioning failure, the system automatically skips the current calibration, retains the existing compensation bias, and continues to operate, ensuring that the core audio positioning and turning functions are unaffected.

[0103] In addition, when the system detects that the robot responds to multiple sound sources from different directions in a single turning action (i.e., triggers multiple turns in a short period of time), it can pause error statistics and calibration updates until the robot re-enters a stable interactive state, in order to eliminate visual positioning interference caused by the robot's own movement.

[0104] Secondly, this application provides a robot steering control system based on multi-dimensional sound source confidence assessment, applied to desktop robots, combined with... Figure 2 As shown, the system includes: an audio acquisition module 10, an audio processing module 20, a main control module 30, and an execution driver module 40.

[0105] The audio acquisition module 10 is used to acquire ambient audio signals.

[0106] The audio processing module 20 is used to locate the sound source of the environmental audio signal to obtain the location information of the sound source relative to the robot, and extract feature confidence scores in multiple dimensions, including voiceprint features.

[0107] The main control module 30 is connected to the audio processing module and is used to execute the above method to perform fusion calculation of feature confidence in multiple dimensions to obtain comprehensive sound source confidence. When the comprehensive sound source confidence meets the preset turning conditions, a turning control command is generated based on the orientation information.

[0108] The execution drive module 40 is connected to the main control module 30 and is used to drive the robot to perform steering actions in response to steering control commands.

[0109] In this embodiment, the core of the audio acquisition module 10 is a microphone array configured on the robot body, which can adopt any configuration of linear array, circular array, or three-dimensional array. Each microphone channel samples synchronously with a sampling rate of not less than 16kHz and a quantization bit depth of not less than 16bit to ensure effective acquisition of the speech signal. The spatial configuration parameters of the microphone array (such as element spacing, array radius, and the three-dimensional coordinates of each microphone in the robot coordinate system) are pre-calibrated and stored in non-volatile memory during system initialization, providing geometric constraints for sound source localization. The hardware architecture of the audio acquisition module 10 includes a multi-channel analog-to-digital converter (ADC) and a front-end amplifier circuit, which converts the analog sound pressure signal picked up by the microphone into a digital audio signal and transmits it through I... 2 The audio is transmitted in real time to the audio processing module 20 via a synchronous serial interface such as S or TDM.

[0110] The audio processing module 20 can be implemented using any one or more combinations of a digital signal processor (DSP), an embedded neural network processor (NPU), or a field-programmable gate array (FPGA). The specific computing power configuration depends on the trade-off between system cost and real-time requirements. Internally, the audio processing module 20 includes a speech activity detection unit, a sound source localization unit, a voiceprint feature extraction unit, and a feature scoring unit. The speech activity detection unit separates valid speech segments from the continuous audio stream transmitted by the audio acquisition module 10; the sound source localization unit calculates the azimuth angle of the sound source relative to the robot based on the valid speech segments; the voiceprint feature extraction unit extracts the voiceprint feature vectors of the valid speech segments; and the feature scoring unit calculates the speech validity score, spatial stability score, signal-to-noise ratio quality score, and voiceprint matching score, and outputs the scores after normalization. Data interaction between the audio processing module 20 and the main control module 30 is achieved through a high-speed serial bus (such as SPI or UART), and the transmitted content includes the timestamp of the valid speech segments, the azimuth angle θ, and the normalized scores in the four dimensions.

[0111] The main control module 30 is the system's decision-making center. Its core is a microcontroller (MCU) or application processor (AP), running a real-time operating system (RTOS) to ensure multi-task scheduling of audio processing, decision calculation, and motor control. The main control module 30 internally has preset confidence threshold T and weight coefficients for each dimension. It also stores a voiceprint template library for registered target users. When the overall sound source confidence level... When the steering control quantity is calculated based on the current azimuth angle θ, the main control module 30 calls a smooth control algorithm (preferably a PID control algorithm) to perform transition processing on the steering control quantity, generating a smooth steering command; when When the current data is discarded, the audio processing module 20 is controlled to proceed to the next round of detection. The main control module 30 also provides a working mode selection interface, through which the external application layer can configure the system to focus mode or public mode. The main control module 30 automatically adjusts the weight coefficients according to the current mode, so that the system exhibits differentiated behavior in different application scenarios. In addition, the main control module 30 is also responsible for executing the visual-assisted calibration algorithm (when the system is equipped with a camera module), calculating the angle error, and updating the compensation bias.

[0112] The execution drive module 40 includes a motor driver, a steering motor, and an angle feedback sensor (such as a magnetic encoder or potentiometer). The motor driver receives smooth steering commands generated by the main control module 30, analyzes parameters such as the target angle, desired speed, and acceleration / deceleration time, and generates corresponding pulse width modulation (PWM) signals to drive the motor. The angle feedback sensor monitors the actual rotation angle of the motor in real time and feeds it back to the main control module 30, forming a closed-loop control to ensure that the steering angle converges precisely to the target value. During rotation, the execution drive module 40 is also responsible for monitoring the motor current and temperature. When a stall or overcurrent abnormality is detected, it reports the fault status to the main control module 30 and executes a safe shutdown to avoid damage to the motor and mechanical structure. The selection of the steering motor depends on the load and response speed requirements of the desktop robot. Stepper motors or brushless DC motors with gearboxes are preferred. The former facilitates open-loop position control, while the latter has higher efficiency and lower operating noise.

[0113] The module division between the audio processing module 20 and the main control module 30 can be flexibly adjusted according to the actual hardware architecture. In low-cost embedded solutions, the audio processing module 20 and the main control module 30 can be integrated into the same system-on-a-chip (SoC), existing as logical functional units rather than physically independent chips. In high-performance solutions, the audio processing module 20 can use a separate DSP chip to handle front-end signal processing, reducing the computational load on the main control module 30. Regardless of the physical implementation, the core processing flow of the system remains consistent: the audio acquisition module 10 continuously acquires data, the audio processing module 20 performs VAD detection, sound source localization, and multi-dimensional feature extraction, the main control module 30 performs confidence fusion and steering decision-making, and the execution drive module 40 executes the steering action. The modules transmit processing results rather than raw data through standardized data interfaces, reducing the coupling between modules and facilitating system maintenance and upgrades.

[0114] In conjunction with the second aspect, the audio processing module 20 includes: a speech activity detection unit, a sound source localization unit, a voiceprint feature extraction unit, and a feature scoring unit.

[0115] The speech activity detection unit is used to separate valid speech segments from the ambient audio signal.

[0116] The sound source localization unit is used to calculate the azimuth angle of the sound source relative to the robot based on the effective speech segment.

[0117] The voiceprint feature extraction unit is used to extract the voiceprint feature vector of the effective speech segment.

[0118] The feature scoring unit is used to calculate the feature confidence scores for the speech validity dimension, spatial stability dimension, signal-to-noise ratio quality dimension, and voiceprint matching dimension, respectively.

[0119] In this embodiment, the speech activity detection unit is used to separate valid speech segments from the environmental audio signal. This unit performs frame-by-frame processing on the input continuous audio stream (frame length 20ms to 30ms, frame shift 10ms), extracting multi-dimensional acoustic features such as short-time energy, zero-crossing rate, and spectral entropy frame by frame. A pre-trained VAD classifier is used for binary classification, determining whether the speech is human or non-human. When several consecutive frames are determined to be human speech, the speech start point is marked; when several consecutive frames are determined to be non-human speech, the end point is marked. The audio segment between the start and end points is the valid speech segment, which is simultaneously distributed to the sound source localization unit, the speaker feature extraction unit, and the feature scoring unit as a unified data source for subsequent processing.

[0120] The sound source localization unit calculates the azimuth angle of the sound source relative to the robot based on the effective speech segment. This unit employs a localization algorithm based on time delay estimation to calculate the time difference of arrival (TDOA) between the effective speech segment and each microphone in the microphone array. The generalized cross-correlation-phase transform (GCC-PHAT) algorithm is preferably used to suppress noise and reverberation interference. After obtaining multiple sets of TDOA, and combining them with pre-calibrated microphone array geometric configuration parameters, the azimuth angle of the sound source is calculated using a geometric localization algorithm. This angle is used for both spatial stability score calculation and as the target angle input for the main control module 30 to generate steering control commands.

[0121] The voiceprint feature extraction unit is used to extract voiceprint feature vectors from valid speech segments. This unit sequentially performs pre-emphasis, framing, and windowing on the valid speech segments to extract the MFCC or Fbank acoustic features of each frame, which are then input into a pre-trained voiceprint feature extraction model. Preferably, this model is an x-vector extractor based on a time-delay neural network (TDNN), which maps variable-length frame-level features to fixed-dimensional speaker embedding vectors (e.g., 512-dimensional or 256-dimensional). This vector represents the current speaker's identity information and is used by the feature scoring unit for voiceprint matching calculations. The model parameters are pre-stored in the unit's memory and can run in real-time on an embedded platform.

[0122] Feature scoring units are used to calculate separately , , and This unit combines the effective speech segments output by the speech activity detection unit and the azimuth angle output by the sound source localization unit. The system, along with the voiceprint feature vector output by the voiceprint feature extraction unit, completes a four-dimensional scoring process: Speech validity score, based on the probability of human voice obtained from MFCC features through a speech / non-speech classifier; Spatial stability score, calculated based on the deviation of the azimuth angle between the current frame and the previous N frames (the smaller the deviation, the higher the score); Signal-to-noise ratio (SNR) quality score, obtained by estimating the SNR of the current frame and mapping it to the 0-1 range; and Voiceprint matching score, obtained by comparing the voiceprint feature vector with the locally stored target user template using cosine similarity or PLDA scoring. Each score is normalized, packaged, and transmitted to the main control module 30.

[0123] In this embodiment, the voice activity detection unit is located at the front end, and its output serves as the common input for the other three units. The sound source localization unit and the voiceprint feature extraction unit work in parallel, extracting location information and identity information respectively. The feature scoring unit aggregates the outputs of the first three units to complete the comprehensive calculation of the confidence scores of the four dimensions of features. In terms of power consumption management, the system adopts an event-driven hierarchical wake-up mechanism: when the voice activity detection unit does not detect a valid voice segment, the sound source localization unit, the voiceprint feature extraction unit, and the feature scoring unit are all in a low-power standby state, with only the voice activity detection unit maintaining low-power listening operation to reduce the overall standby power consumption of the system; when a valid voice segment is detected, each unit is woken up sequentially according to the processing chain order to perform processing. After each unit completes its current task, if there is no new valid voice segment input, it automatically returns to the standby state after a preset timeout. Through the modular division of labor described above, each unit has a single function and a clear interface. The voice activity detection unit focuses on human voice detection, the sound source localization unit focuses on orientation calculation, the voiceprint feature extraction unit focuses on identity embedding and extraction, and the feature scoring unit focuses on multi-dimensional scoring calculation and output. This allows the algorithms of each unit to be iteratively upgraded independently. The units transmit processing results rather than raw data through standardized data interfaces, which reduces the coupling between modules and facilitates the deployment of the system on different hardware platforms and the independent upgrading of algorithms.

[0124] In conjunction with the first aspect, the system also includes: a camera module.

[0125] The camera module is connected to the main control module 30. The camera module is used to collect actual orientation information after the robot performs a turning action and feed it back to the main control module 30. The main control module 30 is also used to calculate the angle error based on the actual orientation information and the orientation information, and to adaptively calibrate the subsequent sound source localization results based on the statistical multiple angle errors.

[0126] When the system is further configured with a camera module, the module is connected to the main control module 30 and is used to collect actual orientation information after the robot performs a turning action and feed it back to the main control module 30. The main control module 30 is also used to calculate the angle error based on the actual angle and the audio positioning orientation angle, and to adaptively calibrate the subsequent sound source positioning results based on the statistical multiple errors.

[0127] The camera module is positioned at the front of the robot, with its optical axis aligned with the robot's forward direction. After turning, the main control module 30 triggers the camera to acquire an image. Target detection locates the speaker's face or body in the image. Based on the pixel distance of the target from the image center and the camera's intrinsic parameters, the actual azimuth angle of the target relative to the robot's current orientation is calculated. .

[0128] After the turn is completed, the camera captures images and calculates the target's actual azimuth angle. The main control module 30 compares it with the audio positioning results. Compare and calculate the angle error. The system stores the error data. It uses a sliding window to statistically analyze the error data: when the average error exceeds a threshold, it calculates a compensation bias and adds it to the subsequent positioning results; when the error dispersion is large, it pauses the compensation.

[0129] Understandably, if visual positioning fails, the calibration is automatically skipped, which does not affect the independent operation of the aforementioned audio positioning and steering functions. Error statistics are paused during multiple consecutive turns to eliminate interference from its own motion.

[0130] Thirdly, embodiments of this application provide an electronic device, combined with Figure 3 As shown, the electronic device includes a memory 131 and a processor 130. The memory 131 stores a computer program, and the processor 130 runs the computer program to make the electronic device perform the above-described method.

[0131] Furthermore, combined Figure 3 The electronic device shown also includes a bus 132 and a communication interface 133, with the processor 130, the communication interface 133 and the memory 131 connected via the bus 132.

[0132] The memory 131 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 133 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 132 may be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0133] Processor 130 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 130 or by instructions in software form. Processor 130 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 131, and processor 130 reads the information in memory 131 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0134] Fourthly, embodiments of this application provide a readable storage medium storing computer program instructions, which are read and executed by a processor to perform the above-described method.

[0135] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0136] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0137] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0138] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0139] Finally, it should be noted that the above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A robot steering control method based on multi-dimensional sound source confidence assessment, characterized in that, The method includes: The ambient audio signal collected by the microphone array is acquired, and the ambient audio signal is preprocessed to determine the effective speech segment; Based on the effective speech segment, the sound source is located to obtain the positional information of the sound source relative to the robot. Extract feature confidence scores from multiple dimensions of the effective speech segment, wherein the multiple dimensions include at least the voiceprint matching dimension; The feature confidence scores of the multiple dimensions are fused and calculated to obtain the comprehensive sound source confidence score; If the overall sound source confidence level meets the preset turning conditions, the robot is controlled to perform a turning action based on the orientation information.

2. The method according to claim 1, characterized in that, The step of preprocessing the ambient audio signal to determine valid speech segments includes: The ambient audio signal is subjected to dereverberation and noise reduction processing to obtain a noise-conditioned audio signal. Speech activity detection is performed on the noise-reduced audio signal, the signal segment corresponding to human voice activity is extracted from the noise-reduced audio signal, and the extracted signal segment is determined as the valid speech segment.

3. The method according to claim 1, characterized in that, The step of locating the sound source based on the effective speech segment and obtaining the sound source's orientation information relative to the robot includes: A positioning algorithm based on time delay estimation is used to calculate the time difference of arrival between each microphone in the microphone array for the effective voice segment; Based on the arrival time difference and the geometry of the microphone array, the azimuth angle of the sound source relative to the robot is calculated.

4. The method according to claim 1, characterized in that, The multiple dimensions also include: speech effectiveness dimension, spatial stability dimension, and signal-to-noise ratio quality dimension; The step of extracting feature confidence scores from multiple dimensions of the effective speech segment, wherein the multiple dimensions include at least a speaker matching dimension, includes: The Mel frequency cepstral coefficients of the effective speech segment are input into the classifier for processing to obtain the feature confidence of the speech validity dimension; Based on the degree of fluctuation of the location information over time, the feature confidence level of the spatial stability dimension is obtained. The signal-to-noise ratio of the current speech frame is mapped to a quality score, and the feature confidence corresponding to the signal-to-noise ratio quality dimension is obtained. Extract the voiceprint feature vector of the current speech segment, and compare the similarity of the voiceprint feature vector with the pre-stored target user voiceprint template to obtain the feature confidence level corresponding to the voiceprint matching dimension.

5. The method according to claim 4, characterized in that, The step of fusing and calculating the feature confidence scores of the multiple dimensions to obtain the comprehensive sound source confidence score includes: The feature confidence scores of the speech validity dimension, spatial stability dimension, signal-to-noise ratio quality dimension, and voiceprint matching dimension are normalized to obtain the normalized confidence scores for each dimension. The normalized confidence scores corresponding to each dimension are fused using a weighted summation method to obtain the comprehensive sound source confidence score.

6. The method according to claim 1, characterized in that, The steps of controlling the robot to perform a turning action based on the orientation information include: Based on the orientation information, a steering control quantity is generated, and a smooth control algorithm is used to process the steering control quantity to generate a smooth steering command. The drive motor runs according to the smooth steering command, causing the robot to rotate to the target direction.

7. The method according to claim 6, characterized in that, After controlling the robot to perform a turning action based on the aforementioned orientation information, the method further includes: The camera captures the actual orientation information after the turn, and the angle error between the actual orientation information and the orientation information is calculated. The angle errors generated by multiple turns are recorded and statistically analyzed. Based on the statistical results, the azimuth information obtained from subsequent sound source localization is adaptively compensated and calibrated.

8. A robot steering control system based on multi-dimensional sound source confidence assessment, applied to a desktop robot, characterized in that, include: The audio acquisition module is used to acquire ambient audio signals. The audio processing module is used to locate the sound source of the environmental audio signal to obtain the location information of the sound source relative to the robot, and extract feature confidence scores in multiple dimensions, including voiceprint features. The main control module, connected to the audio processing module, is used to execute the method as described in any one of claims 1-7, to perform fusion calculation on the feature confidence of the multiple dimensions to obtain a comprehensive sound source confidence, and to generate a turning control command based on the orientation information when the comprehensive sound source confidence meets the preset turning conditions; The execution drive module is connected to the main control module and is used to drive the robot to perform steering actions in response to the steering control command.

9. The robot steering control system based on multi-dimensional sound source confidence assessment according to claim 8, characterized in that, The audio processing module includes: A voice activity detection unit is used to separate valid voice segments from the environmental audio signal; A sound source localization unit is used to calculate the azimuth angle of the sound source relative to the robot based on the effective speech segment; A voiceprint feature extraction unit is used to extract the voiceprint feature vector of the effective speech segment; The feature scoring unit is used to calculate the feature confidence scores for the speech validity dimension, spatial stability dimension, signal-to-noise ratio quality dimension, and voiceprint matching dimension, respectively.

10. The robot steering control system based on multi-dimensional sound source confidence assessment according to claim 8, characterized in that, Also includes: A camera module is connected to the main control module; the camera module is used to collect the actual arrival angle information after the robot performs a turning action and feed it back to the main control module. The main control module is also used to calculate the angle error based on the actual azimuth information and the azimuth information, and to adaptively calibrate the subsequent sound source localization results based on the statistically calculated multiple angle errors.

Images