A mobile robot voice data control method and system based on artificial intelligence

Abstract

The present application relates to the field of data processing technology, and discloses a method and system for controlling speech data of a mobile robot based on artificial intelligence. The method comprises: collecting a three-channel speech data stream through a multi-channel microphone array, and performing time synchronization processing with the robot's motion state data; calculating the motion intensity value based on the motion state data, and performing motion compensation through the MSAP algorithm; inputting the compensated speech signal into a dual-path convolutional neural network to extract motion-aware speech features; jointly encoding speech features and motion state vectors through an MS-LSTM network, and using a CTC loss function to obtain speech recognition text; cross-modally fusing the text with the robot's state information, performing intent recognition through a hierarchical classifier, and obtaining structured control instructions. The present application solves the technical problems of poor voice command recognition accuracy and high control delay of mobile robots in dynamic environments.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and system for controlling voice data of a mobile robot based on artificial intelligence. Background Art

[0002] Existing mobile robot voice control technologies are primarily based on traditional automatic speech recognition (ASR) systems and speech processing methods for static environments. Voice signals are collected through microphones and then preprocessed, with feature extraction and pattern recognition performed. The speech is then converted into text instructions, which are then converted into robot control commands through simple keyword matching or rule parsing. Existing technologies generally employ single-channel voice acquisition, fixed-parameter signal preprocessing, and speech recognition algorithms based on hidden Markov models or deep neural networks. These achieve good recognition results in static or low-speed mobile environments, providing fundamental technical support for the intelligent control of mobile robots.

[0003] However, the existing technology has significant technical defects, mainly manifested in the lack of perception and compensation mechanism for the motion state of the mobile robot. When the robot is in a mobile state, the motion noise such as motor noise, tire friction noise, wind resistance noise, etc. generated by its own movement will seriously interfere with the quality of the voice signal, resulting in a sharp drop in the accuracy of voice recognition. At the same time, the Doppler effect and directional changes caused by the robot's movement further deteriorate the stability of the voice features. The existing static voice processing method cannot adapt to the complex changes of voice signals in a dynamic environment, and the reliability and real-time performance of voice control in a mobile state are seriously insufficient.

[0004] Based on the analysis of the limitations of existing technologies, the core technical problem facing mobile robot voice data control is how to achieve accurate recognition and real-time processing of voice commands in a dynamic mobile environment. Specifically, it includes the adaptive suppression of multi-source noise in motion, the coordinated processing of motion parameters and voice features, the robustness of speech recognition models in dynamic environments, and the voice data control method based on mobile state perception. These technical problems are interrelated and progressive, and require systematic solutions from multiple technical levels such as motion perception, feature fusion, model optimization, and control strategy. Summary of the Invention

[0005] The present application provides a mobile robot voice data control method and system based on artificial intelligence, which is used to solve the technical problems of poor voice command recognition accuracy and high control delay of mobile robots in dynamic environments.

[0006] In a first aspect, the present application provides a method for controlling voice data of a mobile robot based on artificial intelligence, the method comprising: collecting and processing environmental voice signals through a multi-channel microphone array to obtain a three-channel voice data stream, and simultaneously acquiring and processing the robot motion state parameters to obtain motion state data, performing time synchronization processing on the three-channel voice data stream and the motion state data to obtain motion voice synchronization data; calculating and processing the motion noise influencing factor according to the motion state data to obtain a motion intensity value, matching the motion intensity value with the filtering parameter through the MSAP algorithm, and performing motion compensation on the motion voice synchronization data. A motion-compensated speech signal is obtained; the motion-compensated speech signal is input into a dual-path convolutional neural network for feature extraction processing, and static speech features are fused with motion speech features through an attention mechanism to obtain motion-perception speech features; a motion state vector is constructed according to the motion state data, and the motion-perception speech features and the motion state vector are jointly encoded through an MS-LSTM network, and a CTC loss function is used for recognition processing to obtain a speech recognition text sequence; the speech recognition text sequence is cross-modally fused with robot state information to obtain multimodal semantic features, and the multimodal semantic features are subjected to intent recognition processing through a hierarchical classifier to obtain structured control instructions.

[0007] In a second aspect, the present application provides an artificial intelligence-based mobile robot voice data control system, the artificial intelligence-based mobile robot voice data control system comprising:

[0008] An acquisition module is configured to acquire and process ambient voice signals through a multi-channel microphone array to obtain a three-channel voice data stream, simultaneously acquire and process robot motion state parameters to obtain motion state data, and perform time synchronization processing on the three-channel voice data stream and the motion state data to obtain motion voice synchronization data;

[0009] a matching module, configured to calculate and process a motion noise influence factor based on the motion state data to obtain a motion intensity value, match the motion intensity value with a filter parameter using an MSAP algorithm, and perform motion compensation on the motion speech synchronization data to obtain a motion compensated speech signal;

[0010] An extraction module is used to input the motion-compensated speech signal into a dual-path convolutional neural network for feature extraction processing, and to fuse static speech features with motion speech features through an attention mechanism to obtain motion-aware speech features;

[0011] An encoding module is used to construct a motion state vector based on the motion state data, jointly encode the motion-perceived speech features and the motion state vector through an MS-LSTM network, and perform recognition processing using a CTC loss function to obtain a speech recognition text sequence;

[0012] The fusion module is used to perform cross-modal fusion processing on the speech recognition text sequence and the robot state information to obtain multimodal semantic features, and perform intent recognition processing on the multimodal semantic features through a hierarchical classifier to obtain structured control instructions.

[0013] In a third aspect, an artificial intelligence-based mobile robot voice data control device is provided, comprising: a memory and at least one processor, wherein instructions are stored in the memory; the at least one processor calls the instructions in the memory so that the artificial intelligence-based mobile robot voice data control device executes the above-mentioned artificial intelligence-based mobile robot voice data control method.

[0014] In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, which, when executed on a computer, enables the computer to execute the above-mentioned artificial intelligence-based mobile robot voice data control method.

[0015] The technical solution provided in this application solves the problem of incomplete information in traditional single-channel voice collection in mobile environments through the time synchronization processing of multi-channel microphone array and motion state data. The synchronization mechanism of the three-channel voice data stream combined with the motion state data ensures the precise time correspondence between the voice information and the robot's motion state, providing a high-quality data foundation for subsequent processing. The MSAP algorithm realizes adaptive processing of noise characteristics under different motion states based on the innovative design of dynamic matching filter parameters according to the motion intensity value. When the robot is in high-speed motion, it automatically selects a high-order adaptive filter to suppress strong motion noise. When the robot is in low-speed motion, it selects a low-order filter to retain more voice details. This adaptive filtering mechanism significantly improves the quality of voice signals in mobile environments. The parallel feature extraction architecture of the dual-path convolutional neural network retains speech semantic information through the static feature extraction path and specifically processes motion-related acoustic changes through the motion feature extraction path. The feature fusion processing realized by the attention mechanism effectively combines the complementary information of the two paths, and the resulting motion-aware speech features have stronger robustness and discriminability in dynamic environments. The MS-LSTM network's technical solution of jointly encoding motion state vectors and speech features breaks through the technical bottleneck of traditional speech recognition that ignores motion state information. The motion state embedding vector provides important contextual information for speech recognition, enabling the network to adjust recognition strategies based on the robot's motion state. The sequence alignment and probability calculation processing of the CTC loss function ensure the continuity and accuracy of speech recognition during changes in motion state.

[0016] Cross-modal fusion processing achieves deep semantic alignment between voice commands and the robot's actual state by calculating the correlation between the text semantic encoding of the Transformer encoder and the robot's state encoding. The query-key-value weighting of the cross-modal attention mechanism accurately identifies the semantic components of voice commands that are most relevant to the current robot state. This cross-modal understanding capability enables the robot to accurately understand user intent in complex environments. The hierarchical intent recognition processing of the hierarchical classifier decomposes the complex command parsing task into two levels: coarse-grained type recognition and fine-grained parameter extraction. The first-level classification of four command types provides a clear processing framework for subsequent parameter extraction. The second-level extraction of coordinate, angle, and object parameters ensures the integrity and accuracy of control commands. This hierarchical processing strategy significantly improves the parsing accuracy of complex voice commands. The structured control command assembly process converts natural language commands into a standardized control format that can be directly executed by the robot. The formatted encapsulation of command type, target parameters, execution priority, and safety constraints ensures the security and reliability of command execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 This is a schematic diagram of an embodiment of a method for controlling voice data of a mobile robot based on artificial intelligence in an embodiment of the present application;

[0019] Figure 2 This is a schematic diagram of an embodiment of a mobile robot voice data control system based on artificial intelligence in an embodiment of the present application;

[0020] Figure 3 It is a schematic block diagram of the structure of a mobile robot voice data control device based on artificial intelligence in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The embodiments of the present application provide a method and system for controlling voice data of a mobile robot based on artificial intelligence. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0022] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 In the embodiments of the present application, an embodiment of the mobile robot voice data control method based on artificial intelligence includes:

[0023] Step S101: Collect and process ambient voice signals through a multi-channel microphone array to obtain a three-channel voice data stream, and simultaneously acquire and process robot motion state parameters to obtain motion state data. The three-channel voice data stream and the motion state data are time-synchronized to obtain motion voice synchronization data.

[0024] Step S102: Calculate and process the motion noise impact factor based on the motion state data to obtain a motion intensity value, match the motion intensity value with the filter parameter using the MSAP algorithm, and perform motion compensation on the motion speech synchronization data to obtain a motion compensated speech signal;

[0025] Step S103: Input the motion-compensated speech signal into a dual-path convolutional neural network for feature extraction, and fuse the static speech features with the motion speech features through an attention mechanism to obtain motion-aware speech features.

[0026] Step S104: construct a motion state vector based on the motion state data, jointly encode the motion-perceived speech features and the motion state vector through the MS-LSTM network, and perform recognition processing using the CTC loss function to obtain a speech recognition text sequence;

[0027] Step S105: Perform cross-modal fusion processing on the speech recognition text sequence and the robot state information to obtain multimodal semantic features, and perform intent recognition processing on the multimodal semantic features through a hierarchical classifier to obtain structured control instructions.

[0028] It is understandable that the execution subject of this application can be a mobile robot voice data control system based on artificial intelligence, or a terminal or a server, which is not limited here. The embodiment of this application is described by taking the server as the execution subject as an example.

[0029] Specifically, three omnidirectional microphones atop the mobile robot, spaced 120 degrees apart, collect ambient speech signals to generate the first, second, and third audio signals. The sampling frequency is set to 48kHz, with a bit depth of 32 bits. Data streams are combined and processed to generate a three-channel speech data stream. Simultaneously, the IMU sensor collects the robot's three-axis velocity, three-axis angular velocity, and three-axis acceleration data at a frequency of 100Hz, forming the motion state data. A hardware clock interrupt mechanism timestamps the voice data stream and motion data, generating a time-stamped data sequence. Data alignment is then performed based on the timestamps, maintaining the time difference within a ±5ms range, ultimately generating synchronized motion and speech data.

[0030] The MSAP algorithm is the core of the motion state adaptive preprocessing algorithm. It first squares the three-axis velocity, three-axis angular velocity, and three-axis acceleration in the motion state data and then sums them to obtain the motion noise impact factor. The motion noise impact factors are then weighted and summed according to preset weight coefficients α = 0.15, β = 0.25, and γ = 0.1 to calculate the motion intensity value. The MSAP algorithm uses the motion intensity value to determine a threshold: when the value is less than 0.3, a 2nd-order Butterworth filter is selected; when the value is between 0.3 and 0.7, a 4th-order filter is selected; when the value is greater than 0.7, a 6th-order adaptive filter is selected, thereby obtaining dynamic filtering parameters. Frequency-domain filtering is performed on the motion-speech synchronization data to eliminate motion noise, generating de-noised speech data. The Doppler shift is then calculated and compensated based on the robot's motion trajectory, ultimately generating a motion-compensated speech signal.

[0031] The dual-path convolutional neural network consists of two parallel branches: a static feature extraction path and a motion feature extraction path. The static feature extraction path processes the motion-compensated speech signal through a cascade of standard convolutional layers, batch normalization layers, and ReLU activation functions to extract basic semantic information and form static speech features. The motion feature extraction path utilizes a dilated convolutional architecture, using dilated convolutional layers with dilation rates of 2, 4, and 8 to capture motion-related features at different time scales, such as robot motor noise, tire friction, and wind resistance. A residual connection mechanism weightedly superimposes spectral shift information caused by acceleration with the original features to generate velocity-compensated features. The motion state gating mechanism calculates a gating threshold based on the robot's steering angle and movement direction, adaptively filtering features to form direction-aware features. The trajectory prediction module uses information about the robot's future trajectory to preemptively compensate for the impact of the current speech signal, generating trajectory prediction features. Finally, feature encoding is performed based on four motion modes: straight ahead, turning, acceleration, and deceleration, to generate motion speech features. The multi-head attention mechanism performs query-key-value mapping on static speech features and motion speech features to generate a cross-path association matrix, obtains weighted feature vectors through adaptive weight allocation, and forms motion-aware speech features after feature dimension alignment and normalization.

[0032] The MS-LSTM network is a motion-aware long-short-term memory network. It first concatenates the three-axis velocity, three-axis angular velocity, three-axis acceleration, and motion intensity values ​​from the motion state data into vectors to form a 10-dimensional motion state vector. A linear transformation layer maps this vector to the same feature dimension as the speech features, generating a motion state embedding vector. The motion-aware input gate concatenates the motion-aware speech features with the motion state embedding vector, which is then input into the MS-LSTM network for hidden state update and sequence encoding, generating a motion-aware hidden state sequence. The hidden state sequence is dynamically weighted based on the robot's current movement speed, reducing recognition sensitivity to avoid misjudgments at high speeds and improving recognition accuracy at low speeds to ensure accuracy. This results in a speed-adaptive hidden state. The CTC loss function aligns the speed-adaptive hidden state sequence and calculates probabilities, mapping the hidden state sequence to a character probability distribution. A path decoding algorithm is then used to generate a speech recognition text sequence.

[0033] The Transformer encoder semantically encodes the speech recognition text sequence, converting the text into a high-dimensional semantic vector representation to form a text semantic encoding. Robot state information, including current location coordinates, battery charge percentage, task execution status, and environmental map data, is encoded through the state embedding layer to generate a robot state encoding. A cross-modal attention mechanism calculates the correlation between the text semantic encoding and the robot state encoding, and achieves feature fusion through query-key-value attention weight assignment to form cross-modal fusion features. A hierarchical classifier employs a two-layer structure for hierarchical intent recognition. The first layer identifies four command types: navigation, operation, query, and stop. The second layer extracts specific parameter information such as coordinate values, angle parameters, and target objects from the fused features to generate intent classification results. The structured instruction assembly module formats and encapsulates the instruction type, target parameters, execution priority, and safety constraints, ultimately generating structured control instructions containing control information.

[0034] In a specific embodiment, the process of executing step S101 may specifically include the following steps:

[0035] The ambient voice signal is collected and processed by three omnidirectional microphones to obtain a first audio signal, a second audio signal and a third audio signal;

[0036] Performing data stream combination processing on the first audio signal, the second audio signal and the third audio signal to obtain a three-channel voice data stream;

[0037] The robot's three-axis velocity, three-axis angular velocity, and three-axis acceleration are collected and processed through the IMU sensor to obtain motion state data;

[0038] Based on the hardware clock interrupt mechanism, the three-channel voice data stream and motion status data are time-stamped to obtain a time-stamped data sequence;

[0039] The time-stamped data sequence is aligned according to the timestamp to obtain motion speech synchronization data.

[0040] Specifically, three omnidirectional microphones are physically arranged on top of the mobile robot at 120-degree intervals. Each omnidirectional microphone has 360-degree sound reception. The first microphone collects ambient voice signals directly in front of the robot, the second microphone collects voice signals within a 120-degree angle to the left front, and the third microphone collects voice signals within a 120-degree angle to the right front. An analog-to-digital converter within each microphone converts the continuous analog voice signal into a discrete digital signal. The sampling frequency is uniformly set to 48,000 Hz, meaning that the voice signal is sampled 48,000 times per second. The bit depth is set to 32 bits, indicating that each sampling point uses 32 binary bits to represent the signal strength. The first, second, and third audio signals correspond to the digitized output results of the three microphones, respectively. The data stream combination processing stage integrates the three independent audio signals in the time domain. Specifically, the data sequences of the first, second, and third audio signals are arranged one by one according to the sampling time points, forming a three-dimensional data matrix structure. Each row represents the sampling values ​​of the three microphones at the same time, and each column represents the sampling sequence of a single microphone at different times. This matrix data organization generates a three-channel voice data stream. The three-channel voice data stream contains all-round acoustic information about the space surrounding the robot, and has the advantages of directional discrimination and noise suppression compared to single-channel data.

[0041] The IMU sensor, or inertial measurement unit, monitors the robot's motion state in real time through its built-in three-axis gyroscope and three-axis accelerometer. The three-axis gyroscope measures the robot's angular velocity changes around the X, Y, and Z axes in radians per second, while the three-axis accelerometer measures the robot's linear acceleration along the X, Y, and Z axes in meters per second squared. The IMU sensor's sampling frequency is set to 100 Hz, and it outputs a set of motion data containing six parameters every 10 milliseconds. By numerically integrating the position change between two consecutive sampling points, the robot's linear velocity components along the three coordinate axes are calculated. The three-axis velocity, three-axis angular velocity, and three-axis acceleration together constitute a nine-dimensional motion state data vector.

[0042] The hardware clock interrupt mechanism is a key technology for ensuring the time synchronization of voice and motion data. The mobile robot's internal system clock generates clock pulses at a frequency of 1MHz. Whenever the microphone array generates new voice data, the clock interrupt controller automatically assigns a timestamp to the data. The timestamp records the exact moment the data was generated, with microsecond accuracy. Similarly, the IMU sensor also receives a timestamp whenever it outputs motion data. Each three-dimensional data point in the three-channel voice data stream carries a timestamp, as does each nine-dimensional vector in the motion data. All time-stamped data are arranged in chronological order to form a time-stamped data sequence. The time alignment algorithm uses this timestamp information to accurately match the voice and motion data. Since the voice data is sampled at a frequency of 48kHz and the motion data at a frequency of 100Hz, the different sampling intervals require data interpolation. The algorithm first identifies the timestamp of each motion data point, then searches for the data point with the closest timestamp in the voice data sequence, pairing voice and motion data points with a time difference of less than 5 milliseconds. For time differences exceeding 5 milliseconds, the algorithm uses linear interpolation to calculate the data values ​​at intermediate moments. This interpolation is based on the trend of the values ​​between two adjacent sampling points. After time alignment, each moment contains both three-channel speech information and nine-dimensional motion state information, forming 12-dimensional motion-speech synchronized data.

[0043] In a specific embodiment, the process of executing step S102 may specifically include the following steps:

[0044] The three-axis velocity, three-axis angular velocity and three-axis acceleration in the motion state data are squared and calculated to obtain the motion noise impact factor;

[0045] Perform weighted summation on the motion noise influencing factors according to the preset weight coefficients to obtain the motion intensity value;

[0046] The motion intensity value is thresholded using the MSAP algorithm. When the motion intensity value is less than 0.3, a 2nd-order filter is selected. When the motion intensity value is between 0.3 and 0.7, a 4th-order filter is selected. When the motion intensity value is greater than 0.7, a 6th-order adaptive filter is selected to obtain dynamic filtering parameters.

[0047] Perform frequency domain filtering on the motion speech synchronization data based on dynamic filtering parameters to obtain noise-reduced speech data;

[0048] Doppler frequency shift compensation is performed on the noise reduction speech data according to the robot motion trajectory to obtain a motion compensated speech signal.

[0049] Specifically, the motion compensation processing of the MSAP algorithm in the AI-based mobile robot voice data control method is a core technical step in addressing the issue of voice recognition accuracy in mobile environments. The calculation process for the motion noise impact factor first extracts the three-axis velocity components vx, vy, and vz, the three-axis angular velocity components ωx, ωy, and ωz, and the three-axis acceleration components ax, ay, and az from the motion state data. Each component is squared, and then the squared values ​​of the three components of the same type are added together to obtain the three motion noise impact factors: the sum of the squares of the linear velocity, the sum of the squares of the angular velocity, and the sum of the squares of the acceleration. The sum of the squares of the linear velocity reflects the degree of wind noise interference caused by the robot's translational motion on the voice signal, the sum of the squares of the angular velocity reflects the intensity of directional noise caused by the robot's rotational motion, and the sum of the squares of the acceleration represents the level of mechanical noise generated by the motor and transmission during the robot's acceleration and deceleration. The weighted summation process uses empirical data on the impact of different motion types on voice quality in mobile robot voice control applications, with preset weight coefficients α of 0.15, β of 0.25, and γ of 0.1. These coefficients reflect the relative importance of linear velocity, angular velocity, and acceleration in interfering with the voice signal. The calculation process multiplies the sum of the squared linear velocities by a weighting factor α, the sum of the squared angular velocities by a weighting factor β, and the sum of the squared accelerations by a weighting factor γ. The three weighted results are then summed to produce the motion intensity value. The weighting factors are based on noise testing results under different motion states of the mobile robot. Angular velocity is given the highest weight because rotational motion causes the Doppler effect and directional changes, which have the greatest impact on speech recognition.

[0050] The MSAP algorithm, or motion adaptive preprocessing algorithm, uses a threshold judgment mechanism to select the most appropriate filter configuration based on the motion intensity value. When the motion intensity value is less than 0.3, it indicates that the robot is in a low-speed or near-stationary state, with low motion noise. A second-order Butterworth filter is selected for basic noise filtering. The second-order filter has low computational complexity and fast processing speed, making it suitable for handling mild background noise. When the motion intensity value is between 0.3 and 0.7, it indicates that the robot is in a medium-speed state, requiring stronger noise suppression. A fourth-order Butterworth filter is selected. Compared with the second-order filter, the fourth-order filter has a steeper cutoff characteristic and better noise suppression. When the motion intensity value is greater than 0.7, it indicates that the robot is in a high-speed state, with complex and variable motion noise. A sixth-order adaptive filter is selected. The sixth-order filter, combined with an adaptive algorithm, dynamically adjusts the filter parameters based on real-time noise characteristics. Dynamic filter parameters include key technical parameters such as the filter's cutoff frequency, passband gain, and stopband attenuation.

[0051] Frequency-domain filtering first converts the motion-speech synchronization data from the time domain to the frequency domain using a fast Fourier transform (FFT), obtaining the spectral distribution of the speech signal. The corresponding digital filter transfer function is then designed based on the dynamic filtering parameters. The filter's frequency response curve determines the degree to which different frequency components are retained or suppressed. The filtering process multiplies the speech signal spectrum by the filter's frequency response point by point. Motion noise is primarily concentrated in the low and high frequency bands, while human voice signals are primarily distributed in the mid-frequency band of 300Hz to 3400Hz. The filter prioritizes preserving mid-frequency signals while suppressing noise. After frequency-domain filtering, the processed spectrum is converted back to the time domain through an inverse Fourier transform, resulting in noise-reduced speech data.

[0052] Doppler shift compensation corrects the frequency offset caused by the relative motion between the sound source and receiver during robot movement. The Doppler effect can cause the fundamental frequency and harmonic frequencies of the speech signal to shift, affecting the accuracy of speech recognition. The compensation algorithm calculates the relative velocity vector between the sound source and the microphone based on the robot's motion trajectory and uses the Doppler shift formula to calculate the frequency offset. The frequency shift is proportional to the relative velocity: the frequency increases when the motion is toward the microphone and decreases when it is away from the microphone. Frequency shift compensation restores the shifted frequency components to their original positions through frequency domain interpolation and resampling techniques. The interpolation algorithm inserts or deletes sampling points in the spectrum to adjust the frequency distribution. After Doppler shift compensation, the motion-compensated speech signal is obtained, which eliminates the interference of robot motion on speech characteristics.

[0053] In a specific embodiment, the process of executing step S103 may specifically include the following steps:

[0054] The motion-compensated speech signal is input into the static feature extraction path and the motion feature extraction path respectively for parallel processing to obtain a dual-path input signal;

[0055] The static feature extraction path performs a cascade of standard convolution, batch normalization, and ReLU activation functions on the dual-path input signal to obtain static speech features.

[0056] The motion feature extraction path processes the dual-path input signal through dilated convolution, residual connection and gating mechanism to obtain motion speech features;

[0057] Based on the multi-head attention mechanism, query-key-value mapping is performed on static speech features and motion speech features to obtain a cross-path correlation matrix;

[0058] Adaptively weighting the static speech features and the motion speech features according to the cross-path correlation matrix to obtain a weighted feature vector;

[0059] The weighted feature vectors are aligned and normalized to obtain motion-aware speech features.

[0060] Specifically, the dual-path input signal generation process simultaneously replicates and distributes the motion-compensated speech signal to two independent neural network processing branches. The static feature extraction path specifically processes the semantic content of the speech signal, while the motion feature extraction path specifically processes the acoustic variation characteristics associated with the robot's motion state. The two paths operate in parallel without interfering with each other, sharing the same motion-compensated speech signal as input data. The static feature extraction path adopts the traditional deep learning speech processing architecture. It first performs a convolution operation on the input speech signal through a standard convolutional layer. The convolution kernel slides across the speech waveform in the time domain to extract local time-domain feature patterns. The convolution operation converts the time-domain waveform of the speech signal into a feature map representation. Batch normalization normalizes the feature map output by the convolutional layer. The mean and variance of each channel in the feature map are calculated. The feature value is subtracted from the mean and divided by the standard deviation to ensure that the feature distribution has a mean of zero and a variance of one. Batch normalization eliminates amplitude differences between different speech inputs, making neural network training more stable. The ReLU activation function performs a nonlinear transformation on the feature map after batch normalization, setting negative values ​​to zero and keeping positive values ​​unchanged. The ReLU function solves the gradient vanishing problem in deep neural networks. The cascade processing of standard convolution, batch normalization and ReLU activation functions forms a complete feature extraction unit. The cascade of multiple units constitutes a static feature extraction path, which outputs static speech features containing speech semantic information.

[0061] The motion feature extraction path utilizes a specially designed network structure to handle motion-related speech variations. Dilated convolution, also known as atrous convolution, expands the receptive field by inserting zeros between convolution kernel elements. A dilated convolution with a dilation rate of two inserts one zero between adjacent kernel elements, a dilated convolution with a dilation rate of four inserts three zeros, and a dilated convolution with a dilation rate of eight inserts seven zeros. Dilated convolutions with different dilation rates can capture motion noise patterns at different timescales. A dilation rate of two primarily captures high-frequency noise from the robot's motors, a dilation rate of four captures mid-frequency noise from tire-ground friction, and a dilation rate of eight captures low-frequency noise variations caused by wind resistance and environmental reflections. A residual connection mechanism directly adds the input signal of the atrous convolution to the output signal. This residual connection preserves important information from the original speech signal, preventing information loss in deep networks while enabling the network to learn the differential characteristics between motion states and speech signals. The gating mechanism dynamically adjusts the feature transfer weight according to the real-time motion state of the robot. The gating unit contains a sigmoid activation function, which calculates a gating weight value between zero and one based on the robot's steering angle and movement direction. The gating weight is multiplied element-by-element by the feature vector. When the robot is in a straight line motion, the gating weight is close to one to retain most features. When the robot is in a complex turning motion, the gating weight is reduced to suppress noise features. The motion feature extraction path outputs motion speech features that specifically describe the influence of motion.

[0062] The multi-head attention mechanism is a core technology in the Transformer architecture. It is used to establish correlations between static and motion speech features in mobile robot speech processing. The query-key-value mapping process linearly transforms static speech features to generate a query matrix Q, and linearly transforms motion speech features to generate a key matrix K and a value matrix V. The multi-head mechanism splits the Q, K, and V matrices into multiple sub-matrices, with each head computing a set of attention weights in parallel. The attention calculation process first multiplies the query matrix Q by the transpose of the key matrix K to produce an attention score matrix. The attention score reflects the correlation between static and motion features, with higher scores indicating stronger correlation. The attention scores are divided by a scaling factor and normalized using a softmax function to produce an attention weight matrix. The attention weights indicate the importance of the motion feature corresponding to each static feature. The cross-path correlation matrix records the detailed correspondence between two feature paths. Adaptive weight allocation processing calculates the fusion weights of static speech features and motion speech features based on the cross-path correlation matrix. The weight allocation algorithm analyzes the numerical distribution in the correlation matrix. When a static feature has a strong correlation with multiple motion features, the weight of the static feature is reduced accordingly. When a motion feature is only associated with a few static features, the weight of the motion feature is increased accordingly. The weight calculation takes into account the complementarity and redundancy between features. Complementary feature combinations will receive higher fusion weights, while redundant feature combination weights will be suppressed. Static speech features are multiplied by their corresponding weight values, and motion speech features are multiplied by their corresponding weight values. The weighted feature vectors are combined through vector concatenation or vector addition to form a weighted feature vector containing information from both paths.

[0063] Feature dimension alignment addresses the mismatch between the dimensions of static and motion features. When the dimensions of static speech features differ from those of motion speech features, a linear transformation is used to expand the smaller-dimensional feature vector to a larger one, or to compress the larger-dimensional feature vector to a smaller one. Dimension alignment ensures consistency in subsequent processing. Normalization numerically standardizes the weighted feature vectors, calculating their L2 norm and dividing them by their L2 norm to obtain a unit vector. Normalization eliminates the dimensional differences between different feature components, ensuring that all feature components are within the same numerical range. After feature dimension alignment and normalization, motion-perceived speech features are obtained, which comprehensively incorporate both the semantic information of speech and the influence of motion.

[0064] In a specific embodiment, the process of performing the step of performing dilated convolution, residual connection, and gating mechanism on the dual-path input signal through the motion feature extraction path may specifically include the following steps:

[0065] The dual-path input signals are fed into the dilated convolution layer for convolution processing with different dilation rates. The multi-time-scale features of the robot motor noise, tire friction sound, and wind resistance sound are captured through dilated convolution with dilation rates of 2, 4, and 8, and the motion noise separation features are obtained.

[0066] Based on the robot's velocity gradient, the motion noise separation feature is processed by residual connection, and the spectrum drift information caused by acceleration is weighted superimposed with the original feature to obtain the velocity compensation feature.

[0067] The velocity compensation feature is adaptively screened through the motion state gating mechanism, the gating threshold is calculated according to the robot's steering angle and movement direction, and feature suppression processing is performed to obtain the direction perception feature;

[0068] Based on the robot trajectory prediction information, the direction perception features are time-aligned, and the impact of future trajectory changes on the current voice signal is compensated in advance to obtain trajectory prediction features.

[0069] The trajectory prediction features are processed into motion state labels, and feature encoding is performed according to the four motion modes of straight-ahead, turning, acceleration, and deceleration to obtain motion speech features.

[0070] Specifically, refined dilated convolution processing enables accurate separation and identification of different motion noise sources. The dilated convolution layer uses convolution kernels with different dilation rates to extract multi-scale features from dual-path input signals. A dilated convolution kernel with a dilation rate of 2 inserts a zero between adjacent weights, doubling its effective receptive field. This kernel specifically captures high-frequency vibration noise generated by the robot's motors, which typically exhibits fixed frequency characteristics and periodic pulses. A dilated convolution kernel with a dilation rate of 4 inserts three zeros between adjacent weights, quadrupling its effective receptive field. This kernel specifically captures mid-frequency noise generated by tire-ground friction. The spectral characteristics of tire friction noise are closely related to the ground material and the robot's movement speed, exhibiting a wideband noise distribution. A dilated convolution kernel with a dilation rate of 8 inserts seven zeros between adjacent weights, expanding its effective receptive field eightfold. This kernel specifically detects low-frequency ambient noise caused by wind resistance and air flow. Wind resistance noise increases with the square of the robot's movement speed and exhibits both continuous and random characteristics. Three dilation-rate atrous convolutional layers process the input signal in parallel, outputting feature maps corresponding to different noise sources respectively. The three feature maps are concatenated according to the channel dimension through the feature map fusion algorithm to form motion noise separation features containing multi-time scale information.

[0071] The robot's velocity gradient is calculated by numerically differentiating the velocity vector at consecutive time points to obtain an acceleration vector. The velocity gradient reflects the instantaneous trend of the robot's motion state. Positive and negative changes in acceleration will cause corresponding drift in the speech signal spectrum. The residual connection algorithm uses the motion noise separation feature as input and the original dual-path input signal as the residual signal. The residual connection algorithm calculates the correlation coefficient between the input feature and the residual signal, which reflects the degree of influence of acceleration changes on speech features. The weighted superposition process calculates a dynamic weight coefficient based on the magnitude of the velocity gradient. The weight coefficient is close to zero when the robot is in uniform motion and increases when the robot is accelerating or decelerating. The weight coefficient is multiplied by the residual signal and then added element-by-element to the input feature. The weighted superposition eliminates the interference of acceleration-induced spectral drift on speech recognition. The output velocity compensation feature retains the true spectral characteristics of the speech signal while compensating for the effects of motion changes.

[0072] The motion state gating mechanism is an adaptive feature screening technology that dynamically adjusts the strength of feature transmission based on the robot's real-time steering angle and movement direction. The steering angle is calculated using gyroscope data and represents the change in the robot's current direction relative to the previous moment. The movement direction is calculated by fusing accelerometer and magnetometer data to determine the robot's direction vector in the global coordinate system. The gating threshold calculation compares the absolute value of the steering angle with a preset angle threshold. When the steering angle is less than the threshold, the gating weight is set to a higher value, and when the steering angle is greater than the threshold, the gating weight is set to a lower value. The rate of change of the movement direction is also considered in the gating threshold calculation process. Feature suppression processing performs element-by-element multiplication of the gating weight with the velocity compensation feature, suppressing feature components that do not match the current motion state while retaining those consistent with the movement direction. The direction-aware feature records speech feature information related to the robot's current movement direction.

[0073] The robot's trajectory prediction information uses current motion state parameters and historical trajectory data to predict short-term motion trends. The trajectory prediction algorithm analyzes the robot's velocity vector, acceleration vector, and angular velocity vector to predict changes in the robot's position and posture within a certain time window in the future. Timing alignment takes into account the time delay effect of robot motion on voice propagation. When the robot moves rapidly, there is a time difference between the time sound is emitted and the time it is received by the microphone. Trajectory prediction information is used to calculate the compensation for this time delay. Advance compensation adjusts the feature weight distribution at the current moment based on the predicted trajectory changes. For predicted drastic motion changes, the corresponding feature weight is reduced in advance, while for predicted smooth motion, the feature weight remains unchanged. The trajectory prediction feature contains information on expected compensation for future motion state changes.

[0074] The motion state labeling process simplifies the robot's complex motion states into four basic motion modes for classification and labeling. The straight-line mode corresponds to a motion state with a linear velocity greater than zero and an angular velocity close to zero; the turning mode corresponds to a motion state with an angular velocity greater than a set threshold; the acceleration mode corresponds to a motion state with a positive linear acceleration; and the deceleration mode corresponds to a motion state with a negative linear acceleration. The feature encoding process assigns a unique encoding vector to each motion mode: the straight-line mode is encoded as [1, 0, 0, 0], the turning mode as [0, 1, 0, 0], the acceleration mode as [0, 0, 1, 0], and the deceleration mode as [0, 0, 0, 1]. When the robot is in multiple motion states simultaneously, a weighted average method is used to calculate the combined encoding vector. The motion speech feature is generated by performing a tensor product operation on the trajectory prediction feature and the motion state encoding vector. The tensor product operation multiplies each dimension of the feature vector by the encoding vector element-wise, ultimately resulting in a motion speech feature vector containing motion state information.

[0075] In a specific embodiment, the process of executing step S104 may specifically include the following steps:

[0076] Perform vector concatenation processing on the three-axis velocity, three-axis angular velocity, three-axis acceleration and motion intensity value in the motion state data to obtain a ten-dimensional motion state vector;

[0077] The ten-dimensional motion state vector is dimensionally mapped through a linear transformation layer, and the motion parameters are converted into the same feature dimension as the speech features to obtain the motion state embedding vector;

[0078] Based on the motion-aware input gate, the motion-aware speech features and the motion state embedding vector are concatenated and processed, and the sequence encoding is performed through the hidden state update mechanism of the MS-LSTM network to obtain the motion-aware hidden state sequence;

[0079] Dynamically adjust the weight of the motion perception hidden state sequence according to the current movement speed of the robot, reduce the recognition sensitivity at high speed and improve the recognition accuracy at low speed, and obtain the speed-adaptive hidden state;

[0080] The speed-adaptive hidden state is sequence-aligned and probability-calculated using the CTC loss function. The hidden state sequence is mapped to character probability distribution and path decoding is performed to obtain the speech recognition text sequence.

[0081] Specifically, the 10-dimensional motion state vector is constructed by sequentially combining nine basic parameters from the motion state data with motion intensity values. The three-axis velocities vx, vy, and vz represent the robot's instantaneous movement speed in the X, Y, and Z directions, respectively. The three-axis angular velocities ωx, ωy, and ωz represent the robot's rotational speed around the X, Y, and Z axes, respectively. The three-axis accelerations ax, ay, and az represent the robot's acceleration components in the three coordinate axes, respectively. The motion intensity value is a comprehensive motion indicator calculated by the aforementioned MSAP algorithm. The vector concatenation process arranges these ten values ​​into a row vector in a fixed order, forming a 10-dimensional motion state vector that fully describes the robot's motion state. This vector contains complete motion information about the robot in three-dimensional space.

[0082] The linear transformation layer is a fully connected layer structure in a neural network, used to change the dimensionality and feature representation of data. The dimensional mapping process converts the ten-dimensional motion state vector into a higher-dimensional feature representation through matrix multiplication. The number of rows in the linear transformation matrix is ​​equal to the target feature dimension, and the number of columns is equal to the input vector dimension. The weight parameters in the matrix are trained and learned through the backpropagation algorithm. The motion parameter conversion process maps the velocity, angular velocity, and acceleration values ​​with clear physical meaning into abstract feature vector representations. The mapped feature dimensions are consistent with the dimensions of the speech features, ensuring that the data dimensions of subsequent network processing match. The motion state embedding vector is the output of the linear transformation. This vector retains the core information of the original motion state and has the dimensional basis for fusion calculations with speech features.

[0083] The motion-aware input gate is a key innovation in the MS-LSTM network. While traditional LSTM networks only process a single type of sequence data, the MS-LSTM network must process both speech feature sequences and motion state sequences simultaneously. The concatenation operation concatenates the motion-aware speech feature vector and the motion state embedding vector dimensionally, forming a comprehensive feature vector that contains both speech and motion information. The concatenation operation combines the two independent feature vectors into a single, longer feature vector through vector concatenation. The hidden state update mechanism of the MS-LSTM network adds a motion state processing branch to the traditional LSTM. The hidden state update considers not only the previous hidden state and the current speech input, but also the current motion state input. The sequence encoding process selectively retains, forgets, and updates information through the gating mechanism of the LSTM unit. The input gate determines the degree of retention of the current input, the forget gate determines the degree of retention of the previous information, and the output gate determines the degree of output of the current hidden state. The motion-aware hidden state sequence captures the temporal relationship between the speech sequence and the motion sequence.

[0084] Dynamic weight adjustment adaptively adjusts the hidden state sequence based on the robot's current movement speed. This process addresses the impact of varying movement speed on speech recognition accuracy. The robot's current movement speed is calculated using the vector modulus of the three-axis velocities, and the speed value reflects the robot's overall movement state. At high speeds, the ambient noise surrounding the robot increases, making speech signals susceptible to interference. The dynamic weight adjustment algorithm reduces the sensitivity of speech recognition by reducing the hidden state weights to suppress the impact of noise and prevent high-frequency noise from being mistakenly recognized as speech signals. At low speeds, the environment is relatively quiet and the speech signal quality is better. The algorithm improves speech recognition accuracy by increasing the hidden state weights to enhance the recognition of valid speech signals and capture more subtle changes in speech features. The speed-adaptive hidden state sequence is a dynamically weighted hidden state sequence that achieves optimal recognition performance at different movement speeds.

[0085] The CTC loss function, or connectionist temporal classification loss function, specifically addresses the mismatch between the input and output sequence lengths. In speech recognition tasks, the input speech feature sequence is typically longer than the output text sequence. Sequence alignment uses a dynamic programming algorithm to find the optimal alignment path between the hidden state sequence and the target text sequence. This alignment process allows multiple consecutive hidden states to correspond to the same character, and also allows some hidden states to correspond to whitespace. Probability calculation converts each hidden state into a character probability distribution using a softmax function. This probability distribution represents the probability of outputting each character at the current moment, with the sum of all character probabilities equal to one. Hidden state sequence mapping uses a fully connected layer to convert the hidden state features into a probability vector with the number of character dimensions, where each dimension corresponds to a character in the vocabulary. Path decoding uses a forward-backward algorithm to calculate the probabilities of all possible character paths and selects the path with the highest probability as the final recognition result. The decoding process merges consecutive identical characters into a single character and removes whitespace to form the final speech recognition text sequence.

[0086] In a specific embodiment, the process of executing step S105 may specifically include the following steps:

[0087] Perform semantic encoding on the speech recognition text sequence through the Transformer encoder, convert the text sequence into a high-dimensional semantic vector representation, and obtain the text semantic encoding;

[0088] The robot state vector is constructed based on the robot's current position, battery power, task status, and environment map information. Feature encoding is performed through the state embedding layer to obtain the robot state code.

[0089] Based on the cross-modal attention mechanism, the text semantic encoding and the robot state encoding are correlated and processed, and feature fusion processing is performed through query-key-value attention weight allocation to obtain cross-modal fusion features;

[0090] The cross-modal fusion features are input into a hierarchical classifier for hierarchical intent recognition. The first layer identifies four types of instructions: navigation, operation, query, and stop. The second layer extracts coordinates, angles, and object parameters to obtain intent classification results.

[0091] Structured instruction assembly is performed based on the intent classification results, and the instruction type, target parameters, execution priority and safety constraints are formatted and encapsulated to obtain structured control instructions.

[0092] Specifically, when the Transformer encoder performs semantic encoding on speech recognition text sequences, it first decomposes the text sequence into individual characters or word units. Each unit is converted to an initial vector representation through a word embedding layer. The word embedding vector contains the semantic information of the character but lacks context. A positional encoding mechanism adds positional information to each word embedding vector. Positional encoding uses sine and cosine functions to generate positional features of varying frequencies, ensuring that the same word has different vector representations at different positions. A multi-headed self-attention mechanism calculates the association weights between each word in the text sequence. The attention weights reflect the semantic dependencies between words. Attention scores are calculated through matrix operations on the query matrix, key matrix, and value matrix. The attention scores are softmax-normalized and then multiplied by the value matrix to obtain a weighted feature representation. A feedforward neural network applies nonlinear transformations to the attention outputs to enhance the expressive power of the features. The multi-layer Transformer encoder gradually abstracts semantic information through stacking, transforming the text sequence into a high-dimensional semantic vector representation containing complete semantic information, forming a semantic encoding of the text.

[0093] The robot state vector construction process integrates the robot's multidimensional state information. The robot's current position is represented by three-dimensional coordinate data obtained through GPS positioning or indoor positioning algorithms. These coordinates reflect the robot's spatial position in the working environment. The battery management module monitors the remaining battery charge in real time, providing a percentage of the remaining charge. This information directly affects the robot's task execution capability and range of motion. The task state contains information such as the type of task the robot is currently executing, its progress, and its priority. This task state data is maintained and updated by the task scheduling module. The environmental map contains spatial information such as the distribution of obstacles around the robot, traversable areas, and the location of target objects. This map data is created and updated using sensors such as lidar and cameras. The state embedding layer utilizes multiple parallel fully connected networks to process different types of state information. Position coordinates are converted to position feature vectors via a spatial encoding network, battery values ​​are converted to energy feature vectors via an energy encoding network, task information is converted to task feature vectors via a task encoding network, and map information is converted to environment feature vectors via a spatial encoding network. These four feature vectors are combined through vector concatenation or weighted fusion to form a unified robot state representation.

[0094] The cross-modal attention mechanism establishes an associative mapping relationship between text semantics and robot state, solving the problem of semantic alignment between data of different modalities. The correlation calculation process uses the text semantic encoding as the query vector and the robot state encoding as the key vector and value vector. The correlation score between each semantic unit in the text and each dimension of the robot state is calculated through matrix operations. The query-key-value attention weight allocation calculates the attention weight matrix using the scaled dot product attention algorithm. Each element of the weight matrix represents the strength of the association between the corresponding semantic unit and the state dimension. The larger the weight value, the stronger the association. The feature fusion process performs a weighted summation of the attention weight and the value vector of the state encoding. The weighted result is concatenated or added to the original text semantic encoding to form a cross-modal fusion feature that contains both semantic information and state information. This feature vector describes the comprehensive semantic representation of the voice command in the current robot state.

[0095] The hierarchical classifier employs a hierarchical classification strategy to handle the complex task of intent recognition. Hierarchical intent recognition breaks down the complex command parsing problem into multiple subproblems, each solved layer by layer. The first-layer classifier is responsible for coarse-grained command type identification, feeding cross-modal fusion features into a multi-classification neural network for category prediction. The network outputs a probability distribution for four categories: the navigation category corresponds to commands related to robot movement, the operation category corresponds to commands related to robot action execution, the query category corresponds to commands related to information acquisition, and the stop category corresponds to commands that terminate the current task. The second-layer classifier performs fine-grained parameter extraction based on the first-layer classification results. For navigation instructions, it extracts target coordinates and movement path parameters; for operation instructions, it extracts operation angle and target object parameters; for query instructions, it extracts query object and query type parameters; and for stop instructions, it extracts stop range and stop condition parameters. The parameter extraction process uses a sequence labeling algorithm to identify key information in the text and converts the identified parameter values ​​into numerical values ​​or identifiers. The intent classification result includes a command type label and a corresponding set of parameter values.

[0096] The structured instruction assembly process converts intent classification results into a standardized instruction format that the robot control module can understand and execute. This assembly process solves the format conversion problem between natural language instructions and machine control instructions. Instruction type mapping converts natural language instruction categories into instruction codes in the robot control protocol. Navigation instructions are mapped into movement control codes, operation instructions are mapped into action execution codes, query instructions are mapped into information acquisition codes, and stop instructions are mapped into task termination codes. Target parameter conversion converts the extracted parameter values ​​into the data format and value range required by the robot control module. Coordinate parameters are converted into standard coordinate values ​​in the robot coordinate system, angle parameters are converted into angle values ​​for the robot joint control system, and object parameters are converted into identifiers for the object recognition module. Execution priority determines the urgency of instruction execution based on the instruction type and the current robot state. Stop instructions have the highest priority, safety-related instructions have a high priority, general task instructions have a medium priority, and query instructions have a low priority. Safety constraints set instruction execution constraints based on the robot's safety rules and environmental constraints. These constraints include maximum movement speed, safety distance, obstacle avoidance requirements, and power thresholds. Formatting and packaging encapsulates all components according to the standard control protocol format, forming a structured control instruction with complete execution information.

[0097] The above describes the method for controlling the voice data of a mobile robot based on artificial intelligence in the embodiment of the present application. The following describes the control system for the voice data of a mobile robot based on artificial intelligence in the embodiment of the present application. Figure 2 In the embodiments of the present application, an embodiment of a mobile robot voice data control system based on artificial intelligence includes:

[0098] The acquisition module is used to collect and process the environmental voice signal through a multi-channel microphone array to obtain a three-channel voice data stream, and simultaneously obtain and process the robot's motion state parameters to obtain motion state data, and perform time synchronization processing on the three-channel voice data stream and the motion state data to obtain motion voice synchronization data;

[0099] a matching module, configured to calculate and process a motion noise influence factor based on the motion state data to obtain a motion intensity value, match the motion intensity value with a filter parameter using an MSAP algorithm, and perform motion compensation on the motion speech synchronization data to obtain a motion compensated speech signal;

[0100] An extraction module is used to input the motion-compensated speech signal into a dual-path convolutional neural network for feature extraction processing, and to fuse static speech features with motion speech features through an attention mechanism to obtain motion-aware speech features;

[0101] An encoding module is used to construct a motion state vector based on the motion state data, jointly encode the motion-perceived speech features and the motion state vector through an MS-LSTM network, and perform recognition processing using a CTC loss function to obtain a speech recognition text sequence;

[0102] The fusion module is used to perform cross-modal fusion processing on the speech recognition text sequence and the robot state information to obtain multimodal semantic features, and perform intent recognition processing on the multimodal semantic features through a hierarchical classifier to obtain structured control instructions.

[0103] above Figure 2 The artificial intelligence-based mobile robot voice data control system in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The artificial intelligence-based mobile robot voice data control device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0104] Reference Figure 3 In an embodiment of the present invention, there is also provided a mobile robot voice data control device based on artificial intelligence. The mobile robot voice data control device based on artificial intelligence can be a server, and its internal structure can be as follows: Figure 3As shown. The artificial intelligence-based mobile robot voice data control device includes a processor, memory, display screen, input device, network interface and database connected through a system bus. Among them, the computer-designed processor is used to provide computing and control capabilities. The memory of the artificial intelligence-based mobile robot voice data control device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the artificial intelligence-based mobile robot voice data control device is used to store the corresponding data in this embodiment. The network interface of the artificial intelligence-based mobile robot voice data control device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0105] Those skilled in the art will understand that Figure 3 The structure shown in is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the artificial intelligence-based mobile robot voice data control device to which the solution of the present invention is applied.

[0106] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the steps of the artificial intelligence-based mobile robot voice data control method.

[0107] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0108] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing an artificial intelligence-based mobile robot voice data control device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0109] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for controlling voice data of a mobile robot based on artificial intelligence, characterized in that: The method comprises: The environmental voice signal is collected and processed by a multi-channel microphone array to obtain a three-channel voice data stream, and the robot motion state parameters are simultaneously acquired and processed to obtain motion state data, and the three-channel voice data stream is time-synchronized with the motion state data to obtain motion voice synchronization data; Calculating and processing a motion noise influence factor according to the motion state data to obtain a motion intensity value, matching the motion intensity value with a filter parameter using an MSAP algorithm, and performing motion compensation on the motion speech synchronization data to obtain a motion compensated speech signal; Inputting the motion-compensated speech signal into a dual-path convolutional neural network for feature extraction, fusing static speech features with motion speech features through an attention mechanism to obtain motion-aware speech features; Constructing a motion state vector based on the motion state data, jointly encoding the motion-perceived speech features and the motion state vector through an MS-LSTM network, and performing recognition processing using a CTC loss function to obtain a speech recognition text sequence; The speech recognition text sequence is cross-modally fused with the robot state information to obtain multimodal semantic features, and the multimodal semantic features are subjected to intent recognition processing by a hierarchical classifier to obtain structured control instructions.

2. The method for controlling mobile robot voice data based on artificial intelligence according to claim 1, characterized in that: The method collects and processes the ambient voice signal through the multi-channel microphone array to obtain a three-channel voice data stream, simultaneously acquires and processes the robot motion state parameters to obtain motion state data, and performs time synchronization processing on the three-channel voice data stream and the motion state data to obtain motion voice synchronization data, including: The ambient voice signal is collected and processed by three omnidirectional microphones to obtain a first audio signal, a second audio signal and a third audio signal; Performing data stream combination processing on the first audio signal, the second audio signal and the third audio signal to obtain a three-channel voice data stream; The robot's three-axis velocity, three-axis angular velocity, and three-axis acceleration are collected and processed through the IMU sensor to obtain motion state data; Performing time stamp processing on the three-channel voice data stream and motion state data based on a hardware clock interrupt mechanism to obtain a time stamp data sequence; The time-stamped data sequence is aligned according to the timestamp to obtain motion speech synchronization data.

3. The method for controlling mobile robot voice data based on artificial intelligence according to claim 1, characterized in that: The method further comprises: calculating and processing the motion noise influence factor according to the motion state data to obtain a motion intensity value, matching the motion intensity value with a filter parameter through an MSAP algorithm, and performing motion compensation on the motion speech synchronization data to obtain a motion compensated speech signal, including: Performing square sum calculation on the three-axis velocity, three-axis angular velocity and three-axis acceleration in the motion state data to obtain a motion noise impact factor; Performing weighted summation processing on the motion noise impact factors according to a preset weight coefficient to obtain a motion intensity value; The motion intensity value is threshold-judged by the MSAP algorithm, a 2nd-order filter is selected when the motion intensity value is less than 0.3, a 4th-order filter is selected when the motion intensity value is between 0.3 and 0.7, and a 6th-order adaptive filter is selected when the motion intensity value is greater than 0.7, to obtain dynamic filtering parameters; Performing frequency domain filtering on the motion speech synchronization data based on the dynamic filtering parameters to obtain noise-reduced speech data; Doppler frequency shift compensation processing is performed on the noise reduction voice data according to the robot motion trajectory to obtain a motion compensated voice signal.

4. The method for controlling mobile robot voice data based on artificial intelligence according to claim 1, characterized in that: The motion-compensated speech signal is input into a dual-path convolutional neural network for feature extraction, and static speech features and motion speech features are fused through an attention mechanism to obtain motion-perceived speech features, including: Inputting the motion-compensated speech signal into a static feature extraction path and a motion feature extraction path respectively for parallel processing to obtain a dual-path input signal; Performing a cascade of standard convolution, batch normalization, and ReLU activation functions on the dual-path input signal through a static feature extraction path to obtain static speech features; Processing the dual-path input signal through a motion feature extraction path using a dilated convolution, a residual connection, and a gating mechanism to obtain motion speech features; Performing query-key-value mapping processing on the static speech features and the motion speech features based on a multi-head attention mechanism to obtain a cross-path correlation matrix; performing adaptive weight distribution processing on the static speech features and the motion speech features according to the cross-path correlation matrix to obtain a weighted feature vector; The weighted feature vectors are subjected to feature dimension alignment and normalization processing to obtain motion-aware speech features.

5. The method for controlling mobile robot voice data based on artificial intelligence according to claim 4, characterized in that: The motion feature extraction path processes the dual-path input signal through a dilated convolution, a residual connection, and a gating mechanism to obtain motion speech features, including: The dual-path input signal is input into a dilated convolution layer for convolution processing with different dilation rates. The multi-time scale features of the robot motor noise, tire friction sound, and wind resistance sound are captured by dilated convolution with dilation rates of 2, 4, and 8, thereby obtaining motion noise separation features. Based on the robot speed change gradient, the motion noise separation feature is processed by residual connection, and the spectrum drift information caused by acceleration is weighted superimposed with the original feature to obtain the speed compensation feature; The speed compensation feature is adaptively screened through a motion state gating mechanism, a gating threshold is calculated based on the robot's steering angle and movement direction, and feature suppression processing is performed to obtain a direction perception feature; Based on the robot trajectory prediction information, the direction perception features are time-series aligned, and the impact of future trajectory changes on the current voice signal is compensated in advance to obtain trajectory prediction features; The trajectory prediction features are subjected to motion state labeling processing, and feature coding processing is performed according to four motion modes: straight-ahead, turning, acceleration, and deceleration, to obtain motion speech features.

6. The method for controlling mobile robot voice data based on artificial intelligence according to claim 1, characterized in that: The method constructs a motion state vector according to the motion state data, performs joint encoding processing on the motion-perceived speech features and the motion state vector through an MS-LSTM network, and performs recognition processing using a CTC loss function to obtain a speech recognition text sequence, including: Performing vector concatenation processing on the three-axis velocity, three-axis angular velocity, three-axis acceleration and motion intensity value in the motion state data to obtain a ten-dimensional motion state vector; Performing dimension mapping processing on the ten-dimensional motion state vector through a linear transformation layer, converting the motion parameters into the same feature dimension as the speech feature, and obtaining a motion state embedding vector; The motion-aware speech feature and the motion state embedding vector are concatenated in series based on the motion-aware input gate, and sequence encoding is performed through the hidden state update mechanism of the MS-LSTM network to obtain a motion-aware hidden state sequence; Dynamically weighting the motion perception hidden state sequence according to the current movement speed of the robot, reducing recognition sensitivity at high speed and improving recognition accuracy at low speed, thereby obtaining a speed-adaptive hidden state; The speed adaptive hidden state is sequence aligned and probability calculated using a CTC loss function, the hidden state sequence is mapped to a character probability distribution and path decoding is performed to obtain a speech recognition text sequence.

7. The method for controlling mobile robot voice data based on artificial intelligence according to claim 1, characterized in that: The cross-modal fusion processing of the speech recognition text sequence and the robot state information to obtain multimodal semantic features, and the intention recognition processing of the multimodal semantic features by a hierarchical classifier to obtain structured control instructions include: Performing semantic encoding processing on the speech recognition text sequence through a Transformer encoder, converting the text sequence into a high-dimensional semantic vector representation, and obtaining a text semantic encoding; The robot state vector is constructed based on the robot's current position, battery power, task status, and environment map information. Feature encoding is performed through the state embedding layer to obtain the robot state code. Based on the cross-modal attention mechanism, the text semantic encoding and the robot state encoding are correlated and calculated, and feature fusion processing is performed through query-key-value attention weight allocation to obtain cross-modal fusion features; The cross-modal fusion features are input into a hierarchical classifier for hierarchical intent recognition processing. The first layer identifies four types of instructions: navigation, operation, query, and stop. The second layer extracts coordinates, angles, and object parameters to obtain intent classification results. Structured instruction assembly processing is performed based on the intention classification results, and the instruction type, target parameters, execution priority and security constraints are formatted and packaged to obtain structured control instructions.

8. A mobile robot voice data control system based on artificial intelligence, characterized in that: Used to implement the artificial intelligence-based mobile robot voice data control method according to any one of claims 1 to 7, the artificial intelligence-based mobile robot voice data control system includes: An acquisition module is configured to acquire and process ambient voice signals through a multi-channel microphone array to obtain a three-channel voice data stream, simultaneously acquire and process robot motion state parameters to obtain motion state data, and perform time synchronization processing on the three-channel voice data stream and the motion state data to obtain motion voice synchronization data; a matching module, configured to calculate and process a motion noise influence factor based on the motion state data to obtain a motion intensity value, match the motion intensity value with a filter parameter using an MSAP algorithm, and perform motion compensation on the motion speech synchronization data to obtain a motion compensated speech signal; An extraction module is used to input the motion-compensated speech signal into a dual-path convolutional neural network for feature extraction processing, and to fuse static speech features with motion speech features through an attention mechanism to obtain motion-aware speech features; An encoding module is used to construct a motion state vector based on the motion state data, jointly encode the motion-perceived speech features and the motion state vector through an MS-LSTM network, and perform recognition processing using a CTC loss function to obtain a speech recognition text sequence; The fusion module is used to perform cross-modal fusion processing on the speech recognition text sequence and the robot state information to obtain multimodal semantic features, and perform intent recognition processing on the multimodal semantic features through a hierarchical classifier to obtain structured control instructions.

9. A mobile robot voice data control device based on artificial intelligence, characterized in that: It includes a memory and a processor, the memory stores a computer program that can be run on the processor, and the processor implements the artificial intelligence-based mobile robot voice data control method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the processor executes the artificial intelligence-based mobile robot voice data control method according to any one of claims 1 to 7.

Images