Voice interaction intelligent mechanical arm based on large language model

Abstract

The invention discloses a voice interaction intelligent mechanical arm based on a large language model, and relates to the technical field of mechanical arms. Comprising a voice recognition system which mainly completes real-time acquisition, preprocessing, noise reduction, feature extraction, recognition and the like of voice instructions; and the main control system is mainly used for receiving the voice instruction and sending corresponding instruction information to the mechanical arm control end so that the mechanical arm can grab and sort different goods. Meanwhile, when the mechanical arm works, the operation state and position information of the mechanical arm can be displayed on a main control system interface; and the action of the mechanical arm executing mechanism is controlled by the mechanical arm NX100 control cabinet. According to the design method disclosed by the invention, in a noise environment, the accurate understanding and response capability of the mechanical arm to a voice instruction is greatly improved by combining a large-scale pre-training language model and an advanced voice recognition technology.

Description

Technical Field

[0001] The present invention relates to the technical field of robotic arms, and in particular to a voice interactive intelligent robotic arm based on a large language model. Background Art

[0002] In complex and ever-changing industrial and living environments, voice interaction is an important way of human-computer interaction. Its accuracy is often seriously disturbed by environmental noise. Traditional speech recognition technology is difficult to maintain efficient and stable performance in noisy environments. In recent years, with the rapid development of artificial intelligence technology, especially the rise of large-scale pre-trained language models, it has brought revolutionary breakthroughs in the field of speech recognition. This type of model has powerful language understanding and generation capabilities through massive data training. It can effectively extract voice features in high-noise environments and accurately recognize and understand users' voice commands. Researching and applying speech recognition technology based on large language models to intelligent robotic arm control can not only significantly improve the operating accuracy and response speed of robotic arms in complex environments, but also greatly broaden the application scenarios of voice interaction technology, promote the intelligent process in smart manufacturing, smart home and other fields, and provide strong technical support for improving production efficiency and user experience.

[0003] For speech recognition technology, especially in noisy environments, traditional speech recognition technology is difficult to accurately recognize the user's voice commands, which limits the effective control and operation of the robot arm. Specifically, in noisy environments, such as when there is background noise, volume changes, or speech and noise are mixed, traditional speech recognition systems face the challenges of difficulty in extracting speech features and difficulty in accurately distinguishing between speech and noise, which leads to a significant reduction in the accuracy of speech recognition, which in turn affects the reception and execution of commands by the robot arm. At the same time, the speech denoising algorithm has the following problems: it does not fully consider the importance of contextual information, and speech processing may cause the loss of detailed information of the speech signal; it ignores the correlation and potential diversified information between high-dimensional features of different channels. The training of some existing models for speech recognition is unstable, and even causes training crashes as the number of shared layers increases. In addition, the performance is significantly reduced compared to the original model; although the model is compressed, the number of parameters of some models is still large, which is not suitable for use in environments with limited computing resources. Secondly, the intelligent robot arm still has limitations in understanding and executing complex voice commands, especially when the command involves multiple continuous actions or operations in specific situations, the robot arm often cannot accurately parse and respond correctly. Summary of the invention

[0004] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a voice interactive intelligent robotic arm based on a large language model.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] Voice interactive intelligent robotic arm based on large language model, including:

[0007] Speech recognition system: The speech recognition system mainly completes the real-time collection, preprocessing, noise reduction, feature extraction and recognition of voice commands;

[0008] The main control system is mainly used to receive voice commands and send corresponding command information to the robot arm control end so that the robot arm can complete the grabbing and sorting of different goods. At the same time, the robot arm can display its operating status and position information on the main control system interface when it is working;

[0009] The action of the robot arm actuator is controlled by the robot arm NX100 control cabinet. During operation, the PC sends control instructions to the control cabinet, and the control cabinet allocates different pulses to the robot arm AC servo motor according to different instruction information. According to different pulse information, the robot arm can achieve different actions.

[0010] Preferably: when performing speech recognition, the speech recognition system of the intelligent robotic arm first uses a high-precision microphone to collect speech signals, and performs preprocessing, feature extraction, denoising and corresponding algorithms, DFSMN-CTC end-to-end model and lightweight language model, speech recognition algorithm, etc. on the acquired speech signals; specifically including:

[0011] Preprocessing: Use pre-emphasis and windowing to frame. Energy will be lost during the propagation of sound, and the higher the sound wave frequency, the greater the energy loss. MFCC pre-emphasis processing makes up for the energy loss during the propagation of sound and compensates for the high-frequency components. Since the sound emitted by the speaker is an uneven signal, it is necessary to analyze the distribution of each frequency in the speech signal. In the actual research process, the main focus is on analyzing uniform and stable signals. Through the framing operation, the signal can be guaranteed to be stable in a short time. Windowing is to make the signals at both ends of the signal approach 0, avoid spectrum overlap, and reduce spectrum leakage.

[0012] Feature extraction: Before training the acoustic model, it is necessary to extract the acoustic signal features. The present invention adopts the Mel-frequency cepstral coefficient (MFCC) feature extraction method, which can better describe consonants.

[0013] Noise reduction: Noise reduction processing is performed based on the noise reduction algorithm; the design method of the present invention mainly targets additive noise for front-end voice noise reduction. Additive noise refers to a form of noise that exists independently of the signal during the signal acquisition or transmission process. Its characteristic is that it does not change with the signal in time and frequency, such as electronic noise, wind noise, traffic noise, etc.

[0014] Preferably: the noise reduction algorithm is specifically:

[0015] Before the speech signal is input into the speech recognition system, the speech signal is denoised. The speech denoising algorithm CA-DCDCCRN based on the coordinated attention deep complex densely connected convolutional recurrent network is used to perform front-end processing on the noisy speech signal.

[0016] The DCDCCRN network design consists of two parts: a densely connected network and a DCDCCRN network structure; the densely connected DC network consists of densely connected blocks, and the DC network integrates and utilizes more low-level features; when the network is propagating, each layer of the densely connected block will receive the gradient signals of all the previous layers; for the L-layer network, there are L layer connections, and the DC connection network has L(L-1) / 2 inter-layer connections. Each layer is connected to all other layers, obtains additional inputs from all previous layers, and propagates the feature map of this layer to all subsequent layers.

[0017] Preferably: the network structure of the DCDCCRN, the complex codec of the DCDCCRN consists of a convolution block, a complex batch normalization module and a real PReLU module; wherein the convolution block comprises a real module and a complex module, the real module uses real values ​​for calculation, the complex module simulates complex multiplication, and the convolution block uses the relationship between amplitude and phase for modeling without prior knowledge;

[0018] Preferably: the DCDCCRN is a symmetrically designed convolutional encoder-decoder structure, the input is a noisy spectrogram, but the difference is that the DCDCCRN does not perform an STFT inverse transform in the output part, ensuring that the output is a denoised spectrogram; the real convolution module and the imaginary convolution module in the original encoder structure are replaced by DC and complex densely connected CDC blocks; by decomposing a convolution layer into several convolution layers with fewer channels and densely connecting these layers, each layer will be connected to all previous layers in the Channel dimension and serve as the input of the next layer, thereby obtaining all the features of the deeper layer.

[0019] Preferably: the CA-DCDCCRN network consists of 6 complex encoder modules, a complex LSTM module and 6 complex decoder modules; wherein the complex encoding and decoding module uses four layers of DC blocks and CDC blocks to replace the standard convolution in the original encoding and decoding network, and adds a CA module in each encoding block to enhance the network's processing of important features and suppress noise interference; the encoder in the CA-DCDCCRN extracts high-level features from the input features and reduces the resolution, the decoder reduces the resolution and reconstructs the features to the original size of the input, and the LSTM layer captures the long-term correlation of speech features to maintain the causality of the entire framework.

[0020] Preferably, the training target of the CA-DCDCCRN network is to estimate CIRM and optimize it through signal approximation. The loss function used is the scale-invariant source-to-noise ratio SI-SNR. The SI-SNR is calculated as follows:

[0021]

[0022]

[0023]

[0024] Where:

[0025] S. For clean speech, estimate the time domain waveform of speech;

[0026] <·,·> is the dot product between two vectors;

[0027] |||| 2 is the Euclidean distance.

[0028] Preferably: the main control system:

[0029] The computer's built-in microphone collects voice commands in real time. After recognition by the PC, different control information will be sent to the robotic arm based on the recognition results to enable the robotic arm to complete the grabbing and sorting of different types of goods.

[0030] Preferably: the main control system:

[0031] ① Receive the recognition result recognized by the machine hearing system;

[0032] ② According to the received recognition result information, determine the location information of the goods to be grabbed by the robot arm and complete the pulse distribution of the servo motors of each degree of freedom;

[0033] ③ The robot arm control data is sent to the NX100 control cabinet via Ethernet communication, and the control cabinet controls the position of the robot arm to achieve the robot arm's task of grabbing and sorting goods.

[0034] Preferably: when the NX100 control cabinet is controlling the robot arm:

[0035] The control signals in the program are sent to the robot arm to drive and control the servo motors of each joint of the robot arm. The NX100 control cabinet adopts a PC bus structure. The control cabinet includes a servo unit, rectifier, circuit breaker and power connection unit. It also has a cooling system inside. An emergency stop button is set on the outside of the control cabinet.

[0036] The beneficial effects of the present invention are:

[0037] 1. The design method of the present invention greatly improves the robot arm's ability to accurately understand and respond to voice commands in a noisy environment by combining a large-scale pre-trained language model with advanced speech recognition technology.

[0038] 2. The intelligent robotic arm of the present invention can effectively filter background noise in a noisy environment, accurately capture the user's voice commands, and convert them into operations executable by the robotic arm, thereby achieving a more intuitive, efficient and intelligent interactive experience, and improving the operational accuracy and adaptability of the robotic arm in various application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The overall block diagram of the voice interaction intelligent robotic arm based on the large language model proposed by the present invention;

[0040] Figure 2 This is a structural diagram of the DCDCCRN model of the voice interactive intelligent robotic arm based on the large language model proposed in the present invention;

[0041] Figure 3 This is a coding and decoding structure diagram of the DCDCCRN of the voice interactive intelligent robotic arm based on the large language model proposed in the present invention;

[0042] Figure 4 The CA-DCDCCRN model diagram of the voice interactive intelligent robotic arm based on the large language model proposed by the present invention;

[0043] Figure 5 This is a DFSMN model structure diagram of the voice interactive intelligent robotic arm based on a large language model proposed in the present invention;

[0044] Figure 6 This is a CTC model structure diagram of the voice interactive intelligent robotic arm based on a large language model proposed in the present invention;

[0045] Figure 7 A schematic diagram of the WSWTHM weight calculation method of the voice interactive intelligent robotic arm based on a large language model proposed in the present invention;

[0046] Figure 8 This is the Encoder structure diagram of the LLMT of the voice interactive intelligent robotic arm based on the large language model proposed in the present invention;

[0047] Fig. 9 This is a flow chart of the main control system of the voice interactive intelligent robotic arm based on the large language model proposed by the present invention;

[0048] Fig.10 This is a flow chart of the cargo grabbing and sorting program of the robot arm voice control system of the voice interactive intelligent robot arm based on the large language model proposed by the present invention. DETAILED DESCRIPTION

[0049] The technical solution of the present invention is further described in detail below in conjunction with specific implementation methods.

[0050] Embodiment 1:

[0051] The voice interactive intelligent robot arm based on the large language model is mainly divided into the voice recognition system, the main control system, the robot arm execution, etc. The overall block diagram of the control system based on the large model voice is as follows: Figure 1 shown.

[0052] (1) Speech recognition system: The speech recognition system mainly completes the real-time acquisition, preprocessing, noise reduction, feature extraction and recognition of voice commands;

[0053] (2) Main control system: The main control system is mainly used to receive voice commands and send corresponding command information to the robot arm control end so that the robot arm can complete the grabbing and sorting of different goods. At the same time, the operating status and position information of the robot arm can be displayed on the main control system interface when the robot arm is working.

[0054] (3) Robotic arm execution: The action of the robot arm actuator is controlled by the robot arm NX100 control cabinet. During operation, the PC sends control instructions to the control cabinet, which allocates different pulses to the robot arm AC servo motor according to different instruction information. The robot arm can achieve different actions according to different pulse information.

[0055] The present invention utilizes large language model technology to accurately recognize speech even in a noisy environment to control a robotic arm to sort goods.

[0056] For the speech recognition process, we first use a high-precision microphone to collect speech signals, and then perform preprocessing, feature extraction, denoising, and corresponding algorithms, DFSMN-CTC end-to-end models, lightweight language models, speech recognition algorithms, etc. on the acquired speech signals.

[0057] (1) Preprocessing: Use pre-emphasis and windowing to frame. Energy will be lost during the propagation of sound, and the higher the sound wave frequency, the greater the energy loss. MFCC pre-emphasis processing makes up for the energy loss during the sound propagation process and compensates for the high-frequency components. Since the sound emitted by the speaker is an uneven signal, it is necessary to analyze the distribution of each frequency in the speech signal. In the actual research process, the main focus is on analyzing uniform and stable signals. Through the frame operation, the signal can be guaranteed to be stable in a short time. Windowing is to make the signals at both ends of the signal approach 0, avoid spectrum overlap, and reduce spectrum leakage.

[0058] (2) Feature extraction: Before training the acoustic model, it is necessary to extract the acoustic signal features. The present invention adopts the Mel-frequency cepstral coefficient (MFCC) feature extraction method, which can better describe consonants.

[0059] (3) Noise reduction: The design method of the present invention mainly targets additive noise for front-end speech noise reduction. Additive noise refers to a form of noise that exists independently of the signal during the signal acquisition or transmission process. Its characteristic is that it does not change with the signal in time and frequency, such as electronic noise, wind noise, traffic noise, etc.

[0060] In view of the problem that humans can distinguish weak sounds in a quiet environment, but it becomes very difficult in a noisy environment, that is, when a stronger sound (main audio) and a weaker sound (masking audio) exist at the same time, the weaker sound is difficult to be perceived by the human auditory system. The present invention designs a speech noise reduction technology using time-frequency masking estimation, which can better identify and separate different parts of the speech signal through the "masking effect". The specific implementation is: a short-time Fourier transform (STFT) is performed on the signal to obtain a time-frequency spectrum, and then the time-frequency masking function is estimated. Finally, the Jiangyan closed function is applied to the original signal to remove the noise part and retain the speech signal. At the same time, the use of complex ratio mask estimation can improve the intelligibility of human speech under very noisy conditions. For a negative signal-to-noise ratio, the phase of the noise speech can better reflect the phase of the background noise than the phase of the target speech. By operating in the complex domain, the C1RM estimation can simultaneously enhance the amplitude and phase response of the noise speech and produce clean speech, that is, the complex spectrum of the given noise speech is obtained. The complex spectrum of the clean speech is obtained.

[0061] Noise Reduction Algorithm:

[0062] Before the speech signal is input into the speech recognition system, the speech signal is subjected to noise reduction processing to reduce the interference of noise on speech recognition, which can improve the accuracy and robustness of the speech recognition system. The present invention designs and proposes a speech noise reduction algorithm CADCDCCRN based on a deep complex densely connected convolutional recurrent network with coordinated attention to perform front-end processing on noisy speech signals.

[0063] The design of the DCDCCRN network consists of two parts: a densely connected network and a DCDCCRN network structure. The densely connected (DC) network is composed of densely connected blocks. The dense connection mode of the densely connected blocks enables the features in the network to be fully propagated and used. Compared with the features of the general deep network that directly rely on the output of the last layer, the DC network can integrate and utilize more low-level features. When the network is propagating, each layer of the densely connected block will receive the gradient signals of all the previous layers. For an L-layer network, there are L layer connections, and the DC connection network has L(L-1) / 2 inter-layer connections. Each layer is connected to all other layers, and can obtain additional inputs from all previous layers, and propagate the feature map of this layer to all subsequent layers, ensuring that information is transmitted to the maximum extent between layers of the network. This network has the characteristics of feature reuse and enhanced feature transfer.

[0064] Regarding the network structure of DCDCCRN. The complex codec of DCDCCRN is composed of a convolution block, a complex batch normalization module and a real PReLU module. The convolution block consists of two parts: a real module and a complex module. The real module uses real values ​​for calculation, and the complex module simulates complex multiplication. The convolution block can use the relationship between amplitude and phase for modeling without prior knowledge. However, when the DCCRN model extracts spectrogram features, in order to capture speech context information, pooling downsampling is selected to expand the receptive field, and the pooling operation will lose the local features of the spectrogram, resulting in the loss of speech detail information. Therefore, in order to make up for the information loss in the downsampling process, the present invention designs a deep complex densely connected convolutional recursive network (DCDCCRN), and the model structure is as follows: Figure 2 shown.

[0065] Similar to DCCRN, DCDCCRN is also a symmetrically designed convolutional encoder-decoder structure. The input is a noisy spectrogram. The difference is that DCDCCRN does not perform an inverse STFT transform in the output part to ensure that the output is a denoised spectrogram. In addition, the internal structure of the encoder and decoder of the first 5 layers is redesigned, and the real convolution module and imaginary convolution module in the original encoder structure are replaced by DC and complex dense connection (CDC) blocks. This design is based on the idea of ​​reusing feature maps. By decomposing a convolution layer into several convolution layers with fewer channels and densely connecting these layers, each layer will be connected to all previous layers in the Channel dimension and serve as the input of the next layer to obtain all the features of the deeper layer. The encoder-decoder structure of DCDCCRN is shown below. Figure 3 shown.

[0066] Its encoder consists of a convolution module, a complex batch normalization layer, and a real activation function. The convolution module treats the real and imaginary parts as two input channels. The real convolution part consists of 4 DC blocks, and the imaginary convolution part consists of 4 CDC blocks. Each dense block contains a batch normalization layer and a parameter correction linear unit. Assuming that the growth rate of the dense block is set to 8, it means that the number of output channels of the first three convolution layers is 8, and the last layer accepts all previous outputs and performs a normal convolution operation.

[0067] The present invention adds DC, CDC and CA modules to the CA-DCDCCRN model. Compared with the original DCCRN model, CA-DCDCCRN uses the encoder's multi-layer dense connection blocks to capture the detailed information of the spectrogram, and then fuses the attention of the channel dimension and the spatial dimension to learn key information and suppress invalid feature information, increase the attention to the effective features in the spectrogram, and the module has a small amount of calculation and does not increase too many network parameters. The CA-DCDCCRN model structure is as follows: Figure 4 shown.

[0068] The CA-DCDCCRN network consists of 6 complex encoder modules, complex LSTM modules and 6 complex decoder modules. The complex encoder-decoder module uses four layers of DC blocks and CDC blocks to replace the standard convolution in the original encoder-decoder network, deepen the network structure, enhance the information flow between the front and back layers, obtain all the features of the previous layer, and solve the gradient disappearance problem to a certain extent. In addition, a CA module is added to each encoding block to enhance the network's processing of important features, further improve the quality and intelligibility of speech, and suppress noise interference. The encoder in CA-DCDCCRN is designed to gradually extract high-level features from the input features and reduce the resolution, while the decoder is able to reduce the resolution and reconstruct the features to the original size of the input, and the middle LSTM layer undertakes the task of capturing the long-term correlation of speech features and maintaining the causality of the entire framework.

[0069] The training goal of the CA-DCDCCRN network is to estimate CIRM and optimize it through signal approximation. The loss function used is the scale-invariant source-to-noise ratio SI-SNR. The SI-SNR calculation is shown in the following formula:

[0070]

[0071]

[0072]

[0073] Where:

[0074] S. For clean speech, estimate the time domain waveform of speech;

[0075] <·,·> is the dot product between two vectors;

[0076] |||| 2 is the Euclidean distance.

[0077] The speech denoising method based on CA-DCDCCRN converts the noisy speech into a noisy speech spectrogram through STFT during the training phase, with a window length of 20ms and a frame offset of 10ms. The noisy speech spectrogram is used as the input of the CA.DCDCCRN model, and the network parameters are updated through the SI.SNR loss function, and the CIRM value is estimated to obtain the optimal model. The number of output channels of each encoder layer is 16, 32, 64, 128, 256, and 256, respectively. The number of input channels of the CA block is 16, 32, 64, 128, 256, and 256, respectively. The number of units in the two complex LSTM layers is set to 256, and the number of input channels of each decoder layer is 256, 256, 128, 64, 32, and 16, respectively. During the network training process, the size and step size of the convolution kernel are (5, 2) and (2, 1), respectively.

[0078] Experimental data set and evaluation indicators of CA-DCDCCRN model:

[0079] 1. Noise dataset: The clean speech in the experiment comes from the Thchs30 dataset, and the noise comes from noise92. Eight types of environmental noise were selected from noise92, and 20 hours of clean speech were extracted from Thchs30, and mixed into a training set with random signal-to-noise ratio and random speakers. Three types of environmental noise (Babble, Factory, MachineGun) were selected from noisey92, and 200 clean speeches of different speakers were selected from Thchs30. The clean speech was mixed with one of the types of noise at a signal-to-noise ratio of -10, -5, 0, 5, 10, and 15dB to form 1200 noisy speeches, and a total of 3600 noisy speeches were obtained as the test set.

[0080] 2. Evaluation indicators: The evaluation indicators of the experiment are speech perception quality (PESQ), short-term intelligibility (STOI), and output speech signal-to-noise ratio (SNR). The larger the values ​​of PESQ and STOI, the better the speech quality, and the greater the improvement of SNR, the more effective the noise reduction algorithm is.

[0081] (4)DFSMN-CTC end-to-end model:

[0082] The present invention is designed to utilize the DFSMN-CTC end-to-end model, that is, a combination of the DFSMN model and the CTC end-to-end speech recognition model.

[0083] The DFSMN model is an improvement of the Compact Feedforward Sequential Memory Network (CFSMN) model on the basis of the Feedforward Sequential Memory Network (FSMN) model. Although the FSMN model can retain the dependencies between sequences, the parameters of the model increase as the depth of the model increases.

[0084] The DFSMN model introduces skip connections between storage blocks in adjacent layers. These skip connections enable information to flow between different layers, thereby alleviating the gradient vanishing problem when building very deep network structures. The calculation is shown in the following formula:

[0085]

[0086] Where:

[0087] H is a linear or nonlinear function;

[0088] is the characteristic of the radiation layer at time t;

[0089] It is the descending cone mapping layer.

[0090] In addition, by borrowing the idea of ​​dilated convolution, DFSMN introduces a stride factor in the memory module. The DFSMN model structure is as follows: Figure 5 shown.

[0091] The CTC end-to-end model refers to a deep learning model. Since the length of the output label in speech is generally smaller than the length of the input speech frame, the corresponding label needs to be obtained for each frame. Therefore, CTC allows repeated output labels and inserts a placeholder between repeated labels, such as "￡". Suppose the input sequence x is "good", the corresponding output sequence y should be "good", and the sequence after the introduction of the placeholder is "go￡od", and the final output is "good" instead of "god". In CTC, the mapping between the label sequence and the input sequence is learned by maximizing the conditional probability of a given input sequence. The CTC model structure is as follows Figure 6 shown.

[0092] (5) LLMT lightweight language model:

[0093] The present invention proposes a lightweight language model LLMT based on Transfrom. The model first replaces the original FFN structure with a feature-compensated lightweight feedforward network FCLFFN to reduce the model size while ensuring the performance of the model. Then, the WSWTHM weight sharing method is used to increase the diversity of reused weights and filter out invalid parameters in multi-head attention calculations to improve the performance of the language model.

[0094] FCLFFN network structure: The neurons between FFN layers are fully connected to input signals. The calculation results of the MSA layer are residually connected and layer normalized as the input of the FFN layer. The FFN layer performs two linear transformations to extract features more deeply. Through the study of the FFN layer, it is found that the calculation amount of Base FFN is 4,718,592, the calculation amount of LargeFFN is 8,388,608, and the calculation amount of xLarge FFN is 33,554,432. The calculation amount of the FFN layer in the language model is 12 times or 24 times that of Transform. Therefore, it is necessary to reduce the number of parameters of the FFN layer in the language model. In response to the above problems, this chapter proposes a lightweight feedforward network structure FCLFFN with feature compensation to reduce the number of parameter calculations while ensuring model performance. The linear layer in the FCLFFN network structure converts the input features to a high-dimensional space through activation, and then uses a nonlinear activation function to convert the output back to a low-dimensional space. The difference is that the FCLFFN network consists of two branches. The first branch expands the input features to a high-dimensional space and then reduces them to a lower dimension. The second branch uses nonlinear activation to process the input features and combines the processed features with the output of the first branch. Specifically, FCLFFN expands each set of features to four times the original dimension in the first layer and processes the input features using a nonlinear activation function; the second layer reduces the features to 1 / 4 of the initial dimension and concats the dimensionality reduction result with the result of the second branch to compensate for the feature loss problem caused by excessive dimensionality reduction. Compared with the FFN layer of the language model, the number of parameters has been reduced by 37.5%.

[0095] WSWTHM weight calculation method. Weight sharing is a simple and effective method to improve parameter efficiency. The core idea is to share parameters across layers. Mathematically, weight sharing can be formulated as a Transform block. The effectiveness of weight sharing has been explored and proven in the natural language converter model, which can improve parameter efficiency. However, weight sharing leads to the simplification of single-layer parameters of the language model, which damages the model performance to a certain extent. In response to the above problems, the present invention proposes a weight calculation method WSWTHM based on weight transformation and Hadamard matrix. The key idea of ​​weight transformation is to impose linear transformation on shared weights so that different layers have slightly different weights, thereby promoting parameter diversity and improving training stability. The Hadamard matrix is ​​used after the linear transformation to filter the attention weight matrix after the Softmax transformation by setting a threshold. Since the weights below the threshold have a weak effect on the output, they are forced to be converted to 0, and the scores above the threshold are not changed. The WSWTHM weight calculation method is as follows Figure 7 shown.

[0096] LLMT network structure. Through the study of weight reuse, Hadamard matrix, and FFN layer, this paper proposes a lightweight language model LLMT based on Transfrom. The LLMT baseline model is composed of a 12-layer encoder structure stacked together. Its input layer is the word embedding vector of the vocabulary. In order to retain the position information, the position embedding vector is added to the word vector. The encoder is composed of MSA and multi-layer perceptron FCLFFN alternatingly. Layer normalization and residual connection are applied before and after each block respectively. Residual connection can prevent the gradient from disappearing. In the overall calculation process of the LLMT model, LLMT first performs a Softmax transformation and a Hadamand matrix operation on the MSA layer of the first encoder, performs a Softmax transformation on the FCLFFN layer parameters, and uses FCLFFN to integrate multiple features. The transformed parameters are then shared with other encoder modules to increase the diversity of parameters and discard useless attention parameters. The weight reuse method includes the following two points: (1) sharing weights between multiple transformer blocks, which can be regarded as a combination process in reuse; (2) introducing linear transformation and Hadamard matrix operations in each layer, where linear transformation is applied to the attention matrix and FCLFFN layer parameters, so that the weights of each layer are different, thereby improving parameter diversity and model representation capabilities. Weight sharing can restore the pre-trained behavior of the model, so that the instability and performance degradation of training are alleviated. In addition, the Hadamard matrix can force the parameters below the threshold in the matrix A to be set to 0, so that they do not participate in subsequent calculations, thereby reducing the amount of calculation and speeding up the encoding and decoding speed of the language model. The Encoder structure of the LLMT model is as follows: Figure 8 shown.

[0097] Experimental datasets and evaluation indicators of LLMT language model:

[0098] 1. Dataset Introduction

[0099] The Thchs30 dataset is 50 hours of speech recorded in a quiet office environment using a single carbon microphone. Most of the recordings were made by people who are fluent in Mandarin.

[0100] The Aidatatang dataset is a Chinese Mandarin telephone speech database with a corpus of 300 hours. 600 speakers from different key areas of China were invited to participate in the recording. The language materials of the corpus are spoken sentences designed to be phonemically balanced, and the manual transcription accuracy of each sentence is greater than 98%.

[0101] The Magicdata dataset is a conference dataset with a duration of 200 hours and recorded by more than 300 participants, covering general, finance, technology, law, insurance, medical care, education, government, real estate and other fields.

[0102] The Aishell.1 dataset has a speech duration of 1018 hours. The recorded texts cover 12 fields such as wake-up words, voice control words, unmanned driving, and industrial production. Nearly 2,000 people recorded, and the data text accuracy is greater than 96%. The above four datasets were used for language model training, totaling 1573 hours, including 1,249,161 sentences. Among them, there are 1,061,786 training sets, 62,458 validation sets, and 124,916 test sets. The ratio of training sets, validation sets, and test sets is 85:5:10.

[0103] 2. Evaluation indicators

[0104] The evaluation criteria of language models include perplexity and word error rate. Perplexity evaluates the training effect of the model, and word error rate evaluates the actual application effect of the language model. Perplexity is a measure of the degree to which the language model is confused about a sentence. Perplexity is usually expressed as PPL. When PPL = m, it means that there are m choices for the next word of the current word. The smaller the value of m, the higher the accuracy of the language model and the lower the word error rate. The PPL calculation is shown in the following formula:

[0105]

[0106] (6) Speech recognition algorithm:

[0107] The present invention designs and proposes an end-to-end noisy speech recognition algorithm ENSRILLM that integrates a lightweight language model, and constructs two noisy speech recognition models ENSRILLM-S and ENSRILLM-L according to different language model fusion methods.

[0108] ENSRILLM-S model. The present invention is designed on the basis of ShallowFusion fusion method, and builds ENSRILLM-S noise speech recognition model. ENSRILLM-S first builds an acoustic model from speech to pinyin based on DFSMN network and CTC, then builds LLMT language model from pinyin to Chinese characters, and finally decodes the output probability P of CTC. CTC (y) and the probability P of the LLMT model LLMT(y) performs joint prediction to obtain the speech recognition result. The ENSRILLM-S model first performs STFT transformation on the input noisy speech signal to obtain a noisy spectrogram, uses the CA-DCDCCRN denoising module to denoise the noisy spectrogram, and uses the obtained clean spectrogram as the acoustic model input to train the acoustic model. Then the LLMT language model is trained separately using the text corpus, and finally the trained acoustic model and language model are shallowly fused, that is, in the output of each time step of CTC, the probability output of CTC and the probability output of LLMT are fused as the final output to complete the correction of the CTC result. The calculation is shown in the following formula:

[0109] P(y)=logP CTC (y)+λlogP LLMT (y)

[0110] The ENSRILLM-S model combines the acoustic model and the language model for decoding during the recognition phase to fully utilize the linguistic knowledge of the external language model. The model is also pluggable and the language model can be replaced as needed.

[0111] ENSRILLM-L model. The present invention is designed based on the Lean Spelling from Teachers fusion method and constructs the ENSRILLM-L noisy speech recognition model.

[0112] The ENSRILLM-L model is similar to the ENSRILLM-S model. First, the noisy speech information is framed by time window and converted into a spectrogram. The CA-DCDCCRN module performs noise reduction on the noisy spectrogram, and the processed spectrogram is used as the input of the acoustic model. Then, before training the acoustic model, the LLMT language model needs to be used to generate soft labels for speech transcriptions that carry text corpus knowledge, which serve as prior knowledge to guide the training of the end-to-end acoustic model. The encoder network of the acoustic model encodes the sequence into a high-level acoustic representation, providing hard labels for the next step of training. The hard label is a feature vector composed of 0 and 1, while the soft label is composed of a decimal between 0 and 1 and contains more information. Finally, the hard label and soft label are used on a time step of the training sequence to allow the decoder to simultaneously approximate the hard label and soft label.

[0113] Dataset Introduction

[0114] The present invention designs and fuses four datasets of Thchs-30, Aidatatang, Magicdata, and Aishell-1 with eight kinds of environmental noise in noise92 to obtain four kinds of noise speech datasets. Each noise is 2 minutes and 50 seconds. When making the noise dataset, the first 50% is mixed for training set and the last 50% is mixed for test set.

[0115] Main control system:

[0116] The computer's built-in microphone collects voice commands in real time. After recognition by the PC, different control information will be sent to the robotic arm based on the recognition results to enable the robotic arm to complete the grabbing and sorting of different types of goods.

[0117] In the robot arm voice control system, the reception and transmission of voice commands and the operation process of the entire system are completed by the main control system. In order to realize the voice command control of the robot arm to grab and sort the goods, the main control system is designed with the following functions: (1) Receive the recognition results recognized by the machine hearing system; (2) Determine the position information of the goods to be grabbed by the robot arm based on the received recognition result information and complete the pulse distribution of the servo motors of each degree of freedom; (3) Send the robot arm control data to the NX100 control cabinet via Ethernet communication, and the control cabinet controls the posture of the robot arm to realize the robot arm's grabbing and sorting tasks. The flow chart of the main control system for voice command control of the robot arm is shown in the figure below. Fig. 9 shown.

[0118] The robot arm operation control mode is to control the robot arm to grab and sort different goods through voice digital instructions. After receiving the voice instructions recognized by the machine hearing system, the main control system executes the operation function corresponding to the recognized instructions to realize the voice command to control the robot arm. Fig.10 shown.

[0119] Robotic arm movement:

[0120] The present invention is designed to utilize the MOTOMAN robotic arm platform. The MOTOMAN robotic arm hardware system consists of a control cabinet, a teaching programmer, a power supply cable, and a robotic arm body to form the entire robotic arm experimental platform, which also includes an instrument gripper that is closed and opened by an air pump. The functions and features of each part of the entire experimental platform are as follows:

[0121] (1) Robotic arm body

[0122] Each joint of the robot arm is controlled by an independent PLC to control the servo motor of each joint to complete the rotation movement of each joint. Each joint can move independently or in conjunction with other joints. The movement speed and angle of each joint are controlled accordingly by the different drive signals received. The six-degree-of-freedom robot arm simulates the movement of the human waist, back, shoulder, elbow and wrist, meeting the application requirements of most occasions.

[0123] (2) Teaching programming device

[0124] The main function of the teaching pendant is to allow the user of the robot arm to control the robot arm simply and directly. Users can use the teaching pendant to control the robot arm in three ways. One is the teaching mode that directly controls the movement of each axis of the robot arm. In this mode, the user can not only control each joint, but also adjust the movement speed of each axis; the other is the playback mode that directly calls the program. In this mode, the user can call the edited program to control the robot arm, for example, some writing programs can be called to let the robot arm execute writing commands; in remote mode, the robot arm can be connected to an external PC and the PC can be used to control the robot arm. In order to allow users to quickly master the use steps of the teaching pendant, a help wizard will be displayed on the screen of the teaching pendant to help users complete the operation steps of the robot arm.

[0125] (3) NX100 control cabinet

[0126] The NX100 control cabinet controls the robot arm mainly by sending control signals in the program to the robot arm to drive and control the servo motors of each joint of the MOTOMAN robot arm. In order to make it more open and scalable, the NX100 control cabinet adopts a PC bus structure. The main configuration inside the control cabinet includes a servo unit, a rectifier, a circuit breaker, and a power connection unit. At the same time, in order to ensure that the temperature inside the control cabinet does not exceed the standard, a cooling system is also installed inside. There is an emergency stop button in the upper left corner of the control cabinet. When the robot arm makes an incorrect action, you can press the emergency stop button to interrupt the movement of the robot arm.

[0127] The MOTOMAN robot arm can implement five types of instructions: robot arm motion trajectory control function, environmental parameter definition function, program control function, data input and output function, and calculation function. The motion trajectory control function includes continuous trajectory control mode and point-to-point control mode. The point-to-point control mode used by the robot arm controlled by voice commands based on the large model for grabbing and sorting goods in the patent of this invention only focuses on the initial position and end position of the robot arm and does not care about its specific motion trajectory.

[0128] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A voice interactive intelligent robotic arm based on a large language model, characterized by: include: Speech recognition system: The speech recognition system completes the real-time collection, preprocessing, noise reduction, feature extraction and recognition of voice commands; The main control system is used to receive voice commands and send corresponding command information to the robot arm control end, so that the robot arm can complete the grabbing and sorting of different goods; at the same time, the operating status and position information of the robot arm are displayed on the main control system interface when the robot arm is working; The action of the robot arm actuator is controlled by the robot arm NX100 control cabinet. During operation, the PC sends control instructions to the control cabinet, and the control cabinet allocates different pulses of the robot arm AC servo motor according to different instruction information. The robot arm realizes different actions according to different pulse information. When performing speech recognition, the speech recognition system of the intelligent robotic arm first uses a high-precision microphone to collect speech signals, and then preprocesses, extracts features, denoises, and uses corresponding algorithms, DFSMN-CTC end-to-end models, lightweight language models, and speech recognition algorithms to obtain the speech signals.

2. The voice interactive intelligent robotic arm based on a large language model according to claim 1, characterized in that: When performing speech recognition, the speech recognition system of the intelligent robotic arm specifically includes: Preprocessing: Use pre-emphasis and windowing to divide frames; MFCC pre-emphasis processing makes up for the energy loss in the sound propagation process and compensates for high-frequency components; through framing operations, the signal is stabilized; based on windowing, the signals at both ends of the signal are made close to 0 to avoid spectrum overlap and reduce spectrum leakage; Feature extraction: Acoustic signal features need to be extracted before acoustic model training; Noise reduction: Noise reduction processing is performed based on the noise reduction algorithm.

3. The voice interactive intelligent robotic arm based on a large language model according to claim 2, characterized in that: The noise reduction algorithm is specifically: Before the speech signal is input into the speech recognition system, the speech signal is denoised. The speech denoising algorithm CA-DCDCCRN based on the coordinated attention deep complex densely connected convolutional recurrent network is used to perform front-end processing on the noisy speech signal. The DCDCCRN network design consists of two parts: a densely connected network and a DCDCCRN network structure; the densely connected DC network consists of densely connected blocks, and the DC network integrates and utilizes more low-level features; when the network is propagating, each layer of the densely connected block will receive the gradient signals of all the previous layers; for the L-layer network, there are L layer connections, and the DC connection network has L(L-1) / 2 inter-layer connections. Each layer is connected to all other layers, obtains additional inputs from all previous layers, and propagates the feature map of this layer to all subsequent layers.

4. The voice interactive intelligent robotic arm based on a large language model according to claim 3, characterized in that: The network structure of the DCDCCRN, the complex codec of the DCDCCRN consists of a convolution block, a complex batch normalization module and a real PReLU module; wherein the convolution block includes a real module and a complex module, the real module uses real values ​​for calculation, the complex module simulates complex multiplication, and the convolution block uses the relationship between amplitude and phase for modeling without prior knowledge.

5. The voice interactive intelligent robotic arm based on a large language model according to claim 4, characterized in that: The DCDCCRN is a symmetrically designed convolutional encoder-decoder structure. Its input is a noisy spectrogram. The difference is that DCDCCRN does not perform STFT inverse transform in the output part to ensure that the output is a denoised spectrogram. The real convolution module and the imaginary convolution module in the original encoder structure are replaced by DC and complex densely connected CDC blocks. By decomposing a convolution layer into several convolution layers with fewer channels and densely connecting these layers, each layer will be connected to all previous layers in the Channel dimension and serve as the input of the next layer, thereby obtaining all the features of the deeper layer.

6. The voice interactive intelligent robotic arm based on a large language model according to claim 5, characterized in that: The CA-DCDCCRN network consists of 6 complex encoder modules, a complex LSTM module and 6 complex decoder modules; The re-encoding and decoding module uses four layers of DC blocks and CDC blocks to replace the standard convolution in the original encoding and decoding network, and adds a CA module in each encoding block to enhance the network's processing of important features and suppress noise interference; The encoder in CA-DCDCCRN extracts high-level features from the input features and reduces the resolution, the decoder reduces the resolution and reconstructs the features to the original size of the input, and the LSTM layer captures the long-term correlation of speech features and maintains the causality of the entire framework.

7. The voice interactive intelligent robotic arm based on a large language model according to claim 6, characterized in that: The training goal of the CA-DCDCCRN network is to estimate CIRM and optimize it through signal approximation. The loss function used is the scale-invariant source-to-noise ratio SI-SNR. The SI-SNR is calculated as follows: Where: S. For clean speech, estimate the time domain waveform of speech; <·,·> is the dot product between two vectors; || || 2 is the Euclidean distance.

8. The voice interactive intelligent robotic arm based on a large language model according to claim 1, characterized in that: The main control system: The computer's built-in microphone collects voice commands in real time. After recognition by the PC, different control information will be sent to the robotic arm based on the recognition results to enable the robotic arm to complete the grabbing and sorting of different types of goods.

9. The voice interactive intelligent robotic arm based on a large language model according to claim 8, characterized in that: The main control system: ① Receive the recognition result recognized by the machine hearing system; ② According to the received recognition result information, determine the location information of the goods to be grabbed by the robot arm and complete the pulse distribution of the servo motors of each degree of freedom; ③ The robot arm control data is sent to the NX100 control cabinet via Ethernet communication, and the control cabinet controls the position of the robot arm to achieve the robot arm's task of grabbing and sorting goods.

10. The voice interactive intelligent robotic arm based on a large language model according to claim 1, characterized in that: When the NX100 control cabinet is controlling the robot arm: The control signals in the program are sent to the robot arm to drive and control the servo motors of each joint of the robot arm. The NX100 control cabinet adopts a PC bus structure. The control cabinet includes a servo unit, rectifier, circuit breaker and power connection unit. It also has a cooling system inside. An emergency stop button is set on the outside of the control cabinet.

Images