Robot visual interaction control method based on AI large model

Abstract

The invention discloses a robot visual interaction control method based on an AI large model, and the method comprises the steps: firstly carrying out the audio preprocessing of an audio sampling result, then inputting the processed data into a voice recognition model and the AI large model, enabling the AI large model to carry out the classification and analysis of a text, and obtaining a voice recognition result; whether the voice recognition result is an intelligent control problem or an intelligent interaction problem is judged, control or reply is carried out respectively, a control command is sent to the intelligent terminal equipment, a reply text is input into the voice synthesis model, and a synthesized reply audio is obtained; and inputting the synthesized audio into an AI counterpart model to synthesize a digital human interaction video, and broadcasting reply content by an AI digital human. The large model and the natural language audio data are synthesized into the face action of the digital human, the visual interaction effect of the intelligent interaction control system is greatly improved, meanwhile, better scene landing is achieved by adopting the fine-tuning large model, the generalization ability of the system is improved, the multi-platform intelligent home equipment control method is achieved, and the application range is wide. And the method has extremely high expandability.

Description

Technical Field

[0002] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a robot visual interactive control method based on an AI large model. Background Art

[0004] AI big models refer to machine learning models with large-scale parameters and complex computing structures. They are usually built with deep neural networks and have billions or even hundreds of billions of parameters. These models can process massive amounts of data and complete various complex tasks, such as natural language processing, computer vision, and speech recognition. Big models learn complex patterns and features by training large amounts of data, so they have strong generalization capabilities and can make accurate predictions on unseen data. AI big models are widely used in robotics, smart services, smart medical care and other fields. For example, AI-assisted disease diagnosis, smart homes, smart service robots, etc. have extremely high requirements for the model's visual interaction effects and the smoothness of interaction with users. Therefore, combining AI big models with visualization, natural language processing and other technologies is an important means to improve the automation of robot decision-making and humanization of services.

[0005] At present, AI big model services in the field of robotics are mainly focused on text interaction and voice interaction, while there is still little research on integrated digital human interaction technology, and further research on digital interaction and control combined with AI big models has not yet been carried out. For example, the family health service robot and intelligent interaction method based on multimodal big models, published in CN118039057B, sends the text information recognized by voice to the big model to obtain a reply and control the robot, but does not have the ability of visual interaction and intelligent device control, and the user cannot get intuitive device execution feedback; the vast majority of smart home control scenarios lack attention to the interaction effect, such as a robot control method based on multimodal big models, published in CN117944052A, which improves the control effect of the robot through the big model, but ignores the fact that the big model can bring better human-computer interaction capabilities to the intelligent robot. Summary of the invention

[0007] In order to solve the problem that the robot interactive control system lacks large model decision-making and visualized human-computer interaction functions, the present invention provides a robot visualized interactive control method based on an AI large model.

[0008] The technical solution adopted by the present invention is: a robot visual interactive control method based on an AI large model, comprising the following steps:

[0009] Step 1, selecting a suitable sampling frequency for audio sampling based on Nyquist sampling theorem;

[0010] Step 2, use pre-emphasis and filtering methods to perform audio preprocessing, filter out noise in the sampled audio and enhance the high-frequency components in the sound;

[0011] Step 3, input the processed data into the speech recognition model to obtain the recognized content;

[0012] Step 4: Input the speech recognition result into the AI ​​big model. The AI ​​big model will analyze and process the text classification to determine the intelligent control and intelligent interaction categories. If it is a control instruction, it will reply with the number '1'; if it is an interaction instruction, it will reply with the number '2'; if the text content is not related to the above instructions, it will reply with the number '0'. The instruction is determined according to the number in the big model's reply content; if it is an intelligent control problem, go to step 5; if it is an intelligent interaction problem, go to step 6;

[0013] Step 5: When the AI ​​big model identifies an intelligent control problem, the system matches the relevant commands in the pre-prepared command set. If the match is successful, it goes to step 7; if the match fails, it goes to step 6.

[0014] Step 6: When the AI ​​big model identifies an intelligent interaction question, the system obtains the text response from the AI ​​big model and gives a response to the question; then proceed to step 8;

[0015] Step 7, sending the command to the intelligent robot terminal, and controlling the corresponding intelligent device through the intelligent robot terminal; proceed to step 8;

[0016] Step 8: Input the reply text into the speech synthesis model to obtain the synthesized reply audio;

[0017] Step 9, input the synthesized audio into the AI ​​counterpart model to synthesize the digital human interaction video;

[0018] Step 10: The AI ​​digital human announces the reply content.

[0019] Further, the step 2 is specifically as follows:

[0020] Step 2-1, pre-emphasize the sampled voice data:

[0021]

[0022] In the formula, is the input signal sampling points, The value range is 0.9-1;

[0023] Step 2-2: After pre-emphasis, the voice data is divided into frames, each frame is about 20-30ms, and the framed data is windowed:

[0024]

[0025] In the formula, is the sample index, is the frame length;

[0026] Step 2-3, perform fast Fourier transform on the above results:

[0027]

[0028] In the formula, represents the total number of sampling points in the transformation, In the frequency domain, The index of the frequency components;

[0029] Step 2-4, calculate the amplitude spectrum of each frame, convert it into energy spectrum and draw the spectrogram.

[0030] Further, the specific steps of steps 2-4 are:

[0031] Step 2-4-1, by taking the absolute value of the fast Fourier transform result, the amplitude spectrum of each frame is obtained;

[0032] Step 2-4-2, by squaring the amplitude spectrum value of each frequency component, the corresponding energy spectrum value is obtained;

[0033] Step 2-4-3, finally arrange the energy spectrum of each frame along the time axis, use the color depth to represent the energy size, and then draw the spectrogram.

[0034] Further, the step 3 is specifically as follows:

[0035] Step 3-1, input the drawn spectrogram into the VGGish acoustic model based on the Keras and TensorFlow framework and build a deep convolutional neural network with reference to VGG and train it;

[0036] Step 3-2, using the CTC algorithm to merge a large number of consecutively repeated symbols in the output of the acoustic model into the same symbol, and remove the silent separation markers to obtain the final actual phonetic phonetic symbol sequence; the specific steps are:

[0037] For each time step , CTC calculates the probability of each label; let is a possible alignment path, is the input sequence, then the probability formula of CTC is:

[0038]

[0039] In the formula, Indicates that at time step The neural network output of

[0040] The loss function of the CTC algorithm takes into account all valid alignment paths, and its objective function is:

[0041]

[0042] In the formula, Indicates that it can be mapped to the output label All valid paths of

[0043] And use the statistical language model hidden Markov model to convert the pinyin into the final recognized text and output it.

[0044] Furthermore, in step 4, the pre-set smart device control command and text information are synchronously input into the AI ​​big model, and the AI ​​big model determines the degree of relevance between the text information and the above control command, and determines whether the device executes the command or normal text interaction based on the return information of the big model.

[0045] Furthermore, in step 8, the text fed back by the AI ​​big model is preprocessed before the reply text is input into the speech synthesis model, and the special symbols or Arabic numerals in the text are converted into text through text regularization; the rhythm structure of the regularized text is predicted through rhythm detection, and rhythm detection is realized through the hidden Markov model. Its probability calculation includes forward algorithm and backward algorithm, and the core recursive relation function is expressed as:

[0046]

[0047]

[0048] In the formula, and are the forward and backward probabilities, respectively, is the state transition probability, is the observation probability;

[0049] Convert glyphs into phonemes and use a lookup table to map Chinese characters to pinyin;

[0050] Modeling context-dependent pronunciation variation using conditional random fields;

[0051] The estimated conditional probability formula is expressed as:

[0052]

[0053] In the formula, is the normalization factor, is the characteristic function, is the corresponding weight parameter.

[0054] Further, in step 8, the speech synthesis model is a Tacotron2 acoustic model, including a spectrum prediction network and a vocoder, and outputs a Mel spectrum according to the input pinyin text;

[0055] Step 8-1, the spectrogram prediction network first converts the input character into a 512-dimensional character vector through an encoder and passes it through three layers of convolution, each layer of convolution has 512 5×1 convolution kernels, and each convolution kernel spans 5 characters;

[0056] Step 8-2, batch normalization and ReLU activation function are applied after the convolution layer to obtain the large-span context information of the character sequence; the output of the last convolution layer is transmitted to a bidirectional LSTM layer to generate encoding features;

[0057] Step 8-3: The decoder predicts the output spectrum frames one by one from the encoded input sequence. The predicted spectrum frames are first passed into a two-layer fully connected "Pre-Net", each layer consisting of 256 hidden ReLU units; the decoder's Pre-Net output and the attention context vector are concatenated together to form the input for the next decoding step;

[0058] Step 8-4, using the content- and position-based attention mechanism to allow the model to focus on different positions in the input sequence, thereby generating a more accurate spectrum frame;

[0059] The calculation of the attention mechanism includes comparing the target hidden state with the source state, generating attention weights or alignments, and then calculating the weighted average to obtain the context vector;

[0060] First, calculate the similarity. The similarity calculation formula is as follows:

[0061]

[0062] The Attention value is calculated after normalization through the softmax function. The calculation formula of Attention is as follows:

[0063]

[0064] In the formula, represents the corresponding value vector, is the calculated softmax score;

[0065] Step 8-5, using the WaveNet model as a vocoder to convert the Mel spectrum generated by the Tacotron2 acoustic model into an actual speech waveform to achieve speech synthesis; WaveNet is an audio generation model based on the PixelCNN architecture to generate the original speech signal;

[0066] The joint probability of the audio waveform is expressed as the product of the conditional probability distribution, and its formula is expressed as:

[0067]

[0068] In the formula, Indicates the audio sample value at a specific moment;

[0069] Step 8-6, use softmax distribution to model the conditional probability distribution of a single audio sample, and its formula is expressed as:

[0070]

[0071] Perform the data The law is compressed and then quantized into 256 possible values; −1< <1 and =255.

[0072] Furthermore, the step 9 is specifically as follows: realizing the digital human visualization function through the Wav2Lip model, and directly converting the audio waveform into the facial animation;

[0073] Step 9-1, use a 2D convolution layer to encode the input Mel spectrum feature map. The 2D convolution layer calculates the weighted sum of the local area by sliding a fixed-size convolution kernel on the input data;

[0074] Assume that the input data The size is , convolution kernel The size is , then the output after the convolution operation is Calculated by the following formula:

[0075]

[0076] In the formula, Represents the pixel position on the output feature map;

[0077] Step 9-2 is processed through multiple residual blocks. A typical residual block is represented as:

[0078]

[0079] In the formula, and are the input and output of the residual block respectively; represents the residual mapping function; is a set of parameters;

[0080] By adding the original input , forming a residual connection;

[0081] In step 9-3, a transposed convolutional layer is used for upsampling to obtain a facial animation that matches the input video frame. The transposed convolutional layer is used to convert the low-resolution feature map back to a high-resolution feature map, and its formula is expressed as:

[0082]

[0083]

[0084] In the formula, the input feature map The size is ; Convolution kernel The size is ; stride is Fill with ; Output after transposed convolution operation The size is ; The discriminator distinguishes the generated facial animation from the real facial animation.

[0085] Furthermore, the discriminator is a binary classification network used to determine whether the input image is a real image or a generated image; the discriminator is stacked by multiple convolutional layers, each layer uses a Leaky ReLU activation function; the Leaky ReLU activation function is expressed as:

[0086]

[0087] In the formula, yes A coefficient is required when it is a negative number, which controls the angle of the negative slope. The default value is 0.01;

[0088] The discriminator gradually reduces the spatial dimension through downsampling operations and outputs a probability value. Its loss function is usually a binary cross entropy loss expressed as:

[0089]

[0090] In the formula, is the batch size, is the true label, is the predicted probability of the discriminator.

[0091] The beneficial effects of the present invention are as follows: based on the Wav2Lip model, the present invention synthesizes the large model and the audio data after natural language processing into the facial movements of the digital human, which greatly improves the visual interaction effect of the intelligent interactive control system. At the same time, the fine-tuning of the large model is used to achieve better scene implementation, improve the generalization ability of the system, and has a multi-platform smart home device control method with extremely high scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 Design a flow chart for the AI ​​large model visual human-computer interaction control of the present invention;

[0094] Figure 2 It is a flow chart of speech recognition based on ASRT of the present invention;

[0095] Figure 3 This is the AI ​​big model interaction and decision-making flow chart of the present invention;

[0096] Figure 4 A flow chart of the digital human visualization interaction of the present invention;

[0097] Figure 5 This is the AI ​​large model intelligent interactive control application scenario diagram of the present invention;

[0098] Figure 6 This is a hardware architecture diagram of the robot voice control system of the present invention. DETAILED DESCRIPTION

[0100] The present invention will be further described below in conjunction with the accompanying drawings.

[0101] like Figure 1 As shown, the present invention is a robot visual interactive control method based on an AI large model, including a speech recognition part, a large model access and intelligent control part, and a digital human video synthesis part.

[0102] The speech recognition part includes audio sampling, audio preprocessing and speech recognition model.

[0103] The big model access and intelligent control part accesses the cloud big model API and uses cloud services to quickly obtain big model feedback. Based on the AI ​​big model, it optimizes the question content to achieve text classification, question response, command selection and other functions.

[0104] The digital human video synthesis part includes audio synthesis and AI lip-syncing video synthesis. The audio synthesis part will receive the text information returned by the large model, and directly synthesize the text information into audio data through the pre-trained speech synthesis model without processing. The AI ​​lip-syncing video synthesis part inputs the audio information synthesized by the speech synthesis part and the pre-prepared digital human image into the AI ​​lip-syncing model. After the model processing is completed, the corresponding AI broadcast video will be output, and the video can be played to realize voice interaction with the user.

[0105] The present invention comprises the following steps:

[0106] Step 1: Select a suitable sampling frequency for audio sampling based on the Nyquist sampling theorem.

[0107] like Figure 2 As shown, the voice information recorded by the user is sampled under the condition that the Nyquist sampling theorem is satisfied. When the sampling frequency does not satisfy the Nyquist sampling theorem, aliasing will occur, resulting in the inability to correctly reconstruct the original signal. In order to avoid signal aliasing, the sampling frequency must be greater than twice the highest frequency in the signal. When the sampling frequency is lower than twice the highest frequency of the signal, the high-frequency signal component will be incorrectly mapped to the low-frequency range. The Nyquist sampling theorem formula is as follows:

[0108]

[0109] Step 2, use pre-emphasis and filtering methods to pre-process the audio, filter out the noise in the sampled audio and enhance the high-frequency components in the sound, so as to improve the accuracy of speech recognition and the generalization ability of recognition. In speech signals, high-frequency components often contain more information, such as consonants, which have higher frequencies and lower energy. The purpose of pre-emphasis is to enhance these high-frequency components, thereby improving the spectral characteristics of the entire signal and eliminating the influence of lip radiation. Lip radiation is the energy loss of speech signals due to absorption by the medium (such as air) during propagation, especially in the high-frequency part. Pre-emphasis can compensate for this loss and make the signal more realistically reflect the original sound. Speech signals are easily affected by environmental noise during the acquisition process. High-frequency noise can be removed by low-pass filters, while low-frequency interference can be removed by high-pass filters, thereby improving the signal-to-noise ratio of the signal. Filtering can eliminate mutations and spikes in the signal and make it smoother. This is very important for subsequent feature extraction, because spikes may cause errors in spectrum analysis. By filtering the signal, spectrum leakage and boundary effects can also be reduced, so as to obtain more accurate results in spectrum analysis. The specific steps are as follows:

[0110] Step 2-1, pre-emphasize the sampled voice data:

[0111]

[0112] In the formula, is the input signal sampling points; The value range of is 0.9-1, and the value in the present invention is 0.97.

[0113] Step 2-2: After pre-emphasis, the voice data is divided into frames, each frame is about 20-30ms, and the framed data is windowed:

[0114]

[0115] In the formula, is the sample index, is the frame length.

[0116] Step 2-3, perform fast Fourier transform on the above results:

[0117]

[0118] In the formula, represents the total number of sampling points in the transformation, It represents the frequency domain The index of the frequency components;

[0119] Step 2-4, calculate the amplitude spectrum of each frame, convert it into energy spectrum and draw the spectrogram; the specific steps are as follows:

[0120] Step 2-4-1, by taking the absolute value of the fast Fourier transform result, the amplitude spectrum of each frame is obtained;

[0121] Step 2-4-2, by squaring the amplitude spectrum value of each frequency component, the corresponding energy spectrum value is obtained;

[0122] Step 2-4-3, finally arrange the energy spectrum of each frame along the time axis, use the color depth to represent the energy size, and then draw the spectrogram.

[0123] Step 3: Input the processed data into the speech recognition model to obtain the recognized content; the specific steps are as follows:

[0124] Step 3-1, input the drawn spectrogram into the VGGish acoustic model based on the Keras and TensorFlow framework and build a deep convolutional neural network with reference to VGG and train it;

[0125] Step 3-2, by using the concatenated temporal classification (CTC) algorithm, a large number of consecutive repeated symbols in the output of the acoustic model are merged into the same symbol, and the silent separation marker is removed to obtain the final actual speech pinyin symbol sequence; the specific steps are:

[0126] For each time step , CTC calculates the probability of each label; let is a possible alignment path, is the input sequence, then the probability formula of CTC is:

[0127]

[0128] In the formula, Indicates that at time step The neural network output of

[0129] The loss function of the CTC algorithm takes into account all valid alignment paths, and its objective function is:

[0130]

[0131] In the formula, Indicates that it can be mapped to the output label All valid paths of

[0132] The statistical language model hidden Markov model is used to convert pinyin into the final recognized text and output it. The essence of pinyin-to-text conversion is modeled as a hidden Markov chain, and the text output can be obtained by inputting data into the speech model.

[0133] Step 4, such as Figure 3 As shown, the pre-set smart device control commands and text information are synchronously input into the AI ​​big model. The present invention adopts the ERNIE big model API interface. By entering the official website of the cloud big model, the API required for accessing the big model network interface is obtained, and by referring to the official user manual and the sample code, the API information in the code is modified to the API content that has been applied for, and the interaction with the big model can be achieved. By connecting the above code to the core running code of the device and informing the big model of the current interaction environment at the start of the operation stage, the adaptability of the big model to the intelligent control scenario is improved.

[0134] When the speech recognition result is input into the AI ​​big model, the AI ​​big model will analyze and process the text classification, determine the relevance of the text information to the above control commands, and determine whether the speech recognition result is an intelligent control problem or an intelligent interaction problem. For example, two intelligent control functions are currently preset: 1. Turn on the light 2. Turn on the air conditioner. After receiving the speech and recognizing it as text, the text can be processed. You can ask the big model in turn: "The following text: 'Speech recognition text' If it is a light-on command, reply with the number '1', if it is a light-off command, reply with the number '2', if it is an air-conditioning-on command, reply with the number '3', if it is an air-conditioning-off command, reply with the number '4', if the text content is not related to the above commands, reply with the number '0'"; the command can be judged according to the number in the big model's reply content, and the device execution command or normal text interaction can be judged according to the return information of the big model; if it is an intelligent control problem, go to step 5; if it is an intelligent interaction problem, go to step 6.

[0135] The present invention realizes the large model interaction function for specified scenarios through the ERNIE large model. The ERNIE model is mainly based on the Encoder part of the Transformer, and enhances the understanding of language knowledge by introducing a three-level Knowledge Masking strategy. The Transformer model uses a self-attention mechanism to capture the global dependencies in the input sequence, and the calculation formula is:

[0136]

[0137] In the formula, They represent query, key and value respectively. is the dimension of the key vector. Each Encoder layer has a feedforward network, which increases the nonlinear expression ability of the model through two layers of linear transformation and one layer of ReLU activation function. This structure enables the model to learn more complex features and functional relationships. In the feedforward network, the first linear layer increases the dimension of the input, and the second linear layer reduces it, so that the input can be represented more abstractly in a high-dimensional space, thereby capturing more complex semantic information. In each Encoder layer, the input and output of the feedforward network use residual connections and layer normalization, which helps to alleviate the problems of gradient disappearance and explosion, making model training more stable. Unlike the self-attention layer, which mainly focuses on the relative relationship between elements in the sequence, the feedforward network is used to extract features from the global information of the entire sequence, supplementing the information range that the self-attention layer cannot cover. Its formula is expressed as:

[0138]

[0139] In the formula, , is the weight matrix, , is the bias term.

[0140] The pre-training input data of the ERNIE model mainly includes encyclopedia, information and forum conversation data. These data are segmented into different granularities such as characters, words and entities by Baidu's internal lexical analysis tools, and the segmented data is tokenized to obtain the plaintext token sequence and segmentation boundaries. The plaintext data is mapped to ID data according to the dictionary. During the training process, the continuous tokens are randomly masked according to the segmentation boundaries.

[0141] The ERNIE model proposes three levels of Knowledge Masking strategies, namely token level, phrase level and entity level. This hierarchical masking strategy helps the model better understand the vocabulary, syntax and semantic information in the language.

[0142] Token-level masking: Randomly mask certain words and let the model predict the masked words during pre-training.

[0143] Phrase-level masking: Mask the phrases in a sentence.

[0144] Entity-level masking: Mask entities in sentences. The ERNIE model achieves efficient learning and application of complex language knowledge through refined input processing, multi-level masking strategies and pre-training tasks, and a powerful Transformer Encoder structure.

[0145] Step 5: When the AI ​​big model identifies an intelligent control problem, the system matches the relevant commands in the pre-prepared command set. If the match is successful, it goes to step 7; if the match fails, it goes to step 6.

[0146] Step 6: When the AI ​​big model identifies an intelligent interaction question, the system obtains the text response from the AI ​​big model and gives a response to the question; then proceed to step 8;

[0147] Step 7, sending the command to the intelligent robot terminal, and controlling the corresponding intelligent device through the intelligent robot terminal, so as to achieve the purpose of the robot controlling the intelligent device; proceed to step 8;

[0148] Step 8: Preprocess the text fed back by the AI ​​model before inputting the reply text into the speech synthesis model. Convert the special symbols or Arabic numerals in the text into text through text regularization. Predict the rhythmic structure of the regularized text through rhythm detection. Implement rhythmic knot detection through hidden Markov model. The probability calculation includes forward algorithm and backward algorithm. The core recursive relation function is expressed as:

[0149]

[0150]

[0151] In the formula, and are the forward and backward probabilities, respectively, is the state transition probability, is the observation probability;

[0152] Convert glyphs into phonemes and use a lookup table to map Chinese characters to pinyin;

[0153] Modeling context-dependent pronunciation variation using conditional random fields;

[0154] The estimated conditional probability formula is expressed as:

[0155]

[0156] In the formula, is the normalization factor, is the characteristic function, is the corresponding weight parameter.

[0157] Input the reply text into the speech synthesis model to obtain the synthesized reply audio; the speech synthesis model is the Tacotron2 acoustic model, which includes a spectrum prediction network and a vocoder, and outputs a Mel spectrum according to the input pinyin text; the specific steps are as follows:

[0158] Step 8-1, the spectrogram prediction network first converts the input character into a 512-dimensional character vector through the encoder and passes it through three layers of convolution. Each layer of convolution has 512 5×1 convolution kernels, and each convolution kernel spans 5 characters;

[0159] Step 8-2, batch normalization and ReLU activation function are applied after the convolution layer to obtain the large-span context information of the character sequence; the output of the last convolution layer is transmitted to a bidirectional LSTM layer to generate encoding features;

[0160] Step 8-3: The decoder predicts the output spectrum frames one by one from the encoded input sequence. The predicted spectrum frames are first passed into a two-layer fully connected "Pre-Net", each layer consisting of 256 hidden ReLU units; the decoder's Pre-Net output and the attention context vector are concatenated together to form the input for the next decoding step;

[0161] Step 8-4, using the content- and position-based attention mechanism to allow the model to focus on different positions in the input sequence, thereby generating a more accurate spectrum frame;

[0162] The calculation of the attention mechanism includes comparing the target hidden state with the source state, generating attention weights or alignments, and then calculating the weighted average to obtain the context vector;

[0163] First, calculate the similarity. The similarity calculation formula is as follows:

[0164]

[0165] The Attention value is calculated after normalization through the softmax function. The calculation formula of Attention is as follows:

[0166]

[0167] In the formula, represents the corresponding value vector, is the calculated softmax score;

[0168] Step 8-5, using the WaveNet model as a vocoder to convert the Mel spectrum generated by the Tacotron2 acoustic model into an actual speech waveform to achieve speech synthesis; WaveNet is an audio generation model based on the PixelCNN architecture, which can generate original speech signals;

[0169] The joint probability of the audio waveform is expressed as the product of the conditional probability distribution, and its formula is expressed as:

[0170]

[0171] In the formula, Indicates the audio sample value at a specific moment;

[0172] Step 8-6, use softmax distribution to model the conditional probability distribution of a single audio sample, and its formula is expressed as:

[0173]

[0174] Perform the data The law is compressed and then quantized into 256 possible values; −1< <1 and =255.

[0175] Step 9, such as Figure 4 As shown, the present invention obtains audio data output by the speech synthesis model by inputting the text information returned by the large model into a pre-trained speech synthesis model. The audio data output by the model contains less noise and interference than the audio data read by the speech recognition backend. Therefore, the audio data output by the model can be directly input into the digital human AI lip-syncing model as high-quality audio data without audio processing to synthesize digital human interactive video.

[0176] After the audio data is input into the AI ​​lip-syncing model, the model will combine the pre-input pictures or video data containing digital people or real faces and use them as model input. After the audio data is processed by the model, the processed audio data and the digital human image will be synthesized frame by frame to finally generate a digital human interactive video. The digital human's lip movements can be synchronized with the audio to achieve the effect of the digital human speaking autonomously. After the core processing of the device completes the above process, the video will be broadcast to achieve a relatively ideal human-computer interaction effect.

[0177] The present invention realizes the digital human visualization function through the Wav2Lip model, and directly converts the audio waveform into facial animation; the specific steps are as follows:

[0178] Step 9-1, use a 2D convolution layer to encode the input Mel spectrum feature map. The 2D convolution layer calculates the weighted sum of the local area by sliding a fixed-size convolution kernel on the input data;

[0179] Assume that the input data The size is , convolution kernel The size is , then the output after the convolution operation is It can be calculated by the following formula:

[0180]

[0181] In the formula, Represents the pixel position on the output feature map;

[0182] Step 9-2 is processed through multiple residual blocks. A typical residual block is represented as:

[0183]

[0184] In the formula, and are the input and output of the residual block respectively; represents the residual mapping function; is a set of parameters;

[0185] By adding the original input , forming a residual connection;

[0186] In step 9-3, a transposed convolutional layer is used for upsampling to obtain a facial animation that matches the input video frame. The transposed convolutional layer is used to convert the low-resolution feature map back to a high-resolution feature map, and its formula is expressed as:

[0187]

[0188]

[0189] In the formula, the input feature map The size is ; Convolution kernel The size is ; stride is Fill with ; Output after transposed convolution operation The size is ; The discriminator distinguishes the generated facial animation from the real facial animation.

[0190] The discriminator is a binary classification network used to determine whether the input image is a real image or a generated image; the discriminator is composed of multiple convolutional layers stacked together, and each layer uses the Leaky ReLU activation function; the Leaky ReLU activation function is expressed as:

[0191]

[0192] In the formula, yes A coefficient is required when it is a negative number, which controls the angle of the negative slope. The default value is 0.01;

[0193] The discriminator gradually reduces the spatial dimension through downsampling operations and outputs a probability value. Its loss function is usually binary cross entropy loss expressed as:

[0194]

[0195] In the formula, is the batch size, is the true label, is the predicted probability of the discriminator.

[0196] This series of operations together realizes the generation from Mel spectrum feature maps to facial animations, and provides supervision on the authenticity of the generated results through the discriminator; finally, the pre-trained lip discriminator is used to detect the synchronization between the generated frames and the original audio. It evaluates the synchronization accuracy between the generated lip movements and the corresponding audio during the training process and guides the generator to make corresponding adjustments. The combination of these technologies ensures that the model can effectively learn the complex mapping relationship between audio and visual data and generate highly realistic facial animations.

[0197] Step 10: The AI ​​digital human announces the reply content.

[0198] like Figure 5 As shown, the user wakes up the AI ​​digital human through voice input and asks questions or issues relevant instructions for controlling smart devices; the AI ​​digital human obtains responses by recognizing the user's voice information and interacting with the large model, and interacts with the user through the digital human expressing the responses. Based on the interaction content, the user chooses whether to control the smart device through the smart terminal and the specific operation method.

[0199] like Figure 6 As shown, the system platform in the present invention is implemented with a Raspberry Pi 4B development board and a 64G memory card. The Raspberry Pi 4B development board is equipped with a 1.5GHz quad-core 64-bit Cortex-A72 processor, providing powerful computing power. Equipped with Gigabit Ethernet, it supports faster network speeds. Provide multiple USB 3.0 and USB 2.0 interfaces for convenient connection of various peripherals. Support 4K resolution video output, suitable for high-definition display requirements. The operating system and data are stored through a Micro SD card. Since the Raspberry Pi has no built-in sound card system, an external sound card and speakers or headphones are required. At the same time, external peripherals such as a display, mouse, keyboard, etc. can be selected, and the Raspberry Pi program can also be run through a remote connection software at the computer layer through a remote connection.

[0200] The sound card uses the ReSpeaker 2-Mics Pi HAT module. The ReSpeaker 2-Mics Pi HAT is a Raspberry Pi dual-microphone expansion board designed for AI and voice applications. It uses advanced microphone technology, combined with the microcontroller function of RaspberryPi, to provide developers and users with a high-performance, easy-to-use voice processing platform. Raspberry Pi is a widely used microcomputer with powerful computing power and flexible scalability. As an expansion board for Raspberry Pi, the ReSpeaker 2-Mics Pi HAT can capture clearer and more accurate voice signals through the design of a dual microphone array, providing a solid foundation for subsequent voice processing and analysis. ReSpeaker2-Mics Pi HAT also supports a variety of audio codecs and can seamlessly connect with a variety of audio devices and software, making it convenient for users to perform applications such as voice interaction and voice recognition. For other parts, you can choose an HDMI interface display, a USB interface mouse and keyboard, and a 3.5mm interface headset or audio peripherals according to your needs.

Claims

1. A robot visual interactive control method based on AI large model, characterized in that: The following steps are involved: Step 1, selecting a suitable sampling frequency for audio sampling based on Nyquist sampling theorem; Step 2, use pre-emphasis and filtering methods to perform audio preprocessing, filter out noise in the sampled audio and enhance the high-frequency components in the sound; Step 3, input the processed data into the speech recognition model to obtain the recognized content; Step 4: Input the speech recognition result into the AI ​​big model. The AI ​​big model will analyze and process the text classification to determine the intelligent control and intelligent interaction categories. If it is a control instruction, it will reply with the number '1'; if it is an interaction instruction, it will reply with the number '2'; if the text content is not related to the above instructions, it will reply with the number '0'. The instruction is determined according to the number in the big model's reply content; if it is an intelligent control problem, go to step 5; if it is an intelligent interaction problem, go to step 6; Step 5: When the AI ​​big model identifies an intelligent control problem, the system matches the relevant commands in the pre-prepared command set. If the match is successful, it goes to step 7; if the match fails, it goes to step 6. Step 6: When the AI ​​big model identifies an intelligent interaction question, the system obtains the text response from the AI ​​big model and gives a response to the question; then proceed to step 8; Step 7, sending the command to the intelligent robot terminal, and controlling the corresponding intelligent device through the intelligent robot terminal; proceed to step 8; Step 8: Input the reply text into the speech synthesis model to obtain the synthesized reply audio; Step 9, input the synthesized audio into the AI ​​counterpart model to synthesize the digital human interaction video; Step 10: The AI ​​digital human announces the reply content.

2. According to claim 1, a robot visual interactive control method based on AI large model is characterized in that: The step 2 is specifically as follows: Step 2-1, pre-emphasize the sampled voice data: ； In the formula, is the input signal sampling points; The value range is 0.9-1.0; Step 2-2: After pre-emphasis, the voice data is divided into frames, each frame is about 20-30ms, and the framed data is windowed: ； In the formula, is the sample index, is the frame length; Step 2-3, perform fast Fourier transform on the above results: ； In the formula, represents the total number of sampling points in the transformation, In the frequency domain, The index of the frequency components; Step 2-4, calculate the amplitude spectrum of each frame, convert it into energy spectrum and draw the spectrogram.

3. The method for visual interactive control of a robot based on an AI large model according to claim 2, characterized in that: The specific steps of steps 2-4 are: Step 2-4-1, by taking the absolute value of the fast Fourier transform result, the amplitude spectrum of each frame is obtained; Step 2-4-2, by squaring the amplitude spectrum value of each frequency component, the corresponding energy spectrum value is obtained; Step 2-4-3, finally arrange the energy spectrum of each frame along the time axis, use the color depth to represent the energy size, and then draw the spectrogram.

4. The method for visual interactive control of a robot based on an AI large model according to claim 1, characterized in that: The step 3 is specifically as follows: Step 3-1, input the drawn spectrogram into the VGGish acoustic model based on the Keras and TensorFlow framework and build a deep convolutional neural network with reference to VGG and train it; Step 3-2, using the CTC algorithm to merge a large number of consecutively repeated symbols in the output of the acoustic model into the same symbol, and remove the silent separation markers to obtain the final actual phonetic phonetic symbol sequence; the specific steps are: For each time step , CTC calculates the probability of each label; let is a possible alignment path, is the input sequence, then the probability formula of CTC is: ； In the formula, Indicates that at time step The neural network output of The loss function of the CTC algorithm takes into account all valid alignment paths, and its objective function is: ； In the formula, Indicates that it can be mapped to the output label All valid paths of And use the statistical language model hidden Markov model to convert the pinyin into the final recognized text and output it.

5. The method for robot visual interactive control based on AI big model according to claim 1 is characterized in that: In step 4, the pre-set smart device control command and text information are synchronously input into the AI ​​big model. The AI ​​big model determines the relevance of the text information to the above control command, and determines whether the device executes the command or normal text interaction based on the return information of the big model.

6. The method for visual interactive control of a robot based on an AI large model according to claim 1, characterized in that: In step 8, the text fed back by the AI ​​big model is preprocessed before the reply text is input into the speech synthesis model. The special symbols or Arabic numerals in the text are converted into text through text regularization. The rhythm structure of the regularized text is predicted through rhythm detection. The rhythm detection is realized through the hidden Markov model. Its probability calculation includes forward algorithm and backward algorithm. The core recursive relation function is expressed as: ； ； In the formula, and are the forward and backward probabilities, respectively, is the state transition probability, is the observation probability; Convert glyphs into phonemes and use a lookup table to map Chinese characters to pinyin; Modeling context-dependent pronunciation variation using conditional random fields; The estimated conditional probability formula is expressed as: ； In the formula, is the normalization factor, is the characteristic function, is the corresponding weight parameter.

7. The method for visual interactive control of a robot based on an AI large model according to claim 1, characterized in that: In step 8, the speech synthesis model is a Tacotron2 acoustic model, including a spectrum prediction network and a vocoder, and outputs a Mel spectrum according to the input pinyin text; Step 8-1, the spectrogram prediction network first converts the input character into a 512-dimensional character vector through an encoder and passes it through three layers of convolution, each layer of convolution has 512 5×1 convolution kernels, and each convolution kernel spans 5 characters; Step 8-2, batch normalization and ReLU activation function are applied after the convolution layer to obtain the large-span context information of the character sequence; the output of the last convolution layer is transmitted to a bidirectional LSTM layer to generate encoding features; Step 8-3: The decoder predicts the output spectrum frames one by one from the encoded input sequence. The predicted spectrum frames are first passed into a two-layer fully connected "Pre-Net", each layer consisting of 256 hidden ReLU units; the decoder's Pre-Net output and the attention context vector are concatenated together to form the input for the next decoding step; Step 8-4, using the content- and position-based attention mechanism to allow the model to focus on different positions in the input sequence, thereby generating a more accurate spectrum frame; The calculation of the attention mechanism includes comparing the target hidden state with the source state, generating attention weights or alignments, and then calculating the weighted average to obtain the context vector; First, calculate the similarity. The similarity calculation formula is as follows: ； The Attention value is calculated after normalization through the softmax function. The calculation formula of Attention is as follows: ； In the formula, represents the corresponding value vector, is the calculated softmax score; Step 8-5, using the WaveNet model as a vocoder to convert the Mel spectrum generated by the Tacotron2 acoustic model into an actual speech waveform to achieve speech synthesis; WaveNet is an audio generation model based on the PixelCNN architecture to generate the original speech signal; The joint probability of the audio waveform is expressed as the product of the conditional probability distribution, and its formula is expressed as: ； In the formula, Represents the audio sample value at a specific moment; Step 8-6, use softmax distribution to model the conditional probability distribution of a single audio sample, and its formula is expressed as: ； Perform the data The law is compressed and then quantized into 256 possible values; −1< <1 and .

8. The method for robot visual interactive control based on AI big model according to claim 1 is characterized in that: The step 9 specifically includes: realizing the digital human visualization function through the Wav2Lip model, and directly converting the audio waveform into facial animation; Step 9-1, use a 2D convolution layer to encode the input Mel spectrum feature map. The 2D convolution layer calculates the weighted sum of the local area by sliding a fixed-size convolution kernel on the input data; Assume that the input data The size is , convolution kernel The size is , then the output after the convolution operation is Calculated by the following formula: ； In the formula, Represents the pixel position on the output feature map; Step 9-2 is processed through multiple residual blocks. A typical residual block is represented as: ； In the formula, and are the input and output of the residual block respectively; represents the residual mapping function; is a set of parameters; By adding the original input , forming a residual connection; In step 9-3, a transposed convolutional layer is used for upsampling to obtain a facial animation that matches the input video frame. The transposed convolutional layer is used to convert the low-resolution feature map back to a high-resolution feature map, and its formula is expressed as: ； ； In the formula, the input feature map The size is ; Convolution kernel The size is ; stride is Fill with ; Output after transposed convolution operation The size is ; The discriminator distinguishes the generated facial animations from the real ones.

9. The method for robot visual interactive control based on AI big model according to claim 8, characterized in that: The discriminator is a binary classification network used to determine whether the input image is a real image or a generated image; the discriminator is composed of multiple convolutional layers stacked together, and each layer uses the Leaky ReLU activation function; the Leaky ReLU activation function is expressed as: ； In the formula, yes A coefficient is required when it is a negative number, which controls the angle of the negative slope. The default value is 0.01; The discriminator gradually reduces the spatial dimension through downsampling operations and outputs a probability value. Its loss function is usually a binary cross entropy loss expressed as: ； In the formula, is the batch size, is the true label, is the predicted probability of the discriminator.

Images