Humanoid robot language identification model training method and language identification method

Abstract

The application discloses a kind of humanoid robot language identification model training method and language identification method, wherein, the training method obtains multi-modal training sample, the multi-modal training sample includes audio mode sample, oral type mode sample and environment mode sample;The multi-modal training sample is encoded to multi-layer cross-modal, obtains several cross-modal encoding features, each cross-modal encoding feature includes acoustic encoding feature, oral type encoding feature and environment encoding feature;Cross-modal fusion is carried out to all the cross-modal encoding features, obtains first fusion feature;According to the environment mode sample and the first fusion feature, the parameter of initialization humanoid robot language identification model is updated, obtains well-trained humanoid robot language identification model.The training method provides a kind of humanoid robot language identification model, which is beneficial to improve the accuracy and robustness of language identification.The application relates to the technical field of intelligent robot.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot technology, and in particular to a training method and a language recognition method for a humanoid robot language recognition model. Background Technology

[0002] In recent years, humanoid robots have been increasingly used in industrial and home scenarios due to their anthropomorphic movement and interaction capabilities. In industrial scenarios, humanoid robots need to complete tasks such as equipment inspection, fault reporting, and collaboration with workers of multiple nationalities, and need to accurately recognize multiple languages ​​such as Chinese and English. In home scenarios, humanoid robots need to provide services such as companionship and education, and home control, and need to be compatible with commonly used household languages ​​such as Chinese and English.

[0003] Currently, related technologies typically rely on a single acoustic feature or a fixed modal combination (such as fixed acoustic + single visual modality) to train a neural network model, and then use the trained model to achieve language recognition. However, due to the mobility of humanoid robots and the complexity of their environment, the model parameters trained in this way are fixed, making them susceptible to scene interference. Consequently, the anti-interference ability of language recognition is weak, and the accuracy and robustness of language recognition are poor.

[0004] Therefore, the problems with the relevant technologies still need to be solved and optimized. Summary of the Invention

[0005] The purpose of this invention is to at least partially solve one of the technical problems existing in the related art.

[0006] Therefore, one objective of this invention is to provide a training method for a language recognition model of a humanoid robot and a language recognition method, wherein the training method provides a language recognition model for a humanoid robot, which is beneficial to improving the accuracy and robustness of language recognition.

[0007] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of this application include:

[0008] In a first aspect, embodiments of this application provide a method for training a language recognition model for a humanoid robot, comprising:

[0009] Obtain multimodal training samples, which include audio modal samples, lip-sync modal samples, and environmental modal samples;

[0010] The multimodal training samples are subjected to multi-layer cross-modal coding to obtain several cross-modal coding features. Each cross-modal coding feature includes an acoustic coding feature, a lip-sync coding feature, and an environmental coding feature. The coding level of the acoustic coding feature in the first cross-modal feature is different from the coding level of the acoustic coding feature in the second cross-modal feature. The first cross-modal feature is any one of the cross-modal coding features. The second cross-modal feature is any one of the cross-modal coding features other than the first cross-modal feature.

[0011] Cross-modal fusion is performed on all the cross-modal encoded features to obtain the first fused feature;

[0012] Based on the environmental modal samples and the first fusion feature, the parameters of the initialized humanoid robot language recognition model are updated to obtain a trained humanoid robot language recognition model.

[0013] In addition, the method according to the above embodiments of this application may also have the following additional technical features:

[0014] Furthermore, in one embodiment of this application, the method further includes:

[0015] The multimodal training samples are subjected to dynamic scene adaptation data augmentation to obtain augmented multimodal training samples;

[0016] The data augmentation process, which involves dynamic scene adaptation of the multimodal training samples to obtain augmented multimodal training samples, includes:

[0017] Based on a preset distance attenuation function, scene noise is added to the audio modal samples to obtain audio modal samples with added noise;

[0018] The lip-shape modal samples are randomly occluded to obtain enhanced lip-shape modal samples;

[0019] The environmental modal samples are then subjected to illumination adjustment and angle rotation to obtain enhanced environmental modal samples.

[0020] Furthermore, in one embodiment of this application, the step of performing multi-layer cross-modal coding on the multimodal training samples to obtain several cross-modal coding features includes:

[0021] The audio modal samples are subjected to multi-layer acoustic concatenation coding to obtain several acoustic coding features, each of which has a different feature level.

[0022] The lip shape modality samples are visually encoded to obtain the lip shape encoding features;

[0023] The environmental modality samples are subjected to environmental visual encoding to obtain the environmental encoding features;

[0024] Based on the lip shape coding features and the environmental coding features, each of the acoustic coding features is concatenated to obtain several cross-modal coding features.

[0025] Furthermore, in one embodiment of this application, the step of performing multi-layer acoustic concatenation coding on the audio modal samples to obtain several acoustic coding features includes:

[0026] Obtain intermediate features, wherein the intermediate features are the audio modality samples or the acoustic coding features of the previous feature level;

[0027] Acoustic feature extraction is performed on the intermediate features to obtain the acoustic coding features of the current feature level.

[0028] Furthermore, in one embodiment of this application, the step of performing cross-modal fusion on all the cross-modal coding features to obtain a first fused feature includes:

[0029] Cross-attention analysis is performed on all the cross-modal coding features to obtain the attention weight of each cross-modal coding feature;

[0030] Based on each attention weight, the corresponding cross-modal coding features are weighted to obtain several cross-modal weighted features;

[0031] The first fused feature is obtained by fusing all the cross-modal weighted features.

[0032] Further, in one embodiment of this application, the step of updating the parameters of the initialized humanoid robot language recognition model based on the environmental modal samples and the first fusion feature to obtain a trained humanoid robot language recognition model includes:

[0033] The environmental modal samples are classified to obtain environmental prediction labels;

[0034] Based on the environmental prediction labels, dynamic weight analysis is performed to obtain acoustic feature weights and visual feature weights.

[0035] The first fusion feature is weighted and refined based on the acoustic feature weights and the visual feature weights to obtain the target fusion feature;

[0036] Based on the target fusion features, the parameters of the initialized humanoid robot language recognition model are updated to obtain a trained humanoid robot language recognition model.

[0037] Furthermore, in one embodiment of this application, the step of weighting and refining the first fusion feature according to the acoustic feature weight and the visual feature weight to obtain the target fusion feature includes:

[0038] Based on the acoustic feature weights and the visual feature weights, the first fusion feature is weighted and fused to obtain the second fusion feature;

[0039] Temporal attention fusion is performed on the second fusion feature to obtain the third fusion feature;

[0040] Modal attention fusion is performed on the third fusion feature to obtain the target fusion feature.

[0041] Secondly, embodiments of this application provide a language identification method, including:

[0042] Acquire multimodal data for language recognition collected by a humanoid robot;

[0043] The multimodal data is input into the pre-trained humanoid robot language recognition model to perform language recognition and obtain the language recognition result.

[0044] Thirdly, embodiments of this application provide a training system for a language recognition model of a humanoid robot, comprising:

[0045] The first processing unit is used to acquire multimodal training samples, which include audio modal samples, lip-sync modal samples, and environmental modal samples.

[0046] The second processing unit is used to perform multi-layer cross-modal coding on the multimodal training samples to obtain several cross-modal coding features. Each cross-modal coding feature includes an acoustic coding feature, a lip-sync coding feature, and an environment coding feature. The coding level of the acoustic coding feature in the first cross-modal feature is different from the coding level of the acoustic coding feature in the second cross-modal feature. The first cross-modal feature is any one of the cross-modal coding features. The second cross-modal feature is any one of the cross-modal coding features other than the first cross-modal feature.

[0047] The third processing unit is used to perform cross-modal fusion on all the cross-modal encoded features to obtain the first fused feature;

[0048] The fourth processing unit is used to update the parameters of the initialized humanoid robot language recognition model based on the environmental modal samples and the first fusion feature, so as to obtain the trained humanoid robot language recognition model.

[0049] Fourthly, embodiments of this application also provide an electronic device, including:

[0050] At least one processor;

[0051] At least one memory for storing at least one program;

[0052] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0053] Fifthly, embodiments of this application also provide a computer-readable storage medium storing a processor-executable program, which, when executed by the processor, is used to implement the above-described method.

[0054] The advantages and beneficial effects of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application:

[0055] This application discloses a training method and a language recognition method for a humanoid robot language recognition model. The training method acquires multimodal training samples, including audio modal samples, lip-sync modal samples, and environmental modal samples. Multi-layer cross-modal coding is performed on the multimodal training samples to obtain several cross-modal coding features. Each cross-modal coding feature includes an acoustic coding feature, a lip-sync coding feature, and an environmental coding feature. The coding level of the acoustic coding feature in the first cross-modal feature is different from the coding level of the acoustic coding feature in the second cross-modal feature. The first cross-modal feature is any one of the cross-modal coding features. The second cross-modal feature is any one of the cross-modal coding features other than the first cross-modal feature. Cross-modal fusion is performed on all the cross-modal coding features to obtain a first fused feature. Based on the environmental modal samples and the first fused feature, the parameters of the initialized humanoid robot language recognition model are updated to obtain a trained humanoid robot language recognition model. This training method performs multi-layer cross-modal coding on multimodal training samples to obtain several cross-modal coding features containing acoustic coding features at different coding levels. This not only combines the humanoid robot's own visual perception capabilities (such as the user's lip movement patterns and scene environment types) to improve anti-interference capabilities through multimodal information complementarity, but also captures local and global speech information through acoustic coding features at different coding levels, providing multi-granular feature support for subsequent cross-modal fusion, which is beneficial to improving the robustness and accuracy of language recognition. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of this application or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions in this application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0057] Figure 1 A flowchart illustrating a training method for a language recognition model of a humanoid robot provided in an embodiment of this application;

[0058] Figure 2 A schematic diagram of the framework of a training system for a language recognition model of a humanoid robot provided in an embodiment of this application;

[0059] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0060] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0062] Related technologies typically rely on a single acoustic feature or a fixed modality combination (such as fixed acoustics + a single visual modality, where the visual modality could be lip movements) to train a neural network model, and then use the trained model to achieve language recognition. However, due to the mobility of humanoid robots and the complexity of their environments, the parameters of the models trained in this way are fixed, and related technologies face the following significant challenges:

[0063] 1. Severe scene noise interference: Industrial scenes are filled with strong noise such as motor operation, mechanical impact, and assembly line roar (noise intensity can reach 60-80dB), while household scenes are filled with environmental noise such as refrigerator operation, TV background noise, and conversations (noise intensity can reach 40-60dB). These noises will seriously pollute the acoustic features of the speech signal, causing a significant drop in the recognition accuracy of traditional language models based on single acoustic features. For example, single acoustic encoder models often have low language recognition accuracy in industrial scenes.

[0064] 2. Interaction distance and angle fluctuations: Humanoid robots need to move within a range of 1-5 meters during industrial inspections. In home scenarios, users may also interact with the robot in different rooms (1-3 meters away). The voice signal will attenuate with distance (the signal strength decreases by 6-10dB for every 1 meter increase in distance). At the same time, changes in the user's lip-sync angle will lead to incomplete visual information, further reducing the stability of language recognition. For example, static visual aid models are not adapted to distance fluctuations, resulting in large fluctuations in recognition accuracy.

[0065] 3. Weak anti-interference capability of single modality: Related technologies mostly rely on acoustic features or fixed modal combinations for language recognition, without combining the humanoid robot's own visual perception capabilities (such as the user's lip movement patterns and scene environment types), and cannot improve anti-interference capability through multimodal information complementarity;

[0066] 4. Static models cannot adapt to dynamic scenarios: The language recognition model parameters in related technologies are fixed and cannot dynamically adjust feature weights according to the switching between industrial and home scenarios (e.g., industrial scenarios should focus on acoustic noise reduction, while home scenarios should focus on visual assistance), resulting in poor robustness of the model when applied across scenarios.

[0067] The aforementioned challenges result in poor robustness and accuracy of language recognition for humanoid robots in complex scenarios.

[0068] It should be noted that the aforementioned related technologies are only used to assist in understanding the technical solutions of this application and do not mean that they belong to the publicly disclosed prior art.

[0069] In view of this, embodiments of this application provide a training method for a humanoid robot language recognition model and a language recognition method. The training method involves multi-layer cross-modal coding of multimodal training samples to obtain several cross-modal coding features containing acoustic coding features at different coding levels. This not only combines the humanoid robot's own visual perception capabilities (such as user lip movement patterns and scene environment types) to enhance anti-interference capabilities through multimodal information complementarity, but also captures local and global speech information through acoustic coding features at different coding levels, providing multi-granular feature support for subsequent cross-modal fusion, thus improving the robustness and accuracy of language recognition.

[0070] Furthermore, this training method employs data augmentation through dynamic scene adaptation of multimodal training samples. Specifically, it adds noise to audio modal samples using a distance screening function, randomly occludes lip-sync modal samples to simulate occlusions such as side faces or hands caused by distance fluctuations, and simulates perspective changes caused by humanoid robot movement by rotating environmental modal samples and challenging their lighting. This adapts to distance fluctuations, which helps suppress fluctuations in the accuracy of language recognition models, thereby improving the robustness and accuracy of language recognition.

[0071] Furthermore, this training method also obtains acoustic feature weights and visual feature weights for the corresponding scenarios based on environmental prediction label analysis. The first fusion feature is then weighted and refined using the acoustic and visual feature weights. Based on the target fusion feature obtained from the weighted refinement, the humanoid robot language recognition model is trained. This allows the language recognition model to dynamically adjust feature weights according to the switching between industrial scenarios (such as equipment inspection and human-machine collaborative assembly) and home scenarios (such as furniture control and companionship education). For example, industrial scenarios should focus on acoustic noise reduction, while home scenarios should focus on visual assistance. This is beneficial to improving the robustness of the language recognition model when applied across scenarios.

[0072] Reference Figure 1 In this embodiment of the application, a method for training a language recognition model for a humanoid robot includes:

[0073] Step 110: Obtain multimodal training samples, which include audio modal samples, lip-sync modal samples, and environmental modal samples;

[0074] In this application embodiment, the audio modal sample can be multilingual language data in different scenarios (such as industrial scenarios or home scenarios) used for training; the lip-sync modal sample can be the user's lip-sync sequence image when the corresponding audio modal sample is collected; and the environmental modal sample can be the scene environment image when the corresponding audio modal sample is collected.

[0075] It is understandable that, after obtaining the original multimodal training samples, one of the optional methods can also preprocess the original multimodal training samples to obtain preprocessed multimodal training samples. Specifically, preprocessing can involve pre-emphasizing, framing, and windowing (e.g., Hanning windowing) the audio modal samples to extract the preprocessed audio modal samples. Additionally, a machine learning algorithm (Dlib) library is used to extract several facial key points from the lip shape modal samples (focusing on key points around the lips), constructing a lip shape temporal sequence. This lip shape temporal sequence is then input into a 2-layer convolutional neural network (CNN) + a 1-layer long short-term memory network (LSTM), outputting a 256-dimensional feature vector as the preprocessed lip shape modal sample. The CNN has a 3×3 kernel size, a stride of 1, padding="same", and ReLU activation function, while the LSTM has 256 hidden units and tanh activation function. Finally, the environmental modal sample image is scaled to a preset size (e.g., 222×224) and input into the first 10 layers of a ResNet50 network for processing to obtain the preprocessed environmental modal sample.

[0076] In some embodiments, the method further includes:

[0077] The multimodal training samples are subjected to dynamic scene adaptation data augmentation to obtain augmented multimodal training samples;

[0078] The data augmentation process, which involves dynamic scene adaptation of the multimodal training samples to obtain augmented multimodal training samples, includes:

[0079] Based on a preset distance attenuation function, scene noise is added to the audio modal samples to obtain audio modal samples with added noise;

[0080] The lip-shape modal samples are randomly occluded to obtain enhanced lip-shape modal samples;

[0081] The environmental modal samples are then subjected to illumination adjustment and angle rotation to obtain enhanced environmental modal samples.

[0082] In this embodiment, the distance attenuation function can be expressed as:

[0083]

[0084] in, Let be the distance decay function, which represents the interaction distance between the humanoid robot and the user. Speech signal strength at that time (unit: dB); The interaction distance between humanoid robots and users is The voice signal strength (unit: dB) at a given meter can be obtained by actual measurement and calibration of the microphone array of a humanoid robot. This application embodiment uses an industrial scenario as an example. =70-80dB, home use scenario =60-70dB example; The interaction distance between humanoid robots and users; This is the environmental degradation coefficient, which can be calibrated experimentally. The embodiments in this application use an industrial scenario. =0.8, Home use scenario =0.5 Example.

[0085] Understandably, scene noise addition can be achieved by adding scene-specific noise (such as industrial noise library containing 5 types of noise, including motors and impacts, with a signal-to-noise ratio of 10-20dB; household noise library containing 4 types of noise, including refrigerators and televisions, with a signal-to-noise ratio of 15-25dB) to audio modal samples based on distance attenuation function, and simulating speech attenuation of 1-5 meters to obtain audio modal samples with added noise.

[0086] Random occlusion can be achieved by randomly occluding 20%-50% of the key points of the lip-shape image features in the lip-shape modality sample to simulate the occlusion of the user's side face and head, thereby obtaining an enhanced lip-shape modality sample; lighting challenge and angle rotation can be achieved by adjusting the brightness (e.g., brightness ±30% adjustment) and direction (e.g., ±15° rotation) of the environmental image features of the environmental modality sample to simulate the change in perspective caused by the movement of a humanoid robot, thereby obtaining an enhanced environmental modality sample.

[0087] It should be noted that after obtaining the audio modal samples with added noise, the enhanced lip-sync modal samples, and the environmental modal samples, the enhanced multimodal training samples can be easily constructed based on the obtained modal samples.

[0088] Step 120: Perform multi-layer cross-modal coding on the multimodal training samples to obtain several cross-modal coding features. Each cross-modal coding feature includes an acoustic coding feature, a lip-sync coding feature, and an environmental coding feature. The coding level of the acoustic coding feature in the first cross-modal feature is different from the coding level of the acoustic coding feature in the second cross-modal feature. The first cross-modal feature is any one of the cross-modal coding features. The second cross-modal feature is any one of the cross-modal coding features other than the first cross-modal feature.

[0089] In this embodiment, the multimodal training samples can be the original multimodal training samples, the preprocessed multimodal training samples, or the preprocessed and data-augmented multimodal training samples. Multi-layer cross-modal coding can be performed by inputting the multimodal training samples into the multi-layer cross-modal encoder in the humanoid robot language recognition model, and encoding the multimodal training samples through the multi-layer cross-modal encoder to obtain several cross-modal coded features.

[0090] In some embodiments, performing multi-layer cross-modal coding on the multimodal training samples to obtain several cross-modal coding features includes:

[0091] The audio modal samples are subjected to multi-layer acoustic concatenation coding to obtain several acoustic coding features, each of which has a different feature level.

[0092] Furthermore, the multi-layer acoustic concatenation coding of the audio modal samples yields several acoustic coding features, including:

[0093] Obtain intermediate features, wherein the intermediate features are the audio modality samples or the acoustic coding features of the previous feature level;

[0094] Acoustic feature extraction is performed on the intermediate features to obtain the acoustic coding features of the current feature level.

[0095] In this embodiment, the multi-layer cross-modal encoder includes a lip-sync visual encoder, an environmental visual encoder, and a cascaded multi-layer acoustic encoder. Specifically, the multi-layer acoustic cascaded encoding can involve inputting audio modal samples into the cascaded multi-layer acoustic encoder for encoding. The specific number of layers in the multi-layer acoustic encoder can be set according to actual conditions, for example, any number of layers such as 10, 12, or 16. Each layer acoustic encoder is a Wav2Vec2.0 encoder, which includes a multi-head self-attention layer, a convolutional module, and a feedforward neural network, thereby obtaining the acoustic coding features output by each layer acoustic encoder.

[0096] It is understandable that for any acoustic encoder, if it is a first-layer acoustic encoder, the intermediate features input to it can be audio modal samples, and the output features can be acoustic coding features of the first feature layer; or, if it is a second or later layer acoustic encoder, the intermediate features input to it can be acoustic coding features of the previous feature layer, and the output features can be acoustic coding features of the current feature layer. For example, if it is a third-layer acoustic encoder, the intermediate features input to it are acoustic coding features of the second feature layer, and the output features are acoustic coding features of the third feature layer.

[0097] It should be noted that, in this embodiment, a cascaded multilayer acoustic encoder with 12 layers is used as an example. The acoustic coding features output by the acoustic encoders at different layers carry different information. Specifically, the features output by shallow encoders (such as acoustic encoders at layers 1-4) focus more on the local details of speech (such as the acoustic texture of phonemes and syllables), while the features output by deep encoders (such as acoustic encoders at layers 9-12) focus more on the global semantics of speech (such as the prosody at the sentence level and language-specific acoustic patterns). In this embodiment, the acoustic coding features at each feature level not only provide multi-granularity features for the subsequent cross-modal fusion module, but also capture both local and global information of speech.

[0098] It is worth mentioning that, compared to cascaded multilayer acoustic encoders, using "non-cascaded parallel encoders" (such as 12 independent encoders processing acoustic features separately) will result in a lack of progressive abstraction relationships among the various acoustic features, making it impossible to achieve feature extraction "from local to global," thus affecting the discriminative power of subsequent fused features. In addition, compared to cascaded multilayer acoustic encoders, extracting only the "output features of the last encoder layer" (such as extracting only the acoustic coding features of the 12th feature level) will lose language-specific details in shallow local features (such as the differences in phoneme pronunciation between different languages ​​are often reflected in shallow features), resulting in a decrease in the model's ability to distinguish similar languages ​​(such as English, German, and French).

[0099] The lip shape modality samples are visually encoded to obtain the lip shape encoding features;

[0100] The environmental modality samples are subjected to environmental visual encoding to obtain the environmental encoding features;

[0101] Based on the lip shape coding features and the environmental coding features, each of the acoustic coding features is concatenated to obtain several cross-modal coding features.

[0102] In this embodiment, lip shape visual encoding can involve inputting lip shape modality samples into a lip shape visual encoder for feature encoding extraction. This lip shape visual encoder can be a single-layer Long Short-Term Memory (LSTM) network to obtain lip shape encoded features. Environmental visual encoding can involve inputting environmental modality samples into an environmental visual encoder. This environmental visual encoder can be the last five layers of a ResNet50 network to obtain environmental encoded features.

[0103] Understandably, feature concatenation can involve concatenating each acoustic coding feature with lip-sync coding features and environmental coding features to obtain several cross-modal coding features.

[0104] Step 130: Perform cross-modal fusion on all the cross-modal encoded features to obtain the first fused feature;

[0105] In this embodiment, cross-modal fusion can be achieved by inputting all cross-modal encoded features into the attention fusion module in the humanoid robot language recognition model, and then fusing all cross-modal encoded features through the attention fusion module to obtain the first fused feature.

[0106] In some embodiments, the cross-modal fusion of all the cross-modal encoded features to obtain a first fused feature includes:

[0107] Cross-attention analysis is performed on all the cross-modal coding features to obtain the attention weight of each cross-modal coding feature;

[0108] Based on each attention weight, the corresponding cross-modal coding features are weighted to obtain several cross-modal weighted features;

[0109] The first fused feature is obtained by fusing all the cross-modal weighted features.

[0110] In this embodiment, cross-attention analysis can be performed by inputting all cross-modal coding features into an attention fusion module constructed based on the cross-attention mechanism. The attention fusion module then performs scaled dot-product attention calculation on each cross-modal coding feature and the other cross-modal coding features to obtain the attention weight of each cross-modal coding feature.

[0111] It is understandable that feature weighting can be achieved by multiplying each attention weight element-wise with the corresponding cross-modal coding feature to obtain the weighted cross-modal coding feature, which is denoted as cross-modal weighted feature; while feature fusion can be achieved by summing all cross-modal weighted features to obtain the first fused feature.

[0112] Step 140: Based on the environmental modal samples and the first fusion feature, update the parameters of the initialized humanoid robot language recognition model to obtain the trained humanoid robot language recognition model.

[0113] In this embodiment, parameter updating first involves obtaining the ground truth labels corresponding to the multimodal training samples. These ground truth labels record the language category and scene type labels (e.g., industrial scene labels or home scene labels) of the audio modal samples. Then, combining the loss function, a target loss value corresponding to the first fused feature is determined based on the environmental modal samples. Many types of loss functions are commonly used, such as 0-1 loss function, squared loss function, absolute loss function, logarithmic loss function, and cross-entropy loss function, all of which can be used as model loss functions and will not be elaborated upon here. In this embodiment, any one of these loss functions can be selected to determine the training loss value, such as the cross-entropy loss function. Based on the training target loss value, the backpropagation algorithm is used to update the model parameters. After several iterations, a well-trained humanoid robot language recognition model can be obtained. The specific number of iterations can be preset, or training can be considered complete when the test set reaches the required accuracy.

[0114] In some embodiments, updating the parameters of the initialized humanoid robot language recognition model based on the environmental modal samples and the first fusion feature to obtain a trained humanoid robot language recognition model includes:

[0115] The environmental modal samples are classified to obtain environmental prediction labels;

[0116] Based on the environmental prediction labels, dynamic weight analysis is performed to obtain acoustic feature weights and visual feature weights.

[0117] In this embodiment, environment classification can be achieved by inputting features from environmental modal samples into a scene classifier consisting of several fully connected layers (e.g., a scene classifier consisting of two fully connected layers, where the first fully connected layer has 256 neurons and the activation function used can be the ReLU function, and the second fully connected layer has one neuron and a Sigmoid activation function). The scene classifier then predicts the environment indicated by the environmental modal samples to obtain the environment prediction label. This prediction label is used to indicate the scene environment in which the multimodal training samples are located. In this embodiment, the scene environment can be an industrial scene or a home scene.

[0118] Understandably, dynamic weight analysis can determine the acoustic and visual feature weights of corresponding multimodal training samples based on a dynamic weight function and environmental prediction labels. Specifically, the dynamic weight function can be expressed as:

[0119]

[0120] in, For acoustic feature weights in the dynamic weighting function; The visual feature weights in the dynamic weighting function are lip shape feature weights and / or scene feature weights. This is the scene influence coefficient, which can be obtained through training, as shown in the embodiments of this application. ; The basic weight coefficients can also be obtained through training, as shown in the embodiments of this application. .

[0121] The first fusion feature is weighted and refined based on the acoustic feature weights and the visual feature weights to obtain the target fusion feature;

[0122] Further, the step of weighting and refining the first fused feature based on the acoustic feature weights and the visual feature weights to obtain the target fused feature includes:

[0123] Based on the acoustic feature weights and the visual feature weights, the first fusion feature is weighted and fused to obtain the second fusion feature;

[0124] Temporal attention fusion is performed on the second fusion feature to obtain the third fusion feature;

[0125] Modal attention fusion is performed on the third fusion feature to obtain the target fusion feature.

[0126] In this embodiment, the first fusion feature is obtained by concatenating lip-sync features, environment-sync features, and acoustic-sync features, and then performing cross-modal fusion based on an attention mechanism. This first fusion feature includes an acoustically relevant part corresponding to the acoustically sync features and a visually relevant part corresponding to the visually sync features (such as lip-sync features and environment-sync features). Therefore, weighted fusion can be performed by multiplying the acoustically relevant part of the first fusion feature based on acoustic feature weights, and by multiplying the visually relevant part of the first fusion feature based on visual feature weights, thereby obtaining the second fusion feature.

[0127] Understandingly, temporal attention fusion involves sequentially inputting the second fused feature into a first multi-head attention layer and a first attention statistical pooling layer. The first multi-head attention layer performs temporal attention analysis on the second fused feature, and the first attention statistical pooling (ASP) layer fuses the features of the second fused feature at different time steps to obtain the third fused feature. Similarly, modal attention fusion involves sequentially inputting the third fused feature into a second multi-head attention layer and a second attention statistical pooling (ASP) layer. The second multi-head attention layer performs modal attention analysis on the third fused feature, and the second attention statistical pooling layer fuses cross-modal features to obtain the final target fused feature.

[0128] It should be noted that the input to the first attention statistical pooling layer can be the second fusion feature with visual dimension attention weights output by the first multi-head attention layer. The first attention statistical pooling layer can perform weighted statistical aggregation on the second fusion feature based on the normalized visual dimension attention weights to prioritize the retention of time step feature information with higher weights and aggregate it into a globally compact feature. There are various implementation methods for specific weighted statistical aggregation, such as using the "weighted mean + weighted variance concatenation" method. This application will not elaborate further here. Finally, the third fusion feature is obtained by pooling the aggregated second fusion feature.

[0129] The content of the second attention statistical pooling layer is similar to that of the first attention statistical pooling layer, and can be easily deduced. The difference is that the second attention statistical pooling layer performs weighted statistical aggregation on the third fusion feature based on the modality dimension attention weight output by the second multi-head attention layer in order to prioritize the retention of modality feature information with higher weights and aggregate it into a global compact feature. This application will not elaborate further here.

[0130] Based on the target fusion features, the parameters of the initialized humanoid robot language recognition model are updated to obtain a trained humanoid robot language recognition model.

[0131] In this embodiment, after obtaining the target fusion features, the target fusion features can be input into a fully connected layer connected with an activation function (such as the Softmax function) to perform language prediction, thereby obtaining the language prediction probability corresponding to the multimodal training samples; then, based on the language prediction probability, the above-mentioned loss function (such as the cross-entropy loss function) and the language category label, the parameters of the humanoid robot language recognition model are updated, thereby obtaining the trained humanoid robot language recognition model.

[0132] In this embodiment of the application, a language identification method includes:

[0133] Acquire multimodal data for language recognition collected by a humanoid robot;

[0134] The multimodal data is input into the pre-trained humanoid robot language recognition model to perform language recognition and obtain the language recognition result.

[0135] In this embodiment, the humanoid robot can collect voice data during interaction with the user via its onboard microphone array, and capture images of the user's lip movements and the current environment via a camera device (such as a binocular camera), thus forming multimodal data for language recognition. Next, the multimodal data is input into the humanoid robot's language recognition model, which identifies the language spoken by the user, thereby obtaining the language recognition result.

[0136] Understandably, after obtaining the language recognition result, the humanoid robot can load the corresponding speech synthesis model based on that result to achieve human-computer voice interaction between the humanoid robot and the user. Specifically, if the language recognition result is Chinese, the humanoid robot can load a Chinese speech synthesis model (such as TTS-Chinese) and industrial / household Chinese interaction logic (industrial users load professional terminology dictionaries such as "equipment failure" and "inspection task," while household users load everyday phrase dictionaries such as "turn on the lights" and "tell a story"); if the recognition result is English, an English speech synthesis model (such as TTS-English) and corresponding interaction logic are loaded to complete subsequent human-computer voice interaction.

[0137] The following describes in detail, with reference to the accompanying drawings, a training system for a language recognition model of a humanoid robot according to an embodiment of this application.

[0138] Reference Figure 2 The present application proposes a training system for a humanoid robot language recognition model, including...

[0139] The first processing unit 101 is used to acquire multimodal training samples, which include audio modal samples, lip-sync modal samples, and environmental modal samples.

[0140] The second processing unit 102 is used to perform multi-layer cross-modal coding on the multimodal training samples to obtain several cross-modal coding features. Each cross-modal coding feature includes an acoustic coding feature, a lip-sync coding feature, and an environmental coding feature. The coding level of the acoustic coding feature in the first cross-modal feature is different from the coding level of the acoustic coding feature in the second cross-modal feature. The first cross-modal feature is any one of the cross-modal coding features. The second cross-modal feature is any one of the cross-modal coding features other than the first cross-modal feature.

[0141] The third processing unit 103 is used to perform cross-modal fusion on all the cross-modal coding features to obtain the first fused feature;

[0142] The fourth processing unit 104 is used to update the parameters of the initialized humanoid robot language recognition model based on the environmental modal samples and the first fusion feature, so as to obtain a trained humanoid robot language recognition model.

[0143] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0144] Reference Figure 3 This application also provides an electronic device, including:

[0145] At least one processor 201;

[0146] At least one memory 202 is used to store at least one program;

[0147] When the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the method embodiment described above.

[0148] Similarly, it can be understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0149] This application also provides a computer-readable storage medium storing a program executable by a processor 201, which, when executed by the processor 201, is used to implement the above-described method embodiments.

[0150] Similarly, the content of the above method embodiments is applicable to the present computer-readable storage medium embodiments. The specific functions implemented by the present computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0151] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0152] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods.

[0153] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this application are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0154] Furthermore, although this application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding this application. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional technology for an engineer. Therefore, those skilled in the art can implement the application set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of this application, which is determined by the full scope of the appended claims and their equivalents.

[0155] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0156] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0157] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0158] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0159] In the foregoing description of this specification, the references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0160] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0161] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A training method for a language recognition model of a humanoid robot, characterized in that, include: Obtain multimodal training samples, which include audio modal samples, lip-sync modal samples, and environmental modal samples; The multimodal training samples are subjected to multi-layer cross-modal coding to obtain several cross-modal coding features. Each cross-modal coding feature includes an acoustic coding feature, a lip-sync coding feature, and an environmental coding feature. The coding level of the acoustic coding feature in the first cross-modal feature is different from the coding level of the acoustic coding feature in the second cross-modal feature. The first cross-modal feature is any one of the cross-modal coding features. The second cross-modal feature is any one of the cross-modal coding features other than the first cross-modal feature. Cross-modal fusion is performed on all the cross-modal encoded features to obtain the first fused feature; Based on the environmental modal samples and the first fusion feature, the parameters of the initialized humanoid robot language recognition model are updated to obtain a trained humanoid robot language recognition model.

2. The method according to claim 1, characterized in that, The process of performing multi-layer cross-modal coding on the multimodal training samples yields several cross-modal coding features, including: The audio modal samples are subjected to multi-layer acoustic concatenation coding to obtain several acoustic coding features, each of which has a different feature level. The lip shape modality samples are visually encoded to obtain the lip shape encoding features; The environmental modality samples are subjected to environmental visual encoding to obtain the environmental encoding features; Based on the lip shape coding features and the environmental coding features, each of the acoustic coding features is concatenated to obtain several cross-modal coding features.

3. The method according to claim 2, characterized in that, The audio modal samples are subjected to multi-layer acoustic concatenation coding to obtain several acoustic coding features, including: Obtain intermediate features, wherein the intermediate features are the audio modality samples or the acoustic coding features of the previous feature level; Acoustic feature extraction is performed on the intermediate features to obtain the acoustic coding features of the current feature level.

4. The method according to claim 1, characterized in that, The step of fusing all the cross-modal encoded features across modalities to obtain the first fused feature includes: Cross-attention analysis is performed on all the cross-modal coding features to obtain the attention weight of each cross-modal coding feature; Based on each attention weight, the corresponding cross-modal coding features are weighted to obtain several cross-modal weighted features; The first fused feature is obtained by fusing all the cross-modal weighted features.

5. The method according to claim 1, characterized in that, The step of updating the parameters of the initialized humanoid robot language recognition model based on the environmental modal samples and the first fusion feature to obtain a trained humanoid robot language recognition model includes: The environmental modal samples are classified to obtain environmental prediction labels; Based on the environmental prediction labels, dynamic weight analysis is performed to obtain acoustic feature weights and visual feature weights. The first fusion feature is weighted and refined based on the acoustic feature weights and the visual feature weights to obtain the target fusion feature; Based on the target fusion features, the parameters of the initialized humanoid robot language recognition model are updated to obtain a trained humanoid robot language recognition model.

6. The method according to claim 5, characterized in that, The step of weighted refining of the first fused feature based on the acoustic feature weight and the visual feature weight to obtain the target fused feature includes: Based on the acoustic feature weights and the visual feature weights, the first fusion feature is weighted and fused to obtain the second fusion feature; Temporal attention fusion is performed on the second fusion feature to obtain the third fusion feature; Modal attention fusion is performed on the third fusion feature to obtain the target fusion feature.

7. A language identification method, characterized in that, include: Acquire multimodal data for language recognition collected by a humanoid robot; The multimodal data is input into the trained humanoid robot language recognition model as described in any one of claims 1-6 to perform language recognition and obtain the language recognition result.

8. A training system for a language recognition model of a humanoid robot, characterized in that, include: The first processing unit is used to acquire multimodal training samples, which include audio modal samples, lip-sync modal samples, and environmental modal samples. The second processing unit is used to perform multi-layer cross-modal coding on the multimodal training samples to obtain several cross-modal coding features. Each cross-modal coding feature includes an acoustic coding feature, a lip-sync coding feature, and an environment coding feature. The coding level of the acoustic coding feature in the first cross-modal feature is different from the coding level of the acoustic coding feature in the second cross-modal feature. The first cross-modal feature is any one of the cross-modal coding features. The second cross-modal feature is any one of the cross-modal coding features other than the first cross-modal feature. The third processing unit is used to perform cross-modal fusion on all the cross-modal encoded features to obtain the first fused feature; The fourth processing unit is used to update the parameters of the initialized humanoid robot language recognition model based on the environmental modal samples and the first fusion feature, so as to obtain the trained humanoid robot language recognition model.

9. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to implement the method as described in any one of claims 1-7.

Images