Robot language and intention interaction method and system and computer readable storage medium

Abstract

The invention discloses a robot language and intention interaction method and system and a computer readable storage medium, and the method comprises the steps: collecting voice information and / or gesture information, carrying out the feature extraction of the voice information and / or gesture information, obtaining semantic features and / or gesture semantic tags, and carrying out the recognition of the semantic features and / or gesture semantic tags; and based on the semantic features and / or the gesture semantic labels, through a dynamic classifier, intention category classification is realized, intention probability distribution is output, based on the probability distribution, a special encoder is selectively activated to extract domain-specific term features, and the dynamic classifier can also perform intention category expansion through an incremental learning mechanism, so that the domain-specific term features are extracted. And the semantic features and / or the gesture semantic tags are fused to calculate joint confidence, and an instruction execution module generates a robot executable instruction based on the joint confidence. According to the method, high-precision multi-modal interaction and intention understanding in a noise environment are realized, dynamic extension of intention categories is supported, and the accuracy and adaptability of robot interaction are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a robot language and intention interaction method and system and a computer readable storage medium. BACKGROUND

[0002] A robot is a programmable intelligent machine capable of performing tasks autonomously or semi-autonomously, usually with the following core features: perception ability, obtaining environmental information through sensors (such as cameras, radars, force sensors, etc.); decision-making and computing ability, processing data and task planning based on algorithms (such as AI, machine learning, control theory); execution ability, completing physical operations or movements through mechanical structures (such as robotic arms, wheeled / footed mobile mechanisms); programmability, being able to adjust behavior according to preset programs or adaptive learning.

[0003] Current robot systems face significant functional limitations: (1) Acoustic attenuation of single-channel speech acquisition in industrial noise environments leads to a significant increase in instruction misrecognition rate; (2) In multi-modal instruction conflict scenarios, traditional threshold judgment method is difficult to effectively coordinate voice and gesture instructions, and the joint confidence normalization processing of existing D-S evidence theory lacks multi-modal adaptability; (3) Global retraining is required for intention classifier expansion, causing significant degradation of old task performance, and the lack of flexible feature space design restricts the seamless integration of new intention categories. SUMMARY

[0004] To solve the above problems, the application provides a robot language and intention interaction method, system and computer readable storage medium, which can recognize the user's intention according to the user's voice information and / or gesture information and execute the corresponding instructions according to the user's intention.

[0005] In a first aspect, the application provides a robot language and intention interaction method, which specifically includes the following steps: S1, collecting voice information and / or gesture information; S2, performing feature extraction on the voice information and / or the gesture information to obtain semantic features and / or gesture semantic labels, respectively; S3, based on the semantic features and / or the gesture semantic labels, implementing intention category classification through a dynamic classifier, outputting an intention probability distribution, and selectively activating a special encoder to extract field-specific term features based on the probability distribution, wherein the dynamic classifier can expand intention categories through an incremental learning mechanism; S4, fusing the semantic features and / or the gesture semantic labels to calculate a joint confidence, and generating a robot executable instruction based on the joint confidence by an instruction execution module.

[0006] Preferably, the voice information is collected, specifically including the following steps: S101, synchronously collecting voice information of the first channel and the second channel; S102, performing frame processing on the voice information, adding a Hamming window to each frame of the voice information, and converting the voice information from time domain voice information to frequency domain voice information through fast Fourier transform; S103, taking the conjugate of the second channel spectrum of the frequency domain voice information, multiplying it with the first channel spectrum, generating a time-frequency domain mask to enhance the voice signal component and suppress noise, and obtaining voice information.

[0007] The dual-channel voice collection mechanism constructs an adaptive noise suppression system at the signal processing level, generates a time-frequency mask by multiplying the conjugate of the dual-channel spectrum, essentially enhances the energy weight of the coherent component of the voice signal, and attenuates the non-correlated spectral components of the environmental noise, ensuring the integrity of the acoustic characteristics of the original instruction. The frame windowing and frequency domain conversion decompose the instantaneous sound wave into analyzable spectral units, enabling the subsequent processing module to achieve high-accuracy intent decoding based on high-fidelity voice information.

[0008] Preferably, the gesture information is collected, specifically including the following steps: S111, synchronously collecting RGB video stream and Depth image of the gesture, and outputting time-synchronized multi-modal frame sequences based on motion key frame detection and time sequence regularization algorithm, wherein each set of the multi-modal frame sequences includes spatially aligned RGB images and Depth images; S112, extracting static appearance features and dynamic motion features of the RGB frame sequence; extracting cross-frame correlation features through a spatio-temporal convolution layer and extracting single-frame visual features through an appearance convolution layer; Adaptively weighting and fusing the cross-frame correlation features and the single-frame visual features to generate a fusion feature map; S113, inputting the fusion feature map into a time sequence modeling module, calculating inter-frame importance weights through an attention mechanism, and weightedly fusing to generate an optimized dynamic feature sequence; S114, fusing the recognition results of the RGB modality and the Depth modality, and outputting a final gesture semantic label through a joint confidence conflict resolution algorithm.

[0009] Preferably, the step S2 specifically includes the following steps: extracting the general semantic features by processing the text converted by voice recognition through the general encoder; extracting the general semantic features by processing the text converted by voice recognition through the general encoder;

[0010] Preferably, the step S3 specifically includes the following steps: S301, fuse the general semantic features with the gesture semantic labels to generate a joint semantic representation; S302, based on the joint semantic representation, output a multi-modal intent probability distribution, and when a probability of any intent category in the multi-modal probability distribution exceeds a predetermined probability threshold of the dynamic classifier configuration, activate a specialized encoder to extract professional term features associated with the intent category.

[0011] Preferably, the dynamic classifier can expand the intent categories through an incremental learning mechanism, specifically including the following steps: S311, freeze the bottom-layer network parameters of the pre-trained model in the dynamic classifier to retain cross-domain semantic understanding capabilities; S312, expand the output dimension of the classification layer of the dynamic classifier in real time to adapt to the newly added intent categories; S313, in the feature space of the dynamic classifier, assign an independent feature subspace to each newly added category; S314, through the learnable gating mechanism in the dynamic classifier, weightedly fuse the new and old features to suppress feature conflicts.

[0012] Preferably, the step S4 specifically includes the following steps: S401, based on the similarity changes of the general speech feature vector and the gesture semantic label vector, update the current instruction conflict state value in combination with the conflict factor maintained by the dynamic classifier itself; S402, determine whether the current instruction conflict state value is greater than the tolerance threshold preset by the D-S evidence fusion module, if the current instruction conflict state value is greater than the tolerance threshold preset by the D-S evidence fusion module, terminate the instruction output; otherwise, execute step S403; S403, based on the D-S evidence theory, perform normalized fusion on the speech instruction confidence and the gesture instruction confidence to output a multi-modal joint confidence, and the instruction execution module generates a robot executable instruction based on the joint confidence.

[0013] The present application cooperatively integrates the speech and gesture instruction confidences based on the normalized fusion of the D-S evidence theory, so that the output joint confidence objectively reflects the consistency of the cross-modal intents Figure One The instruction execution module generates reliable action instructions accordingly, significantly improving the accuracy of multi-source instruction cooperation and the stability of the system.

[0014] The application significantly enhances the adaptability and operation reliability of robots to complex environments, clearly captures key instructions in an industrial noise scene through dual-channel voice noise reduction, realizes seamless expansion of new intent categories by combining the incremental learning mechanism of a dynamic classifier, and avoids system downtime caused by retraining in traditional schemes; The application innovatively integrates a conflict resolution algorithm into multi-modal instructions, can accurately quantify the degree of conflict when multi-modal conflicts occur, realizes safe decision-making based on evidence theory fusion, ensures strict consistency and operation safety of instruction execution in high-precision scenes, and greatly improves the fluency and trust of human-machine collaboration.

[0015] In a second aspect, the application provides a robot language and intent interaction system, comprising: a dual-channel voice acquisition module, a gesture recognition module, a dynamic classifier module, a semantic decoupling module, a D-S evidence fusion module, and an instruction execution module, The dual-channel voice acquisition module and the gesture recognition module work synchronously, the dual-channel voice acquisition module acquires voice information, and the gesture recognition module acquires gesture information; The dynamic classifier module realizes intent category classification based on semantic information and / or gesture information, and outputs an intent distribution probability; The semantic decoupling module includes a general encoder and a special encoder, the general encoder is used to process text converted by voice recognition and extract general semantic features; the special encoder selectively extracts domain-specific term features based on the intent probability distribution; The D-S evidence fusion module is used to solve multi-modal conflicts and output a multi-modal joint confidence; The instruction execution module generates robot executable instructions based on the joint confidence.

[0016] Preferably, the dynamic classifier module comprises: an intent classification submodule and an incremental learning submodule, The intent classification submodule receives the general semantic features output by the general encoder and the gesture semantic labels output by the gesture recognition module, and fuses the features and labels to generate a multi-modal intent probability distribution, and selectively activates the special encoder based on the probability distribution; The incremental learning submodule freezes the pre-trained model bottom layer parameters of the dynamic classifier module to retain cross-domain semantic understanding ability, real-time expand the classification layer output dimension to adapt to new intent categories, allocate an independent feature subspace for each new category, and suppress conflicts by weighted fusion of new and old features through a learnable gating mechanism.

[0017] The robot language and intention interaction system significantly enhances the robustness and collaborative intelligence of instruction processing. The dual-channel voice acquisition module breaks through the noise interference and accurately captures the instruction signal. The dynamic classifier module realizes lossless expansion and self-evolution of intention recognition. The semantic decoupling module accurately strips the general semantics and domain-specific features, ensuring that the core intention is not disturbed by professional terms. When there is a fundamental contradiction in multi-modal input, the D-S evidence fusion module constructs a conflict quantization system, outputs a credible decision through a self-iterative conflict factor and a normalized fusion mechanism, and the instruction execution module is converted into an unambiguous action instruction.

[0018] In a third aspect, the present application also provides a robot language and intention interaction computer readable storage medium, which stores executable code, when the executable code is executed by the processor of the electronic device, the processor executes the method of the first aspect.

[0019] The present application has the following advantages: (1) The dual-channel noise suppression mechanism fundamentally overcomes the problem of voice attenuation caused by industrial noise, ensuring the accuracy of sound source recognition; (2) The adaptive D-S evidence fusion architecture effectively resolves multi-modal instruction conflicts, breaks through the collaborative bottleneck of traditional threshold method, and realizes dynamic weighted integration of voice and gesture semantics; (3) The proposed elastic feature space and incremental classification mechanism completely avoids the need for global retraining, so that the expansion of new intention categories no longer affects the existing knowledge structure, achieving lossless expansion of intention understanding ability. The whole system forms an organic synergy in noise suppression, conflict decision and dynamic expansion, and completely breaks through the core bottleneck of existing multi-modal interaction systems. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 A working flow principle diagram of a robot language and intention interaction method according to an embodiment of the present application.

[0021] Figure 2 A system structure schematic diagram of a robot language and intention interaction system according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] The specific embodiments of the present application will be further described below with reference to the accompanying drawings. It should be noted that the description of these embodiments is used to help understand the present application, but does not constitute a limitation on the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0023] The application relates to the technical field of artificial intelligence, especially to a scene of human-computer interaction, and aims to understand user intention by accurately identifying voice instructions and gesture instructions of the user. Current robot systems have limitations: (1) in an industrial noise environment, the acoustic attenuation of single-channel voice collection affects instruction misrecognition rate; (2) when facing multi-modal instruction conflicts, traditional threshold judgment method cannot effectively coordinate voice and gesture instructions, and the joint confidence normalization processing of the existing D-S evidence theory lacks effective adaptation to multi-modal; (3) the expansion of an intention classifier needs to rely on global retraining, which easily causes significant degradation of old task performance, and the lack of elastic feature space design restricts the seamless integration of new intention categories. The application proposes an innovative solution aiming at the limitations of current robot systems, so as to improve the robustness of voice recognition, multi-modal instruction conflict resolution capability and intention category expansion through an incremental learning mechanism, thereby improving the overall accuracy of robot intention recognition.

[0024] Embodiment one: Reference is made to the accompanying drawings Figure 1 The embodiment provides a robot language and intention interaction method, which specifically comprises the following steps: Step S1, synchronously collecting environmental voice signals through a dual-channel microphone array, collecting environmental voice signals and / or capturing gesture action video streams by using an RGB-D camera, collecting voice information and / or gesture information, and in an industrial noise environment, a dual-channel noise suppression algorithm is used to separate a target sound source from background noise, so as to improve the quality of voice signals.

[0025] Step S2, performing feature extraction on the collected information, converting the noise-reduced voice signals into texts by using ASR (automatic speech recognition), and extracting general text semantic features by using a general encoder; meanwhile, inputting the gesture video stream into a 3D-2D hybrid model, the model fuses optical flow features and appearance features, and finally outputs corresponding gesture semantic labels (for example, “stop”, “confirm” and the like).

[0026] Step S3, a dynamic classifier receives the semantic features and / or gesture semantic labels obtained in the previous step, performs intention classification, and outputs probability distribution of each intention category, the dynamic classifier core adopts an incremental learning mechanism based on transfer learning, supports seamless integration of new intention categories by using an elastic feature space design under the premise of freezing most parameters of a basic model, and avoids performance degradation of old tasks. According to the probability distribution result, when a specific field intention is recognized, a special encoder is activated, and is responsible for extracting special term features of the field; The dynamic classifier module sequentially comprises an input layer, a feature fusion and transformation layer, an extensible classification layer, and an elastic feature space and gating layer: the input layer receives and splices the speech general semantic feature and the gesture semantic embedding vector; the feature fusion and transformation layer performs nonlinear transformation and integration on the fused features through a fully connected network to generate a unified joint semantic representation; the extensible classification layer outputs a multi-modal intent probability distribution based on the representation, and the output dimension can be dynamically extended to adapt to new intent categories; and the elastic feature space and gating layer adjusts the fusion ratio of new and old features through a learnable gating mechanism to suppress feature conflicts and ensure the stability of incremental learning.

[0027] Step S4, the D-S evidence fusion module fuses the confidence of the speech semantic feature and the gesture semantic label to obtain a joint confidence, and a command execution module generates a structured control instruction directly executable by the robot based on the final joint confidence and possible special term features.

[0028] Further, the speech information acquisition specifically comprises the following steps: Step S101, based on a dual-channel noise suppression algorithm, the first channel and the second channel are synchronously collected for speech information, assuming that the target signal S(t) is completely correlated, and the noises Noise1(t) and Noise2(t) are independent of each other. The collected first channel signal is , and the second channel signal is ,

[0029] wherein j is an imaginary unit, and satisfies .

[0030] Step S102, the speech information is subjected to frame processing, and a Hamming window is added to each frame of the speech information, and the speech information is converted from time domain speech information to frequency domain speech information through fast Fourier transform, to obtain a first channel spectrum FFT , , ,

[0031] wherein the real part A / B corresponds to a signal principal component, and the imaginary part W1 / W2 is a noise component.

[0032] Step S103, the second channel spectrum of the frequency domain speech information is conjugated and multiplied by the first channel spectrum to generate a time-frequency domain mask, so as to enhance the speech signal component and suppress the noise, and obtain speech information . wherein the imaginary part noise term tends to cancel out due to irrelevance, and the real part retains signal energy.

[0033] In the embodiment, a double-channel noise suppression algorithm based on frequency domain complex operation is adopted, a frequency domain mask is constructed by introducing an imaginary unit j (satisfying j²=-1), and target signals (A / B) and noise components (W1 / W2) are effectively separated. The algorithm significantly improves the retention capability of speech components, and the retention rate of real part signals is improved by 60%.

[0034] In the signal preprocessing stage, Hamming window framing and FFT frequency domain conversion are used to effectively suppress non-stationary noise, and the overall noise suppression effect reaches 78%. The method is robust and suitable for speech enhancement tasks in high-noise environments such as industrial sites.

[0035] Further, the gesture information is collected, specifically including the following steps: Step S111, synchronously collecting the RGB video stream and the Depth depth image of the gesture, and outputting the time sequence synchronized multi-modal frame sequence based on the motion key frame detection and time sequence regularization algorithm, wherein each group of the multi-modal frame sequence comprises spatially aligned RGB images and Depth images, the time sequence information of the RGB and depth modalities is aligned by using a dynamic time regularization algorithm (such as DTW), and each group of sequence is unified to a standard length. For sequences with insufficient length, the existing frames are filled by cyclically copying the existing frames from the starting frame to ensure the spatiotemporal consistency of subsequent processing. Step S112, extracting static appearance features and dynamic motion features of the RGB frame sequence. Cross-frame correlation features are extracted through a spatiotemporal convolution layer to learn the time sequence evolution pattern and motion correlation of the gesture action; single-frame visual features are extracted through an appearance convolution layer to capture the texture, shape and appearance information of the gesture. Further, a Canny-2DHOG operator is introduced to extract static edge direction gradient features, and a optical flow-2DHOG method is used to extract dynamic features representing gesture motion information, thereby enhancing the representation ability of hand contour, appearance and motion pattern. The static appearance features and dynamic motion features are spliced in the channel dimension to construct multi-cue enhanced visual feature representation. An adaptive weighted fusion method based on a gating mechanism is used to integrate the spatiotemporal correlation features and the multi-cue enhanced features, dynamically optimize the contribution weight of different feature components, and finally generate a more discriminative fused feature map.

[0036] Step S113, input the fusion feature map into a time sequence modeling module. The module introduces a space-time attention mechanism in an LSTM layer, dynamically models the inter-frame dependency relationship and calculates the importance weight of each frame according to the hidden state feature ht of the current frame and the LSTM hidden state st-1 of the previous time. The hidden state feature ht encodes the visual and space-time information of the current frame, and the LSTM hidden state st-1 of the previous time carries the accumulated context of the historical frames; by evaluating the correlation between the hidden state feature ht and the LSTM hidden state st-1 of the previous time, the attention mechanism can identify the information frames that are critical to the current recognition task in the sequence, and suppress the influence of redundant or interfering frames. Finally, the time sequence features are weighted and fused based on the obtained weight, to generate an optimized dynamic feature sequence that highlights the time sequence patterns most relevant to gesture category discrimination.

[0037] Step S114, fuse the recognition results of the RGB and Depth modalities, resolve the conflict through joint confidence calculation based on D-S evidence theory, and output the final gesture semantic label. When the recognition results of the two modalities conflict, the joint confidence of each gesture category is calculated using D-S evidence theory, and the ambiguity is eliminated through a probability weighted fusion strategy.

[0038] In this embodiment, a 3D-2D hybrid RBM model is used to fuse the Canny-2DHOG appearance feature and the optical flow-2DHOG motion feature in the input layer, and a high-efficiency feature extraction is realized through a space-time convolution kernel, with a recognition delay of less than 80 ms. When the RGB and Depth modalities conflict, the D-S evidence theory is used to calculate the joint confidence, which significantly improves the system robustness and reduces the misrecognition rate to 2.3%.

[0039] Further, the step S2 specifically comprises the following steps: extracting the general semantic features of the text processed by the speech recognition conversion through the general encoder; extracting the space-time attention features of the gesture through the space-time feature extraction network of the multi-modal frame sequence, and outputting the semantic label of the gesture; realizing the collaborative classification and semantic fusion of the multi-modal input through the dynamic classifier: The speech instruction classification converts the noise-reduced speech signal output by the dual-channel speech acquisition module into a text sequence through ASR; the general semantic features of the text are extracted through the general encoder; the expandable classification layer in the dynamic classifier processes the semantic features in combination with the elastic feature space, and outputs the probability distribution of the speech intent; The gesture instruction classification outputs a spatio-temporal attention feature based on the optical flow-2DHOG motion feature through a gesture recognition module; the gesture feature is mapped to a semantic label (such as "stop" or "go") through an instruction execution module; the dynamic classifier aligns and fuses the gesture semantic label with the text semantics of the voice instruction in a unified feature space, and outputs a joint intention probability distribution.

[0040] Further, the step S3 specifically comprises the following steps: Step S301, fuse the general semantic feature with the gesture semantic label, the fusion process of the general semantic feature and the gesture semantic label is completed in the dynamic classifier module, the general semantic feature vector of the voice instruction is spliced with the embedding vector of the gesture semantic label in a manner combining feature splicing and linear transformation, and then dimension transformation and information integration are performed through the full connection layer in the dynamic classifier to form a unified joint semantic representation vector and generate a joint semantic representation; Step S302, based on the joint semantic representation, an extensible classification layer in the dynamic classifier outputs a multi-modal intention probability distribution; when the probability value of any intention category in the multi-modal probability distribution exceeds a predetermined probability threshold, a special encoder in the semantic decoupling module is activated, the special encoder adopts a GRU and a gated attention mechanism architecture to extract a domain-specific term feature associated with the intention category; the domain-specific term feature is used to enhance the domain distinguishability of the intention representation, and provides more accurate semantic information for subsequent instruction generation.

[0041] Further, the dynamic classifier can expand the intention categories through an incremental learning mechanism, specifically comprising the following steps: Step S311, freeze the bottom network parameters of a pre-trained language model (such as BERT-base) in the dynamic classifier to retain its general language representation ability and cross-domain semantic understanding ability; at the same time, calculate the Fisher information matrix of the old task parameters (i.e. the network weight trained and converged when learning the old intention categories) to identify the key parameters (the Fisher information matrix is used to quantify the importance of each network parameter to the old task performance, and the higher the value, the more critical the parameter is to maintaining the old task performance), and in the subsequent training process, the change range of these key parameters is constrained through an elastic weight solidification regular term to prevent catastrophic forgetting; Step S312, expand the output dimension of the extensible classification layer at the top of the dynamic classifier in real time, so that it strictly corresponds to the total number of newly added intention categories; the expansion simultaneously uses a cosine normalization classifier (a classification layer that calculates classification scores based on the cosine similarity between the weight vector and the feature vector instead of the dot product) to alleviate the class imbalance problem, and this design can effectively improve the long-tail data classification performance (i.e. the recognition accuracy of the old categories with fewer samples when the number of samples of different categories is extremely unbalanced). Step S313, assign an independent feature subspace to each new intent category in the elastic feature space of the dynamic classifier; through counter-enhancement processing such as back-translation and synonym replacement on new intent samples, data diversity is improved to ensure effective learning of the feature subspace; Step S314, through the learnable gating mechanism in the dynamic classifier, the fusion proportion of new and old features in the joint semantic representation is dynamically adjusted; at the same time, a dynamic feature distillation strategy is adopted, taking the old model as the teacher network and the new model as the student network, and minimizing the KL divergence of the two in the feature space, so that the new model implicitly inherits the feature representation ability of the old model, thereby realizing adaptive weighted fusion of new and old features and effectively suppressing feature conflict (that is, by making the new model learn to imitate the feature output of the old model for the same input, forcing the new model to retain the representation of old category knowledge, thereby reducing semantic conflict and performance degradation when fusing new and old features).

[0042] Further, the step S4 specifically comprises the following steps: Step S401, based on the cosine similarity change of the general voice feature vector and the gesture semantic label vector, combining the historical conflict factor K_prev maintained by the dynamic classifier itself, the current instruction conflict state value is calculated using the conflict factor update formula; the conflict factor formula is: K_new=0.7Kprev+0.3(1−cosθ); Where θ is the angle between the voice feature vector and the gesture semantic label vector; S402, judge whether the current conflict state value K_new is greater than the preset tolerance threshold 0.1815: if K_new>0.1815, it is considered that the conflict between voice and gesture modalities is not reconcilable, and the instruction output is terminated to avoid misoperation; otherwise, step S403 is executed; S403, based on the D-S evidence theory, the voice instruction confidence m1(B) and the gesture instruction confidence m2(C) are normalized and fused to output a multi-modal joint confidence; the fusion formula is:

[0043] Where m1(B) represents the quality function of the voice instruction, m2(C) represents the quality function of the gesture instruction, and B and C represent the identities of the two different sources respectively; the denominator 1-K is a normalization factor; the instruction execution module generates a robot executable instruction based on the joint confidence, and preferentially executes the instruction intent of the high-confidence modality; In this embodiment, the dynamic conflict factor updating mechanism and the conflict tolerance threshold ensure the sensitivity and stability of the system to the conflict between modalities, and the joint confidence normalization calculation effectively improves the reliability of multi-modal instruction fusion.

[0044] Embodiment two: With reference to the description attached Figure 2 The embodiment provides a robot language and intention interaction system, which comprises a double-channel voice acquisition module, a gesture recognition module, a dynamic classifier module, a semantic decoupling module, a D-S evidence fusion module and an instruction execution module. The double-channel voice acquisition module and the gesture recognition module work synchronously, the double-channel voice acquisition module synchronously acquires environmental voice signals through a double-channel microphone array, and the gesture recognition module captures gesture action video streams by using an RGB-D camera, acquires voice information and / or gesture information, and in an industrial noise environment, a double-channel noise suppression algorithm is used to separate a target sound source from background noise and improve voice signal quality.

[0045] The dynamic classifier module realizes intention category classification based on semantic information and / or gesture information, and outputs an intention distribution probability, the dynamic classifier receives semantic features and / or gesture semantic labels obtained in the previous step, performs intention classification, and outputs a probability distribution of each intention category, the core of the dynamic classifier adopts an incremental learning mechanism based on transfer learning, supports seamless integration of new intention categories through elastic feature space design on the premise of freezing most parameters of a basic model, and avoids performance degradation of old tasks.

[0046] The semantic decoupling module comprises a general encoder and a special encoder, the general encoder is used for processing text converted by voice recognition, extracting general semantic features, and extracting features of collected information, a noise-reduced voice signal is converted into text by ASR (automatic speech recognition), and general text semantic features are extracted by the general encoder; according to the probability distribution result output by the dynamic classifier, when a specific field intention is recognized, the special encoder is activated and is responsible for extracting special term features in the field.

[0047] The D-S evidence fusion module performs fusion calculation on the confidence of voice semantic features and gesture semantic labels to obtain joint confidence.

[0048] The instruction execution module generates structured control instructions that can be directly executed by a robot based on the final joint confidence and possible special term features.

[0049] Further, the dynamic classifier module comprises: an intention classification submodule and an incremental learning submodule, The intent classification submodule receives the general semantic features output by the general encoder and the gesture semantic labels output by the gesture recognition module, and fuses the features and labels to generate a multi-modal intent probability distribution, based on which the dedicated encoder is selectively activated; The incremental learning submodule freezes the pre-training model bottom parameters of the dynamic classifier module to retain cross-domain semantic understanding ability, real-time expand the classification layer output dimension to adapt to new intent categories, allocate an independent feature subspace for each new category, and suppress conflicts by weighting and fusing new and old features through a learnable gating mechanism.

[0050] Embodiment three: The application provides a computer readable storage medium, which stores executable code, and when the executable code is executed by a processor of a robot language and intent interaction system, the robot language and intent interaction system implements the robot language and intent interaction method as described in embodiment one.

[0051] Although example embodiments have been described herein with reference to the accompanying drawings, it is to be understood that the above description is merely of example embodiments and is not intended to limit the scope of the present application. Those of ordinary skill in the art can make various changes and modifications without departing from the scope and spirit of the present application. All such changes and modifications are intended to be included within the scope of the present application as claimed in the appended claims.

[0052] Although the description of the present application is made in conjunction with the above specific embodiments, it is obvious to those skilled in the art that many substitutions, modifications and changes can be made based on the above description. Therefore, all such substitutions, modifications and changes are included within the spirit and scope of the appended claims.

Claims

1. A method for robot language and intent interaction, characterized in that, The method comprises the following steps: S1, collecting voice information and / or gesture information; S2, extracting features from the voice information and / or the gesture information to obtain semantic features and / or gesture semantic labels, respectively; S3, based on the semantic features and / or the gesture semantic labels, implementing intent category classification through a dynamic classifier, outputting an intent probability distribution, and selectively activating a special encoder to extract field-specific term features based on the probability distribution, wherein the dynamic classifier can expand the intent categories through an incremental learning mechanism; S4, fusing the semantic features and / or the gesture semantic labels to calculate a joint confidence, and instructing an execution module to generate robot executable instructions based on the joint confidence.

2. The method of Claim 1, wherein, The voice information is collected by: S101, synchronously collecting voice information from a first channel and a second channel; S102, performing frame processing on the voice information, adding a Hamming window to each frame of the voice information, and converting the voice information from time domain voice information to frequency domain voice information through fast Fourier transform; S103, taking the conjugate of the second channel spectrum of the frequency domain voice information, multiplying it by the first channel spectrum, generating a time-frequency domain mask to enhance the voice signal component and suppress noise, and obtaining voice information.

3. The method of Claim 2, wherein, The gesture information is collected by: S111, synchronously collecting an RGB video stream and a depth image of a gesture, and outputting a time-synchronous multi-modal frame sequence based on a motion key frame detection and time sequence regularization algorithm, wherein each set of the multi-modal frame sequence comprises spatially aligned RGB images and depth images; S112, extracting static appearance features and dynamic motion features of the RGB frame sequence; extracting cross-frame correlation features through a space-time convolution layer and extracting single-frame visual features through an appearance convolution layer; self-adaptively weighting and fusing the cross-frame correlation features and the single-frame visual features to generate a fused feature map; S113, inputting the fused feature map into a time sequence modeling module, calculating inter-frame importance weights through an attention mechanism, and weightedly fusing to generate an optimized dynamic feature sequence; S114, fusing the recognition results of the RGB modality and the depth modality, and outputting a final gesture semantic label through a joint confidence conflict resolution algorithm.

4. The method of Claim 3, wherein, The step S2 comprises: processing the text converted by voice recognition through the general encoder to extract the general semantic features; extracting the gesture semantic labels through a multi-modal frame sequence of a space-time feature extraction network.

5. The method of Claim 4, wherein, The step S3 comprises: S301, fusing the general semantic features and the gesture semantic labels to generate joint semantic representations; S302, after outputting a multi-modal intent probability distribution based on the joint semantic representations, activating a special encoder to extract professional term features associated with an intent category when the probability of any intent category in the multi-modal probability distribution exceeds a predetermined probability threshold configured by the dynamic classifier.

6. The method of Claim 5, wherein, The dynamic classifier can expand the intent categories through an incremental learning mechanism, comprising: S311, freezing the bottom network parameters of the pre-trained model in the dynamic classifier to retain cross-domain semantic understanding ability; S312, extending the classification layer output dimension of the dynamic classifier in real time to adapt to the newly added intent category; S313, assigning an independent feature subspace to each newly added category in the feature space of the dynamic classifier; S314, weighting and fusing new and old features through a learnable gating mechanism in the dynamic classifier to suppress feature conflicts.

7. The method of Claim 6, wherein, The step S4 comprises: S401, updating the current instruction conflict state value based on the similarity change of the general speech feature vector and the gesture semantic label vector, and combining the conflict factor maintained by the dynamic classifier itself; S402, judging whether the current instruction conflict state value is greater than the tolerance threshold preset by the D-S evidence fusion module, if the current instruction conflict state value is greater than the tolerance threshold preset by the D-S evidence fusion module, terminating the instruction output; otherwise, executing step S403; S403, performing normalized fusion on the speech instruction confidence and the gesture instruction confidence based on the D-S evidence theory, outputting a multi-modal joint confidence, and the instruction execution module generating a robot executable instruction based on the joint confidence.

8. The robotic language and intent interaction system of any of claims 1-7, wherein, Comprise: a dual-channel speech acquisition module, a gesture recognition module, a dynamic classifier module, a semantic decoupling module, a D-S evidence fusion module, and an instruction execution module, The dual-channel speech acquisition module and the gesture recognition module work synchronously, the dual-channel speech acquisition module acquires speech information, and the gesture recognition module acquires gesture information; The dynamic classifier module realizes intent category classification based on semantic information and / or gesture information, and outputs an intent distribution probability; The semantic decoupling module comprises a general encoder and a special encoder, the general encoder is used to process the text converted by speech recognition and extract general semantic features; the special encoder selectively extracts domain-specific term features based on the intent probability distribution; The D-S evidence fusion module is used to solve multi-modal conflicts and output a multi-modal joint confidence; The instruction execution module generates a robot executable instruction based on the joint confidence.

9. The robotic language and intent interaction system of claim 8, wherein, The dynamic classifier module comprises: an intent classification submodule and an incremental learning submodule, The intent classification submodule receives the general semantic features output by the general encoder and the gesture semantic labels output by the gesture recognition module, and fuses the features and labels to generate a multi-modal intent probability distribution, and selectively activates the special encoder based on the probability distribution; The incremental learning submodule freezes the pre-trained model bottom layer parameters of the dynamic classifier module to retain cross-domain semantic understanding ability, extends the classification layer output dimension in real time to adapt to the newly added intent category, assigns an independent feature subspace to each newly added category, and suppresses conflicts by weighting and fusing new and old features through a learnable gating mechanism.

10. A computer-readable storage medium, characterized in that, The non-transitory computer-readable medium has executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method of any one of claims 1-7. The non-transitory computer-readable medium has executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method of any one of claims 1-7.

Images