A method for robot intelligent voice dialogue interaction

Abstract

This invention relates to the field of voice dialogue interaction and discloses a robot intelligent voice dialogue interaction method. The method involves collecting and preprocessing user voice, environmental noise, and visual image data using multi-source sensors on a robot, then extracting semantic acoustic, visual emotion, and environmental acoustic features. A context-aware parsing model is used, combined with historical dialogue context, to parse user text content, intent, and key entity slots. Cross-modal fusion analysis of semantic acoustic and visual emotion features yields emotion categories and intensities. An environmental interference coefficient is calculated based on environmental acoustic features to assess the interaction background. Features such as dialogue intent, emotion, and environmental interference are concatenated and input into a dynamic knowledge graph to extract associated features, generating a state fusion vector and retrieving candidate dialogue actions. Finally, a deep reinforcement learning network is used to calculate candidate action strategy matching scores, selecting the optimal action to generate a dynamic response strategy, thus achieving intelligent and personalized voice dialogue interaction.

Description

Technical Field

[0001] This invention relates to the field of voice dialogue interaction technology, and more specifically to a robot intelligent voice dialogue interaction method. Background Technology

[0002] Robotic intelligent voice dialogue interaction technology aims to achieve natural and smooth two-way information communication between humans and devices. Its core process involves parsing and understanding the multimodal perception signals input by users, inferring intent and generating strategies in combination with the current interaction context, and finally converting the feedback results into voice and accompanying physical parameter output to support the autonomous operation of various complex application scenarios such as intelligent customer service, companion services and human-machine collaboration.

[0003] However, existing methods for intelligent voice dialogue interaction with robots still have the following drawbacks: Firstly, existing technologies typically process speech, vision, and ambient sound as independent channels in a single dimension, without performing cross-modal fusion analysis of semantic acoustic features and visual emotional features to output accurate emotional category probability distribution values ​​and emotional intensity scores. At the same time, there is a lack of a mechanism to calculate environmental interference coefficients based on environmental acoustic features, which makes it difficult to accurately identify the user's true emotional tendencies and communication background in complex and noisy scenarios. Secondly, existing technologies, when performing action retrieval, typically rely only on the currently extracted dialogue intent and key entity slots for local matching. They fail to construct a global "state fusion vector" by combining emotional state, environmental interference coefficients, and graph-related features through feature concatenation and cross-weight calculation. This results in the system being unable to accurately map the current real interaction scenario in the node vector space of the dynamic knowledge graph, leading to weak semantic relevance of the retrieved candidate actions. Third, existing technologies often use strategies based on static rules or local similarity to directly output the final response action when selecting it. They do not introduce deep reinforcement learning networks to perform in-depth value calculation and strategy matching score evaluation on the state fusion vector and multiple candidate dialogue actions. As a result, the system cannot select the action in a globally optimal way by combining the complex and ever-changing real-time state, and it is difficult to accurately generate dynamic response strategies that include adaptive voice control parameters. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides a robot intelligent voice dialogue interaction method to solve the problems existing in the background art.

[0005] This invention provides the following technical solution: a robot intelligent voice dialogue interaction method, comprising: S1: Real-time collection of user interaction data through the robot's multi-source sensors, including user voice data, environmental noise data, and visual image data, and preprocessing of the collected user interaction data; S2: By extracting features from the preprocessed user interaction data, semantic acoustic features, visual emotional features, and environmental acoustic features are obtained. S3: Input semantic acoustic features into a pre-trained context-aware parsing model, combine the historical dialogue context of the current dialogue, output the user's text content, calculate the semantic correlation between the text content and the historical dialogue context, and output the dialogue intent and key entity slots based on the calculation results. S4: Perform cross-modal fusion analysis of semantic acoustic features and visual emotional features to obtain the probability distribution value of emotional category and the emotional intensity score; S5: Based on the environmental acoustic characteristics, classify acoustic scenes and calculate acoustic parameters to obtain the environmental interference coefficient, and evaluate the interference level and communication background context of the current human-computer interaction. S6: The dialogue intent, key entity slots, sentiment category probability distribution values, sentiment intensity scores and environmental interference coefficients are concatenated and input into the dynamic knowledge graph for associated feature extraction to obtain the state fusion vector in the current interaction scenario. Based on the state fusion vector, semantic similarity is retrieved in the node vector space of the dynamic knowledge graph to generate multiple candidate dialogue actions that are semantically associated with the current state. S7: Input the state fusion vector and multiple candidate dialogue actions into a deep reinforcement learning network for value calculation, output the policy matching score of each candidate dialogue action, and select the optimal dialogue action based on the policy matching score to generate a dynamic response policy.

[0006] Preferably, step S1 involves real-time acquisition of user interaction data via a multi-source sensor array deployed on the robot body. This includes capturing a mixed audio stream using a microphone array and simultaneously acquiring a video image stream using a visual sensor. Subsequently, the mixed audio stream is processed with sound source localization and beamforming to separate independent user voice data and environmental noise data. At the same time, visual image data containing user micro-expression data is extracted from the video image stream. The user voice data, environmental noise data, and visual image data are then subjected to denoising, normalization, and data alignment processing, respectively.

[0007] Preferably, in step S2, classification features are extracted from the preprocessed user interaction data. Specifically, for the separated user speech data, a fusion vector of local frame-level features and global sentence-level features is calculated using an acoustic pre-trained model, and this fusion vector is used as a semantic acoustic feature. For environmental noise data, the frequency band energy distribution value is calculated by Mel spectrum analysis, and the noise type probability distribution value is calculated by acoustic scene feature mapping network. The frequency band energy distribution value and the noise type probability distribution value are combined as environmental acoustic features. Meanwhile, for visual image data, pixel-level gradient response values ​​and spatial position weight values ​​of key facial regions are calculated through a spatial attention mechanism. The pixel-level gradient response values ​​and spatial position weight values ​​are then concatenated dimensionally to construct a visual spatial feature matrix, which is then used as a visual emotion feature.

[0008] Preferably, in step S3, the fusion vector in the semantic acoustic features is input into the pre-trained context-aware parsing model. The acoustic decoder network of the context-aware parsing model decodes the fusion vector frame by frame, calculates the predicted probability value of each word at each time step, and decodes based on the predicted probability value to determine the user's text content. Meanwhile, the historical dialogue context of the current dialogue is input into the context encoder of the context-aware parsing model. Through multi-layer network mapping calculation, the context feature matrix of the historical dialogue context is output. Then, the context feature matrix is ​​multiplied by the word vector sequence of the determined text content to calculate the context association weight coefficient. Finally, the word vector sequence of the text content is weighted and summed using the context association weight coefficients to obtain the context fusion feature vector. This context fusion feature vector is then input into the intent classifier and the slot labeling network for fully connected mapping calculation, outputting the probability distribution value of the dialogue intent and the start and end position index values ​​of the key entity slots.

[0009] Preferably, in step S4, the fusion vector corresponding to the semantic acoustic features and the visual space feature matrix corresponding to the visual emotion features are projected and mapped to the same cross-modal latent space, respectively, and the visual space feature matrix is ​​dimensionality reduced to calculate the speech modal feature vector and the visual modal feature vector. The speech modality feature vector and the visual modality feature vector are concatenated and calculated using a cross-attention mechanism to obtain a cross-modal fusion feature matrix. Finally, the cross-modal fusion feature matrix is ​​input into the sentiment classifier for fully connected mapping calculation, and the sentiment category probability distribution value and sentiment intensity score are output.

[0010] Preferably, in step S5, the environmental acoustic features are input into a pre-constructed scene classification network and parameter estimation network. The scene classification network outputs the scene confidence of the current environment category, and the parameter estimation network calculates the background noise energy value and the signal-to-noise ratio estimate. The environmental interference coefficient is obtained by weighting and summing the scene confidence level, background noise energy value, and signal-to-noise ratio estimate.

[0011] Preferably, in step S6, the dialogue intent, key entity slots, sentiment category probability distribution values, sentiment intensity scores, and environmental interference coefficients are sequentially concatenated into a multidimensional state array. This multidimensional state array is then mapped to the feature space of a dynamic knowledge graph. Graph nodes with relationships to this array and their node attributes are extracted, and contextual path features between related nodes are obtained. The dialogue intent, key entity slots, sentiment category probability distribution values, sentiment intensity scores, environmental interference coefficients, along with the extracted node attributes and contextual path features, are concatenated in multiple dimensions to obtain a long sequence containing all feature information. Subsequently, cross-weight calculations are performed on the features of each dimension in the long sequence, and the calculation results are remapped into a fixed-length multidimensional numerical sequence, which serves as the state fusion vector for the current interaction scenario. The process involves obtaining each dialogue action node from the dynamic knowledge graph and converting it into a corresponding action vector. The similarity between the state fusion vector and each action vector is calculated to obtain the similarity between each dialogue action. Finally, the similarity values ​​are sorted from largest to smallest, and the top N dialogue action nodes are extracted to generate multiple candidate dialogue actions.

[0012] Preferably, in step S7, the state fusion vector and the action vector corresponding to each candidate dialogue action are input into the deep reinforcement learning network, the policy matching score of each candidate dialogue action is calculated and output, the candidate dialogue action with the highest score is selected as the optimal dialogue action, and the corresponding target response text and target sentiment tag are extracted. The target speech rate and target volume are calculated proportionally based on the magnitude of the environmental interference coefficient, and the target tone is calculated by matching the user's emotional tendency state reflected by the probability distribution value of the emotion category. Finally, the target response text, target sentiment tag, target speech rate, target tone, and target volume are combined and packaged to generate a dynamic response strategy.

[0013] The technical effects and advantages of this invention are as follows: (1) By performing cross-modal fusion analysis of semantic acoustic features and visual emotional features, the probability distribution value of emotional category and the score of emotional intensity are accurately quantified and output. At the same time, the environmental interference coefficient is independently calculated based on the environmental acoustic features, so that the user's true emotional tendency and real-time communication environment can be captured comprehensively and accurately in the context of noisy, changeable and complex interaction.

[0014] (2) By comprehensively splicing the dialogue intent, key entity slots, sentiment probability, intensity score and environmental interference coefficient, and inputting them into the dynamic knowledge graph for related feature extraction and cross weight calculation, a “state fusion vector” containing global information is constructed. This vector can perform high-dimensional semantic similarity retrieval in the graph node vector space, ensuring that the generated candidate dialogue actions are highly consistent with the current real complex interaction scenario.

[0015] (3) By inputting the state fusion vector and multiple candidate dialogue actions into a deep reinforcement learning network, the system outputs a policy matching score through deep value calculation, thereby selecting the globally optimal dialogue action in complex and ever-changing real-time states. Based on this optimal action, the system can accurately generate a dynamic response strategy that includes adaptive voice control parameters (such as tone matching with emotion, speech rate and volume matching with environment), making the robot's expression more intelligent and human-like. Attached Figure Description

[0016] Figure 1 This is a diagram illustrating the method steps of the present invention. Detailed Implementation

[0017] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. In addition, the forms of the various structures described in the following embodiments are merely illustrative. The robot intelligent voice dialogue interaction method involved in the present invention is not limited to the structures described in the following embodiments. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] like Figure 1 The embodiment shown provides a robot intelligent voice dialogue interaction method, including: S1: The robot collects user interaction data in real time through its multi-source sensors. The user interaction data includes user voice data, environmental noise data, and visual image data. The collected user interaction data is then preprocessed.

[0019] In this embodiment, step S1 collects user interaction data in real time through a multi-source sensor array deployed on the robot body. Specifically, a microphone array is used to capture a mixed audio stream, and a visual sensor is used to simultaneously collect a video image stream. Subsequently, the mixed audio stream is processed by sound source localization and beamforming to separate independent user voice data and environmental noise data. At the same time, visual image data containing user micro-expression data is extracted from the video image stream, and the user voice data, environmental noise data, and visual image data are respectively processed by denoising, normalization, and data alignment.

[0020] Specifically, after using a microphone array to capture the mixed audio stream for sound source localization and beamforming to separate independent user voice data and environmental noise data, and using a visual sensor to simultaneously acquire video image streams and extract visual image data containing user micro-expression data, the separated user voice data is further subjected to noise reduction filtering and volume normalization to eliminate residual acoustic interference. The environmental noise data is subjected to frequency band filtering and sound pressure level normalization to retain core environmental features. The visual image data is subjected to face region cropping, illumination compensation, and pixel normalization to eliminate visual background interference. Finally, based on a unified system clock, the processed user voice data, environmental noise data, and visual image data are aligned with multimodal timestamps to ensure that the data frames of each modality at the same interaction moment are strictly synchronized and matched in the time dimension, thereby outputting a standardized, high-quality multi-source interactive data stream.

[0021] S2: By extracting features from the preprocessed user interaction data, semantic acoustic features, visual emotional features, and environmental acoustic features are obtained.

[0022] In this embodiment, S2 performs classification feature extraction on the preprocessed user interaction data. Specifically, for the separated user speech data, a fusion vector of local frame-level features and global sentence-level features is calculated through an acoustic pre-trained model, and the fusion vector is used as a semantic acoustic feature. For environmental noise data, the frequency band energy distribution value is calculated by Mel spectrum analysis, and the noise type probability distribution value is calculated by acoustic scene feature mapping network. The frequency band energy distribution value and the noise type probability distribution value are combined as environmental acoustic features. Meanwhile, for visual image data, pixel-level gradient response values ​​and spatial position weight values ​​of key facial regions are calculated through a spatial attention mechanism. The pixel-level gradient response values ​​and spatial position weight values ​​are then concatenated dimensionally to construct a visual spatial feature matrix, which is then used as a visual emotion feature.

[0023] Specifically, for user speech data, the system performs frame segmentation and windowing according to a preset frame length and frame shift (e.g., frame length of 20-30 milliseconds, stride of 10 milliseconds). It uses a pre-trained Wav2Vec model to extract the acoustic embedding of each frame as local frame-level features. Simultaneously, the entire speech sentence is aggregated into a fixed-dimensional global feature vector through a self-attention pooling layer. The two are then concatenated along the feature channel dimension and mapped through a fully connected layer to output semantic acoustic features of a preset dimension (e.g., 640 dimensions). For environmental noise data, the system converts the noise waveform into a spectrogram containing a preset number (e.g., 40) of Mel filter banks. The spectral amplitude within each filter band is squared and summed to obtain the frequency band energy distribution value for the corresponding frequency band dimension. This spectrogram is then input into a mapping network composed of multiple convolutional layers and processed by Softmax. The live function output includes the probability distribution values ​​of several preset noise categories (such as white noise, sudden impact, background human voice, etc.), and the two are concatenated to form the environmental acoustic features. For visual image data, the system first uses a key point model to locate key facial regions (such as the eyes and mouth) and crop them into local image patches. Then, the image patches are input into the spatial attention module, and gradient operators (such as the Sobel operator) are used to calculate the gradient response of each pixel and calculate the modulus to generate a pixel-level gradient response value map that represents the intensity of muscle deformation. At the same time, a mask is generated by global pooling and activation functions (such as Sigmoid) to assign corresponding adaptive spatial position weights to different facial regions. Finally, the gradient response value map and the weight mask are tensor concatenated in the channel dimension to construct a multi-channel three-dimensional visual spatial feature matrix, which serves as the final visual emotion feature.

[0024] S3: Input semantic acoustic features into a pre-trained context-aware parsing model, combine the current dialogue's historical dialogue context, output the user's text content, calculate the semantic correlation between the text content and the historical dialogue context, and output the dialogue intent and key entity slots based on the calculation results.

[0025] In this embodiment, S3 inputs the fusion vector in the semantic acoustic features into the pre-trained context-aware parsing model. The acoustic decoder network of the context-aware parsing model decodes the fusion vector frame by frame, calculates the predicted probability value of each word at each time step, and decodes based on the predicted probability value to determine the user's text content. Meanwhile, the historical dialogue context of the current dialogue is input into the context encoder of the context-aware parsing model. Through multi-layer network mapping calculation, the context feature matrix of the historical dialogue context is output. Then, the context feature matrix is ​​multiplied by the word vector sequence of the determined text content to calculate the context association weight coefficient. Finally, the word vector sequence of the text content is weighted and summed using the context association weight coefficients to obtain the context fusion feature vector. This context fusion feature vector is then input into the intent classifier and the slot labeling network for fully connected mapping calculation, outputting the probability distribution value of the dialogue intent and the start and end position index values ​​of the key entity slots.

[0026] It should be specifically explained that in the text decoding stage, an acoustic decoder based on the CTC mechanism outputs the word probability distribution frame by frame from the input fusion vector. A beam search algorithm with a preset beam width (e.g., 5) combined with a language model is used for path pruning and merging, decoding and outputting the final user text, such as "How's the weather today?", from multiple candidate paths. In the relevance calculation stage, the system uses a pre-trained language model (such as the BERT model) to encode historical dialogues for a preset number of rounds (e.g., the past three rounds), generating a context feature matrix of size [historical sequence length × feature dimension]. Simultaneously, the current text is converted into a word vector sequence of size [current word count × feature dimension]. The product of the word vector sequence and the transpose of the context feature matrix is ​​calculated and processed by a normalization function (such as Softmax). Output an attention weight matrix of size [current character count × current character count] as context association weight coefficients to characterize the degree to which each character is influenced by its historical context. During the intent and slot output stage, this weight matrix is ​​used to weight and sum the word vector sequence of the current text to obtain a context fusion feature vector of a preset dimension (e.g., 768 dimensions). This vector is then input into an intent classifier based on fully connected layers and a normalization function, outputting a probability distribution such as "[query weather: 0.92, play music: 0.05, ...]" to determine the dialogue intent. Simultaneously, this fusion feature vector is input into a slot labeling network based on a sequence labeling algorithm (e.g., Conditional Random Field, CRF), which outputs a strict BIO entity label sequence "[" for each character in the current text "How's the weather today?" according to character index order. , -weather, -weather, , , ]"(in Represents non-entity external characters. Represents the starting character of an entity. (representing the internal characters of an entity), boundary parsing is performed based on the tag sequence to identify " The starting position is the character index position 2 where "weather" is located. -The character index position 3 where "weather" is located is the termination position, and then the "weather" in the index range [2, 3] is accurately extracted as the key entity slot.

[0027] S4: Perform cross-modal fusion analysis of semantic acoustic features and visual emotional features to obtain the probability distribution value of emotional category and the emotional intensity score.

[0028] In this embodiment, step S4 projects the fusion vector corresponding to the semantic acoustic features and the visual space feature matrix corresponding to the visual emotion features onto the same cross-modal latent space, performs dimensionality reduction on the visual space feature matrix, and calculates the speech modal feature vector and the visual modal feature vector. The speech modality feature vector and the visual modality feature vector are concatenated and calculated using a cross-attention mechanism to obtain a cross-modal fusion feature matrix. Finally, the cross-modal fusion feature matrix is ​​input into the sentiment classifier for fully connected mapping calculation, and the sentiment category probability distribution value and sentiment intensity score are output.

[0029] Specifically, the process involves several steps. First, a semantic acoustic feature fusion vector with a preset dimension (e.g., 640 dimensions) is mapped through a fully connected layer to a speech modality feature vector with a first target dimension (e.g., 256 dimensions). Simultaneously, a visual spatial feature matrix with a high number of channels (e.g., size [64×512]) is sequentially passed through a global average pooling layer (which compresses the feature channels into a single-dimensional vector in the spatial dimension) and another fully connected layer to achieve dimensionality reduction and mapping to a visual modality feature vector with the same first target dimension. This ensures strict alignment of the two dimensions in a unified cross-modal latent space. Subsequently, the two are concatenated into a joint vector with a second target dimension (e.g., 512 dimensions) and input into the cross-attention module. The speech modality feature vector serves as the query, while the visual modality feature vector serves as the input. The visual modality feature vector is used as the key and value. The attention weight of visual expression to speech semantics is calculated and a cross-modal fusion feature matrix of a preset size (e.g., [256×8]) is output. Finally, the matrix is ​​input to an emotion classifier with two parallel branches. The first branch outputs the probability distribution values ​​of emotion categories such as "[happy: 0.85, sad: 0.10, calm: 0.05]" through a fully connected layer with a normalized activation function (e.g., Softmax). The second branch normalizes the absolute value norm of the fusion features to the interval [0, 1] through a mapping activation function (e.g., Sigmoid) and outputs an emotion intensity score such as "0.88" (0.88 represents a very high intensity of the happy emotion), thus completing the dual quantization of the emotion state.

[0030] S5: Based on the environmental acoustic characteristics, perform acoustic scene classification and acoustic parameter calculation to obtain the environmental interference coefficient, and assess the interference level and communication background context of the current human-computer interaction.

[0031] In this embodiment, step S5 inputs the environmental acoustic features into a pre-constructed scene classification network and parameter estimation network. The scene classification network outputs the scene confidence of the current environment category, and the parameter estimation network calculates the background noise energy value and signal-to-noise ratio estimate. The environmental interference coefficient is obtained by weighting and summing the scene confidence level, background noise energy value, and signal-to-noise ratio estimate.

[0032] It should be specifically explained that the environmental acoustic features are simultaneously input into a parallel dual-branch network. The scene classification network branch outputs a vector containing multiple preset environmental probability values ​​(e.g., [quiet indoor: 0.1, busy street: 0.8, construction site: 0.1]) as the scene confidence score through a fully connected layer and a softmax function. The parameter estimation network branch directly maps the background noise energy value (e.g., 65dB) and the signal-to-noise ratio estimate (e.g., 5dB, representing a weak speech signal) in decibels through a regression layer. When calculating the environmental interference coefficient, the system first performs dimension normalization, normalizing the 65dB background noise energy value by applying the maximum value. The noise intensity coefficient (e.g., 0.65) is mapped to the 0-1 interval. The 5dB signal-to-noise ratio estimate is then converted into a signal-to-noise ratio degradation coefficient (e.g., 0.9, representing extremely poor signal-to-noise ratio and extremely high interference) through an inverse proportional function. Subsequently, the normalized three factors are weighted and summed according to preset weight coefficients (e.g., 0.8×0.9 [street confidence] + 0.1×0.65 [noise intensity] + 0.1×0.9 [signal-to-noise ratio degradation] = 0.855). Finally, a dimensionless value of 0.855 in the [0, 1] interval is output as the environmental interference coefficient. The closer this value is to 1, the more severe the environmental interference and the worse the communication background of the current human-computer interaction.

[0033] S6: The dialogue intent, key entity slots, sentiment category probability distribution values, sentiment intensity scores, and environmental interference coefficients are concatenated and input into the dynamic knowledge graph for associated feature extraction. This yields a state fusion vector for the current interaction scenario. Based on the state fusion vector, semantic similarity is retrieved in the node vector space of the dynamic knowledge graph to generate multiple candidate dialogue actions that are semantically associated with the current state.

[0034] In this embodiment, step S6 concatenates the dialogue intent, key entity slots, sentiment category probability distribution values, sentiment intensity scores, and environmental interference coefficients into a multidimensional state array in sequence. The multidimensional state array is then mapped to the feature space of the dynamic knowledge graph. Graph nodes and their attributes that are associated with the array are extracted, and contextual path features between associated nodes are obtained. The dialogue intent, key entity slots, sentiment category probability distribution values, sentiment intensity scores, and environmental interference coefficients, along with the extracted node attributes and contextual path features, are concatenated in multiple dimensions to obtain a long sequence containing all feature information. Subsequently, cross-weight calculations are performed on the features of each dimension in the long sequence, and the calculation results are remapped into a fixed-length multidimensional numerical sequence, which serves as the state fusion vector in the current interaction scenario. The process involves obtaining each dialogue action node from the dynamic knowledge graph and converting it into a corresponding action vector. The similarity between the state fusion vector and each action vector is calculated to obtain the similarity between each dialogue action. Finally, the similarity values ​​are sorted from largest to smallest, and the top N dialogue action nodes are extracted to generate multiple candidate dialogue actions.

[0035] Specifically, the dialogue intent (one-hot encoding of "check the weather"), key entity slots (index values ​​of "weather" [2, 3]), sentiment category probability distribution (3D vector), sentiment intensity score (scalar 0.88), and environmental interference coefficient (scalar 0.85) are first concatenated by channel into an initial state array of, for example, 20 dimensions. This array is then mapped to a query vector in the graph feature space through a fully connected layer. Using this query vector, neighbor nodes are sampled in the dynamic knowledge graph based on an attention scoring mechanism. The attribute vectors of the top-3 nodes with the highest relevance (such as the "local weather bureau" node, the "current date" node, and the "outdoor noise" node) and the contextual path vectors connecting these three nodes are extracted. Subsequently, this 20-dimensional initial state array is concatenated with the extracted node and path vectors into an array of, for example, 80 dimensions. An 80-dimensional sequence is input into a multi-head self-attention network for cross-weight calculation, enabling the intent feature to dynamically focus on relevant graph path features. Finally, a pooled fully connected layer compresses the 80-dimensional sequence into a fixed-length (e.g., 128-dimensional) state fusion vector. Then, 128-dimensional action vectors corresponding to all preset dialogue action nodes (e.g., "report the weather", "turn up the volume", "ask for confirmation") in the dynamic knowledge graph are obtained. The dot product spatial similarity between the state fusion vector and each action vector is calculated using the cosine similarity formula, yielding similarity scores for each action (e.g., 0.92, 0.85, 0.60 respectively). These scores are sorted from largest to smallest, and the top N (e.g., N=3) are truncated, ultimately generating a list of multiple candidate dialogue actions containing "report the weather", "turn up the volume", and "ask for confirmation".

[0036] S7: Input the state fusion vector and multiple candidate dialogue actions into a deep reinforcement learning network for value calculation, output the policy matching score of each candidate dialogue action, and select the optimal dialogue action based on the policy matching score to generate a dynamic response policy.

[0037] In this embodiment, step S7 inputs the state fusion vector and the action vector corresponding to each candidate dialogue action into the deep reinforcement learning network, calculates and outputs the policy matching score of each candidate dialogue action, selects the candidate dialogue action with the highest score as the optimal dialogue action, and extracts the corresponding target response text and target sentiment tag. The target speech rate and target volume are calculated proportionally based on the magnitude of the environmental interference coefficient, and the target tone is calculated by matching the user's emotional tendency state reflected by the probability distribution value of the emotion category. Finally, the target response text, target sentiment tag, target speech rate, target tone, and target volume are combined and packaged to generate a dynamic response strategy.

[0038] It should be specifically noted that, in the policy score calculation stage, the system concatenates the 128-dimensional state fusion vector with the 128-dimensional action vectors corresponding to each candidate dialogue action along the feature channel dimension, and inputs this concatenation into the evaluation fully connected layer of a pre-trained Deep Q-Network (DQN). Through forward propagation, the system directly maps and outputs the Q-values ​​of each candidate action as the policy matching score (e.g., "report the weather" scores 8.5, "turn up the volume" scores 6.2). The system selects "report the weather," corresponding to the highest score of 8.5, as the optimal dialogue action, and extracts the associated target response text "Today is sunny, temperature 25 degrees" and the target sentiment tag "peaceful" from the knowledge graph. In the acoustic parameter mapping stage, the system uses a piecewise linear mapping function to handle the environmental interference coefficient (assuming the current coefficient is 0.85, representing high interference). The system increases the baseline volume (e.g., 70 dB) to the target volume of 84 dB by a direct proportionality factor of 1.2, and decreases the baseline speech rate (e.g., 4 words per second) to the target speech rate of 3.2 words per second by an inverse proportionality factor of 0.8. In the tone matching stage, the system extracts the emotion tendency with the highest probability in the emotion category probability distribution (e.g., "happy" corresponds to a probability of 0.85), and converts this emotion tendency into target tone parameters through a preset tone mapping table or regression network (e.g., increasing the average value of the fundamental frequency F0 in speech synthesis by 15% to express a cheerful feeling). Finally, the text "Today is sunny, temperature 25 degrees Celsius", the emotion tag "peaceful", the speech rate "3.2 words / second", the tone "fundamental frequency +15%" and the volume "84 dB" are combined and packaged according to a predetermined protocol format to generate the final dynamic response strategy and sent to the speech synthesis module for execution.

[0039] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0040] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for a robotic intelligent voice dialog interaction, the method comprising: include: S1: Real-time collection of user interaction data through the robot's multi-source sensors, including user voice data, environmental noise data, and visual image data, and preprocessing of the collected user interaction data; S2: By extracting features from the preprocessed user interaction data, semantic acoustic features, visual emotional features, and environmental acoustic features are obtained. S3: Input semantic acoustic features into a pre-trained context-aware parsing model, combine the historical dialogue context of the current dialogue, output the user's text content, calculate the semantic correlation between the text content and the historical dialogue context, and output the dialogue intent and key entity slots based on the calculation results. S4: Perform cross-modal fusion analysis of semantic acoustic features and visual emotional features to obtain the probability distribution value of emotional category and the emotional intensity score; S5: Based on the environmental acoustic characteristics, classify acoustic scenes and calculate acoustic parameters to obtain the environmental interference coefficient, and evaluate the interference level and communication background context of the current human-computer interaction. S6: The dialogue intent, key entity slots, sentiment category probability distribution values, sentiment intensity scores and environmental interference coefficients are concatenated and input into the dynamic knowledge graph for associated feature extraction to obtain the state fusion vector in the current interaction scenario. Based on the state fusion vector, semantic similarity is retrieved in the node vector space of the dynamic knowledge graph to generate multiple candidate dialogue actions that are semantically associated with the current state. S7: Input the state fusion vector and multiple candidate dialogue actions into a deep reinforcement learning network for value calculation, output the policy matching score of each candidate dialogue action, and select the optimal dialogue action based on the policy matching score to generate a dynamic response policy.

2. The robot intelligent voice dialogue interaction method according to claim 1, characterized in that, The S1 collects user interaction data in real time through a multi-source sensor array deployed on the robot body. Specifically, a microphone array is used to capture a mixed audio stream, and a visual sensor is used to simultaneously collect a video image stream. Subsequently, the mixed audio stream is processed by sound source localization and beamforming to separate independent user voice data and environmental noise data. At the same time, visual image data containing user micro-expression data is extracted from the video image stream, and the user voice data, environmental noise data, and visual image data are respectively processed by denoising, normalization, and data alignment.

3. The robot intelligent voice dialogue interaction method according to claim 2, characterized in that, S2 performs classification feature extraction on the preprocessed user interaction data. Specifically, for the separated user speech data, a fusion vector of local frame-level features and global sentence-level features is calculated through an acoustic pre-trained model, and this fusion vector is used as semantic acoustic features. For environmental noise data, the frequency band energy distribution value is calculated by Mel spectrum analysis, and the noise type probability distribution value is calculated by acoustic scene feature mapping network. The frequency band energy distribution value and the noise type probability distribution value are combined as environmental acoustic features. Meanwhile, for visual image data, pixel-level gradient response values ​​and spatial position weight values ​​of key facial regions are calculated through a spatial attention mechanism. The pixel-level gradient response values ​​and spatial position weight values ​​are then concatenated dimensionally to construct a visual spatial feature matrix, which is then used as a visual emotion feature.

4. The robot intelligent voice dialogue interaction method according to claim 3, characterized in that, S3 inputs the fusion vector from the semantic acoustic features into the pre-trained context-aware parsing model. The acoustic decoder network of the context-aware parsing model decodes the fusion vector frame by frame, calculates the predicted probability value of each word at each time step, and decodes based on the predicted probability value to determine the user's text content. Meanwhile, the historical dialogue context of the current dialogue is input into the context encoder of the context-aware parsing model. Through multi-layer network mapping calculation, the context feature matrix of the historical dialogue context is output. Then, the context feature matrix is ​​multiplied by the word vector sequence of the determined text content to calculate the context association weight coefficient. Finally, the word vector sequence of the text content is weighted and summed using the context association weight coefficients to obtain the context fusion feature vector. This context fusion feature vector is then input into the intent classifier and the slot labeling network for fully connected mapping calculation, outputting the probability distribution value of the dialogue intent and the start and end position index values ​​of the key entity slots.

5. The robot intelligent voice dialogue interaction method according to claim 4, characterized in that, S4 projects the fusion vector corresponding to the semantic acoustic features and the visual space feature matrix corresponding to the visual emotion features onto the same cross-modal latent space, respectively, performs dimensionality reduction on the visual space feature matrix, and calculates the speech modal feature vector and the visual modal feature vector. The speech modality feature vector and the visual modality feature vector are concatenated and calculated using a cross-attention mechanism to obtain a cross-modal fusion feature matrix. Finally, the cross-modal fusion feature matrix is ​​input into the sentiment classifier for fully connected mapping calculation, and the sentiment category probability distribution value and sentiment intensity score are output.

6. The robot intelligent voice dialogue interaction method according to claim 5, characterized in that, S5 inputs the environmental acoustic features into a pre-constructed scene classification network and parameter estimation network. The scene classification network outputs the scene confidence of the current environment category, and the parameter estimation network calculates the background noise energy value and signal-to-noise ratio estimate. The environmental interference coefficient is obtained by weighting and summing the scene confidence level, background noise energy value, and signal-to-noise ratio estimate.

7. A robot intelligent voice dialogue interaction method according to claim 6, characterized in that, S6 sequentially concatenates the dialogue intent, key entity slots, sentiment category probability distribution values, sentiment intensity scores, and environmental interference coefficients into a multidimensional state array. This multidimensional state array is then mapped to the feature space of a dynamic knowledge graph. Graph nodes and their attributes that are associated with this array are extracted, and contextual path features between associated nodes are obtained. The dialogue intent, key entity slots, sentiment category probability distribution values, sentiment intensity scores, and environmental interference coefficients, along with the extracted node attributes and contextual path features, are then concatenated in multiple dimensions to obtain a long sequence containing all feature information. Subsequently, cross-weight calculations are performed on the features of each dimension in the long sequence, and the calculation results are remapped into a fixed-length multidimensional numerical sequence, which serves as the state fusion vector for the current interaction scenario. The process involves obtaining each dialogue action node from the dynamic knowledge graph and converting it into a corresponding action vector. The similarity between the state fusion vector and each action vector is calculated to obtain the similarity between each dialogue action. Finally, the similarity values ​​are sorted from largest to smallest, and the top N dialogue action nodes are extracted to generate multiple candidate dialogue actions.

8. A robot intelligent voice dialogue interaction method according to claim 7, characterized in that, S7 inputs the state fusion vector and the action vector corresponding to each candidate dialogue action into the deep reinforcement learning network, calculates and outputs the policy matching score of each candidate dialogue action, selects the candidate dialogue action with the highest score as the optimal dialogue action, and extracts the corresponding target response text and target sentiment tag. The target speech rate and target volume are calculated proportionally based on the magnitude of the environmental interference coefficient, and the target tone is calculated by matching the user's emotional tendency state reflected by the probability distribution value of the emotion category. Finally, the target response text, target sentiment tag, target speech rate, target tone, and target volume are combined and packaged to generate a dynamic response strategy.

Images