Method and device for robot dog dangerous scene recognition and interaction based on multi-modal attention fusion

Abstract

The application discloses a kind of based on multimodal attention fusion's robot dog dangerous scene identification and interaction method and device, the method includes: through the sensing system of deployment on intelligent robot dog, the speech signal and gesture image signal of user are collected;Voice signal is converted into text in real time;Lightweight semantic analysis is carried out to the text converted, at least including one structured semantic object and corresponding confidence is obtained, and the confidence corresponding to structured semantic is as first semantic information;Based on gesture image signal, obtain single frame RGB image;Based on single frame RGB image, the 3D coordinates of key point are obtained;Based on the 3D coordinates of key point corresponding to multiple frame RGB image, the probability distribution of each gesture category is obtained and as second semantic information;Based on first semantic information and second semantic information, using semantic enhanced double attention mechanism, the final scene category is identified.The application can make correct dangerous scene identification according to speech signal and gesture image signal.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction and intelligent robot technology, and in particular to a method, device, terminal equipment, computer-readable storage medium and computer program product for dangerous scene recognition and interaction of robot dogs based on multimodal attention fusion. Background Technology

[0002] With the continuous development of robotics technology, intelligent robot dogs (or robots) are increasingly being used in security patrols, disaster relief, and hazardous environment operations. In these complex and potentially dangerous scenarios, achieving natural, efficient, and reliable human-robot interaction (HRI) is crucial to ensuring mission success and personnel safety.

[0003] Currently, most intelligent robot dogs use a single modality for interaction, mainly including the following two categories: (I) Voice-based interaction methods. This method allows users to control the robot dog through natural voice commands, offering the advantages of natural interaction and no need for physical contact. However, its application in real-world dangerous scenarios has significant limitations: First, background noise (such as wind noise, machine noise, and alarm sounds) severely interferes with the clarity of the voice signal, leading to a sharp drop in recognition rate; second, voice commands have inherent ambiguity in conveying precise spatial and directional information, making it difficult to determine solely by voice. (II) Visual gesture-based interaction methods. This method uses a camera to capture the user's hand or body movements for recognition, providing intuitive and accurate spatial pointing information. However, its performance is highly dependent on environmental conditions: drastic changes in lighting, complex background interference, partial occlusion, and non-standardized user gestures can all lead to recognition failure or misrecognition. Existing dynamic gesture recognition technologies mainly suffer from two problems: First, end-to-end deep learning models have complex structures and large parameters, making it difficult to achieve real-time operation on embedded platforms, resulting in excessively high response latency; second, although traditional manual feature methods have advantages in computational efficiency, they lack robustness under complex lighting or background interference.

[0004] Therefore, in order to meet the needs of intelligent robot dogs for adapting to dangerous environments and for accurate semantic matching, there is an urgent need for a multimodal fusion method that is lightweight, dynamic, and semantically consistent, so that it can adapt to environmental changes, make full use of the complementary advantages of voice and gesture modalities, and achieve high-precision and high-robust recognition in dangerous scenarios. Summary of the Invention

[0005] This invention provides a method, device, terminal equipment, computer-readable storage medium, and computer program product for dangerous scene recognition and interaction of robot dogs based on multimodal attention fusion, aiming to solve the problems of low recognition reliability and high response delay in existing technologies in dangerous environments.

[0006] The first objective of this invention is to provide a method for dangerous scene recognition and interaction of robot dogs based on multimodal attention fusion.

[0007] The second objective of this invention is to provide a robot dog dangerous scene recognition and interaction device based on multimodal attention fusion.

[0008] The third objective of this invention is to provide a terminal device.

[0009] A fourth objective of this invention is to provide a computer-readable storage medium.

[0010] The fifth objective of this invention is to provide a computer program product.

[0011] The first objective of this invention can be achieved by adopting the following technical solution:

[0012] A method for dangerous scene recognition and interaction of a robot dog based on multimodal attention fusion, the method comprising:

[0013] The system collects the user's voice signals and gesture image signals through a sensing system deployed on the intelligent robot dog;

[0014] The speech signal is converted into text in real time; the converted text is subjected to lightweight semantic parsing to obtain at least one structured semantic object and its corresponding confidence score, and the confidence score corresponding to the structured semantic object is used as the first semantic information.

[0015] Based on the gesture image signal, a single-frame RGB image is obtained; based on the single-frame RGB image, the 3D coordinates of key points are obtained.

[0016] Based on the 3D coordinates of key points corresponding to multiple frames of RGB images, the probability distribution of each gesture category is obtained and used as the second semantic information;

[0017] Based on the first and second semantic information, a semantically enhanced dual attention mechanism is employed to identify the final scene category, including:

[0018] Based on the first semantic information and the second semantic information, calculate the speech feature vector and gesture confidence.

[0019] By employing confidence gating, the initial confidence weights are generated by sequentially applying linear transformations and Sigmoid function mappings to the first semantic information and the gesture confidence. ;

[0020] By using semantic consistency gating, a matrix M is obtained by performing a dot product operation on the speech feature vector and the second semantic information. Then, global max pooling is applied to matrix M, and semantic relevance is generated by mapping using the Sigmoid function. ;

[0021] Calculate the scene adaptation coefficient based on the first semantic information and gesture confidence. ;

[0022] Calculate fusion weights ;

[0023] Based on fusion weight Obtain the fusion probability vector; based on the fusion probability vector, map the final scene recognition result.

[0024] Preferably, the step of obtaining the probability distribution of each gesture category based on the 3D coordinates of key points corresponding to multiple RGB images includes:

[0025] Based on the 3D coordinates of key points in a single frame of continuously input RGB image, the set of key frames is determined by the Euclidean distance threshold between adjacent frames.

[0026] The coordinates of the hand joints in the set of keyframes are normalized.

[0027] Geometric features are extracted from keyframes based on normalized gesture joint coordinates.

[0028] Spatiotemporal feature extraction is performed on the extracted geometric features;

[0029] The extracted spatiotemporal features are input into a residual fully connected network for gesture classification, and then the probability distribution of each gesture category is output through the Softmax function.

[0030] Preferably, the spatiotemporal feature extraction of the extracted geometric features includes:

[0031] Geometric features are reconstructed into tensors of a set dimension, padding with zeros if necessary. Then, a lightweight (2+1)D convolutional structure is used for spatiotemporal feature extraction, including:

[0032] In the spatial convolution stage, use Convolutional kernels extract spatial correlation patterns of features within a single frame:

[0033] ;

[0034] in, The spatial convolution weight matrix, For the extracted geometric features, For bias terms, The spatial characteristics of the output;

[0035] In the temporal convolution stage, the following is adopted: The size of the convolutional kernel captures dynamic changes between frames:

[0036] ;

[0037] in, The time convolution weight matrix is... For bias terms, This is the output spatiotemporal fusion feature vector.

[0038] Preferably, the residual fully connected network includes three residual blocks, wherein the first residual block maps the extracted spatiotemporal features to the hidden space through a fully connected layer, and then reduces the dimensionality after processing by the PReLU function to obtain the spatiotemporal fusion feature vector. The second residual block will fuse the spatiotemporal feature vectors. First expand, then compress to obtain the spatiotemporal fusion feature vector. The third residual block will fuse the spatiotemporal feature vectors. Mapping to an L-dimensional classification space yields a spatiotemporal fusion feature vector. ; L represents the type of gesture to be output.

[0039] Preferably, the geometric features include joint rotation features and fingertip distance features;

[0040] The normalized gesture joint coordinates are used to extract geometric features from keyframes, including:

[0041] Extracting joint rotation features: Joint rotation features are characterized by the real part of the quaternion rotation angle to represent the local bending degree of the joint.

[0042] , ;

[0043] in, Features of joint rotation; This refers to the joint rotation angle; and They are respectively , The corresponding length-normalized bone vector; and All are skeletal vectors, derived from the hand joints in the set of keyframes. relative coordinates Calculated;

[0044] Extracting fingertip distance features: , These represent the distance between adjacent fingertips and the distance between each fingertips relative to the root node of the wrist, respectively.

[0045] The extracted geometric features are obtained by concatenating the joint rotation features and the fingertip distance features.

[0046] Preferably, obtaining the 3D coordinates of key points based on a single-frame RGB image includes:

[0047] The single-frame RGB image is sequentially compressed and layer normalized.

[0048] Multi-level feature extraction is performed on the feature map after layer normalization. The extracted features are then input into the two-dimensional coordinate branch and the relative depth branch, respectively. The two-dimensional coordinate branch generates a heatmap through 3×3 convolution, and then outputs normalized coordinates through the differentiable maximum index. The relative depth branch sequentially passes through small convolutional kernels and global spatial average pooling to obtain the relative depth of each keypoint.

[0049] Based on normalized coordinates The 3D coordinates of the keypoints are obtained by calculating their relative depth.

[0050] Preferably, for the feature map after layer normalization, a multi-level feature extraction module is used to perform multi-level feature extraction. The multi-level feature extraction module includes three sets of downsampling modules and three sets of upsampling modules. The three sets of downsampling modules are connected in series, and the three sets of upsampling modules are connected in series. Each set of downsampling modules consists of a downsampling module and two series-connected feature modules, and each set of upsampling modules consists of two series-connected feature modules and an upsampling module. Both the downsampling modules and the upsampling modules are implemented using basic convolutional units. The three sets of downsampling modules and the three sets of upsampling modules are fused through residual connections, and the feature map output by the third set of upsampling modules is used as the extracted features.

[0051] The three sets of downsampling modules and the three sets of upsampling modules are connected via residual connections for feature fusion, specifically:

[0052] Connect the first set of downsampling modules to the third set of upsampling modules, connect the second set of downsampling modules to the second set of upsampling modules, and connect the third set of downsampling modules to the first set of upsampling modules.

[0053] Preferably, the real-time conversion of speech signals into text includes:

[0054] Preliminary filtering of ambient noise or other speech signals from the speech signal;

[0055] The initially filtered speech signal is purified and enhanced;

[0056] High-dimensional acoustic feature vector sequences are extracted from the enhanced speech signal; these sequences are then input into an ASR (Automatic Speech Recognition) model to convert the speech stream into text in real time.

[0057] The second objective of this invention can be achieved by adopting the following technical solution:

[0058] A robotic dog dangerous scene recognition and interaction device based on multimodal attention fusion, the device comprising:

[0059] The acquisition module is used to acquire the user's voice signals and gesture image signals through the sensing system deployed on the intelligent robot dog;

[0060] The conversion and semantic parsing module is used to convert speech signals into text in real time; it performs lightweight semantic parsing on the converted text to obtain at least one structured semantic object and its corresponding confidence score, and uses the confidence score corresponding to the structured semantic object as the first semantic information.

[0061] The feature extraction module is used to obtain a single-frame RGB image based on the gesture image signal; and to obtain the 3D coordinates of key points based on the single-frame RGB image.

[0062] The dynamic feature modeling module is used to obtain the probability distribution of each gesture category based on the 3D coordinates of key points corresponding to multiple frames of RGB images and use it as the second semantic information.

[0063] The scene category recognition module, based on first and second semantic information, employs a semantically enhanced dual attention mechanism to identify the final scene category, including:

[0064] Based on the first semantic information and the second semantic information, calculate the speech feature vector and gesture confidence.

[0065] By employing confidence gating, the initial confidence weights are generated by sequentially applying linear transformations and Sigmoid function mappings to the first semantic information and the gesture confidence. ;

[0066] By using semantic consistency gating, a matrix M is obtained by performing a dot product operation on the speech feature vector and the second semantic information. Then, global max pooling is applied to matrix M, and semantic relevance is generated by mapping using the Sigmoid function. ;

[0067] Calculate the scene adaptation coefficient based on the first semantic information and gesture confidence. ;

[0068] Calculate fusion weights ;

[0069] Based on fusion weight Obtain the fusion probability vector; based on the fusion probability vector, map the final scene recognition result.

[0070] The third objective of this invention can be achieved by adopting the following technical solution:

[0071] A terminal device includes a processor and a memory for storing processor-executable programs. When the processor executes the program stored in the memory, it implements the above-described method for identifying and interacting with dangerous scenes in a robot dog based on multimodal attention fusion.

[0072] The fourth objective of this invention can be achieved by adopting the following technical solution:

[0073] A computer-readable storage medium storing a program that, when executed by a processor, implements the above-described method for dangerous scene recognition and interaction in a robot dog based on multimodal attention fusion.

[0074] The fifth objective of this invention can be achieved by adopting the following technical solution:

[0075] A computer program product includes a computer program that, when executed by a processor, implements the aforementioned method for identifying and interacting with dangerous scenes in a robot dog based on multimodal attention fusion.

[0076] The present invention has the following advantages over the prior art:

[0077] This invention employs a semantically enhanced dual attention mechanism. Through a dual-weight generation logic of confidence gating and semantic consistency gating, it leverages both speech confidence and the maximum probability of gestures to ensure modal reliability, while capturing the inherent semantic connections through a dynamic correlation matrix of speech feature vectors and gesture probability distributions, effectively avoiding single-modal misjudgments. Simultaneously, by introducing a scene adaptation coefficient, it supports flexible adjustment of the dual attention weights according to the scene and smooth transitions during modal conflicts, significantly improving the naturalness of interaction and system adaptability. This invention is particularly suitable for real-time human-computer interaction in dangerous scenarios for intelligent robot dogs, combining high accuracy, low latency, and strong anti-interference capabilities. Attached Figure Description

[0078] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0079] Figure 1 This is a flowchart of the dangerous scene recognition and interaction method for robot dogs based on multimodal attention fusion according to Embodiment 1 of the present invention;

[0080] Figure 2 This is a schematic diagram illustrating the process of preprocessing the acquired speech signal and training and real-time recognition of the end-to-end streaming speech recognition model in Embodiment 1 of the present invention.

[0081] Figure 3 This is a schematic diagram of the semantic parsing and speech confidence score generation process in Embodiment 1 of the present invention;

[0082] Figure 4 This is a schematic diagram of the gesture estimation subnet of Embodiment 1 of the present invention;

[0083] Figure 5 This is a schematic diagram of the gesture recognition network in Embodiment 1 of the present invention;

[0084] Figure 6 This is an architecture diagram of the semantically enhanced dual-attention gating multimodal fusion module of Embodiment 1 of the present invention;

[0085] Figure 7 This is a structural block diagram of the robot dog dangerous scene recognition and interaction device based on multimodal attention fusion according to Embodiment 2 of the present invention;

[0086] Figure 8 This is a structural block diagram of the terminal device according to Embodiment 3 of the present invention. Detailed Implementation

[0087] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be understood that the specific embodiments described are merely used to explain this application and are not intended to limit this application.

[0088] This embodiment provides a method for dangerous scene recognition and interaction in robot dogs based on multimodal attention fusion. Through speech recognition, gesture recognition, and multimodal attention allocation technology, it enables intelligent robot dogs to perceive user voice and visual commands, understand scene context, and make correct dangerous scene recognition interactions. (Refer to...) Figure 1 This includes the following steps:

[0089] S101. Collect user's voice signals and gesture image signals through a sensing system deployed on the intelligent robot dog; preprocess and extract features from the voice signals, and then convert the voice stream into text in real time through an ASR speech recognition model;

[0090] S102. Perform lightweight semantic parsing on the original text to obtain at least one structured semantic object and its corresponding confidence level; use the confidence level corresponding to the structured semantic object as the first semantic information.

[0091] S103. Based on the gesture image signal, obtain a single-frame RGB image; based on the single-frame RGB image, obtain the 3D coordinates of the key points;

[0092] S104. Based on the 3D coordinates of key points in multiple RGB images, obtain the probability distribution of each gesture category and use it as the second semantic information;

[0093] S105. Based on the first semantic information and the second semantic information, a semantically enhanced dual attention mechanism is adopted to identify the final scene category.

[0094] Further, refer to Figure 2 Step S101 includes:

[0095] (1) Collect the voice signal emitted by the user through the sensing system deployed on the intelligent robot dog; preprocess the voice signal.

[0096] (1-1) Collect the voice signal sent by the user and perform preliminary filtering of the voice signal.

[0097] When the user issues a voice command, the ring microphone array on the robot dog's head starts working to collect multi-channel synchronous audio.

[0098] Specifically, the ring microphone array has a sound source localization function and uses beamforming to enhance the user's sound source in the main direction, thereby initially filtering out irrelevant environmental noise or other people's voices.

[0099] (1-2) The preliminarily filtered speech signal is purified and enhanced.

[0100] Purification and enhancement include:

[0101] Endpoint detection involves deploying the SiloVAD module on the server side to analyze the incoming audio stream in real time and accurately distinguish between human speech, silence, and background noise. Based on a deep learning model, this module can precisely handle different volumes, speech rates, and environmental noise, accurately marking the start and end points of each speech segment to remove silent sections.

[0102] Noise reduction and echo cancellation employ adaptive filtering to suppress steady-state noise and echo cancellation to remove sound from the robot's own speaker, thus suppressing environmental noise and eliminating echo interference generated by the robot's own motor movement. Effective suppression of steady-state noise is achieved through an adaptive filtering algorithm: noise samples are collected using a reference microphone, and the filter coefficients are updated in real time using an adaptive filter to make its transfer function approximate the characteristics of the noise channel, thereby canceling noise components from the main microphone signal. This is particularly suitable for periodic steady-state noise generated by motor operation, etc.

[0103] For echo cancellation, an adaptive filtering-based echo canceller is employed: using the device's speaker output signal as a reference input, an adaptive algorithm simulates the acoustic echo path to generate an echo estimate, which is then subtracted in real-time from the mixed signal acquired by the main microphone, thereby eliminating echo interference generated by the playback sound. The output is a clear and clean speech signal.

[0104] (2) Extract the high-dimensional acoustic feature vector sequence from the enhanced speech signal; input the high-dimensional acoustic feature vector sequence into the speech recognition ASR model to convert the speech stream into text in real time.

[0105] An acoustic feature extraction algorithm is used to extract a high-dimensional acoustic feature vector sequence from effective audio segments, namely, extracting Mel frequency cepstral coefficients and filter bank features to obtain a representation that is more consistent with the characteristics of human hearing. The high-dimensional acoustic feature vector sequence is then input into a deep learning-based end-to-end automatic speech recognition model (a specially trained ASR model) to output the original text sequence.

[0106] The ASR model converts speech streams into text in real time. To meet the stringent requirements of real-time interaction, the adopted ASR model supports streaming recognition mode, which means that while receiving the audio stream, it outputs intermediate recognition results in real time, and continuously corrects and optimizes the final text as more speech information is input, thereby providing users with an instant feedback experience.

[0107] Specifically, the automatic speech recognition model adopts an end-to-end deep learning model architecture that integrates an acoustic model, a language model, and a dictionary. The acoustic model is responsible for analyzing the acoustic features of the audio and determining the phonemes or sub-words corresponding to each time frame. The language model is trained on a large-scale domain text corpus and is responsible for predicting the most probable word sequence based on the contextual semantics, ensuring that the converted text conforms to grammatical rules and scene semantic logic.

[0108] The training process for an end-to-end automatic speech recognition model includes the following key stages:

[0109] (2-1) Preparation and preprocessing of training data: Obtain a large-scale speech-text pairing dataset, with particular emphasis on introducing noise samples and domain-specific instruction data for dangerous scenarios.

[0110] (2-2) Standardize and preprocess the training data: standardize the format and normalize the volume of the speech data; then enhance the data by adding environmental noise and simulated reverberation to improve the robustness of the model; and extract the Mel frequency cepstral coefficients and the acoustic features of the filter bank.

[0111] (2-3) Normalize the text labels corresponding to the speech data: including regularizing the text and constructing a dictionary set for specific dangerous scenarios.

[0112] (2-4) Constructing an end-to-end automatic speech recognition model: Based on the Transformer encoder-decoder structure as the main body of the model, the model architecture deeply integrates the acoustic model, pronunciation dictionary and language model in the traditional speech recognition process into a unified neural network, and directly realizes the mapping from acoustic feature sequence to text sequence through a single model.

[0113] (2-5) An end-to-end joint optimization strategy is adopted: the preprocessed acoustic feature sequence is used as input, and the corresponding text sequence is used as the target label to train the deep learning model based on the Transformer architecture in an end-to-end manner. This process uses the backpropagation algorithm and the gradient descent optimization algorithm to optimize parameters by minimizing the difference loss between the model output and the true label.

[0114] (2-6) Fine-tuning the model: Through domain adaptive fine-tuning technology, the model is fine-tuned using a professional dataset rich in keywords of dangerous interaction scenarios, which significantly improves its recognition accuracy and response speed of key instructions.

[0115] (2-7) Evaluate the performance of the trained model: Use an independent validation dataset to evaluate the performance of the trained model. Key metrics include word error rate, real-time rate, and recall rate for key instructions.

[0116] After export, the model combines high accuracy, strong anti-interference ability and excellent streaming recognition performance, which can fully meet the real-time interaction needs of intelligent robot dogs in complex environments.

[0117] Further, refer to Figure 3 Step S102 includes:

[0118] Lightweight semantic parsing is performed on the original text identified in step S101, and it is formatted into standardized structured data containing intent and key parameters.

[0119] Lightweight semantic parsing, based on predefined grammar rules and a keyword dictionary, uses pattern matching and finite state machine techniques to quickly classify user commands by intent and extract parameters. Its output includes at least a structured semantic object and a confidence score corresponding to that semantic object. .

[0120] (1) Structured semantic objects.

[0121] This object is a standardized machine-readable data format used to accurately represent the semantic core of user instructions. Its structure is shown below, encapsulating at least two levels of information:

[0122]

[0123] Intent: An abstract label mapped from a predefined intent library, used to identify the fundamental purpose or type of operation of a user instruction.

[0124] The key parameters are a set of parameters associated with the intent, used to specifically define the execution object, attributes, and target value of the operation. This set consists of zero or more parameter items, each of which is a key-value pair. The "key" is a unique identifier for the parameter type, and the "value" is a specific numerical value or string extracted from the original text and normalized.

[0125] (2) Confidence level corresponding to structured semantic objects .

[0126] The confidence generation mechanism completely avoids the computational complexity and consistency risks brought about by introducing an independent semantic understanding model. Instead, it adopts a source fusion method to directly fuse and calculate the underlying output probabilities of the core model in the upstream automatic speech recognition (ASR) process.

[0127] Specifically, The calculation is based on the word sequence W=(w1,w2,...,w) that constitutes the original text. n ), where n∈N, is the number of lexical units in the lexical sequence W, and its generation process includes the following steps:

[0128] (2-1) Obtaining probabilistic information from the original text: For each word unit wᵢ in the sequence W, where i∈N and i≤n, the following two core probability indicators are obtained simultaneously:

[0129] Acoustic score (A(wᵢ)): Derived from the acoustic model, it represents the log-likelihood probability of the word wᵢ occurring given the input audio frame sequence, reflecting the matching confidence at the acoustic level.

[0130] Language model probability (P(wᵢ|Context)): Derived from the language model, it represents the prior probability of the word wᵢ appearing given the historical word context Context, reflecting the confidence level of language fluency and grammatical rationality.

[0131] (2-2) Calculate the confidence score RawScore based on the core probability index.

[0132] The serialization probability information obtained above is used to calculate a global confidence score through a predefined fusion function F(·). The fusion function F(·) is implemented using the joint likelihood method, which calculates the normalized joint log-likelihood value based on the overall generation probability of sequence W.

[0133]

[0134] (2-2) The confidence score RawScore is mapped by the Sigmoid function to obtain the standardized confidence score.

[0135] The original values ​​obtained from the above fusion calculation are mapped to a fixed interval ([0,1]) using the Sigmoid function to form the final, standardized confidence score.

[0136]

[0137] Among them, parameters (scaling factor) and The offset factor is used to perform a linear transformation on the original scores to adapt to the output distribution characteristics of the ASR model and to fine-tune the confidence level.

[0138] Specifically, and The value can be determined through empirical assignment, optimization based on the development set, or online adaptive methods.

[0139] Further, step S103 includes:

[0140] User gesture image signals are acquired to obtain a single-frame RGB image; this single-frame RGB image is then input into the trained gesture estimation subnetwork, which outputs the 3D coordinates of hand key points. (Reference) Figure 4 The gesture estimation subnet includes an input compression unit, a layer normalization processing layer, a multi-level feature extraction module, and an output unit.

[0141] (1) The single-frame RGB image is processed sequentially through a compression unit and a layer normalization processing layer.

[0142] The preprocessed single-frame RGB image is input to the compression unit, where a 4×4 convolutional kernel with a stride of 4 is used to compress the image resolution. The output feature map is... The size is 1 / 4 of the input, followed by a normalization operation:

[0143]

[0144] in, The feature map has undergone layer normalization. and To output the mean and variance of the feature map, and For learnable parameters, It is a tiny constant used to avoid a denominator of 0.

[0145] Specifically, Usually taken arrive .

[0146] (2) The normalized feature map is processed by a multi-level feature extraction module and an output unit to obtain the relative depth of each key point.

[0147] This step is mainly implemented through a multi-level feature extraction module and an output unit, where:

[0148] The multi-level feature extraction module comprises three groups of downsampling modules and three groups of upsampling modules. Each downsampling module group consists of a downsampling module and two cascaded feature modules, while each upsampling module group consists of two cascaded feature modules and an upsampling module. Both downsampling and upsampling modules are implemented using basic convolutional units, and the feature modules have the same structure as the basic convolutional units. Feature fusion between the downsampling and upsampling stages is achieved through residual connections, outputting the extracted feature result.

[0149] After obtaining the initial feature map, the hand's features at different levels are learned through a multi-level feature extraction module, which can be divided into two stages:

[0150] Downsampling stage: This stage comprises three groups of downsampling modules, each consisting of one downsampling module and two feature modules connected in series. With each group, the feature map size is halved, but the number of channels increases, thus extracting higher-level information.

[0151] Upsampling stage: This stage includes three groups of upsampling modules, each consisting of two feature modules and one upsampling module connected in series. After each group, the feature map size doubles, gradually restoring the spatial resolution.

[0152] During the downsampling stage, the feature maps output by modules 1, 2, and 3 are passed to modules 3, 2, and 1 in the upsampling stage via residual connections (correspondence: downsampling module 1 connects to upsampling module 3, module 2 connects to module 2, and module 3 connects to module 1). In the upsampling stage, while recovering details, the semantic information preserved in the downsampling stage is also referenced to more accurately locate key hand points. After passing through all upsampling module groups, the feature map output by the third upsampling module group is used as the final extraction result.

[0153] The extracted feature map is input into the output unit, which includes two parallel branches: a two-dimensional coordinate branch and a relative depth branch. The two-dimensional coordinate branch generates a heatmap through a 3×3 convolution and outputs normalized coordinates after passing through a differentiable maximum index (Soft-Argmax). The relative depth branch sequentially uses small convolutional kernels and global spatial average pooling (mean) to obtain the relative depth of each keypoint. .

[0154] (3) Based on the relative depth, the 3D coordinates of the final 21 key points are obtained.

[0155] Combined with the hand reference length obtained through scene priors Calculate absolute depth :

[0156]

[0157] in, The absolute depth of the root critical point is obtained by solving the following equation:

[0158]

[0159] Where K is the camera intrinsic parameter matrix, used for the conversion between 2D coordinates and 3D information; the normalized coordinates output by the 2D coordinate branch. The two-dimensional coordinates of the key points at the base of the hand serve as the fundamental reference points for the hand in the image; the normalized coordinates output by the two-dimensional coordinate branch. The two-dimensional coordinates of the palmar node of the middle finger are used in conjunction with the root point to define the two ends of the reference length of the hand; It is the normalized relative depth of the middle finger metacarpophalangeal joint, which helps to describe the three-dimensional positional relationship of the middle finger metacarpophalangeal joint.

[0160] The final output of the 3D coordinates of the 21 key points is as follows:

[0161]

[0162] In this embodiment, the 21 key points include one wrist root node and four key points for each of the five fingers: metacarpophalangeal joint (MCP), proximal interphalangeal joint (PIP), distal interphalangeal joint (DIP), and fingertip (TIP).

[0163] This embodiment trains the gesture estimation subnetwork, including:

[0164] During the training phase, random scaling and padding are performed on the participating images to complete image preprocessing, data augmentation is performed, and annotation preprocessing is performed to make the 2D annotations of key points change synchronously with the image rotation, and equation (2) is used to convert the absolute depth annotations into normalized relative depths.

[0165] The Smooth-L1 loss is applied to the two-dimensional coordinates and relative depth. The Smooth-L1 loss is defined as follows:

[0166]

[0167] in, The difference between the prediction and the annotation is used for the two regression heads: coordinates and depth.

[0168] Smooth-L1 loss is used for backpropagation to update network parameters, ensuring the accuracy of coordinate and depth regression.

[0169] The SGD optimizer is used to iteratively train the input image in batches, updating the network parameters until the loss function value no longer decreases, thus completing the training of the gesture estimation subnetwork.

[0170] Further, step S104 includes:

[0171] Based on the 3D coordinates of key points in multiple RGB images, a gesture recognition network is used to output the probability distribution of each gesture category, specifically including:

[0172] (1) Based on the 3D coordinates of key points of a single frame of RGB image with continuous input, the set of key frames is determined by the Euclidean distance threshold between adjacent frames.

[0173] Given a continuous sequence of T frames of keypoints (T>6), the Euclidean distance threshold between adjacent frames is as follows:

[0174]

[0175] in, It is the first The key point is in the first Frame coordinates, abbreviated as ; It is the first The key point is in the first Frame coordinates, abbreviated as The above formula can be written as:

[0176]

[0177] As an empirical threshold, define an action segment that satisfies The maximum continuous interval of time ,Will The corresponding frame is determined to be the starting frame of the action. The corresponding frame is determined to be the end frame of the action.

[0178] In one embodiment, in the interval inside, take The largest value A key moment, serving as an index set for keyframes.

[0179] (2) Normalize the coordinates of the hand joints in the set of keyframes.

[0180] As the root node of the wrist For the hand joints in the set of keyframes, perform relative coordinate transformation to eliminate spatial position differences:

[0181]

[0182] in, for The relative coordinates.

[0183] Normalize bone length to eliminate size differences, define and For skeletal vectors, and Depend on It is calculated; among them, The parent bone vector points from the root node of the wrist to the fingertip. Let be the sub-bone vector, with its direction pointing from the fingertip node to the wrist root node; its length is standardized as follows:

[0184]

[0185] in, and They are respectively , The corresponding length-normalized bone vector, and This is the standard finger joint length; you can refer to the ACT hand joint segment to determine the length.

[0186] Normalize the coordinates of the hand nodes. This represents the number of bones after normalization of bone length. Take the coordinates of each joint point. The remaining joints are summed along the bone vector according to the normalized joint length to obtain the remaining joints. .

[0187] (3) Extract geometric features from keyframes based on normalized gesture joint coordinates.

[0188] This mainly includes the extraction of joint rotation features and fingertip distance features.

[0189] (3-1) Extracting joint rotation features:

[0190] Joint rotation characteristics are represented by the real part of the quaternion rotation angle to characterize the local bending degree of the joint:

[0191] ,

[0192] The joint rotation angle, This corresponds to the 15 hand joints excluding the 5 fingertip nodes and the root node. For convenience, the joint rotation feature is... Represented as .

[0193] (3-2) Extracting fingertip distance features:

[0194] use Characterizing the normalized fingertip (TIP) joints, , These represent the distance between adjacent fingertips and the distance of each fingertips relative to the wrist root node, respectively. The characteristics of these two fingertip distances are as follows:

[0195]

[0196] (3-3) Concatenate the joint rotation features and fingertip distance features into a 24-dimensional feature vector. :

[0197]

[0198] (4) The extracted geometric features are extracted using a lightweight (2+1)D convolutional structure for spatiotemporal feature extraction.

[0199] Reconstruct the 6-frame 24-dimensional geometric feature vector sequence as follows: The tensor is padded with zeros if the dimension is insufficient, and then a lightweight (2+1)D convolutional structure is used for spatiotemporal feature extraction.

[0200] Spatial convolution stage uses Convolutional kernels extract spatial correlation patterns of features within a single frame:

[0201]

[0202] in, The spatial convolution weight matrix, As the bias term, output 64-channel spatial features. .

[0203] In the temporal convolution stage, the following is adopted: The size of the convolutional kernel captures dynamic changes between frames:

[0204]

[0205] in, The time convolution weight matrix is... As a bias term, the output is a 64-dimensional spatiotemporal fusion feature vector. .

[0206] The number of parameters in this (2+1)D convolution structure is greatly reduced compared to traditional 3D convolution.

[0207] (5) Input the extracted spatiotemporal features into a three-layer residual fully connected network for gesture classification.

[0208] The first residual block will input 64 dimensions. The hidden space is mapped to a 2000-dimensional space through a fully connected layer, and then reduced to 800 dimensions after processing with the PReLU activation function.

[0209]

[0210] in, The output is an 800-dimensional spatiotemporal fusion feature vector. for A fully connected layer maps 64 dimensions to an 800-dimensional feature vector. , The weight matrix is ​​used; the PReLU function is defined as:

[0211]

[0212] The second residual block first expands the 800-dimensional features to 1500 dimensions, and then compresses them to 256 dimensions:

[0213]

[0214] in, The output is a 256-dimensional spatiotemporal fusion feature vector. , This is the weight matrix.

[0215] Assuming the final output needs to be a classification of 6 gestures, the third residual block will ultimately map the 256-dimensional features to a 6-dimensional classification space:

[0216]

[0217] in, The output is a 6-dimensional spatiotemporal fusion feature vector. , This is the weight matrix.

[0218] Finally, the probability distribution of each gesture category is output through the Softmax function:

[0219]

[0220] in, express The c-th element, where c is the index of the actual gesture category; express The k-th element in the array, where k ranges from 1 to 6, traversing all gesture categories.

[0221] This embodiment uses the cross-entropy loss function to perform end-to-end optimization of the gesture recognition network:

[0222]

[0223] in, This is a one-hot real tag.

[0224] In this embodiment, the gesture estimation subnet alleviates the gradient vanishing problem through residual connections, and the deep dimensionality reduction design significantly reduces the number of parameters, meeting the requirements of embedded deployment.

[0225] The structure of the gesture recognition network can be referenced from Figure 5 .

[0226] Further, step S105 includes:

[0227] The first and second semantic information are input into a semantically enhanced dual-attention gating multimodal fusion module to obtain a fusion probability vector, which is then used to identify the final scene category. (Reference) Figure 6 The semantically enhanced dual-attention gating multimodal fusion module includes a modality feature processing unit, a dual-attention gating unit, and a fusion decision unit.

[0228] (1) Input the first semantic information and the second semantic information into the modal feature processing unit to calculate the speech feature vector and gesture confidence.

[0229] (1-1) Based on the first semantic information, generate a speech feature vector with reliability weights. .

[0230] The text commands output from speech recognition are input into the pre-trained lightweight language model Sentence-BERT (SBERT), which outputs a semantic vector of dimension d. This vector can capture deep semantic information from text instructions. (The vector is then used to...) With voice confidence Perform element-wise multiplication to generate a speech feature vector with reliability weights:

[0231]

[0232] (1-2) Based on the second semantic information, the gesture confidence score is obtained. .

[0233] definition (k is the number of preset gesture categories for the home scene) is the Softmax probability vector output by the gesture module (22), where each element in the vector represents the probability of each corresponding gesture. The gesture probability vector is directly... As a gesture feature vector ,Right now:

[0234]

[0235] Define gesture confidence , which is the maximum value in the gesture probability vector, characterizes the reliability of gesture recognition.

[0236] (2) Input the first semantic information, gesture confidence, second semantic information and speech feature vector into the dual attention gating unit to obtain the initial confidence weight. and semantic relevance .

[0237] The dual attention gating unit includes confidence gating and semantic consistency gating.

[0238] (2-1) Input the first semantic information and gesture confidence into the confidence gating to generate the initial confidence weights. .

[0239] Based on speech confidence With gesture confidence Initial confidence weights are generated through linear transformation and the Sigmoid activation function. The formula is as follows:

[0240]

[0241] in, For learnable weight matrix, The bias term was obtained through training with a large number of labeled samples in dangerous scenarios. This is a concatenated vector of speech confidence and gesture confidence; the Sigmoid function will... Mapped to the [0,1] interval. A larger value indicates a higher reliability for the speech modality, while a smaller value indicates a more reliable gesture modality.

[0242] (2-2) Input the speech feature vector and the second semantic information into the confidence gating and calculate the semantic relevance. .

[0243] speech feature vectors With gesture feature vector The transpose of is used to perform a dot product operation to generate a d×k-dimensional dynamic correlation matrix M, as shown in the following formula:

[0244]

[0245] Matrix elements This represents the semantic correlation between the i-th dimension feature of speech and the j-th type of gesture.

[0246] Global max pooling is performed on matrix M to extract the most significant semantic association features, which are then activated by the Sigmoid function to generate... The formula is as follows:

[0247]

[0248] (3) Calculate the scene adaptation coefficient based on the first semantic information and gesture confidence. .

[0249] Introducing scene adaptation coefficient This is used to control the relative contributions of the first and second semantic information in the final fusion. The system constructs a training sample set by collecting recognition results and confidence scores of speech and gesture images in different scenarios, and trains the model using a sigmoid regression model. Two model parameters. Based on training. Combining real-time voice and gesture confidence, the following calculation formula is used to obtain... :

[0250]

[0251] (4) Combine the scene adaptation coefficient and the initial confidence weight. and semantic relevance Input the fusion decision unit and calculate the fusion weights. .

[0252] Obtain the fusion weights :

[0253]

[0254] (5) Based on fusion weight The fusion probability vector is obtained and mapped to the final scene recognition result.

[0255] (5-1) Transform the speech feature vector As the probability distribution vector of speech ; through dynamic fusion weights right and We perform weighted fusion to obtain the fusion probability vector:

[0256]

[0257] (5-2) Based on the fusion probability vector, map the final scene recognition result.

[0258] Establish a scene category mapping table to clarify the correspondence between each dimension of the fusion probability vector and the home scene category, and calculate the fusion probability vector. Maximum value and corresponding index:

[0259]

[0260] in, It is a mathematical function that returns the independent variable (parameter) that maximizes the objective function, selecting the most likely class from the probability distribution output by the model.

[0261] Based on the scene category mapping table, the corresponding scene categories are... This serves as the final scene recognition result.

[0262] The semantically enhanced dual-attention gating multimodal fusion module in this embodiment achieves efficient and accurate fusion of voice and gesture data through a three-level architecture of "modal feature processing - dual attention weight generation - dynamic fusion decision", and identifies accurate dangerous scenarios.

[0263] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware, and the corresponding program can be stored in a computer-readable storage medium.

[0264] It should be noted that although the method operations of the above embodiments are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the order of execution of the described steps may be changed. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0265] Example 2:

[0266] like Figure 7 As shown, this embodiment provides a robot dog dangerous scene recognition and interaction device based on multimodal attention fusion. The device includes a data acquisition module 701, a conversion and semantic parsing module 702, a feature extraction module 703, a dynamic feature modeling module 704, and a scene category recognition module 705, wherein:

[0267] The acquisition module 701 is used to acquire the user's voice signals and gesture image signals through the sensing system deployed on the intelligent robot dog;

[0268] The conversion and semantic parsing module 702 is used to convert speech signals into text in real time; perform lightweight semantic parsing on the converted text to obtain at least one structured semantic object and its corresponding confidence level, and use the confidence level corresponding to the structured semantic object as the first semantic information;

[0269] The feature extraction module 703 is used to obtain a single-frame RGB image based on the gesture image signal; and to obtain the 3D coordinates of key points based on the single-frame RGB image.

[0270] The dynamic feature modeling module 704 is used to obtain the probability distribution of each gesture category based on the 3D coordinates of key points corresponding to multiple frames of RGB images and use it as the second semantic information.

[0271] The scene category recognition module 705 is used to identify the final scene category based on the first semantic information and the second semantic information, using a semantically enhanced dual attention mechanism, including:

[0272] Based on the first semantic information and the second semantic information, calculate the speech feature vector and gesture confidence.

[0273] By employing confidence gating, the initial confidence weights are generated by sequentially applying linear transformations and Sigmoid function mappings to the first semantic information and the gesture confidence. ;

[0274] By using semantic consistency gating, a matrix M is obtained by performing a dot product operation on the speech feature vector and the second semantic information. Then, global max pooling is applied to matrix M, and semantic relevance is generated by mapping using the Sigmoid function. ;

[0275] Calculate the scene adaptation coefficient based on the first semantic information and gesture confidence. ;

[0276] Calculate fusion weights ;

[0277] Based on fusion weight Obtain the fusion probability vector; based on the fusion probability vector, map the final scene recognition result.

[0278] The specific implementation of each module in this embodiment can be found in Embodiment 1 above, and will not be repeated here. It should be noted that the device provided in this embodiment is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above.

[0279] Example 3:

[0280] This embodiment provides a terminal device, which can be a computer, such as... Figure 8 As shown, the processor 802, memory, input device 803, display 804, and network interface 805 are connected via system bus 801. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium 806 and an internal memory 807. The non-volatile storage medium 806 stores the operating system, computer programs, and database. The internal memory 807 provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. When the processor 802 executes the computer programs stored in the memory, it implements the multimodal attention fusion-based dangerous scene recognition and interaction method for robot dogs in Embodiment 1 described above.

[0281] Example 4:

[0282] This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the method for dangerous scene recognition and interaction of a robot dog based on multimodal attention fusion as described in Embodiment 1 above.

[0283] It should be noted that the computer-readable storage medium in this embodiment can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0284] Example 5:

[0285] This embodiment provides a computer program product, including a computer program that, when executed by a processor, implements the multimodal attention fusion-based method for dangerous scene recognition and interaction of a robot dog in Embodiment 1 described above.

[0286] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope disclosed in the present invention, based on the technical solution and inventive concept of the present invention, shall fall within the scope of protection of the present invention.

Claims

1. A method for dangerous scene recognition and interaction in robotic dogs based on multimodal attention fusion, characterized in that, The method includes: The system collects the user's voice signals and gesture image signals through a sensing system deployed on the intelligent robot dog; The speech signal is converted into text in real time; the converted text is subjected to lightweight semantic parsing to obtain at least one structured semantic object and its corresponding confidence score, and the confidence score corresponding to the structured semantic object is used as the first semantic information. Based on the gesture image signal, a single-frame RGB image is obtained; based on the single-frame RGB image, the 3D coordinates of key points are obtained. Based on the 3D coordinates of key points corresponding to multiple frames of RGB images, the probability distribution of each gesture category is obtained and used as the second semantic information; Based on the first and second semantic information, a semantically enhanced dual attention mechanism is employed to identify the final scene category, including: Based on the first semantic information and the second semantic information, calculate the speech feature vector and gesture confidence. By employing confidence gating, the initial confidence weights are generated by sequentially applying linear transformations and Sigmoid function mappings to the first semantic information and the gesture confidence. ; By using semantic consistency gating, a matrix M is obtained by performing a dot product operation on the speech feature vector and the second semantic information. Then, global max pooling is applied to matrix M, and semantic relevance is generated by mapping using the Sigmoid function. ; Calculate the scene adaptation coefficient based on the first semantic information and gesture confidence. ; Calculate fusion weights ; Based on fusion weight Obtain the fusion probability vector; based on the fusion probability vector, map the final scene recognition result.

2. The method for identifying and interacting with dangerous scenarios using a robotic dog according to claim 1, characterized in that, The probability distribution of each gesture category is obtained based on the 3D coordinates of key points corresponding to multiple RGB images, including: Based on the 3D coordinates of key points in a single frame of continuously input RGB image, the set of key frames is determined by the Euclidean distance threshold between adjacent frames. The coordinates of the hand joints in the set of keyframes are normalized. Geometric features are extracted from keyframes based on normalized gesture joint coordinates. Spatiotemporal feature extraction is performed on the extracted geometric features; The extracted spatiotemporal features are input into a residual fully connected network for gesture classification, and then the probability distribution of each gesture category is output through the Softmax function.

3. The method for dangerous scene recognition and interaction of a robot dog according to claim 2, characterized in that, The extraction of spatiotemporal features from the extracted geometric features includes: Geometric features are reconstructed into tensors of a set dimension, padding with zeros if necessary. Then, a lightweight (2+1)D convolutional structure is used for spatiotemporal feature extraction, including: In the spatial convolution stage, use Convolutional kernels extract spatial correlation patterns of features within a single frame: ； in, The spatial convolution weight matrix, For the extracted geometric features, For bias terms, The spatial characteristics of the output; In the temporal convolution stage, the following is adopted: The size of the convolutional kernel captures dynamic changes between frames: ； in, The time convolution weight matrix is... For bias terms, This is the output spatiotemporal fusion feature vector.

4. The method for dangerous scene recognition and interaction of a robot dog according to claim 2, characterized in that, The residual fully connected network comprises three residual blocks. The first residual block maps the extracted spatiotemporal features to the hidden space through a fully connected layer, and then reduces the dimensionality after processing by the PReLU function to obtain the spatiotemporal fused feature vector. The second residual block will fuse the spatiotemporal feature vectors. First expand, then compress to obtain the spatiotemporal fusion feature vector. The third residual block will fuse the spatiotemporal feature vectors. Mapping to an L-dimensional classification space yields a spatiotemporal fusion feature vector. ; L represents the type of gesture to be output.

5. The method for dangerous scene recognition and interaction of a robot dog according to claim 2, characterized in that, The geometric features include joint rotation features and fingertip distance features; The normalized gesture joint coordinates are used to extract geometric features from keyframes, including: Extracting joint rotation features: Joint rotation features are characterized by the real part of the quaternion rotation angle to represent the local bending degree of the joint. ， ； in, Features of joint rotation; This refers to the joint rotation angle; and They are respectively , The corresponding length-normalized bone vector; and All are skeletal vectors, derived from the hand joints in the set of keyframes. relative coordinates Calculated; Extracting fingertip distance features: , These represent the distance between adjacent fingertips and the distance between each fingertips relative to the root node of the wrist, respectively. The extracted geometric features are obtained by concatenating the joint rotation features and the fingertip distance features.

6. The method for identifying and interacting with dangerous scenarios using a robot dog according to claim 1, characterized in that, The process of obtaining the 3D coordinates of key points based on a single-frame RGB image includes: The single-frame RGB image is sequentially compressed and layer normalized. Multi-level feature extraction is performed on the feature map after layer normalization. The extracted features are then input into the two-dimensional coordinate branch and the relative depth branch, respectively. The two-dimensional coordinate branch generates a heatmap through 3×3 convolution, and then outputs normalized coordinates through the differentiable maximum index. The relative depth branch sequentially passes through small convolutional kernels and global spatial average pooling to obtain the relative depth of each keypoint. Based on normalized coordinates The 3D coordinates of the keypoints are obtained by calculating their relative depth.

7. The method for dangerous scene recognition and interaction of a robot dog according to claim 6, characterized in that, For the feature map after layer normalization, a multi-level feature extraction module is used for multi-level feature extraction. The multi-level feature extraction module includes three sets of downsampling modules and three sets of upsampling modules. The three sets of downsampling modules are connected in series, and the three sets of upsampling modules are also connected in series. Each set of downsampling modules consists of a downsampling module and two series-connected feature modules, and each set of upsampling modules consists of two series-connected feature modules and an upsampling module. Both the downsampling and upsampling modules are implemented using basic convolutional units. The three sets of downsampling modules and the three sets of upsampling modules are fused through residual connections, and the feature map output by the third set of upsampling modules is used as the extracted features. The three sets of downsampling modules and the three sets of upsampling modules are connected via residual connections for feature fusion, specifically: Connect the first set of downsampling modules to the third set of upsampling modules, connect the second set of downsampling modules to the second set of upsampling modules, and connect the third set of downsampling modules to the first set of upsampling modules.

8. The method for dangerous scene recognition and interaction of a robot dog according to any one of claims 1 to 7, characterized in that, The real-time conversion of speech signals into text includes: Preliminary filtering of ambient noise or other speech signals from the speech signal; The initially filtered speech signal is purified and enhanced; High-dimensional acoustic feature vector sequences are extracted from the enhanced speech signal; these sequences are then input into an ASR (Automatic Speech Recognition) model to convert the speech stream into text in real time.

9. A robot dog dangerous scene recognition and interaction device based on multimodal attention fusion, characterized in that, The device includes: The acquisition module is used to acquire the user's voice signals and gesture image signals through the sensing system deployed on the intelligent robot dog; The conversion and semantic parsing module is used to convert speech signals into text in real time; it performs lightweight semantic parsing on the converted text to obtain at least one structured semantic object and its corresponding confidence score, and uses the confidence score corresponding to the structured semantic object as the first semantic information. The feature extraction module is used to obtain a single-frame RGB image based on the gesture image signal; and to obtain the 3D coordinates of key points based on the single-frame RGB image. The dynamic feature modeling module is used to obtain the probability distribution of each gesture category based on the 3D coordinates of key points corresponding to multiple frames of RGB images and use it as the second semantic information. The scene category recognition module, based on first and second semantic information, employs a semantically enhanced dual attention mechanism to identify the final scene category, including: Based on the first semantic information and the second semantic information, calculate the speech feature vector and gesture confidence. By employing confidence gating, the initial confidence weights are generated by sequentially applying linear transformations and Sigmoid function mappings to the first semantic information and the gesture confidence. ; By using semantic consistency gating, a matrix M is obtained by performing a dot product operation on the speech feature vector and the second semantic information. Then, global max pooling is applied to matrix M, and semantic relevance is generated by mapping using the Sigmoid function. ; Calculate the scene adaptation coefficient based on the first semantic information and gesture confidence. ; Calculate fusion weights ; Based on fusion weight Obtain the fusion probability vector; based on the fusion probability vector, map the final scene recognition result.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot dog dangerous scene recognition and interaction method according to any one of claims 1 to 8.

Images