Robot indoor easily-confused behavior class identification method based on audio-visual fusion
Through deep learning, combining video space and time difference characteristics with audio signals, a category-adaptive audio-visual fusion model is constructed, which solves the problem of insufficient accuracy of human confusing behavior recognition in complex indoor environments, and realizes efficient identification of pathological behaviors and Noise adaptation.
Patent Information
- Application Number
- CN202411085491.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-11-26
AI Technical Summary
The prior art has insufficient accuracy in identifying human confusing behaviors in complex indoor environments. Visual information is affected by environmental occlusion and light changes, auditory information is easily disturbed by background noise, and single modal identification is difficult to distinguish between pathological and normal behavior.
Using deep learning methods, the audio signal is processed by combining the spatial distribution and temporal differential characteristics of video, and the audio category information is combined with the expansion causal convolution, and a category-adaptive feature fusion network is introduced, and the loss function is constructed to train the audio-visual fusion model to deal with noise. Mirmur filtering, and a distraction-free behavior data set in indoor scenarios is constructed to train the behavior recognition model.
It improves the accuracy of identification of confusing indoor behaviors, enhances the model's adaptability to environmental noise and dynamic changes, avoids information loss, and makes full use of the complementary advantages of audio-visual information.
Smart Images

Figure CN119028016A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of service robot perception technology, and in particular to the fusion processing of visual and auditory information of indoor service robots. The present invention is a robot indoor confusable behavior recognition method based on visual and audio fusion, which is used to improve the recognition accuracy of confusable human behaviors in complex indoor environments. Background Art
[0002] Service robots are increasingly used in indoor environments such as hospitals, nursing homes, and homes. They need to be able to accurately recognize human behavior in order to provide timely responses and services. In medical and nursing scenarios, timely detection and response to pathological behaviors (such as sneezing and coughing) are crucial to monitoring the health of patients; at the same time, for groups such as the elderly and children, rapid recognition and response to sudden unexpected behaviors (such as falls and collisions) can greatly reduce the risk of injury.
[0003] At present, the main technologies used by service robots in the field of pathological behavior and accident detection include: (1) Posture estimation based on human key points: Posture estimation technology usually relies on the human posture estimation algorithm in computer vision, using convolutional neural network (CNN) to extract features of people in the image, and predicting the location of human key points through recurrent neural network (RNN) or graph convolutional network (GCN). The model needs to be trained on a dataset containing various pathological behavior postures, such as coughing, sneezing, falling, etc. By analyzing the changes in key points, the robot can identify abnormalities in human posture, such as violent coughing movements or unstable gait, and thus determine whether there is pathological behavior. (2) Facial recognition and expression analysis: This type of algorithm first needs to use a facial detection algorithm to locate the face in the video, and then use a deep learning algorithm to analyze the facial video stream to identify specific facial expressions or movements that may be related to pathological behavior. For example, yawning usually involves opening the mouth and stretching facial muscles. This type of algorithm enables service robots to process video streams in real time and respond quickly to detected expression changes. (3) Sound recognition and acoustic event detection: This type of algorithm requires the use of a high-quality microphone array to capture environmental sounds, and uses acoustic feature extraction methods such as Mel-frequency cepstral coefficients (MFCC) to convert sound signals into machine-understandable feature vectors, which are then classified. Sound recognition technology is used to detect specific sounds, such as coughing or sneezing. Acoustic event detection algorithms can be designed to identify sudden sound events, such as the sounds of falls or collisions. (4) Multimodal fusion technology: At the decision-making level, voting mechanisms, Bayesian methods, or deep learning models can be used to integrate the recognition results of different modalities. At the feature level, specific fusion mechanisms can be designed, such as weighted averaging, feature concatenation, or the use of attention mechanisms to selectively emphasize certain features. For example, facial expression analysis and sound recognition can be combined to more accurately detect coughing behavior.
[0004] However, existing technologies face challenges in recognizing these behaviors in complex indoor environments. Visual information may be inaccurate due to environmental occlusion and lighting changes, while auditory information may be disturbed by background noise. In addition, pathological behaviors and some normal behaviors are highly similar visually, making it difficult for recognition systems that rely only on single-modality information to distinguish them.
[0005] Although multimodal learning can improve recognition accuracy in theory, there are many challenges in its actual implementation. Most systems use only one data source, video or audio, and cannot fully utilize multimodal information to improve recognition accuracy. Some systems that attempt to fuse multimodal data have relatively simple feature fusion strategies and fail to fully utilize the complementary advantages of audio-visual information. Existing systems have poor adaptability to environmental noise and dynamic changes, and are easily disturbed, which affects recognition results. Summary of the invention
[0006] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a robot indoor confusable behavior recognition method based on audio-visual fusion, which can be used to solve the common false detection problem of visually confusing classes in behavior recognition in indoor scenes, and through deep learning, enable service robots to understand human behavior more accurately.
[0007] In order to achieve the above object, the technical solution adopted by the present invention is:
[0008] The present invention uses the information on the spatial distribution of the video, the feature information on the difference, and the category information in the audio to comprehensively identify the behavior class in the video. In the processing of audio information, the original audio information is directly processed using the dilated causal convolution, thereby avoiding information loss in the process of converting the audio into other information formats. Considering that the behavioral features are unevenly distributed in vision and sound, the present invention introduces a category-adaptive feature fusion network in the decoder. The fusion network constructs a loss function by introducing a category confusion table to form an adaptive training paradigm. In view of the problem that there is a large amount of interfering noise in the sound of the existing behavior recognition video data set, the present invention uses a deep network to filter out noise, and through appropriate manual screening, constructs a video data set for visually confusing behaviors in indoor scenes. The behaviors in the data set have typical sounds without interference, and can train a convergent audio-visual fusion behavior recognition model.
[0009] Specifically, the process of the robot indoor confusing behavior recognition method based on audio-visual fusion of the present invention can be described as follows:
[0010] For video data, two branches with different frame sampling rates and different image resolutions are used to perform feature extraction at two different levels. The first branch is the time stream branch, which samples the video at a higher frame sampling rate, and obtains the features of the frame image in time difference by performing continuous three-dimensional convolution on the obtained frame image; the second branch is the space stream branch, which samples the video at a lower frame sampling rate, and extracts the spatial features retained in the frame image by performing continuous three-dimensional convolution on the obtained frame image, and reduces the frame image of the time stream branch by equal proportion to ensure efficiency synchronization, thereby improving the speed of three-dimensional convolution to be on par with the space stream branch.
[0011] That is, for the video data, image sampling is performed at the frame sampling rate of the spatial stream branch to obtain a plurality of image frames constituting a spatial information stream, wherein each sampling frame has the image resolution of the spatial stream branch, and the spatial stream branch features are extracted from the spatial information stream; and, for the video data, image sampling is performed at the frame sampling rate of the temporal stream branch to obtain a plurality of image frames constituting a temporal information stream, wherein each sampling frame has the image resolution of the temporal stream branch, and the temporal stream branch features are extracted from the temporal information stream; wherein the frame sampling rate of the temporal stream branch is greater than the frame sampling rate of the spatial stream branch, and the image resolution of the temporal stream branch is less than the image resolution of the spatial stream branch.
[0012] Performing audio sampling on the video data at a fixed sampling rate to obtain an audio information stream composed of a plurality of audios, and extracting final sound features from the audio information stream;
[0013] The spatial stream branch features and the temporal stream branch features are combined to obtain the final visual features;
[0014] The final sound feature and the final visual feature are fused to obtain a final feature, and the final feature is used to perform behavior recognition.
[0015] In a specific implementation, the frame sampling rate of the spatial stream branch is 2 frames per second, the frame sampling rate of the temporal stream branch is 16 frames per second, the image resolution of the spatial stream branch is 256×256, and the image resolution of the temporal stream branch is 224×224.
[0016] In a specific implementation, the image sampling is performed at the frame sampling rate of the spatial stream branch to obtain a plurality of image frames to form a spatial information stream, and the spatial stream branch features are extracted from the spatial information stream. The implementation method is as follows:
[0017] For video data I, the image frame is sampled at the frame sampling rate of the spatial stream branch to obtain the spatial stream input V s :
[0018]
[0019] Among them, V s,i represents the i-th frame in the spatial stream input, Indicates the first frame, represents rounding down, c1 represents the frame sampling rate of the spatial stream branch; the spatial stream input V is convolved using residual 3D convolution s Extract features and obtain spatial flow branch features F vs,L , where L is the number of convolution operations;
[0020] The image sampling is performed at the frame sampling rate of the time stream branch to obtain a plurality of image frames to form a time information stream, and the time stream branch features are extracted from the time information. The implementation method is as follows:
[0021] For video data I, the image frame is sampled at the frame sampling rate of the time stream branch to obtain the time stream input V t :
[0022]
[0023] Among them, V t,i represents the i-th frame in the time stream input, Indicates the first Frame, c2 represents the frame sampling rate of the time stream branch; residual 3D convolution is used to input V t Extract features and obtain the time stream branch feature F vt,L .
[0024] In a specific implementation, the time stream is input into V t The video frames in the image are divided into smaller slices to improve the processing speed. Before the operation, the dimension of each video frame is H×W×3. By slicing a frame image into 4 equal parts and splicing the sliced image blocks onto the channel dimension, a new input format is obtained:
[0025] H×W×3=H / 2×W / 2×12
[0026] Where H is the height of a single frame image before slicing, and W is the width of a single frame image before slicing.
[0027] In a specific implementation, the residual 3D convolution is used to input the spatial stream V s Extract features and obtain spatial flow branch features F vs,L , the implementation method is as follows:
[0028] For the spatial stream input V s Perform multiple steps of continuous residual 3D convolution to extract its features. For the convolution of the lth layer, the visual feature vector F is obtained. vs,l :
[0029] F vs,l =ReLU(Conv3D(F vs,l-1 , W vs,l )+b vs,l )
[0030] Among them, the 3D convolution operation is represented as F conv =Conv3D(I, W, P)+b, Conv3D represents 3D convolution, W represents convolution kernel or filter, P represents padding, b is bias term, and activation function ReLU is represented by F relu=max(0, F conv ), W vs,l is the convolution kernel of the lth layer, b vs,l is the bias term of the lth layer;
[0031] Inter-layer residual feature F res Extraction is expressed by the following formula:
[0032] F res =F in +F relu
[0033] where F in is the input feature;
[0034] The residual 3D convolution is used to input the temporal stream V t Extract features and obtain the time stream branch feature F vt,L , the implementation method is as follows:
[0035] F vt,l =ReLU(Conv3D(F vt,l-1 , W vt,l )+b vt,l )
[0036] Among them, W vt,l is the convolution kernel of the lth layer, b vt,l is the bias term of the lth layer;
[0037] After each convolution, the temporal stream branch features are directly added to the spatial stream as supplementary information, and the spatial stream branch features remain unchanged, expressed as:
[0038] F′ vs,l =F vs,l +F vt,l
[0039] After L convolution operations, the spatial stream branch feature F is obtained vs,L and the time stream branch feature F vt,L , fused to obtain the final visual feature F V , expressed as:
[0040] F V =F vs,L +F vt,L
[0041] In a specific implementation, the video data is sampled at a fixed sampling rate to obtain an audio information stream composed of several audios, and the implementation method is as follows:
[0042] A one-dimensional audio signal vector a1(t) is extracted from the video data, a1(t) is input into the Open-Unmix network to obtain an audio set S, and background noise is filtered out to obtain an audio input a b (t);
[0043] A b (t) synthesize into a mono channel to obtain an audio signal, and normalize the audio data so that its range is locked between 0 and 1 to obtain audio a norm (t), and then use μ-law encoding to convert a norm (t) Dynamic compression, get a μ (t), and finally μ (t) is quantized to L' levels to obtain the final audio vector a(t), that is, the audio information stream.
[0044] In a specific implementation, a residual neural network structure based on dilated causal convolution is used to extract the final sound features from the audio information stream, and the implementation method is as follows:
[0045] The audio information stream is divided into t time steps, and the one-dimensional sampled data A of the t time step is processed using causal convolution based on the dilation rate d t , get the input sequence A t:t-d+1 The original features F at time step t t ; Among them A t:t-d+1 Represents all samples from the current time step t back to t-d+1;
[0046] Through K convolutional layers with different expansion rates, features of different scales are extracted. The output of each convolutional layer is expressed as Finally, the residual outputs of all layers are combined to form the final sound feature F A .
[0047] In the specific implementation, make a classification confusion table T F , weighting each row in the table according to different weights, and finally making the fused T F The sum of the elements on the off-diagonal line is minimized, and the optimal category fusion weight of the final sound feature and the final visual feature is obtained;
[0048] The classification confusion table T is made F , the process is as follows:
[0049] Visual prediction confidence vector S of the predicted video V and the sound prediction confidence vector S A ;
[0050] According to the confidence vector, make a visual flow confusion table T V And audio stream confusion table T A , TV The i-th row and T A The i-th row of is represented as:
[0051] T V,i =S ci,V
[0052] T A,i =S ci,A
[0053] Where S ci,V For class i Video I ci Visual prediction confidence, S ci,A For class i Video I ci The audio prediction confidence of
[0054] Fusion T v and T A Get the classification confusion table T F as follows:
[0055] T F =T V ×W V +T A ×W A
[0056] Where W V , W A Respectively represent the weights of the fully connected layer to be trained;
[0057] Using the confusion C F The quantitative measurement prediction effect is expressed as follows:
[0058]
[0059] The calculation of confusion is T F The sum of the values on the off-diagonal lines. Ideally, all elements on the diagonal lines are 1, while the sum of the elements on the off-diagonal lines is 0, which means that there is no confusion in the classification of the data by the model.
[0060] In a specific implementation, the fused T F The sum of the elements on the off-diagonal line is the smallest. The implementation method is as follows:
[0061] Construct a fusion network, the input of which is T during training v and T A , the output is an n×n category confusion table T F The training goal is to make T F The sum of the elements on the off-diagonal line is the smallest, and the loss function L″ of the entire training is as follows:
[0062]
[0063] Take T when predicting F The maximum value of each column in constructs an S F vector, as the final fusion prediction score.
[0064] In one embodiment, a visually confusing behavior recognition dataset D for indoor scenes is constructed to train the recognition model, which is expressed as:
[0065] D=Clean(Open-Unmix(D open )∪D camera )
[0066] Clean means that the data set will be manually screened in the end. open Denotes the typical sound of the behavior obtained by filtering out the noise of the easily confused behavior classes selected from the public dataset using the open-unmix network, D camera It means that certain behavior categories are manually photographed and labeled.
[0067] Compared with the prior art, the present invention has the following beneficial effects:
[0068] The present invention uses a residual neural network structure of dilated causal convolution to extract audio features, specifically adding the residuals of each residual neural part as the final feature, so as to retain as many audio features as possible. Compared with converting audio into a time-frequency spectrum or a waveform, it can avoid the loss of certain features during the conversion process.
[0069] At the same time, since the information distribution of many categories in sound and visual information is not uniform, it is unreasonable to blindly use the same set of weights to fuse sound and visual information. Considering that different behaviors will produce different combinations of misjudgments when using a single information stream, if the prediction results of n categories are arranged in the order of categories, then the confusion table formed will have non-zero values outside the diagonal. The present invention weights each row of the confusion table according to different weights, and finally makes the fused table T F The sum of the elements on the off-diagonal line is the smallest, thus obtaining the best category fusion weight, better integrating visual features and sound features, and improving the ability to recognize confusing behaviors. BRIEF DESCRIPTION OF THE DRAWINGS
[0070] Figure 1 The framework diagram of the identification method of the present invention
[0071] Figure 2 This is a flow chart of audio processing of the present invention.
[0072] Figure 3Schematic diagram for comparing the results of the embodiment of the present invention with those of different methods. DETAILED DESCRIPTION
[0073] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.
[0074] The present invention proposes an action recognition framework based on audio-visual fusion, which fully utilizes the multimodal information in the scene video by combining the three information branches of original audio, video spatial stream and video temporal stream, and obtains excellent indoor easily confused class recognition effect through the category fusion adaptive training paradigm.
[0075] refer to Figure 1 As shown, the information flow of the framework consists of the following three independent but collaboratively processed data streams: spatial information stream, temporal information stream and audio information stream. The spatial information stream is composed of a video image stream, which is sampled at a lower frame sampling rate of the spatial stream branch, but each sampled image frame has a larger resolution of the spatial stream branch to capture detailed information in the environment. The temporal information stream also comes from the video image stream, but the stream is sampled at a higher frame sampling frequency of the temporal stream branch, although each sampled image frame has a lower image resolution of the temporal stream branch, thereby focusing on capturing the temporal dynamic characteristics of the behavior. The audio information stream is independent of the video stream, and the noise is filtered through a deep network, and the audio signal sampling of the video data is processed at a fixed sampling rate to obtain a number of audios to form an audio information stream to extract the final sound features. Finally, the fusion of different modal information is completed through an adaptive training paradigm. Specifically, the spatial stream branch features and the temporal stream branch features are first fused to obtain the final visual features, and then the final sound features and the final visual features are fused to obtain the final features. In order to achieve the above-mentioned purpose, the present invention specifically includes the following steps:
[0076] Step 1: Convert the audio signal into a vector representation a(t).
[0077] See also Figure 2 The audio extraction algorithm first extracts a one-dimensional audio signal vector a1(t) from the RGB video data I to be identified, and then inputs a1(t) into the Open-Unmix network, which processes the input audio into a four-part output set S: {s1(t), s2(t), s3(t), s4(t)}, which represent human voice, drum sound, low-frequency sound, and background sound respectively. The background noise is filtered out to obtain the audio input a b (t), expressed as:
[0078] a b (t) = s1(t) + s2(t) + s3(t)
[0079] Then a b (t) synthesize into mono channel to get audio signal amono (t), then a mono (t) Normalize the audio data to lock its range between 0 and 1 to obtain a norm (t), and then use μ-law coding to convert the audio a norm (t) Dynamic compression, get a μ (t), expressed as:
[0080]
[0081] Finally, a μ (t) is quantized to L' levels to obtain the final audio vector a(t), that is, the audio information stream, which is expressed as:
[0082]
[0083] Wherein μ is a coding parameter, which is usually set to 255, and L' can be set to 256. In this embodiment, the audio sampling rate is 16 kHz.
[0084] Step 2: Sample the spatial stream and temporal stream image frames in the video data I.
[0085] The method of the present invention selects the frame sampling rate c1 of the spatial stream branch of 2 frames per second as the spatial information stream sampling image frame, and obtains the spatial stream input V s :
[0086]
[0087] Among them, V s,i represents the i-th frame in the spatial stream input, Indicates the first frame, represents rounding down, c1 represents the frame sampling rate of the spatial stream branch, and the image resolution of the spatial stream branch of the obtained image frame is 256×256.
[0088] The time stream input V is obtained by sampling the image frames at the frame sampling rate c2 of the time stream branch of 16 frames per second as the time information stream. s :
[0089]
[0090] Among them, V t,i represents the i-th frame in the time stream input, Indicates the first frame, represents rounding down, c2 represents the frame sampling rate of the time stream branch, and the image resolution of the time stream branch of the obtained image frame is 224×224.
[0091] Step 3: Input V to the time stream t Slice segmentation.
[0092] Input the time stream into V t The video frames in the image are divided into smaller parts (slices) to improve the processing speed. Before the operation, the dimension of each video frame is H×W×3. By slicing a frame image into 4 equal parts and splicing the sliced image blocks into the channel dimension, a new input format is obtained:
[0093] H×W×3=H / 2×W / 2×12
[0094] Where H is the height of a single frame image before slicing, and W is the width of a single frame image before slicing.
[0095] Step 4: Extract the final visual feature representation F V .
[0096] The present invention uses residual 3D convolution to transform the spatial stream input V s Extract features and obtain spatial flow branch features F vs,L ; Use residual 3D convolution to input V into the temporal stream t Extract features and obtain the time stream branch feature F vt,L . Where L is the number of convolution operations. The specific extraction process can be described as follows:
[0097] Step 41: The lth layer visual feature vector F vs,l Extraction.
[0098] For spatial stream input V s , the method of the present invention performs multiple steps of continuous residual 3D convolution to extract its features. For the convolution of the lth layer, the visual feature vector F vs,l :
[0099] F vs,l =ReLU(Conv3D(F vs,l-1 , W vs,l )+b vs,l )
[0100] Among them, the 3D convolution operation is represented as F conv =Conv3D(I, W, P)+b, Conv3D represents 3D convolution, W represents convolution kernel or filter, P represents padding, b is bias term, and activation function ReLU is represented by F relu =max(0,F conv ), W vs,l is the convolution kernel of the lth layer, b vs,l is the bias term of the lth layer.
[0101] Step 42: Inter-layer residual feature F resExtraction is expressed by the following formula:
[0102] F res =F in +F relu
[0103] where F in is the input feature. For the time stream input V t , using the same process, we get feature F vt,l .
[0104] Step 43: Temporal and spatial branch features F vt,L 、F vs,L Fusion.
[0105] After each convolution, the temporal stream branch features need to be directly added to the spatial stream as supplementary information, and the spatial stream branch features remain unchanged, expressed as:
[0106] F′ vs,l =F vs,l +F vt,l
[0107] After L convolution operations, the spatial stream branch feature F is obtained vs,L and the time stream branch feature F vt,L In this embodiment, L=4. The spatial stream branch feature F vs,L and the time stream branch feature F vt,L The final fusion is performed to obtain the final visual feature F v , expressed as:
[0108] F V =F vs,L +F vt,L
[0109] Step 5: Extract the final feature representation F of the sound signal A .
[0110] Step 51: Audio features F based on dilated causal convolution t extract.
[0111] like Figure 1 As shown in the lower part, the method of the present invention uses a residual neural network structure based on dilated causal convolution to extract the final sound features from the audio information stream. For the one-dimensional sampling result of the audio signal, that is, the audio information stream A = a(t), it is usually divided into t time steps for processing, where the one-dimensional sampling data of each part is composed of A t The present invention uses causal convolution based on dilation rate d to process one-dimensional sampled data A at time step t t , get the input sequence A t:t-d+1 The original features F at time step tt Specifically, the dilation rate controls the spacing between one-dimensional elements in the convolution kernel. By increasing the dilation rate, the convolution kernel can operate on a wider input sequence while maintaining a smaller size, which helps the network capture longer-distance dependencies and increase the receptive field. The principle can be described as follows:
[0112] F t =Conv1D causal (A t:t-d+1 ,W d )
[0113] The above formula represents the residual output F at time step t t is obtained through a causal convolution operation that considers all samples A from the current time step t back to t-d+1 t:t-d+1 , and use a specific set of convolution kernel weights W d , Conv1D causal () represents one-dimensional causal convolution W d is a set of convolution kernel weights. In order to maintain the causality of time series data, causal convolution ensures that the convolution kernel only performs weighted summation on the past and current time steps of A, without information about future time steps.
[0114] Step 52: Extract the residual feature R of K-layer convolution t .
[0115] Residual connections allow the network to directly add input and output features, which helps train deep networks and prevents gradient vanishing or gradient exploding problems. t It is expressed as:
[0116] R t =F t +Conv1D causal (A t:t-d+1 ,W d )
[0117] where F t is the input sequence A t:t-d+1 The original features at time step t.
[0118] Through K convolutional layers with different expansion rates, features of different scales can be extracted. The output of each convolutional layer is expressed as Finally, the residual outputs of all layers are combined to form the final sound feature F A , as shown below:
[0119]
[0120] In the formula, combine means directly adding K residual features; represents the residual output of the Kth convolutional layer at time step t.
[0121] In the above formula, R t It is to represent the structure of the resnet network, that is, the addition across time steps. In fact, in one-dimensional causal convolution, repeated K resnet networks are used to represent the residual sum of all networks as input features. Therefore, R t represents the residual in a single-layer network, It represents the residual of a single layer network in the Kth network.
[0122] This step extracts features of the audio signal by dilating the causal convolutional network, uses convolutional layers with different expansion rates to capture key features in the audio signal, and enhances the training of the deep network through residual connections. It is one of the core steps of the present invention.
[0123] Step 6: Obtain the classification results of visual and sound features CLASS = {class1, class2,…, classn}.
[0124] Input Fv into the visual classifier C V The output prediction confidence vector S is obtained from V , F A Input audio classifier C A The output prediction confidence vector S is obtained from A , where the structure of the visual classifier is consistent with that of the audio classifier, and can be expressed as follows:
[0125]
[0126] Where F is the feature tensor of the input classifier, and W, b are the parameters of the network;
[0127] Step 7: Create a classification confusion table T F .
[0128] The goal of this step is to convert the classification confusion table T F Each row in the matrix is weighted according to different weights, so that the fused T F The sum of the elements on the off-diagonal line is the smallest, and the optimal category fusion weight of the final sound feature and the final visual feature is obtained. Classification confusion table T F The production process can be described as follows:
[0129] Step 71: Predict the visual prediction confidence vector S of the video V and the sound prediction confidence vector S A .
[0130] For a set of n classes CLASS = {class1, class2, ..., classn}, the method of the present invention will take one video for each category in the set, and a total of n videos will be obtained, namely I c1 ,I c2 ,…,I cn Then, these n videos are subjected to feature extraction through the feature extraction process of step 4 and step 5 to obtain the output feature F c1,V 、F c1 ,A,…,I ci,V ,I ci ,A,…,I cn,V ,I cn,A Then, feature classification is performed in step 6 to obtain 2n groups of scores S c1,V , S c1,A ,…,S ci,V , S ci,A ,…,S cn,V , S cn,A . Where classn indicates that there are n classes in total. That is, based on the final visual feature F c1,V ,…,I ci,V ,…,I cn,V And the final sound characteristics F c1,A ,…,I ci,A ,…,I cn,A , respectively using the visual classifier C V and audio classifier C A Get the classification results of visual and sound features CLASS = {class1, class2, ..., classn}, and get the visual prediction confidence vector S V =S c1,V ,…,S ci,V ,…,S cn,V and the sound prediction confidence vector S A =S c1,A ,…,S ci,A ,…,S cn,A .
[0131] Step 72: Create visual stream and audio stream confusion table T V .
[0132] Make a confusion table based on the confidence vector. For the visual flow confusion table T V , for each class i Video I ci , whose prediction confidence vector S ci,V Each element in represents a video I ci The confidence level of each category is predicted, so the visual flow confusion table T V The i-th row of is:
[0133] TV,i =S ci,V
[0134] Similarly, the audio stream confusion table T A The production process is also similar to T V Consistent, T A The i-th row of is represented as:
[0135] T A,i =S ci,A
[0136] Where S ci,V For class i Video I ci Visual prediction confidence, S ci,A For class i Video I ci The audio prediction confidence.
[0137] Step 73: Fusion confusion table T F .
[0138] The final classification confusion table T F By T V and T A The fusion is expressed as:
[0139] T F =T V ×W V +T A ×W A
[0140] Where W V , W A They represent the weights of the fully connected layers to be trained.
[0141] Step 74: Confusion C F .
[0142] The classification confusion table can measure the prediction effect of the overall model on the data set. This prediction effect is determined by the confusion C F To quantify the measurement, the calculation formula is as follows:
[0143]
[0144] This means that the value of the confusion is T F The sum of the values on the off-diagonal lines. Ideally, all elements on the diagonal lines are 1, while the sum of the elements on the off-diagonal lines is 0, which means that there is no confusion in the classification of the data by the model.
[0145] This step selects representative videos and classifies them to construct confusion tables for the visual and audio streams to evaluate the prediction performance of the model. The calculation of confusion provides a quantitative measure of the accuracy of model fusion classification and is one of the cores of the method of the present invention.
[0146] Step 8: Design the loss function L of the fusion network.
[0147] The fusion network is trained with the prediction confusion table T consisting of the prediction results of the video image stream and the audio stream. V , T A , the output is an n×n category confusion table T F The training goal is to make T F The sum of the elements on the off-diagonal line is the smallest, and the loss function L″ of the entire training is as follows:
[0148]
[0149] This minimizes the confusion of the model for the category, and T is taken when predicting. F The maximum value of each column in constructs an S F vector, as the final fusion prediction score.
[0150] Step 9: Construct the indoor scene visually confusing action recognition dataset ICAD (Indoor Confusion Action Dataset).
[0151] When robot vision algorithms are used to identify easily confused classes in indoor scenes, they need a clean indoor behavior dataset for training. However, there is no dataset for easily confused classes in the existing public behavior recognition datasets, and most of them have interfering background noise. The method framework of the present invention uses an open-unmix network to filter out the noise of easily confused behavior classes such as coughing and sneezing selected from the public dataset, thereby obtaining the typical sounds of the behavior as the first part of the dataset D open For some behavior categories, such as falling, manual photography and annotation are performed to obtain D camera , in order to increase the diversity and coverage of the data set, the process of finally obtaining the indoor scene visual confusable behavior recognition data set D is as follows:
[0152] D=Clean(Open-Unmix(D open )∪D camera )
[0153] Clean means that the data set will be manually screened in the end. openDenotes the typical sound of the behavior obtained by filtering out the noise of the easily confused behavior classes selected from the public dataset using the open-unmix network, D cameea It means that certain behavior categories are manually photographed and labeled.
[0154] This step creates a dataset for easily confusing indoor behaviors. By filtering out noise and artificial photography in the public dataset and supplementing specific behavior categories, a high-quality and diverse dataset is constructed for training and optimizing robot vision algorithms, which is a key part of the present invention.
[0155] Step 10: Use the visually confusing behavior recognition dataset to pre-train the network of the visual image stream and the audio stream (i.e., the stacked 3D resnet convolutional network) separately, and finally fuse the network of the visual image stream and the audio stream with the category-adaptive decoder (i.e., the fusion network trained based on the classification confusion table) to continue training as a whole until it converges;
[0156] Step 11: Use the trained fusion network to predict and fuse multi-stream information of the video, and finally predict the test set, using TOP1 and TOP5 as measurement indicators, which respectively represent the proportion of videos with the largest confidence values corresponding to the correct class in the test set and the proportion of the top 5.
[0157] See also Figure 3 There are four experimental methods in the figure, all of which are trained with the same data set and parameters until convergence. The first group of results represents the TOP1 and TOP5 accuracy of the data set using only the audio stream, the second group of results represents the indicators of recognition using only the video stream, the third group represents the best results of fusing all categories in the data set using the same linear weighting value, and the fourth group represents the fusion recognition results obtained by the method of the present invention using the confusion table for training. From the experimental results, it can be seen that the effect of the method of the present invention is much better than other methods in both indicators.
Claims
1. A robot indoor confusing behavior recognition method based on audio-visual fusion, Features: For video data, two branches with different frame sampling rates and different image resolutions are used to perform feature extraction at two different levels. The first branch is the time stream branch, which samples the video at a higher frame sampling rate, and obtains the features of the frame image in time difference by performing continuous three-dimensional convolution on the obtained frame image; the second branch is the space stream branch, which samples the video at a lower frame sampling rate, and extracts the spatial features retained in the frame image by performing continuous three-dimensional convolution on the obtained frame image, and reduces the frame image of the time stream branch by equal proportion to ensure efficiency synchronization, thereby improving the speed of three-dimensional convolution to be on par with the space stream branch; Performing audio sampling on the video data at a fixed sampling rate to obtain an audio information stream composed of a plurality of audios, and extracting final sound features from the audio information stream; The final visual features are obtained by fusing the spatial stream branch features and the temporal stream branch features; The final sound feature and the final visual feature are fused to obtain a final feature, and the final feature is used to perform behavior recognition.
2. According to the method for identifying confusing indoor robot behaviors based on audio-visual fusion according to claim 1, It is characterized in that For feature extraction of video modality, the frame sampling rate of the spatial stream branch is 2 frames per second, the frame sampling rate of the temporal stream branch is 16 frames per second, the image resolution of the spatial stream branch is 256×256, and the image resolution of the temporal stream branch is 224×224.
3. The method for identifying indoor confusing behaviors of robots based on audio-visual fusion according to claim 1 or 2, It is characterized in that The image sampling is performed at the frame sampling rate of the spatial stream branch to obtain a plurality of image frames to form a spatial information stream, and the spatial stream branch features are extracted from the spatial information stream. The implementation method is as follows: For video data I, the image frame is sampled at the frame sampling rate of the spatial stream branch to obtain the spatial stream input V s : Among them, V s,i represents the i-th frame in the spatial stream input, Indicates the first frame, represents rounding down, c1 represents the frame sampling rate of the spatial stream branch; the spatial stream input V is convolved using residual 3D convolution s Extract features and obtain spatial flow branch features F vs,L , where L is the number of convolution operations; The image sampling is performed at the frame sampling rate of the time stream branch to obtain a plurality of image frames to form a time information stream, and the time stream branch features are extracted from the time information. The implementation method is as follows: For video data I, the image frame is sampled at the frame sampling rate of the time stream branch to obtain the time stream input V t : Among them, V t,i represents the i-th frame in the time stream input, Indicates the first Frame, c2 represents the frame sampling rate of the time stream branch; residual 3D convolution is used to input V t Extract features and obtain the time stream branch feature F vt,L .
4. According to claim 3, the method for identifying robot indoor confusing behaviors based on audio-visual fusion, It is characterized in that Input the time stream into V t The video frames in the image are divided into smaller slices to improve the processing speed. Before the operation, the dimension of each video frame is H×W×3. By slicing a frame image into 4 equal parts and splicing the sliced image blocks onto the channel dimension, a new input format is obtained: H×W×3=H / 2×W / 2×12 Where H is the height of a single frame image before slicing, and W is the width of a single frame image before slicing.
5. According to claim 3, the method for identifying robot indoor confusing behaviors based on audio-visual fusion, It is characterized in that The residual 3D convolution is used to input the spatial stream V s Extract features and obtain spatial flow branch features F vs,L , the implementation method is as follows: For the spatial stream input V s Perform multiple steps of continuous residual 3D convolution to extract its features. For the convolution of the lth layer, we get the visual feature vector F vs,l : F vs,l =ReLU(Conv3D(F vs,l-1 ,IN vs,l )+b vs,l ) Among them, the 3D convolution operation is represented as F conv =Conv3D(I,W,P)+b, Conv3D represents 3D convolution, W represents convolution kernel or filter, P represents padding, b represents bias term, and activation function ReLU is represented by F relu =max(0,F conv ), W vs,l is the convolution kernel of the lth layer, b vs,l is the bias term of the lth layer; Inter-layer residual feature F res Extraction is expressed by the following formula: F res =F in +F relu Among them, F in is the input feature; The residual 3D convolution is used to input the temporal stream V t Extract features and obtain the time stream branch feature F vt,L , the implementation method is as follows: F vt,l =ReLU(Conv3D(F vt,l-1 ,IN vt,l )+b vt,l ) Among them, W vt,l is the convolution kernel of the lth layer, b vt,l is the bias term of the lth layer; After each convolution, the temporal stream branch features are directly added to the spatial stream as supplementary information, and the spatial stream branch features remain unchanged, expressed as: F′ vs,l =F vs,l +F vt,l After L convolution operations, the spatial stream branch feature F is obtained vs,L and the time stream branch feature F vt,L , fused to obtain the final visual feature F V , expressed as: F V =F vs,L +F vt,L 。 6. According to claim 1, the method for identifying robot indoor confusing behaviors based on audio-visual fusion, It is characterized in that The video data is sampled at a fixed sampling rate to obtain an audio information stream composed of several audios, and the implementation method is as follows: A one-dimensional audio signal vector a is extracted from the video data. 1 (t), a 1 (t) Input the Open-Unmix network to obtain the audio set S, and filter out the background noise to obtain the audio input a b (t); A b (t) synthesize into a mono channel to obtain an audio signal, and normalize the audio data so that its range is locked between 0 and 1 to obtain audio a norm (t), and then use μ-law encoding to convert a norm (t) Dynamic compression, get a μ (t), and finally μ (t) is quantized to L' levels to obtain the final audio vector a(t), that is, the audio information stream.
7. According to claim 1, the method for identifying indoor confusing behaviors of robots based on audio-visual fusion, It is characterized in that The final sound features are extracted from the audio information stream using a residual neural network structure based on dilated causal convolution, and the implementation method is as follows: The audio information stream is divided into t time steps, and the one-dimensional sampled data A of the t time step is processed using causal convolution based on the dilation rate d t , get the input sequence A t:t-d+1 The original features F at time step t t ; Among them A t:t-d+1 Represents all samples from the current time step t back to t-d+1; Through K convolutional layers with different expansion rates, features of different scales are extracted. The output of each convolutional layer is expressed as Finally, the residual outputs of all layers are combined to form the final sound feature F A .
8. According to claim 1, the method for identifying indoor confusing behaviors of robots based on audio-visual fusion, It is characterized in that Create a classification confusion table T F , weighting each row in the table according to different weights, and finally making the fused T F The sum of the elements on the off-diagonal line is minimized, and the optimal category fusion weight of the final sound feature and the final visual feature is obtained; The classification confusion table T is made F , the process is as follows: Visual prediction confidence vector S of the predicted video V and the sound prediction confidence vector S A ; According to the confidence vector, make a visual flow confusion table T V And audio stream confusion table T A , T V The i-th row and T A The i-th row of is represented as: T V,i =S ci,V T A,i =S ci,A Where S ci,V For class i Video I ci The visual prediction confidence, S ci,A For class i Video I ci The audio prediction confidence of Fusion T V and T A Get the classification confusion table T F as follows: T F =T V ×W V +T A ×W A Where W V , W A Respectively represent the weights of the fully connected layers to be trained; Using the confusion C F The quantitative measurement prediction effect is expressed as follows: The calculation of confusion is T F The sum of the values on the off-diagonal lines. Ideally, all elements on the diagonal lines are 1, while the sum of the elements on the off-diagonal lines is 0, which means that there is no confusion in the classification of the data by the model.
9. According to claim 8, the method for identifying indoor confusing behaviors of robots based on audio-visual fusion, It is characterized in that The fusion of T F The sum of the elements on the off-diagonal line is the smallest. The implementation method is as follows: Construct a fusion network, the input of which is T during training V and T A , the output is an n×n category confusion table T F The training goal is to make T F The sum of the elements on the off-diagonal line is the smallest, and the loss function L″ of the entire training is as follows: Take T when predicting F The maximum value of each column in constructs an S F vector, as the final fusion prediction score.
10. According to claim 1, the method for identifying indoor confusing behaviors of robots based on audio-visual fusion, It is characterized in that Construct an indoor scene visually confusing behavior recognition dataset D to train the recognition model, which is expressed as: D=Clean(Open-Unmix(D open )∪D camera ) Clean means that the data set will be manually screened in the end. open Denotes the typical sound of the behavior obtained by filtering out the noise of the easily confused behavior classes selected from the public dataset using the open-unmix network, D camera It means that certain behavior categories are manually photographed and labeled.
