Sound production object multi-classification method and device based on multiple modes and computer equipment

Abstract

The invention discloses a sound production object multi-classification method and device based on multiple modes and computer equipment. The method comprises the following steps: acquiring audios and videos of sounding objects to be classified to obtain audio information and video information; inputting the audio information and the video information into a classification model for classification to obtain a classification result; obtaining the classification result; wherein the classification model is obtained by taking a plurality of pieces of audio and video information with category labels as a sample set to train a deep learning model. By implementing the method provided by the invention, the multi-class sound classification problem can be effectively solved, the classification accuracy can be enhanced in combination with multi-modal information, and the method has strong noise filtering capability to improve the recognition precision in a noisy environment; the problem that sound classification recognition accuracy is not high only from a pure voice mode due to strong noise interference is solved.

Description

Technical Field

[0001] The present invention relates to artificial intelligence, and more specifically to a multi-modal sound object multi-classification method, device and computer equipment. Background Art

[0002] With the rapid development of information technology, audio and sound classification technologies have been widely used in various fields, such as smart homes, voice assistants, and environmental monitoring. However, existing sound classification technologies face multiple challenges, including but not limited to low efficiency in multi-category sound classification, lack of multimodal support, and sensitivity to noise.

[0003] Some existing methods can only recognize and classify sounds of a specific category. For multi-category sound classification tasks, their effectiveness is limited and they are not flexible enough. Other solutions attempt to improve the efficiency of sound classification by processing two-dimensional spectrograms using variational autoencoders. However, this method relies primarily on the information of the sound signal itself and lacks the support of other auxiliary information, which affects the prediction accuracy. In addition, the lack of noise filtering further increases the difficulty of model prediction. There are also deep learning-based sound classification methods that use convolutional neural network models to extract time and frequency domain features from samples, improving the recognition of frequency energy changes and overall sound information. However, these methods still exhibit high error rates when faced with strong noise interference and lack sufficient robustness for effective noise removal.

[0004] In summary, although the above methods each solve some problems, they still have significant limitations in practical applications and cannot solve the problem of low accuracy of sound classification and recognition based only on pure speech modality due to strong noise interference.

[0005] Therefore, it is necessary to design a new method that can not only effectively handle multi-category sound classification problems, but also enhance classification accuracy by combining multimodal information, and have strong noise filtering capabilities to improve recognition accuracy in noisy environments, and solve the problem of low accuracy of sound classification and recognition from pure speech modality due to strong noise interference. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the prior art and provide a multi-modal sound object multi-classification method, device and computer equipment.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a multi-modal sound object multi-classification method, comprising:

[0008] Obtaining audio and video of the sound object to be classified to obtain audio information and video information;

[0009] Inputting the audio information and the video information into a classification model for classification to obtain a classification result;

[0010] outputting the classification result;

[0011] The classification model is obtained by training a deep learning model using a number of audio and video information with category labels as a sample set.

[0012] Its further technical solution is: the classification model includes an audio processing branch, a video processing branch, a convolutional layer and a multi-layer perceptron; wherein, the audio processing branch includes a Mel-frequency cepstral coefficient layer, a gated recurrent unit, a Conformer module, a Sub layer, an EfficientNet module and a multi-layer perceptron; the video processing branch includes a time shift module, a global pooling network, a multi-layer perceptron layer, a Transformer module, a spatial pyramid pooling network, and a spatial attention module.

[0013] A further technical solution is: inputting the audio information and the video information into a classification model for classification to obtain a classification result, including:

[0014] Inputting the audio information and the video information into a classification model;

[0015] Predicting noise characteristics of the audio information, removing noise, and generating an audio feature matrix;

[0016] Extracting and enhancing the video temporal features of the video information, and extracting the spatial information to generate a video feature matrix;

[0017] The audio feature matrix and the video feature matrix are fused and classified to obtain a classification result.

[0018] A further technical solution is: predicting noise characteristics of the audio information, removing noise, and generating an audio feature matrix includes:

[0019] Performing Fourier transform on the audio information and extracting audio features through Mel-frequency cepstral coefficients and a gated recurrent unit network;

[0020] Input the audio features into the Conformer module for processing to obtain predicted noise features;

[0021] Extracting audio features containing noise from the audio information through a convolution operation;

[0022] Based on a SUB operation, subtracting the predicted noise feature from the audio feature containing noise to obtain an audio feature after noise removal;

[0023] Inputting the de-noised audio features into an EfficientNet module to obtain a new feature representation, and concatenating the new feature representation with the de-noised audio features to obtain a concatenated feature;

[0024] The concatenated features are converted into a format used by a matrix through a multi-layer perceptron to obtain an audio feature matrix.

[0025] A further technical solution is: extracting the video information and enhancing the video temporal features, and extracting the spatial information to generate a video feature matrix, including:

[0026] Segmenting the video information into picture frames and extracting the time series features of the video using a time shift module network;

[0027] Inputting the time series features into a global pooling network and connecting multiple layers of perceptrons in series to generate time series information;

[0028] Extracting an intermediate frame from the video information as a static image, inputting it into a spatial pyramid pooling network, and extracting semantic information through a convolutional block attention mechanism to obtain spatial information;

[0029] Inputting the temporal information and the spatial information into a spatial attention module to obtain a final spatial feature representation;

[0030] The temporal information and the final spatial feature representation are combined to form a video feature matrix.

[0031] A further technical solution is: fusing and classifying the audio feature matrix and the video feature matrix to obtain a classification result, including:

[0032] The audio feature matrix and the video feature matrix are fused using a Transformer module to obtain fused features;

[0033] The fused features are sequentially input into the convolution layer, multiple multi-layer perceptrons and the softmax function for processing to obtain the classification result.

[0034] Its further technical solution is: the loss function adopted in the classification model training process includes a function formed by weighted addition of three auxiliary loss functions, and the three auxiliary loss functions respectively include the MSE loss for calculating the noise label, the MSE loss for calculating the denoising label, and the cross entropy loss of the classification result.

[0035] Its further technical solution is: when training the classification model, a visual source is masked for several audio and video information with category labels, and an enhanced sample set is produced by combining different audio and noise data in a mosaic manner, and a deep learning model is trained based on the enhanced sample set.

[0036] The present invention also provides a multi-modal sound object multi-classification device, comprising:

[0037] An acquisition unit, configured to acquire audio and video of the sound object to be classified to obtain audio information and video information;

[0038] a classification unit, configured to input the audio information and the video information into a classification model for classification to obtain a classification result;

[0039] an output unit, configured to output the classification result;

[0040] The classification model is obtained by training a deep learning model using a number of audio and video information with category labels as a sample set.

[0041] The present invention further provides a computer device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0042] The beneficial effects of the present invention compared with the existing technology are: the present invention obtains audio and video data of the sound object to be classified to simultaneously obtain audio and video information, and inputs these multimodal information into a deep learning model trained with a large number of labeled audio and video samples for classification processing, and finally outputs the classification results; this method utilizes the complementarity of audio and video information, enhances the model's ability to extract features of sound-emitting objects, and not only effectively solves the problem of low recognition accuracy of a single audio modality in a noisy environment, but also significantly improves the classification accuracy and noise filtering ability by combining the additional prior information provided by the visual modality, thereby achieving more accurate sound classification and recognition in a noisy environment; this method makes full use of the synergistic effect between different modalities, and improves the robustness and reliability of the overall system.

[0043] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 A schematic diagram of an application scenario of the multi-modal sound object multi-classification method provided in an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of a flow chart of a multi-modal sound object multi-classification method provided in an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of a sub-process of a multi-modal sound object multi-classification method provided in an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of a sub-process of a multi-modal sound object multi-classification method provided in an embodiment of the present invention;

[0049] Figure 5 A schematic diagram of a sub-process of a multi-modal sound object multi-classification method provided in an embodiment of the present invention;

[0050] Figure 6 A schematic diagram of a sub-process of a multi-modal sound object multi-classification method provided in an embodiment of the present invention;

[0051] Figure 7 A schematic block diagram of a multi-modal sound object multi-classification device provided by an embodiment of the present invention;

[0052] Figure 8 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0054] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0055] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the present invention. As used in the specification and appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0056] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0057] See also Figure 1 and Figure 2 , Figure 1 Schematic diagram of an application scenario of the multi-modal sound object multi-classification method provided in an embodiment of the present invention. Figure 2 A schematic flowchart of a multimodal sound classification method provided by an embodiment of the present invention is provided. This multimodal sound classification method is applied to a server that interacts with a terminal. By combining multimodal audio and video information, the method utilizes deep learning models to process and enhance the audio and video data from the sound source. First, the audio processing branch uses Mel-frequency cepstral coefficients, gated recurrent units, and Conformer modules to extract and remove noise features, generating a clear audio feature matrix. The video processing branch uses a time shift module and a global pooling network to extract temporal features and spatial information, forming a video feature matrix. The two are then fused using a Transformer module, and classification is performed using convolutional layers and a multi-layer perceptron to improve classification accuracy. Furthermore, the application of specific loss functions and sample augmentation techniques, such as masking visual sources and mosaic data augmentation, during training effectively enhances the model's noise filtering capabilities and robustness in complex environments, thereby addressing the low recognition accuracy of a single speech modality in noisy environments. This method not only efficiently handles multi-category sound classification but also significantly improves classification accuracy and reliability.

[0058] Figure 2 FIG. 1 is a flow chart of a multi-modal sound object classification method provided by an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S130.

[0059] S110 : Acquire audio and video of the sound object to be classified to obtain audio information and video information.

[0060] In this embodiment, acquiring the audio and video of the sound object to be classified is the first and most important step of the entire method. This step is to ensure that the audio and video data of the sound object can be accurately captured for subsequent feature extraction and classification processing.

[0061] Audio information refers to the digital signal converted from the sound signal emitted by the sound-emitting object after being captured by a collection device (such as a microphone). Specifically, in the method of the present invention, audio information includes multiple levels of acoustic features:

[0062] Time domain features: directly reflect the information of how the sound waveform changes over time.

[0063] Frequency domain features: Time domain signals are converted to the frequency domain through methods such as Fourier transform, revealing the energy distribution of different frequency components. For example, Mel-Frequency Cepstral Coefficients (MFCCs) are a commonly used frequency domain feature representation method that can effectively simulate the human ear's perception of sounds of different frequencies.

[0064] Semantic features: High-level features extracted by deep learning models (such as GRU and Conformer modules). These features can represent the semantic information in the sound and help distinguish different sound categories.

[0065] In addition, the audio information also includes noise features, that is, interference signals generated by non-target sound sources in the environment. This embodiment specifically designs a denoising module to identify and remove these noises, thereby improving the quality of the audio features and the accuracy of classification.

[0066] Video information refers to a sequence of images captured by a camera (such as a camera). It includes not only static visual content (single-frame images) but also dynamic time series information. In the present invention, the role of video information is mainly to provide additional contextual information to assist in sound classification. Specifically:

[0067] Single-frame visual information: This refers to each frame extracted from a video. It can be used to analyze the appearance, location, and relationship of the sound-producing object to the surrounding environment. This information is crucial for understanding the source of the sound.

[0068] Temporal visual information: By analyzing the differences between consecutive frames, we can gain knowledge about the motion patterns or behavioral habits of the sound-making object. This temporal information can help distinguish between sound-making objects with similar appearance but different motion patterns.

[0069] To fully leverage these two types of visual information, this paper employs a Temporal Shift Module Network (TSMNet) to extract temporal features from videos, and a Spatial Pyramid Pooling Network (SPPNet) combined with a Convolutional Block Attention Mechanism (CBAM) to enhance spatial feature extraction from single frames. Ultimately, these rich visual features are combined with audio features and fed into an Image-Voice-Transformer Model for comprehensive analysis and classification.

[0070] In summary, step S110 provides a solid data foundation for subsequent steps by obtaining high-quality audio and video data of the sound-emitting object, thereby achieving more accurate sound category judgment.

[0071] S120: Input the audio information and the video information into a classification model for classification to obtain a classification result.

[0072] In this embodiment, classification information refers to the sound categories predicted by the model based on the input audio and video data. Each category corresponds to a specific type of sound source or event, such as human speech, animal sounds, or mechanical noise. The classification results include not only the most likely sound category but also the probability distribution for each category, allowing users to understand the model's confidence in different categories.

[0073] The classification model is obtained by training a deep learning model using a number of audio and video information with category labels as a sample set.

[0074] The classification model includes an audio processing branch, a video processing branch, a convolutional layer and a multi-layer perceptron; wherein the audio processing branch includes a Mel-frequency cepstral coefficient layer, a gated recurrent unit, a Conformer module, a Sub layer, an EfficientNet module and a multi-layer perceptron; the video processing branch includes a time shift module, a global pooling network, a multi-layer perceptron layer, a Transformer module, a spatial pyramid pooling network, and a spatial attention module.

[0075] In this embodiment, the classification model is trained using a large number of audio and video sample sets with category labels. Its main architecture consists of the following key components:

[0076] Audio processing branch:

[0077] Mel-frequency cepstral coefficient layer (MFCCs Layer): used to extract frequency domain features of audio signals.

[0078] Gated Recurrent Unit (GRU): Capturing temporal dependencies in audio sequences.

[0079] Conformer module: combines the self-attention mechanism with convolution operations to further refine audio features.

[0080] Sub layer: performs subtraction operations between features for denoising.

[0081] EfficientNet module: Improves feature extraction efficiency and enhances feature expression capabilities.

[0082] Multi-layer Perceptron (MLP): concatenates the above features and feeds them into the Transformer module as a query matrix.

[0083] Video processing branch:

[0084] Temporal Shift Module (TSMNet): processes the temporal sequence information of the video and captures dynamic changes.

[0085] Global Pooling Network (GAP): Reduces feature dimensions while retaining important information.

[0086] Multi-layer perceptron (MLP) layer: Combined with the GAP output and input to the Transformer module.

[0087] Transformer module: Fusion of information from audio and video to achieve cross-modal interaction.

[0088] Spatial Pyramid Pooling Network (SPPNet): Extracts spatial features of single-frame images.

[0089] Spatial Attention Module: Enhances attention to important spatial regions.

[0090] Convolutional layer: used at different stages throughout the model to extract local features.

[0091] Multilayer Perceptron (MLP): used for the final classification decision, usually at the end of the model.

[0092] First, the audio information is extracted through the MFCCs layer and GRU to extract primary features, and then further processed by the Conformer module; the video information is captured through TSMNet to capture temporal features, while the SPPNet+Spatial Attention combination is used for spatial feature extraction.

[0093] In the audio branch, the Sub layer is used to remove noise and then combined with the features of the EfficientNet module; the video branch prepares the v matrix for input to the Transformer through the GAP and MLP layers.

[0094] The audio and video features are input into the Transformer module as q (query), k (key), and v (value), respectively, and the bimodal information is integrated through the self-attention mechanism.

[0095] Finally, a series of MLP layers and Softmax functions are used to output the probability distribution of each category to determine the final classification result.

[0096] In summary, step S120 effectively integrates information from both audio and video modalities through a carefully designed deep learning model, significantly improving the accuracy and robustness of sound classification. This approach not only addresses the problem of inaccurate classification caused by environmental interference in single-modality scenarios, but also provides new insights and technical means for multimodal data analysis.

[0097] In one embodiment, see Figure 3 , the above-mentioned step S120 may include steps S121 to S124.

[0098] S121. Input the audio information and the video information into a classification model.

[0099] In this embodiment, the processed audio and video data are first input into a pre-built Image-Voice-Transformer Model, which is specially designed to use bimodal (image and sound) information to determine the sound category.

[0100] Using visual information to assist in sound classification improves the accuracy and robustness of the model in identifying sound categories.

[0101] S122: Predict noise characteristics of the audio information, remove noise, and generate an audio feature matrix.

[0102] In this embodiment, the audio feature matrix refers to matrix data used to represent audio signal features obtained after a series of processing (including Fourier transform, noise prediction and removal, EfficientNet feature extraction and MLP conversion).

[0103] In one embodiment, see Figure 4 , the above-mentioned step S122 may include steps S1221 to S1226.

[0104] S1221. Perform Fourier transform on the audio information, and extract audio features through Mel-frequency cepstral coefficients and a gated recurrent unit network.

[0105] In this embodiment, Fourier transform is used to capture the frequency domain characteristics of the audio signal, and MFCC is combined to extract higher-level audio features. GRU can effectively capture the temporal dependencies of audio sequences.

[0106] S1222: Input the audio features into a Conformer module for processing to obtain predicted noise features.

[0107] In this embodiment, the Conformer module can more accurately identify and separate noise components through the self-attention mechanism and convolution operation.

[0108] S1223. Extract audio features containing noise from the audio information through a convolution operation.

[0109] In this embodiment, the understanding of noise patterns is enhanced to provide a basis for further denoising.

[0110] S1224 . Based on a SUB operation, subtract the predicted noise feature from the audio feature containing noise to obtain the audio feature after noise removal.

[0111] In this embodiment, background noise is effectively removed, audio quality is improved, and the accuracy of subsequent classification tasks is improved.

[0112] S1225. Input the audio features after noise removal into the EfficientNet module to obtain a new feature representation, and splice the new feature representation with the audio features after noise removal to obtain a spliced ​​feature.

[0113] In this embodiment, EfficientNet optimizes the feature extraction efficiency, and the splicing operation helps to retain the original information while introducing new features.

[0114] S1226. Convert the concatenated features into a matrix format using a multi-layer perceptron to obtain an audio feature matrix.

[0115] In this embodiment, the feature representation is standardized to facilitate subsequent processing and analysis.

[0116] S123 . Extract the video information and enhance the video temporal features, and extract the spatial information to generate a video feature matrix.

[0117] In this embodiment, the video feature matrix refers to matrix data obtained by extracting time series features and spatial information of a video and integrating the information to express multi-dimensional features of the video content.

[0118] In one embodiment, see Figure 5 , the above-mentioned step S123 may include steps S1231 to S1235.

[0119] S1231. Segment the video information into picture frames, and use a time shift module network to extract time series features of the video.

[0120] In this embodiment, the time shift module effectively captures the timing information in the video and enhances the model's understanding of dynamic changes.

[0121] S1232: Input the time series features into a global pooling network and connect multiple layers of perceptrons in series to generate time series information.

[0122] In this embodiment, the global pooling network reduces the feature dimension while retaining important information; the MLP further refines the features and enhances the expressive power.

[0123] In this embodiment, the time sequence information refers to time series feature data extracted from video frames that represents the dynamic change characteristics of the video.

[0124] S1233. Extract an intermediate frame from the video information as a static image, input it into a spatial pyramid pooling network, and extract semantic information through a convolutional block attention mechanism to obtain spatial information.

[0125] In this embodiment, spatial information refers to feature data about the spatial structure and layout of a scene or object extracted from static image frames of a video.

[0126] SPPNet (Spatial Pyramid Pooling Network) and CBAM (Convolutional Block Attention Mechanism) work together to improve the ability to capture spatial details and enrich feature representation.

[0127] S1234. Input the temporal information and the spatial information into a spatial attention module to obtain a final spatial feature representation.

[0128] In this embodiment, the final spatial feature representation refers to a comprehensive representation that combines the spatial information optimized by the convolutional block attention mechanism and the spatial detail features further refined by the spatial attention module.

[0129] The spatial attention module emphasizes important spatial regions and improves the relevance and discrimination of features.

[0130] S1235: Combine the temporal information and the final spatial feature representation to form a video feature matrix.

[0131] In this embodiment, both temporal and spatial information are integrated to provide a more comprehensive video feature representation.

[0132] S124: Fusing and classifying the audio feature matrix and the video feature matrix to obtain a classification result.

[0133] In one embodiment, see Figure 6 , the above-mentioned step S124 may include steps S1241 to S1242.

[0134] S1241. Fusing the audio feature matrix and the video feature matrix using a Transformer module to obtain fused features.

[0135] In this embodiment, the fused feature refers to a new feature representation that is generated by cross-modal fusion of the audio feature matrix and the video feature matrix through the Transformer module and can capture both audio and video information.

[0136] Transformer realizes cross-modal information interaction through the self-attention mechanism, enhancing the effect of feature fusion.

[0137] S1242: Input the fusion features into the convolution layer, multiple multi-layer perceptrons, and softmax function in sequence for processing to obtain a classification result.

[0138] In this embodiment, this series of processing steps gradually refines the features, and finally outputs the probability distribution of each classification through the softmax function, thereby improving the accuracy and reliability of the classification.

[0139] In summary, the above process not only systematically addresses the issues of feature extraction, denoising, and enhancement for audio and video information, but also significantly improves overall classification performance through an effective cross-modal fusion strategy. This multi-layered, multi-faceted approach is crucial for achieving high-precision sound classification.

[0140] Specifically, the audio is first Fourier transformed and then features are derived using MFCCs and GRU. These features are then fed into the Conformer module to generate predicted noise features, representing the predicted noise characteristics in the audio. A convolution operation is then performed to obtain the noise-containing audio features, representing the extracted audio features. A SUB operation is then performed to subtract the noise-containing audio features from the predicted noise features to obtain the de-noised features. The de-noised audio features are then fed into the EfficientNet module to generate a new feature representation, which is then concatenated with the de-noised audio features. The resulting q matrix is ​​then passed through the MLP to the Transformer module. The Denoising Module then subtracts the mixed sound from the noise information to eliminate the noise, significantly improving the audio's purity and the accuracy of subsequent sound classification.

[0141] For visual modality information, the video segments are fed into a TSMNet to extract temporal features. This feature is then fed into the global pooling network (GAP) module. The MLP layer is then connected in series to serve as the V matrix input to the transformer module, which incorporates more temporal information. To better extract spatial information from the visual modality, the intermediate frames of the video are extracted as static images and fed into an SPPNet. The CBAM attention mechanism is then connected in series to facilitate the extraction of more semantic information and temporal information. This feature is then fed into the SpatialAttention module to obtain richer spatial information features. This feature is then fed into the transformer module as the k matrix to obtain a fusion matrix. This feature is then passed through convolution, multiple MLP layers, and a softmax function to classify feature categories. A category threshold is then set to classify different sound categories. This solution uses audio information as a query feature to search for video information and static image information. Furthermore, better multimodal fusion improves the model's accuracy in sound classification.

[0142] In short, the audio signal is first Fourier transformed, processed using MFCCs, and extracted using a GRU network. It is then fed into the Conformer module to predict noise features. A convolution operation is then performed to obtain the audio features containing noise. The two features are then subtracted using a SUB operation to remove noise and obtain pure audio features. This feature is then fed into the EfficientNet module for further refinement, concatenated with the previous audio features, and converted via the MLP into the transformer's q matrix input. Through this process, the Denoising Module effectively eliminates noise, improving audio purity and the accuracy of subsequent sound classification.

[0143] For visual information, the video is decomposed into image frames and features are extracted using TSMNet. These features are then processed through a global pooling network (GAP) module and connected in series with an MLP layer as the v matrix input to the transformer. This step incorporates more temporal information. To better capture spatial information in the visual modality, the middle frame of the video is selected as the static image input to SPPNet. The CBAM attention mechanism is then applied to enhance semantic information extraction. Finally, a spatial attention module (SpatialAttention) is used to obtain richer spatial information features. These features are used as the k matrix input to the transformer, completing multimodal feature fusion. Finally, convolutional layers, multiple MLP layers, and a softmax function are used to classify feature categories, and a category threshold is set to accurately classify different sound categories. By using audio information as the query feature, this solution searches for video information and static image information, achieving efficient multimodal fusion and significantly improving the accuracy of sound category classification.

[0144] By leveraging dual-modal data, image and audio, we enhanced our ability to identify sound categories. We designed a special Image-Voice-Transformer model, or classification model, which significantly improved classification performance. Furthermore, we introduced a denoising module to enhance noise filtering and optimize sound classification. In terms of visual modality, we combined the temporal information of the video with the spatial information of a single frame, enabling the model to fully utilize the rich temporal and spatial features.

[0145] For the classification model, the loss function used in the training process includes a function formed by weighted addition of three auxiliary loss functions. The three auxiliary loss functions include the MSE loss calculated for noise labels, the MSE loss calculated for denoising labels, and the cross entropy loss of the classification results.

[0146] During training, the visual source is masked by several audio and video information with category labels, and an enhanced sample set is produced by combining different audio and noise data in a mosaic manner. The deep learning model is trained based on the enhanced sample set.

[0147] Specifically, in terms of loss function design, multiple auxiliary loss functions were used to optimize model performance. Specifically, three loss functions were designed: the first, l1, is used to calculate the mean squared error (MSE) loss between the noisy labels and the predicted values; the second, l2, is used to calculate the MSE loss between the denoised audio labels and the predicted values; and the third, l3, uses the cross-entropy (CE) loss function to evaluate the classification results. The final loss function, Loss, is obtained by adding the corresponding weights alpha, beta, and gamma to each loss, i.e., Loss = alpha*l1+beta*l2+gamma*l3.

[0148] For sample production, we first collected a large amount of single-category sound audio, and then used the mosaic method to perform data enhancement by collecting different audio data and noise data to improve the robustness of the model.

[0149] The specific training process is as follows:

[0150] Collect audio and video data. The video data must contain the sound source and be synchronized with the audio. To increase the robustness of the model, the visual sound source is appropriately masked. The data is then divided into training, validation, and test sets.

[0151] Based on the structure of the classification model, an evolutionary algorithm is used to search for optimal hyperparameters, such as the learning rate, number of training rounds, and loss function weight ratio. The optimal parameter combination is selected based on the validation set. Here, the F1 score is used as the classification evaluation metric, and the parameter configuration with the highest F1 score is selected for subsequent testing.

[0152] Model inference is performed using the optimal parameters determined from the validation set. Unlike the training phase, which requires calculating a loss function, the inference process directly outputs the probabilities of different sound source categories. These probabilities are normalized using a sigmoid function, and the category with the highest probability is selected as the prediction result. This not only improves model accuracy but also ensures its practicality.

[0153] S130, outputting the classification result;

[0154] Output the classification results to the terminal for display.

[0155] The method of this embodiment addresses the problem of decreased accuracy when relying solely on the audio modality for classification and recognition due to external environmental interference such as noise. Specifically, by improving the audio recognition algorithm and introducing the visual modality as a guide for the audio modality, additional prior information is provided for audio processing. This approach not only enables the model to make judgments based on auditory information, but also gives it the ability to utilize visual information, thereby more effectively improving the accuracy of audio content judgment.

[0156] In addition, the method of this embodiment provides a solution specifically for the problem that it is difficult to achieve efficient sound classification and recognition by relying solely on the voice modality in a strong noise environment. By combining the visual modality with the voice modality and based on the information fusion between the two, richer vocal features can be extracted. This method makes full use of the complementarity between different modalities and significantly improves the accuracy and reliability of sound classification. Therefore, the present invention effectively overcomes the challenge of limited performance of a single audio modality in a noisy environment through the integration of multimodal information, and greatly improves the accuracy of sound recognition and classification.

[0157] The above-mentioned multi-classification method of multi-modal sound objects obtains audio and video data of the sound objects to be classified to simultaneously obtain audio and video information, and inputs these multi-modal information into a deep learning model trained with a large number of labeled audio and video samples for classification processing, and finally outputs the classification results; this method utilizes the complementarity of audio and video information, enhances the model's ability to extract features of sound objects, and not only effectively solves the problem of low recognition accuracy of a single audio modality in a noisy environment, but also significantly improves the classification accuracy and noise filtering ability by combining the additional prior information provided by the visual modality, thereby achieving more accurate sound classification and recognition in noisy environments; this method makes full use of the synergistic effect between different modalities and improves the robustness and reliability of the overall system.

[0158] Figure 7 FIG is a schematic block diagram of a multi-modal sound object classification device 300 provided by an embodiment of the present invention. Figure 7 As shown, corresponding to the above multi-modal sound object multi-classification method, the present invention also provides a multi-modal sound object multi-classification device 300. The multi-modal sound object multi-classification device 300 includes a unit for executing the above multi-modal sound object multi-classification method, and the device can be configured in a server. Specifically, please refer to Figure 7 The multi-modal sound object multi-classification device 300 includes an acquisition unit 301, a classification unit 302 and an output unit 303.

[0159] The acquisition unit 301 is used to acquire the audio and video of the sound object to be classified to obtain audio information and video information; the classification unit 302 is used to input the audio information and the video information into the classification model for classification to obtain a classification result; the output unit 303 is used to output the classification result;

[0160] The classification model is obtained by training a deep learning model using a number of audio and video information with category labels as a sample set.

[0161] In one embodiment, the classification unit 302 includes:

[0162] The input subunit is used to input the audio information and the video information into the classification model; the audio processing subunit is used to predict the noise characteristics of the audio information, remove the noise, and generate an audio feature matrix; the video processing subunit is used to extract and enhance the video temporal characteristics of the video information, and extract spatial information to generate a video feature matrix; the fusion classification subunit is used to fuse and classify the audio feature matrix and the video feature matrix to obtain a classification result.

[0163] In one embodiment, the audio processing subunit includes:

[0164] The first extraction module is used to perform Fourier transform on the audio information and extract audio features through Mel-frequency cepstral coefficients and gated recurrent unit networks; the prediction module is used to input the audio features into the Conformer module for processing to obtain predicted noise features; the second extraction module is used to extract audio features containing noise from the audio information through convolution operations; the subtraction module is used to subtract the predicted noise features from the audio features containing noise based on the SUB operation to obtain audio features after noise removal; the splicing module is used to input the audio features after noise removal into the EfficientNet module to obtain new feature representations, and splice the new feature representations with the audio features after noise removal to obtain spliced ​​features; the conversion module is used to convert the spliced ​​features into a format used by a matrix through a multi-layer perceptron to obtain an audio feature matrix.

[0165] In one embodiment, the video processing subunit includes:

[0166] A segmentation and extraction module is used to segment the video information into picture frames and use a time shift module network to extract the time series features of the video; a first generation module is used to input the time series features into a global pooling network and connect a multi-layer perceptron layer in series to generate timing information; a third extraction module is used to extract intermediate frames from the video information as static images, input them into a spatial pyramid pooling network, and extract semantic information through a convolutional block attention mechanism to obtain spatial information; an attention processing module is used to input the timing information and the spatial information into a spatial attention module to obtain a final spatial feature representation; a combination module is used to combine the timing information and the final spatial feature representation to form a video feature matrix.

[0167] In one embodiment, the fusion classification subunit includes:

[0168] A fusion module is used to fuse the audio feature matrix and the video feature matrix using a Transformer module to obtain fusion features; a classification processing module is used to input the fusion features into a convolutional layer, multiple multi-layer perceptrons and a softmax function in sequence for processing to obtain classification results.

[0169] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned multi-modal sound-emitting object multi-classification device 300 and each unit can refer to the corresponding description in the aforementioned method embodiment. For the convenience and brevity of the description, it will not be repeated here.

[0170] The multi-modal sound object classification device 300 can be implemented as a computer program. Figure 8 Runs on the computer equipment shown.

[0171] See also Figure 8 , Figure 8 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a server, wherein the server may be an independent server or a server cluster composed of multiple servers.

[0172] See Figure 8 The computer device 500 includes a processor 502 , a memory, and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0173] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, which, when executed, can enable the processor 502 to perform a multi-modal sound object classification method.

[0174] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500.

[0175] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a multi-modal sound object multi-classification method.

[0176] The network interface 505 is used to communicate with other devices through the network. Figure 8The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0177] The processor 502 is configured to execute a computer program 5032 stored in the memory to implement the following steps:

[0178] Acquiring audio and video of the sound object to be classified to obtain audio information and video information; inputting the audio information and the video information into a classification model for classification to obtain a classification result; and outputting the classification result;

[0179] The classification model is obtained by training a deep learning model using a number of audio and video information with category labels as a sample set.

[0180] The classification model includes an audio processing branch, a video processing branch, a convolutional layer and a multi-layer perceptron; wherein the audio processing branch includes a Mel-frequency cepstral coefficient layer, a gated recurrent unit, a Conformer module, a Sub layer, an EfficientNet module and a multi-layer perceptron; the video processing branch includes a time shift module, a global pooling network, a multi-layer perceptron layer, a Transformer module, a spatial pyramid pooling network, and a spatial attention module.

[0181] The loss function used in the classification model training process includes a function formed by weighted addition of three auxiliary loss functions, and the three auxiliary loss functions include the MSE loss for calculating the noise label, the MSE loss for calculating the denoising label, and the cross entropy loss of the classification result.

[0182] When training the classification model, a visual source is masked for a number of audio and video information with category labels, and an enhanced sample set is produced by combining different audio and noise data in a mosaic manner. The deep learning model is trained based on the enhanced sample set.

[0183] In one embodiment, when the processor 502 implements the step of inputting the audio information and the video information into the classification model for classification to obtain a classification result, the processor 502 specifically implements the following steps:

[0184] The audio information and the video information are input into a classification model; the noise characteristics of the audio information are predicted and the noise is removed to generate an audio feature matrix; the video information is extracted and the video temporal characteristics are enhanced, and the spatial information is extracted to generate a video feature matrix; the audio feature matrix and the video feature matrix are fused and classified to obtain a classification result.

[0185] In one embodiment, when the processor 502 implements the steps of predicting noise features of the audio information, removing noise, and generating an audio feature matrix, the processor 502 specifically implements the following steps:

[0186] The audio information is Fourier transformed, and audio features are extracted through Mel-frequency cepstral coefficients and a gated recurrent unit network; the audio features are input into a Conformer module for processing to obtain predicted noise features; audio features containing noise are extracted from the audio information through a convolution operation; based on a SUB operation, the predicted noise features are subtracted from the audio features containing noise to obtain audio features after noise removal; the audio features after noise removal are input into an EfficientNet module to obtain a new feature representation, and the new feature representation is spliced ​​with the audio features after noise removal to obtain a spliced ​​feature; the spliced ​​feature is converted into a format used by a matrix through a multi-layer perceptron to obtain an audio feature matrix.

[0187] In one embodiment, when the processor 502 extracts the video information, enhances the video temporal features, extracts the spatial information, and generates the video feature matrix, it specifically implements the following steps:

[0188] The video information is divided into picture frames, and the time series features of the video are extracted using a time shift module network; the time series features are input into a global pooling network, and a multi-layer perceptron layer is connected in series to generate timing information; an intermediate frame is extracted from the video information as a static image, input into a spatial pyramid pooling network, and semantic information is extracted through a convolutional block attention mechanism to obtain spatial information; the timing information and the spatial information are input into a spatial attention module to obtain a final spatial feature representation; the timing information and the final spatial feature representation are combined to form a video feature matrix.

[0189] In one embodiment, when the processor 502 implements the step of fusing and classifying the audio feature matrix and the video feature matrix to obtain a classification result, the processor 502 specifically implements the following steps:

[0190] The audio feature matrix and the video feature matrix are fused using a Transformer module to obtain fused features; the fused features are sequentially input into a convolutional layer, multiple multi-layer perceptrons, and a softmax function for processing to obtain a classification result.

[0191] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0192] Those skilled in the art will appreciate that all or part of the steps in the method of the above-described embodiment can be implemented by instructing the relevant hardware through a computer program. The computer program includes program instructions, which can be stored in a storage medium that is computer-readable. The program instructions are executed by at least one processor in the computer system to implement the steps in the method of the above-described embodiment.

[0193] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein when the computer program is executed by a processor, the processor performs the following steps:

[0194] Acquiring audio and video of the sound object to be classified to obtain audio information and video information; inputting the audio information and the video information into a classification model for classification to obtain a classification result; and outputting the classification result;

[0195] The classification model is obtained by training a deep learning model using a number of audio and video information with category labels as a sample set.

[0196] The classification model includes an audio processing branch, a video processing branch, a convolutional layer and a multi-layer perceptron; wherein the audio processing branch includes a Mel-frequency cepstral coefficient layer, a gated recurrent unit, a Conformer module, a Sub layer, an EfficientNet module and a multi-layer perceptron; the video processing branch includes a time shift module, a global pooling network, a multi-layer perceptron layer, a Transformer module, a spatial pyramid pooling network, and a spatial attention module.

[0197] The loss function used in the classification model training process includes a function formed by weighted addition of three auxiliary loss functions, and the three auxiliary loss functions include the MSE loss for calculating the noise label, the MSE loss for calculating the denoising label, and the cross entropy loss of the classification result.

[0198] When training the classification model, a visual source is masked for a number of audio and video information with category labels, and an enhanced sample set is produced by combining different audio and noise data in a mosaic manner. The deep learning model is trained based on the enhanced sample set.

[0199] In one embodiment, when the processor executes the computer program to implement the step of inputting the audio information and the video information into the classification model for classification to obtain a classification result, the processor specifically implements the following steps:

[0200] The audio information and the video information are input into a classification model; the noise characteristics of the audio information are predicted and the noise is removed to generate an audio feature matrix; the video information is extracted and the video temporal characteristics are enhanced, and the spatial information is extracted to generate a video feature matrix; the audio feature matrix and the video feature matrix are fused and classified to obtain a classification result.

[0201] In one embodiment, when the processor executes the computer program to implement the steps of predicting noise characteristics of the audio information, removing noise, and generating an audio feature matrix, the processor specifically implements the following steps:

[0202] The audio information is Fourier transformed, and audio features are extracted through Mel-frequency cepstral coefficients and a gated recurrent unit network; the audio features are input into a Conformer module for processing to obtain predicted noise features; audio features containing noise are extracted from the audio information through a convolution operation; based on a SUB operation, the predicted noise features are subtracted from the audio features containing noise to obtain audio features after noise removal; the audio features after noise removal are input into an EfficientNet module to obtain a new feature representation, and the new feature representation is spliced ​​with the audio features after noise removal to obtain a spliced ​​feature; the spliced ​​feature is converted into a format used by a matrix through a multi-layer perceptron to obtain an audio feature matrix.

[0203] In one embodiment, when the processor executes the computer program to implement the steps of extracting the video information, enhancing the video temporal features, extracting the spatial information, and generating the video feature matrix, the processor specifically implements the following steps:

[0204] Segmenting the video information into picture frames and extracting the time series features of the video using a time shift module network;

[0205] The time series features are input into a global pooling network, and a multi-layer perceptron layer is connected in series to generate time series information; an intermediate frame is extracted from the video information as a static image, input into a spatial pyramid pooling network, and semantic information is extracted through a convolutional block attention mechanism to obtain spatial information; the time series information and the spatial information are input into a spatial attention module to obtain a final spatial feature representation; the time series information and the final spatial feature representation are combined to form a video feature matrix.

[0206] In one embodiment, when the processor executes the computer program to implement the step of fusing and classifying the audio feature matrix and the video feature matrix to obtain a classification result, the processor specifically implements the following steps:

[0207] The audio feature matrix and the video feature matrix are fused using a Transformer module to obtain fused features; the fused features are sequentially input into a convolutional layer, multiple multi-layer perceptrons, and a softmax function for processing to obtain a classification result.

[0208] The storage medium may be any computer-readable storage medium that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk.

[0209] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0210] In the several embodiments provided herein, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the various units is merely a logical functional division, and actual implementation may employ other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be omitted or not implemented.

[0211] The steps in the methods of the embodiments of the present invention may be adjusted in order, combined, or deleted as needed. The units in the devices of the embodiments of the present invention may be combined, divided, or deleted as needed. Furthermore, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit.

[0212] If this integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the existing technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present invention.

[0213] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A multi-modal sound classification method, characterized in that: include: Obtaining audio and video of the sound object to be classified to obtain audio information and video information; Inputting the audio information and the video information into a classification model for classification to obtain a classification result; outputting the classification result; The classification model is obtained by training a deep learning model using a number of audio and video information with category labels as a sample set.

2. The multi-modal sound object multi-classification method according to claim 1, characterized in that: The classification model includes an audio processing branch, a video processing branch, a convolutional layer and a multi-layer perceptron; wherein the audio processing branch includes a Mel-frequency cepstral coefficient layer, a gated recurrent unit, a Conformer module, a Sub layer, an EfficientNet module and a multi-layer perceptron; the video processing branch includes a time shift module, a global pooling network, a multi-layer perceptron layer, a Transformer module, a spatial pyramid pooling network, and a spatial attention module.

3. The multi-modal sound object multi-classification method according to claim 2, characterized in that: The inputting the audio information and the video information into a classification model for classification to obtain a classification result includes: Inputting the audio information and the video information into a classification model; Predicting noise characteristics of the audio information, removing noise, and generating an audio feature matrix; Extracting and enhancing the video temporal features of the video information, and extracting the spatial information to generate a video feature matrix; The audio feature matrix and the video feature matrix are fused and classified to obtain a classification result.

4. The multi-modal sound object multi-classification method according to claim 3, characterized in that: The step of predicting noise characteristics of the audio information, removing noise, and generating an audio feature matrix includes: Performing Fourier transform on the audio information and extracting audio features through Mel-frequency cepstral coefficients and a gated recurrent unit network; Input the audio features into the Conformer module for processing to obtain predicted noise features; Extracting audio features containing noise from the audio information through a convolution operation; Based on a SUB operation, subtracting the predicted noise feature from the audio feature containing noise to obtain an audio feature after noise removal; Inputting the de-noised audio features into an EfficientNet module to obtain a new feature representation, and concatenating the new feature representation with the de-noised audio features to obtain a concatenated feature; The concatenated features are converted into a format used by a matrix through a multi-layer perceptron to obtain an audio feature matrix.

5. The multi-modal sound object multi-classification method according to claim 3, characterized in that: The extracting and enhancing the video temporal features of the video information and extracting the spatial information to generate a video feature matrix includes: Segmenting the video information into picture frames and extracting time series features of the video using a time shift module network; Inputting the time series features into a global pooling network and connecting multiple layers of perceptrons in series to generate time series information; Extracting an intermediate frame from the video information as a static image, inputting it into a spatial pyramid pooling network, and extracting semantic information through a convolutional block attention mechanism to obtain spatial information; Inputting the temporal information and the spatial information into a spatial attention module to obtain a final spatial feature representation; The temporal information and the final spatial feature representation are combined to form a video feature matrix.

6. The multi-modal sound object multi-classification method according to claim 3, characterized in that: The fusing and classifying the audio feature matrix and the video feature matrix to obtain a classification result includes: The audio feature matrix and the video feature matrix are fused using a Transformer module to obtain fused features; The fused features are sequentially input into the convolution layer, multiple multi-layer perceptrons and softmax function for processing to obtain the classification result.

7. The multi-modal sound object multi-classification method according to claim 1, characterized in that: The loss function used in the classification model training process includes a function formed by weighted addition of three auxiliary loss functions, and the three auxiliary loss functions include the MSE loss for calculating the noise label, the MSE loss for calculating the denoising label, and the cross entropy loss of the classification result.

8. The multi-modal sound object multi-classification method according to claim 7, characterized in that: When training the classification model, a visual source is masked for a number of audio and video information with category labels, and an enhanced sample set is produced by combining different audio and noise data in a mosaic manner. The deep learning model is trained based on the enhanced sample set.

9. A multi-modal sound classification device, characterized in that: include: An acquisition unit, configured to acquire the audio and video of the sound object to be classified to obtain audio information and video information; a classification unit, configured to input the audio information and the video information into a classification model for classification to obtain a classification result; an output unit, configured to output the classification result; The classification model is obtained by training a deep learning model using a number of audio and video information with category labels as a sample set.

10. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

Images