A method and system for audio-video joint scene classification based on transformer

Abstract

The application belongs to the technical field of deep learning, and particularly relates to a Transformer network model for classification, and discloses a method for audio-video joint scene classification based on Transformer, which comprises the following steps: early fusion and feature extraction of noisy audio, video and audio-video are respectively performed by using a Transformer unit; a pre-training network on the video side is performed on the fused feature representation by using an EfficientNetV2_S unit; and the features obtained by weighted summation of audio features, video features and audio-video joint features are input into a classifier for scene classification. The application replaces the original attention mechanism with a Transformer structure, and replaces the original pre-training model from ResNet50 to EfficientNetV2_S, thereby improving the classification accuracy, improving the scene classification accuracy by using the Transformer unit, and improving the correlation between multi-modalities.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, and particularly relates to a Transformer network model for classification. BACKGROUND

[0002] Multi-modal machine learning aims to establish a model capable of processing and associating multi-modal information. Currently, common classification models include a classification model based on a ResNet network and an attention-based classification model. The ResNet network can obtain a deep network with good performance, but if a model with the best performance matching the limited hardware resources is to be obtained, three parameters need to be considered in coordination, namely, the resolution of the input image, the network depth and the network width. At this time, the ResNet network is helpless, that is, it is difficult to achieve optimal performance under limited hardware resources. The attention obtains the internal connection of the sequence by calculating the attention of the sequence itself and assigning different weight information to different elements. The attention model can well capture the characteristics and improve the classification accuracy. However, the current network using the attention model alone often has low classification accuracy and cannot exert the performance of the attention model.

[0003] The present application adopts a model for audio-video joint scene classification. For various chaotic urban scenes, the Transformer unit is used to extract features from the noisy audio and video respectively. After early fusion of the extracted features, the Transformer unit is used again for deep feature extraction. Finally, a learnable weight parameter is assigned to the audio feature, the video feature and the audio-video joint feature, and the features obtained by weighted summation are input into a classifier for scene classification. The present model improves the accuracy of scene classification and enhances the correlation between multi-modalities by using the Transformer unit. SUMMARY

[0004] This section is intended to summarize some aspects of the embodiments of the present application and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of the specification to avoid obscuring the purpose of this section, the abstract and the title, and such simplifications or omissions cannot be used to limit the scope of the present application.

[0005] In view of the above existing problems, the present application is proposed.

[0006] Therefore, a method for audio-video joint scene classification based on a Transformer is provided, which aims to replace the original attention mechanism with a Transformer structure and replace the original pre-training model with an EfficientNetV2_S, thereby improving the classification accuracy and enhancing the correlation between audio and video modalities.

[0007] To solve the above technical problems, the present application provides the following technical solutions, a method for audio-video joint scene classification based on a Transformer, comprising:

[0008] The Transformer unit is used to perform early fusion and feature extraction on noisy audio, video and audio-video respectively, the EfficientNetV2_S unit is used to perform a pre-training network on the video side for the fused feature representation, and the classification unit is used to perform weighted summation on the audio feature, the video feature and the audio-video joint feature, and then input the obtained feature into a classifier for scene classification.

[0009] As a preferred scheme of the method for audio-video joint scene classification based on a Transformer, the Transformer unit includes feature extraction on noisy audio and video respectively, and deep feature extraction is performed on the extracted features by using the Transformer unit again after early fusion.

[0010] The audio, video and audio-video feature extraction includes obtaining an audio hidden layer vector through an OpenL3 network, obtaining a video feature vector through an EfficientNetV2_S pre-training network, and inputting a multi-modal feature vector obtained after splicing the audio-video vector into the Transformer unit.

[0011] As a preferred scheme of the method for audio-video joint scene classification based on a Transformer, the Encoder layer includes an Encoder layer part of the Transformer unit, and the specific steps are as follows,

[0012] First, the audio signal is extracted from the original signal and input into the Encoder part of the Transformer unit to convert the input sequence into a high-dimensional representation, and the output result is introduced into the Add&Norm layer to perform the following operation:

[0013] LayerNorm(X+Multi-HeadAttention(X))

[0014] Secondly, the result input FeedForward part after the fast convergence of the Norm layer is a two-layer fully connected layer, the input of the first layer is x, the output is max(0, x) using the activation function Relu, when x is less than 0, the output is 0, otherwise the output is x, after entering the second layer, no activation function is used, the original value of the result is output, and then the Add&Norm layer is entered:

[0015] LayerNorm(X+FeedForward(X))

[0016] The output of the Encoder part is obtained, and is input into the Classification layer classification unit to complete the task.

[0017] As a preferred scheme of the audio-video joint scene classification method based on the Transformer, the Decoder layer includes using the Multi-HeadAttention layer to adopt the Masked operation, and using the K and V matrices of the Attention to obtain an output matrix.

[0018] The Masked operation includes shielding each feature after t time, setting the attention score to 0, the sequence length of the input is T, and the Mask matrix is represented as an upper triangular matrix M with a size of (T, T), wherein:

[0019] M[i, j] = 0, i <= j

[0020] M[i, j] = -inf, i > j

[0021] Where -inf is negative infinity, and i and j are indexes of sequence positions numbered from 1.

[0022] As a preferred scheme of the audio-video joint scene classification method based on the Transformer, the K and V matrices include the following steps: first, using the K and V matrices of the Attention to calculate the output matrix of the Encoder, calculating K and V according to the output matrix of the Encoder, and calculating Q according to the output of the previous Add&Norm layer of the Decoder.

[0023] The output matrix of the K and V matrices of the Encoder is represented as:

[0024] H e R n×d

[0025]

[0026]

[0027] where n denotes the sequence length, d denotes the vector dimension at each position, W k is a trainable weight matrix, d k is the dimension of Q, K, V vectors in self-attention mechanism, W v is a trainable weight matrix, d v is the dimension of V vector in self-attention mechanism.

[0028] Secondly, the output of the Multi-HeadAttention layer continues to circulate the Add&Norm layer, the FeedForward layer, the Add&Norm layer, and the Linear layer, and the final output OutputProbabilities is obtained by the Softmax layer and input into the Classificationlayer classification unit to complete the task.

[0029] As a preferred scheme of the audio-video joint scene classification method based on the Transformer, the EfficientNetV2_S unit includes the following steps,

[0030] Firstly, a pre-training network on the video side is performed, secondly, the pre-training network is divided into Stage0 to Stage7, and Operator represents a module used in the current stage. Firstly, video side data is received and input into Stage0 to obtain a feature representation TO of the first frame image. Secondly, the TO is processed by the Operator in Stage1 to obtain a high-level feature representation T1 of the first frame image. Thirdly, the T1 is processed by the Operator in Stage2 to obtain a higher-level feature representation T2 of the first frame image with more output channel numbers. For each subsequent frame image, the Operator in Stage3 to Stage7 is sequentially processed to obtain feature representations of different abstraction levels T3-T7. Finally, T0-T7 of each step distance Stride are spliced together to obtain a three-dimensional tensor, the first dimension is the time step, and the last two dimensions are the feature representation. The three-dimensional tensor is sent into a Transformer sequence model for processing, and the output result OutputProbabilities obtained by the Softmax layer is sent into a Classificationlayer classification unit for classification.

[0031] As a preferred scheme of the audio-video joint scene classification method based on the Transformer, wherein: the classification unit includes, the pre-training network set on the video side of the EfficientNetV2_S unit classifies different videos into the Classification layer classification unit, and the features are divided into A1-A10, the feature mean of each obtained category is taken as the feature representation of this category, and the clustering algorithm K-Means is used to cluster the features into three clusters, which respectively represent indoor B1, outdoor B2 and traffic places B3.

[0032] When the video classification is A1 and is divided into the B1 category, the feature vector A1 is allocated to the cluster representing the B1 scene category, and the center of the B1 cluster is selected as the final classification result, and the nearest category of the cluster center is selected as the final scene classification result.

[0033] Another object of the present application is to provide an audio-video joint scene classification system based on the Transformer, which replaces the original attention mechanism with the Transformer structure and replaces the original pre-training model from ResNet50 to EfficientNetV2_S, thereby improving the classification accuracy and improving the correlation between audio and video modalities.

[0034] An audio-video joint scene classification system based on the Transformer, characterized by comprising a data preprocessing module, a feature extraction and fusion module, a model training module, and a scene classification module.

[0035] The data preprocessing module pre-processes the audio and video data, including data cleaning and data format conversion.

[0036] The feature extraction and fusion module includes a Transformer unit for extracting features of audio and video, and also for secondary feature extraction on the early fused features.

[0037] The model training module includes an EfficientNetV2_S unit for pre-training network on the video side.

[0038] The scene classification module classifies the network features by using a Classification layer classification unit.

[0039] A computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the audio-video joint scene classification method based on the Transformer when executing the computer program.

[0040] A computer readable storage medium, wherein a computer program is stored, and when the computer program is executed by a processor, steps of a method for audio-video joint scene classification based on a Transformer are implemented.

[0041] The application provides a model for audio-video joint scene classification, for various chaotic urban scenes, the noisy audio and video are respectively extracted by a Transformer unit, early fusion is performed, deep feature extraction is performed by a Transformer unit, audio features, video features and audio-video joint features are weighted and summed to obtain features, and the features are input into a classifier for scene classification. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor under the premise of the drawings.

[0043] Figure 1 A working method flowchart of a method for audio-video joint scene classification based on a Transformer is provided for an embodiment of the application.

[0044] Figure 2 A running example of a method for audio-video joint scene classification based on a Transformer is provided for an embodiment of the application.

[0045] Figure 3 A standard version of a Transformer network structure of a method for audio-video joint scene classification based on a Transformer is provided for an embodiment of the application.

[0046] Figure 4 A network structure of an EfficientNetV2_S unit of a method for audio-video joint scene classification based on a Transformer is provided for an embodiment of the application.

[0047] Figure 5 An audio output result of a method for audio-video joint scene classification based on a Transformer is provided for an embodiment of the application.

[0048] Figure 6The concat output result of the audio-video joint scene classification method based on the Transformer provided by one embodiment of the present application.

[0049] Figure 7 The video output result of the audio-video joint scene classification method based on the Transformer provided by one embodiment of the present application.

[0050] Figure 8 The working flow diagram of the audio-video joint scene classification system based on the Transformer provided by one embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the above objectives, features and advantages of the present application more apparent, clear and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work should belong to the protection scope of the present application.

[0052] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the concept of the present application, so the present application is not limited to the specific embodiments disclosed below.

[0053] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. The "in one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is the embodiment independent or selectively excluded from other embodiments.

[0054] The present application is described in detail in conjunction with the schematic diagram, and in the detailed description of the embodiments of the present application, the cross-sectional view of the device structure is locally enlarged without the general proportion for the convenience of description, and the schematic diagram is only an example, which should not limit the scope of protection of the present application herein. In addition, the three-dimensional spatial dimensions of length, width and depth should be included in actual manufacturing.

[0055] Meanwhile, in the description of the present application, it should be noted that the terms "upper, lower, inner and outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first, second or third" are only for the purpose of description and cannot be understood as indicating or implying relative importance.

[0056] Unless otherwise expressly specified and limited, the terms "mounting, connecting, connecting" in the present application should be understood broadly, for example: it can be fixed connection, detachable connection or integral connection; It can also be mechanical connection, electrical connection or direct connection, it can also be indirectly connected through intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0057] Embodiment 1

[0058] Reference Figures 1-4 For the first embodiment of the present application, the embodiment provides a method for audio and video joint scene classification based on Transformer, comprising:

[0059] S1: Early fusion and feature extraction are performed on noisy audio, video and audio-video respectively by using a Transformer unit.

[0060] Further, the audio hidden layer vector obtained by the OpenL3 network, the video feature vector obtained by the EfficientNetV2_S pre-training network, and the multi-modal feature vector obtained by splicing the audio and video vectors are input into the Transformer unit, and then enter the Multi-HeadAttention part. The Transformer unit is used three times in the model, the video side is set to double-head self-attention mechanism, the Encoder and Decoder parts are repeated once, the audio side is set to three-head self-attention mechanism, the Encoder and Decoder parts are repeated once, and the fused Transformer unit is set to double-head self-attention mechanism, and the Encoder and Decoder parts are repeated once.

[0061] It should be noted that the three feature vectors enter the Encoder part and enter the Add&Norm layer, which is composed of Add and Norm two parts, and the specific steps are as follows:

[0062] The audio signal is extracted from the original signal and input into the Encoder part of the Transformer unit for input sequence conversion to a high-dimensional representation. The output result is introduced into the Add&Norm layer, and the operation is:

[0063] LayerNorm(X+Multi-HeadAttention(X))

[0064] Add refers to X+MultiHeadAttention(X), which is a residual connection to solve the problem of multi-layer network training, allowing the network to only focus on the difference between the current part; Norm refers to LayerNormalization, which converts the input of each layer of neurons to the same mean and variance to speed up convergence.

[0065] After the Norm layer converges quickly, the result is input into the FeedForward part, which is a two-layer fully connected layer. The input of the first layer is x, and the activation function is Relu, which outputs max(0, x). When x is less than 0, the output is 0, otherwise the output is x. After entering the second layer, no activation function is used, and the original value of the result is output and then looped into the Add&Norm layer:

[0066] LayerNorm(X+FeedForward(X))

[0067] The output of the Encoder part is obtained and input into the Classificationlayer classification unit to complete the task.

[0068] It should also be noted that the first layer Multi-HeadAttention layer of the Decoder part of the Transformer unit uses Masked operation. The Mask matrix blocks each feature after t time, sets the attention score to 0, and the input sequence length is T. The Mask matrix is represented as an upper triangular matrix M with a size of (T, T), where:

[0069] M[i, j] = 0, i ≤ j

[0070] M[i, j] = -inf, i > j

[0071] Where -inf is negative infinity, and i and j are the indices of the sequence positions, numbered from 1.

[0072] It should also be noted that after the Masked operation, the K and V matrices of the Attention in the Multi-HeadAttention layer are calculated using the output matrix of the Encoder, K and V are calculated according to the output matrix of the Encoder, and Q is calculated according to the output of the previous Add&Norm layer of the Decoder;

[0073] The output matrix of the K and V matrices of the Encoder is represented as:

[0074] H∈R n×d

[0075]

[0076]

[0077] where n represents the sequence length, d represents the vector dimension of each position, W k is a trainable weight matrix, d k is the dimension of the Q, K, and V vectors in the self-attention mechanism, W v is a trainable weight matrix, d v is the dimension of the V vector in the self-attention mechanism;

[0078] After the output of the Multi-HeadAttention layer, the Add&Norm layer, the FeedForward layer, the Add&Norm layer, and the Linear layer are continued to loop, and the final output OutputProbabilities is obtained by the Softmax layer and input into the Classification layer classification unit to complete the task.

[0079] S2: The fused feature representation is input into the video-side pre-training network using the EfficientNetV2_S unit.

[0080] Further, the pre-training network is as follows: a pre-training network on the video side; the pre-training network is divided into Stage0 to Stage7 by eight layers, Operator represents a module used in the current Stage, first receives video side data and inputs into Stage0 to obtain a feature representation S0 of the first frame image; S0 is processed by the Operator in Stage1 to obtain a high-level feature representation S1 of the first frame image, S1 is processed by the Operator in Stage2 to obtain a higher-level feature representation S2 of the first frame image, and there are more output channels; for each subsequent frame image, the Operators in Stage3 to Stage7 are sequentially processed to obtain feature representations S3-S7 of different abstraction levels; S0-S7 of each step distance Stride are spliced together to obtain a three-dimensional tensor, the first dimension is the time step, and the last two dimensions are the feature representation; the three-dimensional tensor is sent into a Transformer sequence model for processing, and the output result OutputProbabilities obtained by a Softmax layer is sent into a Classification layer classification unit for classification.

[0081] It should be noted that, as shown in Figure 3 Stage0 is a Conv3x3 module, which is a common 3x3 convolution + activation function + BN; Stage1 to Stage3 are Fused-MBConv modules, and the 1 and 4 followed by the module name represent the expansion ratio, and k3x3 represents the kernel size of 3x3; Stage4 to Stage6 are MBConv modules, and the 4 and 6 followed by the module name represent the expansion ratio, and SE0.25 represents that an SE module is used, and 0.25 represents that the number of nodes of the first fully connected layer in the SE module is 1 / 4 of the channels of the feature matrix input into the MBConv module; Stage7 is composed of Conv1x1, Pooling and fully connected FC; Stride represents the step distance, and the Operator module is repeatedly stacked in each Stage, and the step distance of only the first Operator module in each Stage is set as Stride, and the Stride of the other repeated Operator modules is 1 by default; Channels represents the Channels of the feature matrix output by the Stage, and Layers represents the number of times of repeating the stacking of the Operator in the Stage.

[0082] S3: input the feature obtained by weighted sum of the audio feature, the video feature and the audio-video joint feature into the classifier by using the classification unit for scene classification.

[0083] Further, the pre-training network set on the video side of the EfficientNetV2_S unit classifies different videos into the Classification layer classification unit, divides the features into ten categories A1-A10, takes the mean of the features of each category as the feature representation of this category, and uses the clustering algorithm K-Means to cluster the features into three clusters, representing indoor B1, outdoor B2, and traffic site B3 respectively:

[0084] When the video classification is selected as one of A1-A10 and divided into the B1 category, the feature vector is assigned to the cluster representing the B1 scene category, and the center of the B1 cluster is selected as the final classification result, and the nearest category of the cluster center is selected as the final scene classification result.

[0085] When the audio classification is selected as one of A1-A10 and divided into the B2 category, the feature vector is assigned to the cluster representing the B2 scene category, and the center of the B2 cluster is selected as the final classification result, and the nearest category of the cluster center is selected as the final scene classification result.

[0086] When the audio-video classification is selected as one of A1-A10 and divided into the B3 category, the feature vector is assigned to the cluster representing the B3 scene category, and the center of the B3 cluster is selected as the final classification result, and the nearest category of the cluster center is selected as the final scene classification result.

[0087] Embodiment 2

[0088] Reference Figures 5-7 For an embodiment of the present application, a method for audio-video joint scene classification based on Transformer is provided. In order to verify the beneficial effects of the present application, scientific demonstration is carried out through experiments.

[0089] The traditional method is: S1: using attention mechanism for feature extraction and early fusion, the original pre-training network model uses ResNet50 to preprocess the audio-video data, including image cropping, scaling, and normalization, and audio sampling and preprocessing operations.

[0090] S2: using convolutional neural network (CNN) to extract features from images, and using short-time Fourier transform (STFT) method to extract features from audio. These methods can capture important information in the original data.

[0091] S3: Generally, the features of the image and the audio are extracted independently, and then the two are fused. The traditional method is to fuse the feature vectors of the two modalities by weighted summation or direct splicing. This simple fusion method ignores the relationship between the two modalities, which may cause information loss. The fused feature vector is input into a classifier for classification, and algorithms such as support vector machine (SVM), random forest (RandomForest) or artificial neural network are usually used.

[0092] The present application uses a transformer unit to extract features from noisy audio and video respectively, and then uses the transformer unit again for deep feature extraction after early fusion of the extracted features. Finally, a learnable weight parameter is assigned to the audio feature, the video feature and the audio-video joint feature, and the features obtained by weighted summation of the three are input into a classifier for scene classification. Through no less than 10 test experiments on audio, video and audio-video, the result model is as shown in Figure 5 、 Figure 6 、 Figure 7 It can be seen that:

[0093] Classification accuracy Conventional benchmark model 90.26％ Transformer architecture model 91.34％

[0094] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the present application, and they should be covered in the scope of the claims of the present application.

[0095] Embodiment 3

[0096] The third embodiment of the present application is different from the first two embodiments in that:

[0097] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application or the parts that essentially contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes instructions for making a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, etc. various program code storage media.

[0098] The logic and / or steps represented in flow diagrams or otherwise described herein, for example, can be considered as a sequence of executable instructions for implementing logic functions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or a combination of them. The "computer-readable medium" can be any media that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electronic), a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber (optical), and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example via the optical scanner of a device, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and stored in a computer memory.

[0099] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electronic), a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber (optical), and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example via the optical scanner of a device, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and stored in a computer memory.

[0100] It should be understood that portions of the present application can be implemented in hardware, software, firmware, or combinations thereof. In the above embodiments, a number of steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and as in another embodiment, it can be implemented using any or a combination of the following technologies, which are well known in the art: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0101] Embodiment 4

[0102] Reference Figure 8 For the fourth embodiment of the present application, the embodiment provides an audio and video joint scene classification system based on Transformer, which includes a data preprocessing module, a feature extraction and fusion module, a model training module, and a scene classification module.

[0103] The data preprocessing module pre-processes the audio and video data, including data cleaning and data format conversion.

[0104] The feature extraction and fusion module includes a Transformer unit for extracting features of audio and video, and also for secondary feature extraction on the early fused features.

[0105] The model training module includes an EfficientNetV2_S unit for a pre-training network on the video side.

[0106] The scene classification module classifies the network features using a Classification layer classification unit.

[0107] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application, and they should be covered in the scope of the claims of the present application.

Claims

1. A method for audio-video joint scene classification based on a Transformer, characterized in that: The application relates to a scene classification method and device based on audio, video and audio-video fusion. The noisy audio, video and audio-video are respectively subjected to early fusion and feature extraction by utilizing a Transformer unit in an Encoder layer and a Decoder layer; The fused feature representation is subjected to a video-side pre-training network by utilizing an EfficientNetV2_S unit; The audio feature, video feature and audio-video joint feature are weighted and summed to obtain a feature which is input into a classifier for scene classification by utilizing a classification unit; The Transformer unit comprises the following steps of extracting features from the noisy audio and video, and performing deep feature extraction on the extracted features by utilizing the Transformer unit again after early fusion. The audio, video and audio-video feature extraction comprises the following steps of obtaining an audio hidden layer vector by utilizing an OpenL3 network, obtaining a video feature vector by utilizing an EfficientNetV2_S pre-training network, and inputting a multi-modal feature vector obtained by splicing the audio-video vector into the Transformer unit to enter a Multi-HeadAttention part; The EfficientNetV2_S unit comprises the following steps, Firstly, a video-side pre-training network is performed, and secondly, the pre-training network is divided into Stage0 to Stage7, and Operator represents a module used in the current stage, the video-side data is first received and input into Stage0 to obtain a feature representation T0 of a first frame image, T0 is processed by the Operator in Stage1 to obtain a high-level feature representation T1 of the first frame image, T1 is processed by the Operator in Stage2 to obtain a higher-level feature representation T2 of the first frame image, and the number of output channels is more, for each frame image, the Operator in Stage3 to Stage7 is sequentially processed to obtain feature representations T3-T7 of different abstraction levels, finally, T0-T7 of each step distance Stride are spliced together to obtain a three-dimensional tensor, the first dimension is a time step, and the last two dimensions are feature representations, the three-dimensional tensor is input into a Transformer sequence model for processing, and an output result OutputProbabilities obtained by utilizing a Softmax layer is input into a Classification layer classification unit for classification.

2. The method for audio-visual joint scene classification based on Transformer according to claim 1, wherein: The Encoder layer comprises an Encoder layer part of a Transformer unit, and the specific steps are as follows, Firstly, an audio signal is extracted from an original signal, input sequence conversion into a high-dimensional representation is performed by utilizing the Encoder part of the Transformer unit, and the output result is introduced into an Add&Norm layer to perform the following operation: LayerNorm(X+Multi-HeadAttention(X)) Secondly, the result input FeedForward part after the fast convergence of the Norm layer is a two-layer fully connected layer, the input of the first layer is x, the output is max(0, x) using the activation function Relu, when x is less than 0, the output is 0, otherwise the output is x, after entering the second layer, no activation function is used, the original value of the result is output, and then the cycle enters the Add&Norm layer: LayerNorm(X+FeedForward(X)) The output of the Encoder part is obtained and input into the Classificationlayer classification unit to complete the task.

3. The method of claim 2, wherein the method is based on a Transformer for audio-visual joint scene classification. The Decoder layer includes using the Multi-HeadAttention layer to adopt the Masked operation, and using the K and V matrices of Attention to obtain the output matrix. The Masked operation includes shielding each feature after t time, setting the attention score to 0, the input sequence length is T, and the Mask matrix is represented as an upper triangular matrix M with a size of (T, T), wherein: M[i, j] = 0, i <= j M[i, j] = -inf, i > j Where -inf is negative infinity, and i and j are sequence position indexes numbered from 1.

4. The method of claim 3, wherein the method is based on a Transformer. The K and V matrices include the following steps: first, using the K and V matrices of Attention to calculate the output matrix of the Encoder, K and V are calculated according to the output matrix of the Encoder, and Q is calculated according to the output of the previous Add&Norm layer of the Decoder; The output matrix of the K and V matrices of the Encoder is represented as: H e R n×d where n denotes the sequence length, d denotes the vector dimension at each position, W k is a trainable weight matrix, d k is the dimension of Q, K, V vectors in the self-attention mechanism, W v is a trainable weight matrix, d v is the dimension of V vectors in the self-attention mechanism; Secondly, the Multi-HeadAttention layer output continues to cycle the Add&Norm layer, the FeedForward layer, the Add&Norm layer, and the Linear layer, and the final output OutputProbabilities is obtained by the Softmax layer and input into the Classificationlayer classification unit to complete the task.

5. The method of claim 4, wherein the method is based on a Transformer for audio-visual joint scene classification. The classification unit includes inputting different videos into the Classificationlayer classification unit through the pre-training network set on the video side of the EfficientNetV2_S unit, classifying the features into A1-A10, taking the average of the features of each category as the feature representation of this category, and using the clustering algorithm K-Means to cluster the features into three clusters, representing indoor B1, outdoor B2 and traffic place B3 respectively; When the video classification is A1, which is divided into the B1 category, the feature vector A1 is assigned to the cluster representing the B1 scene category, and the center of the B1 cluster is selected as the final classification result, and the nearest category of the cluster center is selected as the final scene classification result.

6. A system for audio-visual joint scene classification using the method according to any one of claims 1-5. It includes a data preprocessing module, a feature extraction and fusion module, a model training module, and a scene classification module. The data preprocessing module preprocesses audio and video data, including data cleaning and data format conversion. The feature extraction and fusion module includes a Transformer unit for extracting features of audio and video, and also for secondary feature extraction on the early fused features; The model training module includes an EfficientNetV2_S unit for a pre-training network on the video side. The scene classification module uses a Classification layer classification unit to classify network features. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-6 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 5.

Images