A multimodal scene content understanding system carried by drones

Abstract

A multimodal scene content understanding system, mounted on an unmanned aerial vehicle (UAV), relates to the fields of deep learning and multimodal content understanding. The system comprises a sequentially connected signal acquisition unit, a data sampling and processing unit, a spatiotemporal feature extraction unit, a bimodal feature fusion unit, and a scene content classification unit. The signal acquisition unit is used to acquire audio and video data; the data sampling and processing unit is used to process the audio data to form a spectrogram and perform frame sampling on the video data; the spatiotemporal feature extraction unit is used to extract features from the continuous spectrograms of the audio modality and to extract spatiotemporal features from the continuous frames of the video modality. The entire system combines audio and video to understand scene content. Compared to single-modal or single-frame plus logic system designs, the present invention can still output robust conclusions when single-modal information is insufficient, and its inter-frame variation and motion feature extraction capabilities can understand more dynamic scene content types.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and multimodal content understanding, and in particular to a multimodal scene content understanding system carried by an unmanned aerial vehicle. Background Art

[0002] With the continuous innovation of deep learning algorithms and the continued growth of chip computing power, the trend of adapting server-side algorithms to lighter-weight edge devices is becoming increasingly apparent, especially on drones. However, existing algorithms installed on drones are typically classification, detection, and other algorithms applied to single frames. Determining the state of an object of interest within a given timeframe can only be achieved through manually designed rules. Single-frame algorithms, due to their difficulty utilizing inter-frame information, are unable to determine the target's motion characteristics, resulting in inaccurate understanding of scene content.

[0003] Audio information is also an effective complement to models, especially when image modality is insufficient to identify scene content, such as in dense smoke or when the target is obscured by obstacles in search and rescue scenarios. Audio can play a key role in effectively determining the scene's condition based solely on images. However, due to the difficulties of drones collecting noise-free audio and algorithms fusing multimodal inputs, audio modality is often overlooked.

[0004] Therefore, existing airborne algorithms have the problem of inaccurate scene content judgment due to their difficulty in fully utilizing audio and video information. Summary of the Invention

[0005] To address the above problems, the present invention provides a multimodal scene content understanding system carried by an unmanned aerial vehicle.

[0006] The technical solution of the present invention is: a multimodal scene content understanding system carried by an unmanned aerial vehicle, comprising a signal acquisition unit, a data sampling and processing unit, a spatiotemporal feature extraction unit, a bimodal feature fusion unit and a scene content classification unit connected in sequence.

[0007] The signal acquisition unit is used to collect audio data and video data;

[0008] The data sampling and processing unit is used to process the audio data to form a spectrogram and perform frame sampling on the video data;

[0009] The spatiotemporal feature extraction unit is used to extract features from continuous spectrograms of the audio modality and to extract spatiotemporal features from continuous frames of the video modality;

[0010] The dual-modal feature fusion unit is used to fuse the features extracted by the spatiotemporal feature extraction unit;

[0011] The scene content classification unit is used to perform content classification on the output feature vector of the bimodal feature fusion unit, obtain the category number corresponding to the feature vector, and map it to the content category to which the original audio and video belong.

[0012] The signal acquisition unit includes:

[0013] Two microphones are used to collect audio data. One microphone is hung under the drone, away from the fuselage, to collect scene sounds. The other microphone is fixed near the bottom of the fuselage to collect ambient noise generated by the rotor.

[0014] Video data uses a visible light camera to obtain a continuous video stream.

[0015] The data sampling and processing unit includes:

[0016] Audio processing: The time-domain audio is windowed and then subjected to a short-time Fourier transform to obtain a spectrum. A Mel filter bank is then used to calculate a stable spectrum. The frames are grouped into frames of a certain length and stacked in chronological order in the batch dimension to obtain a multi-batch single-channel four-dimensional array. Scene sounds and ambient noise are processed separately using the above method. The two sets of results are stacked in chronological order in the channel dimension to form a multi-batch dual-channel four-dimensional array as the model input.

[0017] Video processing: Capture video stream data through the camera, convert the video stream into a frame sequence, sample T frames at a certain interval, scale them to the appropriate size and stack them in the batch dimension as the input of the model.

[0018] The spatiotemporal feature extraction unit includes:

[0019] The audio and video modalities have independent feature extraction branch networks. Each feature extraction branch network is based on the backbone network of the general classification or detection model composed of convolution, and an adaptive time domain attention module is inserted. On the basis of the original network extracting spatial dimensional features, the model is enhanced with the ability to extract temporal variation features. The two modalities output their own one-dimensional feature vectors, which represent the global and local features of the continuous input of each modality in the time and space dimensions.

[0020] The adaptive time domain attention module includes a time dimension adaptive local feature extraction submodule and a time dimension global attention submodule. The arrays or tensors input to the two modules are first dimensionally transformed to split the time series of the same sample from the batch dimension, from [B, C, H, W] to [B / T, T, C, H, W], which is used to extract time dimension features within the sample.

[0021] The time dimension adaptive local feature extraction submodule includes:

[0022] In each channel, global pooling is first performed on the spatial dimension, followed by a one-dimensional convolution on the temporal dimension. A one-dimensional BatchNorm layer and a ReLU activation layer are used to extract features between adjacent moments. The fully connected layer is then used to reduce the dimensionality. Finally, a Sigmoid layer is used to adjust the value range and generate an adaptive convolution kernel, which serves as the weight for subsequent 2D convolution. The value of the convolution kernel comes from the input itself. Different inputs will generate different convolution kernels, making the model weights adaptive to the input.

[0023] The temporal global attention submodule extracts feature changes in continuous time by subtracting spatial feature maps of adjacent times, thereby capturing high-dimensional motion information of the target or shot.

[0024] Specifically, the minuend branch goes from t1 to tn in the time dimension, the subtrahend branch passes through one more 2D convolution layer, and then moves t1 to the end of the time dimension to form the order of t2, t3,.., tn, t1. The feature maps of the corresponding positions of the two branches are subtracted to obtain the difference feature map of the spatial dimension. After global pooling and Softmax, the weight of the time dimension is obtained, which is multiplied by the input and then added to the input and output.

[0025] The output of the time-dimension global attention submodule is subjected to 2D convolution using the convolution kernel obtained from the time-dimension adaptive local feature extraction submodule to complete the extraction of global and local features.

[0026] The adaptive temporal attention module is a plug-in module that can be applied anywhere in the backbone network of most 2D convolutional feature extraction models.

[0027] To extract features from continuous spectrograms, insert an adaptive temporal attention module into each module of any pre-trained audio classification model composed of 2D convolutions that uses feature spectrograms as input (such as the VGGish model). After inserting the adaptive temporal attention module, the model needs to be retrained.

[0028] Extract spatiotemporal features from consecutive frames,

[0029] An adaptive temporal attention module is inserted into each submodule of the backbone network of any pre-trained image classification or detection model composed of 2D convolution (such as ResNet50). After inserting the adaptive temporal attention module, the model needs to be retrained on the target task.

[0030] The bimodal feature fusion unit includes:

[0031] After adding position embedding to the one-dimensional feature vectors extracted from each of the two modalities, the features between the two vectors are extracted and fused through a multi-head self-attention module. The two vectors are then concatenated into one vector, and the features are extracted again through a feedforward network and the vector dimension is adjusted. The one-dimensional vectors of each of the two modalities are then fused to finally output a one-dimensional feature vector. This feature vector covers the global and local spatiotemporal features of the audio and video streams, so it can be used to understand the video content.

[0032] The scene content classification unit includes:

[0033] Through the classification layer, content classification is performed based on the one-dimensional feature vector output by the bimodal feature fusion unit to obtain the content category to which the audio and video to be classified belongs.

[0034] Use a fully connected layer to adjust the dimension of the feature vector to the same number of classification categories, then connect it to the Softmax layer to select the category with the highest score and output it to achieve single classification;

[0035] or

[0036] After using the fully connected layer to adjust the dimension of the feature vector to the same number of classification categories, the Sigmoid layer is connected to select categories with scores higher than the threshold to achieve multi-label.

[0037] During operation, the present invention can reduce the interference of the drone's own noise during the audio collection process through the design of dual-channel audio collection and dual-channel spectrogram, and obtain robust audio classification results. The proposed adaptive time domain attention module is a plug-in design and can be easily inserted into various common ordinary image convolutional networks to make up for the feature extraction capability of the time dimension, so as to capture the changing features and motion features in continuous sequences. The application of audio and video modalities has achieved improved classification accuracy compared to existing algorithms. The entire system combines audio and video to complete the understanding of scene content. Compared with the system design of single modality or single frame plus logic, the present invention can still output robust conclusions when single modality information is insufficient, and with its inter-frame change and motion feature extraction capabilities, it can understand more dynamic scene content types. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a system block diagram of the present invention,

[0039] Figure 2 This is a flowchart of the multimodal scene content understanding algorithm.

[0040] Figure 3 This is a schematic diagram of two audio signals converted into spectrum diagrams and stacked in the channel dimension.

[0041] Figure 4This is a flowchart of dual-stream spatiotemporal feature extraction, fusion and classification.

[0042] Figure 5 This is the flowchart of the adaptive temporal domain attention module.

[0043] Figure 6 Flowchart for adaptive convolution kernel generation in the time-dimensional adaptive local feature extraction submodule.

[0044] Figure 7 Flowchart for the calculation of attention weights for the time-dimensional global attention submodule.

[0045] Figure 8 Schematic diagram of how to insert the adaptive temporal attention module into the core modules of VGGish, ResNet, and YOLOv5;

[0046] Figure 9 This is the model structure diagram of the bimodal feature fusion unit. DETAILED DESCRIPTION

[0047] The present invention Figure 1 As shown, a multimodal scene content understanding system carried by a drone is provided, comprising a signal acquisition unit, a data sampling and processing unit, a spatiotemporal feature extraction unit, a bimodal feature fusion unit and a scene content classification unit connected in sequence.

[0048] The signal acquisition unit is used to collect audio data and video data;

[0049] The data sampling and processing unit is used to process the audio data to form a spectrogram and perform frame sampling on the video data;

[0050] The spatiotemporal feature extraction unit is used to extract features from continuous spectrograms of the audio modality and to extract spatiotemporal features from continuous frames of the video modality;

[0051] The dual-modal feature fusion unit is used to fuse the features extracted by the spatiotemporal feature extraction unit;

[0052] The scene content classification unit is used to perform content classification on the output feature vector of the bimodal feature fusion unit, obtain the category number corresponding to the feature vector, and map it to the content category to which the original audio and video belong.

[0053] like Figure 2 As shown, specific applications include:

[0054] S100, signal acquisition unit:

[0055] For example, the S101 uses two microphones to collect two audio signals. One microphone is suspended under the drone, away from the fuselage, for collecting scene sounds. A soft data cable can be used for suspension, and the motor drives the shaft to rotate, control the suspension length, and retract the device. It can also be fixed with a hard cable, as long as it can be away from the drone body and closer to the ground target, to capture as much ground scene sound as possible and reduce the interference of the drone's own noise. The second microphone is fixed near the bottom of the fuselage to collect the ambient noise emitted by the rotor when it is working, which is used as input to the model for algorithm noise reduction.

[0056] S102, video data acquisition, uses a visible light camera to obtain a continuous video stream.

[0057] S200, data sampling and processing unit: frequency sampling of audio data to obtain spectrum data; frame sampling of video data to reduce redundant information.

[0058] S103, a specific implementation of audio processing is as follows. The same processing method is used for scene sound and environmental noise. Taking the processing of scene sound as an example, first, a 16s audio stream is intercepted according to the timestamp and stored as a .wav format file locally. After loading it into the memory, the audio is resampled to 16kHz mono audio. A 25ms Hann time window and a 10ms frame shift are used to perform short-time Fourier transform on the audio to obtain a spectrogram. The stable logarithmic Mel spectrum is calculated by mapping the spectrogram to a 64-order Mel filter bank; then these features are framed with a duration of 1s, and there is no frame overlap. Each frame contains 64 Mel frequency bands with a duration of 10ms, for a total of 100 frames, stacking a 100*64 two-dimensional array. The 16s audio is processed and stacked into a 16*1*100*64 four-dimensional array; the environmental noise is processed in the same way into an array of the same dimension, and a 16*2*100*64 four-dimensional array is obtained by splicing in the channel dimension as the input of the audio branch model. Figure 3 The above-mentioned audio capture time, windowing range, and number of recorder groups can be adjusted according to the actual scenario. This is just an example.

[0059] S104, a specific implementation of video processing is as follows: the video stream data captured by the camera is converted into a video frame sequence through Opencv or ffmpeg tools, and is evenly sampled according to the time interval or frame number interval. For example, a 16-second video is sampled at one frame per second, and a total of 16 frames of images are extracted.

[0060] S300, spatiotemporal feature extraction unit: Figure 4As shown in the figure, a specific model structure diagram for dual-stream spatiotemporal feature extraction is described. The dual streams refer to audio stream and video stream. The two streams use different model structures to extract spatiotemporal features between continuous audio spectra and video frames respectively. Both use the same adaptive time domain attention module, which is inserted into each sub-module of the feature extraction backbone network of the two streams.

[0061] The adaptive temporal attention module consists of two parts: the temporal adaptive local feature extraction submodule and the temporal global attention submodule. The two submodules constitute the adaptive temporal attention module as follows: Figure 5 As shown in the figure, the input of the entire module is replicated into four copies: one for the temporal adaptive local feature extraction submodule and three for the temporal global attention submodule. The temporal adaptive local feature extraction submodule generates an adaptive convolution kernel; the temporal global attention submodule extracts attention weights from the temporal dimension of one input and applies these weights to all elements of the corresponding temporal dimension of the second input by multiplication. The result is then added to the third input at the corresponding position as the output of this submodule. The final part of the entire module is a 2D convolution. The input of the convolution is the output of the temporal global attention submodule, and the convolution kernel of this convolution is the output of the temporal adaptive local feature extraction submodule. The result of this 2D convolution serves as the output of the entire module.

[0062] Time dimension adaptive local feature extraction submodule, such as Figure 6 The flowchart for adaptive convolution kernel generation in the temporal adaptive local feature extraction submodule shows this. This submodule's input is the input of the entire adaptive temporal attention module, and its output serves as the convolution kernel for the next convolution. One implementation involves performing global pooling on the spatial dimension within each channel. A one-dimensional convolution with a kernel size of 5 is then performed on the temporal dimension. This convolution is then passed through a one-dimensional BatchNorm layer and a ReLU activation layer to extract features between adjacent moments. A fully connected layer reduces the dimensionality to 5, and a Sigmoid layer adjusts the value range to generate a 5x1 convolution kernel, which serves as the weight for the subsequent 2D convolution. The kernel value is derived from the input itself; different inputs generate different kernels, enabling model weights to adapt to the input. The kernel size for the one-dimensional convolution can be any value smaller than the temporal dimension. A larger kernel size increases the receptive field and correspondingly increases the computational effort. The fully connected layer reduces the dimensionality to the size of the output kernel. The target kernel size can be any value; it is controlled by the output dimension of the fully connected layer.

[0063] The temporal global attention submodule, such as Figure 7As shown in the flowchart of the attention weight calculation of the global attention submodule in the time dimension, by subtracting the spatial dimension feature maps of adjacent times, the feature changes in continuous time are extracted, and the high-dimensional motion information of the target or shot is captured. A specific implementation method is that the minuend branch goes from t1 to tn in the time dimension, and the subtrahend branch passes through one more 2D convolution layer, and then moves t1 to the end of the time dimension to form the order of t2, t3,.., tn, t1. The feature maps of the corresponding positions of the two branches are subtracted to obtain the difference feature map of the spatial dimension. After global pooling and Softmax, the weight of the time dimension is obtained. After broadcasting and replicating the weight in the spatial dimension, the weight is multiplied with the module input, added to the input, and output.

[0064] S105, extract features from the continuous spectrogram by constructing a model with an adaptive time domain attention module. One implementation method is to insert an adaptive time domain attention module in the middle of each module of the pre-trained VGGish model, such as Figure 8 As shown in (a). After inserting the adaptive temporal attention module, the model needs to be retrained.

[0065] S106, extracting spatiotemporal features from consecutive frames and constructing a model with an adaptive temporal attention module. One implementation method is to insert an adaptive temporal attention module into each module of the pre-trained ResNet50, such as Figure 8 As shown in (b), inserting the module at a location with fewer channels can reduce computational complexity and improve runtime speed. The model needs to be retrained after inserting the adaptive temporal attention module.

[0066] S106, another implementation method is to insert an adaptive time domain attention module into each CSP module of the pre-trained detection model YOLOv5, such as Figure 8 As shown in (c), compared with the classification model, the backbone network of the detection model will have additional target position features when extracting features, which is beneficial to the training of downstream video scene content understanding tasks. After inserting the adaptive temporal domain attention module, the model needs to be retrained on the target task.

[0067] The present invention, in application, includes:

[0068] Establish a database and model training. The training database includes two channels of audio and video data, which are manually annotated and proportionally divided into training, validation, and test sets. The audio and video data are processed and the model is trained using the annotated labels. After testing on the test set, the trained model is converted and deployed to drone devices to perform inference on the real-time audio and video content, achieving content recognition.

[0069] During the training process, the feature extraction models of audio and video are first pre-trained independently and then fused together for training.

[0070] During independent training, the model selects the backbone network of a pre-trained classification model or detection model, and inserts an adaptive temporal attention module into each sub-module; it first uses a large amount of public training data sets with rich categories for supervised training, and then fine-tunes using manually labeled data of target scenes.

[0071] One way to train the audio stream is to insert the adaptive temporal attention module into the pre-trained VGGish, retrain it on the AudioSet dataset, and then fine-tune it on the labeled data for scene content understanding.

[0072] One way to train the video stream is to insert the adaptive temporal attention module into a pre-trained ResNet50, retrain the classification task on the ImageNet dataset, and then fine-tune it on labeled data for scene content understanding.

[0073] Another way to train the video stream is to insert the adaptive temporal attention module into the pre-trained detection model YOLOv5s, retrain the backbone network for the classification task on the ImageNet dataset, and then fine-tune it on labeled data for scene content understanding.

[0074] During inference, you can directly load the trained weights.

[0075] It also includes S107, bimodal feature fusion and classification.

[0076] S400, dual-modal feature fusion unit: Figure 9 This is the model structure of the bimodal feature fusion unit. Its input is the one-dimensional feature vectors extracted from each of the two modal branches of the spatiotemporal feature extraction unit. After adding positional embedding, the multi-head self-attention module extracts and fuses the features between the two vectors. The two vectors are then concatenated into a single vector. A feedforward network extracts features again and adjusts the vector dimension. The 1D vectors of the two modalities are then fused to produce a single one-dimensional feature vector. This feature vector captures both the global and local spatiotemporal characteristics of the audio and video streams, enabling video content understanding.

[0077] One implementation of fusion training is to load the model weights obtained from independent training into the two feature extraction models. All BatchNorm parameters except the first BatchNorm are frozen, and the parameters of the dual-modal feature fusion unit and the scene content classification unit are randomly initialized. During training, data from both modalities must be prepared. During a forward pass, data from the corresponding modality must be input to each tributary. To improve the model's robustness and modality independence, there is a certain probability during training that the input of one modality is randomly set to 0. This allows the fused model to infer the correct category labels based solely on a single modality, making the dual-stream model more balanced and less dependent on a single tributary.

[0078] S500, scene content classification unit: One specific implementation method is to use a fully connected layer to adjust the dimension of the feature vector to be consistent with the number of classification categories, and then connect it to a Softmax layer to select the category with the highest score for output to achieve single classification; another implementation method is to use a fully connected layer to adjust the dimension of the feature vector to be consistent with the number of classification categories, and then connect it to a Sigmoid layer to select the category with a score higher than the threshold to achieve multi-label.

[0079] S108, obtain the category label corresponding to the video, map the category number output by the model to the corresponding category label for system output, and facilitate human understanding.

[0080] The present invention studies and designs audio and video acquisition processing and fusion processing algorithms for drones, effectively utilizes multimodal information, and solves the problem of scene content understanding within continuous time periods.

[0081] Regarding the content disclosed in this case, the following points need to be explained:

[0082] (1) The drawings of the embodiments disclosed in this case only involve the structures involved in the embodiments disclosed in this case. Other structures can refer to the general design;

[0083] (2) In the absence of conflict, the embodiments and features of the embodiments disclosed in this case may be combined with each other to form new embodiments;

[0084] The above are only specific implementation methods disclosed in this case, but the protection scope of this disclosure is not limited thereto. The protection scope disclosed in this case should be based on the protection scope of the claims.

Claims

1. A multimodal scene content understanding system mounted on an unmanned aerial vehicle, characterized in that: It includes a signal acquisition unit, a data sampling and processing unit, a spatiotemporal feature extraction unit, a bimodal feature fusion unit and a scene content classification unit, which are connected in sequence. The signal acquisition unit is used to collect audio data and video data; the audio data uses two microphones to collect two audio signals, one of which is hung under the drone away from the fuselage to collect scene sounds and reduce interference from the drone's own noise, and the other is fixed near the bottom of the fuselage to collect ambient noise emitted when the rotor is working; The data sampling and processing unit is used to process the audio data to form a spectrogram and perform frame sampling on the video data; The audio processing is implemented as follows: scene sounds and ambient noise are processed in the same way. First, the audio stream is intercepted by timestamp and stored locally. After loading into memory, the audio is resampled to mono audio. Using a Hann time window, the audio is short-time Fourier transform with a 10ms frame shift, and the logarithmic Mel spectrum is calculated. The features are then framed without overlap and stacked into a four-dimensional array. The ambient noise is processed in the same way into an array of the same dimension, and the resulting four-dimensional array is spliced ​​in the channel dimension as the input to the audio branch model. The spatiotemporal feature extraction unit is used to extract features from continuous spectrograms of the audio modality and to extract spatiotemporal features from continuous frames of the video modality; The spatiotemporal feature extraction unit includes: The audio and video modalities have independent feature extraction branch networks. Each feature extraction branch network is based on the backbone network of a general classification or detection model composed of convolution, with an adaptive time domain attention module inserted. The two modalities output their own one-dimensional feature vectors, representing the global and local features of the continuous input of each modality in the time and space dimensions. The adaptive time-domain attention module includes a time-dimensional adaptive local feature extraction submodule and a time-dimensional global attention submodule; the input of the adaptive time-domain attention module is copied into four copies, one for the time-dimensional adaptive local feature extraction submodule and three for the time-dimensional global attention submodule; the time-dimensional adaptive local feature extraction submodule generates an adaptive convolution kernel; the time-dimensional global attention submodule extracts the attention weight in the time dimension of one of the inputs, and applies the weight to all elements of the corresponding time dimension of the second input by multiplication, and then adds the result to the third input at the corresponding position as the output of the time-dimensional global attention submodule; the last part of the adaptive time-domain attention module is a 2D convolution, the input of the convolution is the output of the time-dimensional global attention submodule, the convolution kernel of the convolution is the output of the time-dimensional adaptive local feature extraction submodule, and the result of the 2D convolution is the output of the adaptive time-domain attention module; The temporal dimension adaptive local feature extraction submodule takes as input the input of the adaptive temporal domain attention module and outputs the convolution kernel for the next convolution. This is achieved by first performing global pooling on the spatial dimension in each channel, then performing a one-dimensional convolution with a kernel size of 5 on the temporal dimension. This is then followed by a 1-dimensional BatchNorm layer and a ReLU activation layer to extract features between adjacent moments. The dimension is then reduced to 5 using a fully connected layer, and finally a Sigmoid layer is used to adjust the value range to generate a 5*1 convolution kernel, which serves as the weight for the subsequent 2D convolution. The value of the convolution kernel comes from the input itself. Different inputs will generate different convolution kernels, making the model weights adaptive to the input. The temporal global attention submodule extracts feature changes in continuous time by subtracting spatial feature maps of adjacent time intervals, thereby capturing high-dimensional motion information of the target or shot. The implementation is as follows: the minuend branch goes from t1 to tn in the time dimension, and the subtrahend branch passes through one more 2D convolution layer. Then, t1 is moved to the end of the time dimension, forming the order of t2, t3, .., tn, t1. The feature maps of the corresponding positions of the two branches are subtracted to obtain the difference feature map of the spatial dimension. After global pooling and Softmax, the weight of the time dimension is obtained. After broadcasting and replicating the weight in the spatial dimension, the weight is multiplied with the module input, added to the input, and output; The dual-modal feature fusion unit is used to fuse the features extracted by the spatiotemporal feature extraction unit; The scene content classification unit is used to perform content classification on the output feature vector of the bimodal feature fusion unit, obtain the category number corresponding to the feature vector, and map it to the content category to which the original audio and video belong.

2. The multimodal scene content understanding system mounted on an unmanned aerial vehicle according to claim 1, characterized in that: The signal acquisition unit includes: Video data uses a visible light camera to obtain a continuous video stream.

3. The multimodal scene content understanding system mounted on an unmanned aerial vehicle according to claim 1, characterized in that: The data sampling and processing unit includes: Video processing: Capture video stream data through the camera, convert the video stream into a frame sequence, and sample some video frames at equal intervals.

4. The multimodal scene content understanding system mounted on an unmanned aerial vehicle according to claim 1, characterized in that: To extract features from continuous spectrograms, an adaptive temporal attention module is inserted into each module of any pre-trained audio classification model consisting of 2D convolutions that takes feature spectrograms as input. After inserting the adaptive temporal attention module, the model needs to be retrained.

5. The multimodal scene content understanding system mounted on an unmanned aerial vehicle according to claim 1, characterized in that: Extract spatiotemporal features from consecutive frames, An adaptive temporal attention module is inserted into each submodule of any pre-trained image classification or detection model backbone network composed of 2D convolution. After inserting the adaptive temporal attention module, the model needs to be retrained on the target task.

6. The multimodal scene content understanding system mounted on an unmanned aerial vehicle according to claim 1, characterized in that: The bimodal feature fusion unit includes: After adding position embedding to the one-dimensional feature vectors extracted from each of the two modalities, the multi-head self-attention module extracts and fuses the features between the two vectors. The two vectors are then concatenated into one vector, and the features are extracted again and the vector dimension is adjusted through the feedforward network. The one-dimensional vectors of each of the two modalities are then fused to finally output a one-dimensional feature vector.

7. The multimodal scene content understanding system mounted on an unmanned aerial vehicle according to claim 1, characterized in that: The scene content classification unit includes: Through the classification layer, content classification is performed based on the one-dimensional feature vector output by the bimodal feature fusion unit to obtain the content category to which the audio and video to be classified belongs.

8. The multimodal scene content understanding system mounted on an unmanned aerial vehicle according to claim 7, characterized in that: Use a fully connected layer to adjust the dimension of the feature vector to the same number of classification categories, then connect it to the Softmax layer to select the category with the highest score and output it to achieve single classification; or After using the fully connected layer to adjust the dimension of the feature vector to the same number of classification categories, the Sigmoid layer is connected to select categories with scores higher than the threshold to achieve multi-label.

Images