Video recording equipment identification system based on multi-modal fusion

Abstract

The invention discloses a video recording equipment identification system based on multi-modal fusion. The video recording equipment identification system comprises a data preprocessing module, a self-adaptive frame screening module, an image feature extraction module, an audio feature extraction module, a cross-modal fusion module and a classification and discrimination module. According to the method, the image and audio modal information is jointly utilized, and a synchronous-deformable bidirectional cross-modal attention mechanism (SD-BCA) is introduced, so that high-robustness recognition of different recording devices under the multi-noise and multi-compression conditions is realized, and the video traceability in a complex scene is remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of audio and video forensics and multimodal recognition technology, and in particular relates to a video recording device recognition system based on multimodal fusion, which can be used in scenarios such as video source device forensics, intelligent source tracing and multimedia security detection. Background Technology

[0002] With the widespread use of recording devices such as smartphones, surveillance cameras, and drones, the amount of video content has exploded. Determining the true source of a video has become a key issue in video security forensics, privacy protection, and copyright tracking.

[0003] Traditional video device identification methods often rely on image-level sensor fingerprints (PRNU) or compressed features. While these methods can achieve high recognition accuracy under ideal conditions, their performance significantly degrades when the video is compressed and re-encoded multiple times or when the signal quality deteriorates after transmission through a network platform. Furthermore, environmental noise, low light, motion blur, and other interferences in complex scenes can also compromise the stability of image fingerprints, and single-modal features (image only) are insufficient to fully characterize the imaging and recording characteristics of the device.

[0004] Existing research generally neglects the device fingerprint information (such as microphone frequency response characteristics, ADC sampling noise characteristics, etc.) contained in audio signals, resulting in insufficient recognition accuracy in low signal-to-noise ratio environments. Summary of the Invention

[0005] To address the issues of low accuracy and noise sensitivity in existing video device identification methods under complex environments, this invention provides a video recording device identification system based on multimodal fusion to improve device fingerprint extraction and classification performance. This invention achieves highly robust identification of different recording devices under conditions of high noise and high compression by jointly utilizing image and audio modal information and introducing a synchronous-deformable bidirectional cross-modal attention mechanism (SD-BCA), significantly improving video source tracing capabilities in complex scenarios.

[0006] In a first aspect, embodiments of this application provide a video recording device identification system based on multimodal fusion, comprising the following modules:

[0007] Data preprocessing module: used to separate and extract the audio stream and intra-coded frame sequence from the input video.

[0008] Adaptive Frame Filtering Module: Used to filter the most representative set of frames from the preprocessed intra-coded frame sequence.

[0009] By comprehensively analyzing inter-frame motion changes, texture complexity, and sensor fingerprint energy, a comprehensive score is calculated, and a greedy strategy is used to select the representative set of frames with the most information.

[0010] Image Feature Extraction Module: This module extracts visual device features from representative frames output by the adaptive frame filtering module. It combines an image fingerprint enhancement structure with an improved Transformer network to extract visual features from representative frames, thereby obtaining visual device features that reflect differences in imaging sensors.

[0011] Audio Feature Extraction Module: This module extracts acoustic features of the recording device from the audio stream. It integrates convolutional structures and attention mechanisms to simultaneously capture local spectral features and global temporal dependencies, extracting acoustic device features that reflect the microphone and analog-to-digital conversion characteristics.

[0012] Cross-modal fusion module: used to fuse visual device features and acoustic device features in the temporal and spatial dimensions.

[0013] This module employs a Sync-Deformable Bidirectional Cross-Modal Attention (SD-BCA) structure. Through temporal synchronization, spatially deformable sampling, and bidirectional cross-modal attention mechanisms, it enables the interaction and complementarity of audio and video information. This allows acoustic device features to guide visual feature representation, while visual features in turn enhance acoustic feature representation, thereby generating multimodal device feature representations and providing a unified input for the classification and discrimination module.

[0014] Classification and discrimination module: Used to discriminate the multimodal device features obtained after processing by the cross-modal fusion module, so as to realize the category identification and confidence assessment of the recording device.

[0015] In one possible implementation, the data preprocessing module is specifically implemented as follows:

[0016] The video is demultiplexed using a multimedia analysis tool to obtain the audio stream and the intra-coded frame sequence. The intra-coded frame sequence is then subjected to denoising, normalization, and brightness equalization. Simultaneously, the audio stream is formatted and subjected to spectral transformation.

[0017] In one possible implementation, the adaptive keyframe selection module is specifically implemented as follows:

[0018] First, optical flow is calculated for adjacent frames. The Farneback algorithm is used to obtain the motion vector of each pixel, and its average amplitude is used as the inter-frame motion score. Second, the ORB (Oriented Fast and Rotated BRIEF) operator is used to detect the number of feature points within the frame, and this is used as a texture complexity index. Then, the sensor fingerprint energy of the frame is calculated. Specifically, the optical response non-uniformity (PRNU) noise residual is extracted from the intra-coded frame; a high-pass filter is applied to the residual image to suppress low-frequency brightness components, and its pixel standard deviation is calculated as an energy metric to obtain the sensor fingerprint energy of the frame. After obtaining the inter-frame motion score, texture complexity, and sensor fingerprint energy, the three indices are linearly normalized, and the importance of the frame is calculated according to the following comprehensive scoring function. After comprehensive scoring, the system sorts the frames from high to low scores and uses a greedy algorithm to select representative frames, ensuring that the time interval between any two adjacent representative frames is not less than a set threshold.

[0019] In one possible implementation, the image feature extraction module is specifically implemented as follows:

[0020] The input to the Integrated Fingerprint Enhancement Module (IFEM) is a representative frame image output by the Adaptive Frame Selection Module. This module first extracts the sensor fingerprint noise component from the image using a high-pass filter, and then performs multi-scale feature extraction through three parallel 3×3 dilated convolution branches with different dilation rates. Subsequently, a 1×1 convolutional layer is used to fuse the multi-scale features, and these features are then element-wise superimposed with the input residual to obtain the enhanced image feature representation, denoted as the fingerprint enhancement feature map.

[0021] The fingerprint-enhanced feature map serves as the input to the improved Swin Transformer. The improved Swin Transformer network, based on the original window multi-head self-attention (W-MSA) and shifted window attention (SW-MSA) structures, introduces a convolutional pre-embedding module and a hierarchical channel attention mechanism. The convolutional pre-embedding module is placed at the input of each stage, between the patch block and the window attention module (W-MSA / SW-MSA); the hierarchical channel attention mechanism is placed at the output of each stage, after the feedforward fusion module (FFN). Structurally, the improved Swin Transformer network follows the original Swin Transformer's four-stage hierarchical feature extraction framework, with each stage including a patch block, window attention (W-MSA / SW-MSA), and a feedforward fusion module (FFN).

[0022] During image feature extraction, the network generates two-dimensional visual feature maps layer by layer. Global average pooling and linear mapping are performed on the high-level two-dimensional visual feature maps of each frame to obtain the corresponding frame-level visual feature vectors. These vectors are then stacked chronologically to form a frame-level visual feature sequence, while simultaneously recording timestamp information. Global average pooling and linear mapping are then performed on the high-level two-dimensional visual feature maps of all frames to generate visual device feature vectors, which serve as the final output of the module.

[0023] In one possible implementation, the audio feature extraction module is specifically implemented as follows:

[0024] The input audio signal is first converted into a time-spectrum image through a Short-Time Fourier Transform (STFT), and after logarithmic power normalization, it is sequentially input into three one-dimensional convolutional layers. The stride of the convolutional layers is set to 1, 2, and 2, respectively, followed by batch normalization and a ReLU activation function after each layer. Subsequently, the output of the convolutional layers, as intermediate acoustic features, is flattened and linearly transformed before being input into a two-layer Transformer encoder. A multi-head self-attention mechanism is used to capture global temporal dependencies, and positional encoding is used to preserve temporal order information. Dropout and layer normalization structures are introduced in the flattening and linear transformation stages to prevent feature overfitting. The output sequence of the two-layer Transformer encoder is constructed into a frame-level acoustic feature sequence in temporal order; simultaneously, this frame-level acoustic feature sequence is globally aggregated and linearly mapped to obtain the acoustic device feature vector, which serves as the final output of the module.

[0025] In one possible implementation, the cross-modal fusion module is specifically implemented as follows:

[0026] After feature input, the module first uses a linear mapping layer to unify the visual device feature vector and the acoustic device feature vector to the same feature dimension as a global semantic embedding.

[0027] Subsequently, the module enters the temporal-spatial alignment stage. A learnable temporal alignment network (TAN) is introduced, which takes frame-level acoustic feature sequences and video frame temporal distribution information as input. It extracts short-term contextual relationships through one-dimensional convolution and uses gated recurrent units (GRUs) to model long-term dependencies to align the temporal distribution of the audio stream with the intra-coded frame sequence, and outputs a temporally aligned acoustic feature sequence.

[0028] After completing temporal synchronization, the module further introduces a deformable sampling mechanism. The input to this mechanism includes the acoustic feature sequence aligned by the temporal synchronization network TAN and a two-dimensional visual feature map from the image feature extraction module. Based on the response intensity distribution at each spatial location in the two-dimensional visual feature map, key regions with response intensities exceeding an adaptive threshold are dynamically selected as sampling points. The offset of each sampling point is generated by learnable parameters. After deformable sampling processing, the module outputs a spatially reweighted visual feature representation, i.e., the spatially aligned visual features.

[0029] In the channel dimension, a learnable weighted linear fusion operation is used to integrate the spatially reweighted visual feature representation with the temporally aligned acoustic feature sequence to form a preliminary multimodal joint feature representation.

[0030] Subsequently, the multimodal joint feature representation is input into a bidirectional cross-modal attention mechanism (BCA) to further achieve deep semantic interaction and information complementarity between visual and acoustic features. This mechanism consists of two structurally symmetrical attention blocks, which perform feature interaction operations in the Audio→Image and Image→Audio directions, respectively.

[0031] In the Audio→Image attention block, the temporally aligned acoustic feature sequence is used as the query term, and the spatially aligned visual features are used as the key and value. The attention weight of audio on visual features is calculated to achieve cross-modal perception of "audio-guided vision" and to enhance visual features in the spatial region related to audio.

[0032] In the Image→Audio attention block, spatially aligned visual features are used as queries, and temporally aligned acoustic feature sequences are used as keys and values ​​to achieve reverse guidance of "visual feedback audio," thereby enhancing the acoustic features in the time segments corresponding to the visual events.

[0033] Finally, the attention outputs from both directions are concatenated with the global semantic embedding along the channel dimension and input to the feedforward network for nonlinear feature transformation and semantic compression. The output is then superimposed on the input features through residual connections to form the final multimodal device feature representation.

[0034] In one possible implementation, the classification and discrimination module is specifically implemented as follows:

[0035] The identification of recording devices is achieved through two fully connected layers and a Softmax classifier. The fully connected layers are used for non-linear feature mapping (each layer is followed by a ReLU activation function and a Dropout layer). Finally, the Softmax classifier outputs the predicted probability distribution of each recording device category.

[0036] Secondly, embodiments of this application provide a video recording device identification method based on multimodal fusion, comprising the following modules:

[0037] Step 1: Data preprocessing, separating and extracting the audio stream and intra-coded frame sequence from the input video.

[0038] The video is demultiplexed using a multimedia analysis tool to obtain the audio stream and the intra-coded frame sequence. The intra-coded frame sequence is then subjected to denoising, normalization, and brightness equalization. Simultaneously, the audio stream is formatted and subjected to spectral transformation.

[0039] Step 2: Select the most representative set of frames from the preprocessed intra-coded frame sequence.

[0040] By comprehensively analyzing inter-frame motion changes, texture complexity, and sensor fingerprint energy, a comprehensive score is calculated, and a greedy strategy is used to select the representative set of frames with the most information.

[0041] Step 3: Image Feature Extraction

[0042] This method is used to extract visual device features from representative frames. By combining an image fingerprint enhancement structure with an improved Transformer network, visual features are extracted from representative frames to obtain visual device features that reflect differences in imaging sensors.

[0043] Step 4: Audio Feature Extraction

[0044] Extracting acoustic features of recording devices from audio streams. By fusing convolutional structures and attention mechanisms, local spectral features and global temporal dependencies are captured simultaneously to extract acoustic device features reflecting microphone and analog-to-digital conversion characteristics.

[0045] Step 5: Integrate visual device features with acoustic device features in the temporal and spatial dimensions.

[0046] A Sync-Deformable Bidirectional Cross-Modal Attention (SD-BCA) structure is adopted. Through temporal synchronization, spatially deformable sampling and bidirectional cross-modal attention mechanism, the interaction and complementarity of audio and video information are realized. This allows acoustic device features to guide visual feature representation, while visual features in turn enhance acoustic feature representation, thereby generating multimodal device feature representation.

[0047] Step 6: Discriminate the multimodal device features obtained after processing by the cross-modal fusion module to achieve category identification and confidence assessment of the recording device.

[0048] The beneficial effects of this invention are as follows:

[0049] This invention performs cross-modal joint modeling of the visual device feature representation of video images and the acoustic device feature representation of audio streams, and utilizes an alignment mechanism and a bidirectional attention structure to achieve deep fusion of audio and video features. Compared with traditional single-modal recognition methods, this invention can fully utilize multi-source information, improve the completeness and discriminativeness of feature representations, and thus significantly improve the accuracy of recording device recognition.

[0050] In the vision section, a device fingerprint enhancement structure is introduced. Multi-scale convolution and channel attention mechanisms are used to strengthen the feature representation of the imaging sensor, effectively suppressing compression artifacts and highlighting imaging differences between different devices. In the acoustic section, a feature extraction module fusing convolutional structures and attention mechanisms is designed to capture microphone frequency response characteristics and sampling noise distribution, thereby constructing acoustic feature representations that reflect the differences between recording devices.

[0051] Furthermore, the dynamic cross-modal fusion mechanism proposed in this invention achieves audio-video alignment and bidirectional interactive fusion through time synchronization and sparse spatial sampling strategies. This enables acoustic feature representations to guide visual attention areas, while visual feature representations conversely enhance acoustic semantic expressions, thereby achieving adaptive complementarity at the feature level. This mechanism significantly improves the model's robustness to recognition under complex noise, illumination variations, and multiple compression conditions.

[0052] In summary, this invention effectively overcomes the problem of decreased recognition accuracy in complex environments using traditional methods, achieving high robustness and high-precision recognition of recording devices, and possesses broad application value in fields such as video source forensics and multimedia security detection. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0054] Figure 1 This is a schematic diagram of the overall structure and workflow of the system according to an embodiment of the present invention.

[0055] Figure 2 This is a schematic diagram of the image feature extraction module in an embodiment of the present invention.

[0056] Figure 3This is a schematic diagram of the audio feature extraction module in an embodiment of the present invention.

[0057] Figure 4 This is a schematic diagram of the cross-modal fusion module structure according to an embodiment of the present invention.

[0058] Figure 5 This is a comparison of the recognition accuracy of the embodiments of the present invention and existing methods. Detailed Implementation

[0059] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0060] Most existing video recording device identification methods rely on a single visual modality, identifying devices through sensor fingerprints or compressed features. While these methods can achieve high accuracy under ideal conditions, they still have significant shortcomings in real-world, complex environments. On the one hand, video images are severely distorted after multiple compressions, re-encodings, or transmission via network platforms, weakening the device fingerprint features. On the other hand, existing methods generally neglect the microphone frequency response characteristics and analog-to-digital conversion noise features inherent in audio signals, resulting in a significant decrease in recognition performance and insufficient robustness in low signal-to-noise ratio and strong interference scenarios.

[0061] To address the aforementioned issues, this invention proposes a video recording device recognition system based on a dynamic cross-modal fusion mechanism. Starting from visual and acoustic dual-channel information, a device feature representation model with greater discriminative power and environmental adaptability is established.

[0062] refer to Figure 1 As shown in the figure, this application provides a video recording device recognition system based on multimodal fusion, which includes the following modules:

[0063] Data Preprocessing Module: This module extracts the audio stream and intra-coded frame sequence from the input video, providing a unified input format for subsequent feature analysis. The video file is demultiplexed using FFmpeg to extract the intra-coded frame sequence and the original audio stream. For the intra-coded frames, Gaussian filtering (3×3 kernel size, standard deviation σ=1.0) is first applied for denoising to remove compression artifacts and random noise. Subsequently, the intra-coded frames are uniformly adjusted to 256×256 pixels and subjected to Z-score normalization and luminance normalization. The normalization formula is:

[0064]

[0065] in The original pixel intensity, μ, These are the mean and standard deviation, respectively. The values ​​are normalized pixel intensity values. The audio signal is converted to 44.1 kHz / 16 bit PCM format and divided into frames with a frame length of 25 ms and a frame shift of 10 ms. After Hamming window weighting, a Short Time Fourier Transform (STFT) is performed to generate a time-spectrum. After processing by this module, the system can synchronously obtain the audio stream and the intra-coded frame sequence.

[0066] Adaptive keyframe selection module: This module is used to automatically select the most representative frames with the most device characteristics from the preprocessed intra-coded frame sequence in order to reduce redundant data and enhance the stability of subsequent identification.

[0067] First, optical flow is calculated for adjacent frames. The Farneback algorithm is used to obtain the motion vector of each pixel, and its average amplitude is used as the inter-frame motion score. A higher motion score indicates more significant dynamic changes in the image, which helps capture motion blur and exposure response differences related to device imaging characteristics. Secondly, the ORB (Oriented FAST and Rotated BRIEF) operator is used to detect the number of intra-frame feature points, and this is used as an indicator of texture complexity. In other words, the more feature points a frame has, the richer the image details; the fewer feature points a frame has, the smoother the image or the less texture information it contains. The sensor fingerprint energy of the subsequent frames is then calculated. Specifically, the optical response nonuniformity (PRNU) noise residual is extracted from the intra-coded frame. This residual can be considered as subtle fixed-pattern noise introduced by the device's imaging sensor during the shooting process. By applying a high-pass filter to the residual image to suppress low-frequency brightness components and calculating its pixel standard deviation as an energy metric, the sensor fingerprint energy of the frame can be obtained. The higher the energy value, the more obvious the device feature signals retained in the frame. After obtaining the inter-frame motion score, texture complexity, and sensor fingerprint energy, the three indicators are linearly normalized, and the importance of the frame is calculated according to the following comprehensive scoring function:

[0068]

[0069] in The weighting coefficients are set to 0.3, 0.3, and 0.4 respectively. After comprehensive scoring, the system sorts the frames from highest to lowest score and uses a greedy algorithm to select representative frames, ensuring that the time interval between any two adjacent representative frames is not less than a set threshold. (e.g., 0.5 seconds). In this embodiment, the final output representative frame count accounts for approximately 5% of the total number of frames. This percentage is not a fixed value, but rather an empirical parameter determined based on extensive experiments: performance evaluations under different video resolutions and scene types revealed that when the representative frame percentage is set between 3% and 7%, the system achieves a better balance between recognition accuracy and computational efficiency. A percentage of 5% significantly reduces redundant data while ensuring the integrity of device features. For videos with drastic motion changes or frequent scene transitions, this percentage can be appropriately increased; while for static or low-dynamic videos, it can be appropriately decreased.

[0070] Image Feature Extraction Module: This module consists of an Integrated Fingerprint Enhancement Module (IFEM) and an improved SwingTransformer network.

[0071] The IFEM module takes as input a representative frame image from the adaptive frame filtering module. This module first extracts the sensor fingerprint noise component from the image using high-pass filtering, and then performs multi-scale feature extraction through three parallel 3×3 dilated convolution branches with different dilation rates. In this embodiment, the dilation rates are 1, 2, and 3, respectively, to capture device noise patterns under different receptive fields. Subsequently, a 1×1 convolutional layer is used to fuse the multi-scale features, and then element-wise superimposed with the input residual to obtain the enhanced image feature representation, denoted as the fingerprint enhancement feature map. This map is used to characterize the visual device features after noise enhancement and structure preservation processing. This feature map serves as the input to the improved Swin Transformer. The improved Swin Transformer network introduces a convolutional pre-embedding module and a hierarchical channel attention mechanism based on the original window multi-head self-attention (W-MSA) and shifted window attention (SW-MSA) structures. The improved Swin Transformer network employs a four-stage hierarchical feature extraction framework. Each stage includes a patch block, window attention (W-MSA / SW-MSA), and a feedforward fusion module (FFN). The pre-convolutional embedding module is positioned at the input of each stage, between the patch block and the window attention module (W-MSA / SW-MSA). It performs local convolutional enhancement before feature processing, extracting fine-grained texture information and improving spatial awareness. The hierarchical channel attention mechanism is positioned at the output of each stage, after the feedforward fusion module (FFN). It adaptively weights channel-dimensional features to selectively enhance global semantic information. The improved Swin Transformer network retains the original Swin Transformer's four-stage hierarchical feature extraction framework. Each stage includes a patch block, window attention (W-MSA / SW-MSA), and a feedforward fusion module (FFN). The pre-convolutional embedding module and the hierarchical channel attention mechanism are then embedded to enhance features. During image feature extraction, the network generates two-dimensional visual feature maps during layer-by-layer computation to characterize the spatial distribution features of the device fingerprint. The two-dimensional visual feature maps from the high-level layer of the improved SwinTransformer (referring to the output of the last stage (Stage 4) of the improved SwinTransformer network, which contains global semantic information and spatial distribution patterns) are used for two types of tasks simultaneously: On the one hand, the two-dimensional visual feature maps serve as input to the deformable sampling mechanism of the subsequent cross-modal fusion module, used to select and enhance key visual regions in the spatial dimension; on the other hand, the network first performs global average pooling and linear mapping on the high-level two-dimensional visual feature maps of each frame to obtain the corresponding frame-level visual feature vectors, and stacks them in chronological order to form a frame-level visual feature sequence, while recording timestamp information for use by the temporal synchronization network in the cross-modal fusion module.Subsequently, the network performs global average pooling and linear mapping on the high-level 2D visual feature maps of all frames, further generating a 512-dimensional visual device feature vector as the final output of the module. This global visual device feature can be regarded as a temporal aggregation result of the frame-level visual feature sequence, used to provide an overall device semantic representation.

[0072] Audio Feature Extraction Module: This module uses a one-dimensional convolution-Transformer hybrid structure to extract the acoustic features of the recording device from the audio stream.

[0073] The input audio signal is first converted into a time-spectrum image through a Short-Time Fourier Transform (STFT), and after logarithmic power normalization, it is sequentially input into three one-dimensional convolutional layers. The stride of the convolutional layers is set to 1, 2, and 2 sequentially. Each layer is followed by batch normalization and a ReLU activation function to capture local spectral features and suppress background noise. Subsequently, the output of the convolutional layers, as intermediate acoustic features, is flattened and linearly transformed before being input into a two-layer Transformer encoder. A multi-head self-attention mechanism is used to capture global temporal dependencies, and positional encoding is used to maintain temporal order information. To improve the stability and generalization ability of feature extraction, Dropout and layer normalization structures are introduced in the flattening and linear transformation stages to prevent feature overfitting. The output sequence of the two-layer Transformer encoder is constructed into a frame-level acoustic feature sequence in temporal order for subsequent cross-modal synchronization and fusion. At the same time, this frame-level acoustic feature sequence is globally aggregated and linearly mapped to obtain a 256-dimensional acoustic device feature vector, which serves as the final output of the module.

[0074] Cross-modal fusion module: This module designs a Sync-Deformable Bidirectional Cross-Modal Attention (SD-BCA) structure to fuse visual device features and acoustic device features in the temporal and spatial dimensions.

[0075] The module's main input includes three types of features:

[0076] (1) The 512-dimensional visual device feature vector from the image feature extraction module and the 256-dimensional acoustic device feature vector from the audio feature extraction module are used as global semantic embeddings to provide overall semantic context information within the modality in the subsequent attention and feature integration stages.

[0077] (2) Frame-level visual feature sequences from the image feature extraction module and frame-level acoustic feature sequences from the audio feature extraction module are used for temporal synchronization and cross-modal alignment;

[0078] (3) Two-dimensional visual feature maps from the image feature extraction module are used in the subsequent spatial deformable sampling stage to extract discriminative key regions from the spatial dimension.

[0079] After feature input, the module first unifies the feature vectors of the visual device and the acoustic device to the same feature dimension (256 dimensions) through a linear mapping layer, so as to facilitate subsequent cross-modal fusion and attention calculation.

[0080] Subsequently, the module enters the time-space alignment phase. To address the timing offset issue caused by the mismatch between audio and video sampling rates, the module introduces a learnable Temporal Alignment Network (TAN). Figure 4 The Synchronizer (Δt) network takes frame-level acoustic feature sequences and video frame temporal distribution information as input, extracts short-term contextual relationships through one-dimensional convolution, and uses gated recurrent units (GRUs) to model long-term dependencies to align the temporal distribution of the audio stream with the intra-coded frame sequence, and outputs a time-aligned acoustic feature sequence.

[0081] After completing timing synchronization, the module further introduces a deformable sampling mechanism (corresponding to...). Figure 4 The DeformableSampler mechanism takes as input an acoustic feature sequence aligned by a temporal synchronization network (TAN) and a two-dimensional visual feature map from an image feature extraction module. Its core idea is to dynamically select key regions with response intensities exceeding an adaptive threshold τ as sampling points based on the response intensity distribution at each spatial location in the two-dimensional visual feature map, thereby capturing spatial patterns with high discriminative power in device features. The response intensity is calculated from the network's activation values ​​and used to quantify the information contribution of that region in the device recognition task; the threshold τ is a learnable parameter that is adaptively adjusted by the network during training. The offset of each sampling point... Generated by learnable parameters, it achieves adaptive focusing on key visual regions. After deformable sampling, the module outputs a spatially reweighted visual feature representation (i.e., spatially aligned visual features). This representation establishes a one-to-one correspondence with the temporally aligned acoustic features in the semantic space, providing input for subsequent cross-modal fusion.

[0082] TAN and the deformable sampling mechanism together constitute the temporal-spatial alignment unit of the module. The former ensures the temporal consistency of acoustic features, while the latter achieves spatial adaptive alignment of visual features based on this.

[0083] After temporal synchronization and deformable sampling, visual and acoustic device features are aligned in both temporal and spatial dimensions. Subsequently, the module integrates the spatially reweighted visual feature representation output by the deformable sampling mechanism with the temporally aligned acoustic feature sequence output by the temporal synchronization network through a learnable weighted linear fusion operation in the channel dimension, forming a preliminary multimodal joint feature representation. The fusion weights are adaptively learned by the network during training to balance the contributions of the two modalities to the overall feature representation.

[0084] Subsequently, the multimodal joint feature representation is input into a bidirectional cross-modal attention mechanism (BCA) to further achieve deep semantic interaction and information complementarity between visual and acoustic features. This mechanism consists of two structurally symmetrical attention blocks, which perform feature interaction operations in the Audio→Image and Image→Audio directions respectively (as shown in the right half of Figure 4).

[0085] In the Audio→Image attention block, the time-aligned acoustic feature sequence output by TAN is used as the query term, and the spatially aligned visual features output by the deformable sampling mechanism are used as the key and value. The attention weight of audio on visual features is calculated to achieve cross-modal perception of "audio-guided vision" and to enhance visual features in the spatial region related to audio.

[0086] In the Image→Audio attention block, the spatially aligned visual features output by the deformable sampling mechanism are used as the query, and the temporally aligned acoustic feature sequence output by TAN is used as the key and value. This achieves reverse guidance of "visual feedback audio", which enhances the acoustic features in the time segment corresponding to the visual event.

[0087] Finally, the attention outputs from both directions are concatenated with the global semantic embeddings (visual device feature vectors and acoustic device feature vectors) along the channel dimension and then input into the Feed-Forward Network (FFN) for nonlinear feature transformation and semantic compression. The FFN consists of two fully connected layers and a ReLU activation function, used to fuse fine-grained cross-modal interaction features and global contextual information. The output is superimposed on the input features through residual connections to form the final multimodal device feature representation, which serves as the input to the classification module. The attention calculation formula is as follows:

[0088]

[0089] in These represent the query, key, and value matrices, respectively. The key vector dimension is used. The fused output is a 256-dimensional joint embedding feature.

[0090] Classification and discrimination module: The multimodal device feature representation obtained after processing by the cross-modal fusion module is input into the classification and discrimination module to realize the category identification and confidence assessment of the recording device.

[0091] This module identifies recording devices using two fully connected layers and a Softmax classifier. The fully connected layers are used for non-linear feature mapping (each layer is followed by a ReLU activation function and a Dropout layer). Finally, the Softmax classifier outputs the predicted probability distribution for each recording device category. Assume the system needs to identify the number of device categories... Then the first The predicted probability of a class can be expressed as:

[0092]

[0093] in For the first The network output corresponding to the class. During the training phase, the entire multimodal device recognition model adopts an end-to-end joint training approach, with the optimization objective being to minimize the cross-entropy loss function:

[0094]

[0095] In the formula, The one-hot encoding for the true label represents the expression where, when the sample belongs to the first... Class Time ,otherwise ; For the model to the first The model maximizes the predicted probability of the true class by minimizing this loss function, thus achieving accurate identification of the recording device. The Adam optimizer is used during training, and weight decay is implemented to prevent overfitting.

[0096] To verify the effectiveness of the multimodal device feature fusion video recording device identification method proposed in this invention, the publicly available QUFVD dataset and the self-built Daxing dataset were selected for experiments, and noise-enhanced versions (QUFVD-NA, Daxing-NA) were constructed to simulate real complex acoustic environments.

[0097] The experimental comparison methods include:

[0098] (1) Traditional methods based on spatial noise pattern (SPN) and waveform consistency (WCS).

[0099] (2) A method based on optical response nonuniformity (PRNU) and multi-feature fusion (MFM+GA+PCE).

[0100] (3) A method based on the fusion strategy of convolutional neural network (CNN) and patch selection (PatchSel);

[0101] (4) Deep fingerprint network (PRNU-Net) model.

[0102] (5) The multimodal fusion device identification method (MFF-RVRDI) proposed in this invention.

[0103] The recognition accuracy of each method on different datasets is as follows: Figure 5 As shown.

[0104] from Figure 5 As can be seen, the method of this invention significantly outperforms the comparative methods on all datasets, especially on the noise-enhanced versions (QUFVD-NA and Daxing-NA), where the accuracy is improved by approximately 25% to 40%, indicating that the method of this invention can maintain high recognition performance even under low signal-to-noise ratio and multiple compression conditions. This result verifies that the present invention, through a dual-modal feature fusion mechanism of visual and acoustic modes, can effectively enhance the robustness and generalization ability of the model, achieving high-precision recognition of video recording devices.

[0105] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0106] The various embodiments in this specification are described in a related manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other.

[0107] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A video recording device recognition system based on multimodal fusion, characterized in that, Includes the following modules: Data preprocessing module: used to separate and extract the audio stream and intra-coded frame sequence from the input video; Adaptive frame filtering module: used to filter the most representative set of frames from the preprocessed intra-coded frame sequence; By comprehensively analyzing inter-frame motion changes, texture complexity, and sensor fingerprint energy, a comprehensive score is calculated, and a greedy strategy is used to select the representative set of frames with the most information. Image feature extraction module: used to extract visual device features from representative frames output by the adaptive frame filtering module; this module combines image fingerprint enhancement structure and improved Transformer network to extract visual features from representative frames, thereby obtaining visual device features that can reflect the differences in imaging sensors; Audio feature extraction module: used to extract acoustic features of recording devices from audio streams; this module integrates convolutional structure and attention mechanism, which can simultaneously capture local spectral features and global temporal dependencies, and extract acoustic device features that reflect the characteristics of microphones and analog-to-digital conversion; Cross-modal fusion module: used to fuse visual device features and acoustic device features in the temporal and spatial dimensions; This module adopts a synchronous-deformable bidirectional cross-modal attention structure. Through temporal synchronization, spatially deformable sampling, and bidirectional cross-modal attention mechanism, it realizes the interaction and complementarity of audio and video information, enabling acoustic device features to guide visual feature representation, while visual features inversely enhance acoustic feature expression, thereby generating multimodal device feature representation and providing a unified input for the classification and discrimination module. Classification and discrimination module: Used to discriminate the multimodal device features obtained after processing by the cross-modal fusion module, so as to realize the category identification and confidence assessment of the recording device.

2. The video recording device recognition system based on multimodal fusion according to claim 1, characterized in that, The data preprocessing module is implemented as follows: The video is demultiplexed using a multimedia parsing tool to obtain the audio stream and the intra-coded frame sequence; then, the intra-coded frame sequence is subjected to denoising, normalization, and brightness equalization; at the same time, the audio stream is formatted and subjected to spectral transformation.

3. The video recording device recognition system based on multimodal fusion according to claim 1, characterized in that, The adaptive keyframe selection module is implemented as follows: First, optical flow is calculated for adjacent frames. The Farneback algorithm is used to obtain the motion vector of each pixel, and its average amplitude is used as the inter-frame motion score. Second, the ORB operator is used to detect the number of feature points within the frame, and this is used as a texture complexity index. Then, the sensor fingerprint energy of the frame is calculated. Specifically, the light response non-uniformity noise residual is extracted from the intra-coded frame. By applying a high-pass filter to the residual image to suppress low-frequency brightness components, and calculating its pixel standard deviation as an energy metric, the sensor fingerprint energy of the frame can be obtained. After obtaining the inter-frame motion score, texture complexity, and sensor fingerprint energy, the three indices are linearly normalized respectively, and the importance of the frame is calculated according to the following comprehensive scoring function. After comprehensive scoring, the system sorts the frames from high to low scores and uses a greedy algorithm to select representative frames, ensuring that the time interval between any two adjacent representative frames is not less than a set threshold.

4. A video recording device recognition system based on multimodal fusion according to claim 1 or 3, characterized in that, The image feature extraction module is implemented as follows: The input to the integrated fingerprint enhancement module is a representative frame image output by the adaptive frame filtering module. The module first uses high-pass filtering to extract the sensor fingerprint noise component in the image, and then performs multi-scale feature extraction through three parallel 3×3 dilated convolution branches with different dilation rates. Subsequently, a 1×1 convolutional layer is used to fuse the multi-scale features, and the feature is superimposed element-wise with the input residual to obtain the enhanced image feature representation, denoted as the fingerprint enhancement feature map. The fingerprint-enhanced feature map serves as the input to the improved Swing Transformer. Based on the original multi-head self-attention and shifted window attention structures, the improved Swing Transformer network introduces a convolutional pre-embedding module and a hierarchical channel attention mechanism. The convolutional pre-embedding module is placed at the input of each stage, between the patch block and the window attention module; the hierarchical channel attention mechanism is placed at the output of each stage, after the feedforward fusion module. Structurally, the improved Swing Transformer network follows the original Swing Transformer's four-stage hierarchical feature extraction framework, with each stage including patch block, window attention, and a feedforward fusion module. During image feature extraction, the network generates two-dimensional visual feature maps in layer-by-layer computation. Global average pooling and linear mapping are performed on the high-level two-dimensional visual feature maps of each frame to obtain the corresponding frame-level visual feature vectors. These vectors are then stacked in chronological order to form a frame-level visual feature sequence, while recording timestamp information. Overall average pooling and linear mapping are performed on the high-level two-dimensional visual feature maps of all frames to generate visual device feature vectors, which serve as the final output of the module.

5. The video recording device recognition system based on multimodal fusion according to claim 1, characterized in that, The audio feature extraction module is implemented as follows: The input audio signal is first transformed into a time-spectrum image through a short-time Fourier transform, and then logarithmically normalized before being sequentially input into three one-dimensional convolutional layers. The stride of each convolutional layer is set to 1, 2, and 2, respectively, followed by batch normalization and ReLU activation function after each layer. Subsequently, the output of the convolutional layers, as intermediate acoustic features, is flattened and linearly transformed before being input into a two-layer Transformer encoder. A multi-head self-attention mechanism is used to capture global temporal dependencies, and positional encoding is used to maintain temporal order information. Dropout and layer normalization structures are introduced in the flattening and linear transformation stages to prevent feature overfitting. The output sequence of the two-layer Transformer encoder is constructed into a frame-level acoustic feature sequence in temporal order. At the same time, the frame-level acoustic feature sequence is globally aggregated and linearly mapped to obtain the acoustic device feature vector, which is the final output of the module.

6. The video recording device recognition system based on multimodal fusion according to claim 1, characterized in that, The cross-modal fusion module is implemented as follows: After feature input, the module first unifies the visual device feature vector and the acoustic device feature vector to the same feature dimension through a linear mapping layer as a global semantic embedding; Subsequently, the module enters the temporal-spatial alignment stage; a learnable temporal synchronization network TAN is introduced, which takes frame-level acoustic feature sequences and video frame temporal distribution information as input, extracts short-term contextual relationships through one-dimensional convolution, and uses gated recurrent units to model long-term dependencies to align the temporal distribution of the audio stream with the intra-frame coded frame sequence, and outputs the temporally aligned acoustic feature sequence. After completing the temporal synchronization, the module further introduces a deformable sampling mechanism. The input of this mechanism includes the acoustic feature sequence aligned by the temporal synchronization network TAN and the two-dimensional visual feature map from the image feature extraction module. Based on the response intensity distribution of each spatial location in the two-dimensional visual feature map, key regions with response intensity exceeding an adaptive threshold are dynamically selected as sampling points. The offset of each sampling point is generated by learnable parameters. After deformable sampling processing, the module outputs the spatially reweighted visual feature representation, i.e., the spatially aligned visual features. In the channel dimension, a learnable weighted linear fusion operation is used to integrate the spatially reweighted visual feature representation with the temporally aligned acoustic feature sequence to form a preliminary multimodal joint feature representation. Subsequently, the multimodal joint feature representation is input into a bidirectional cross-modal attention mechanism to further achieve deep semantic interaction and information complementarity between visual and acoustic features. This mechanism consists of two structurally symmetrical attention blocks, which perform feature interaction operations in the Audio→Image and Image→Audio directions, respectively. In the Audio→Image attention block, the temporally aligned acoustic feature sequence is used as the query term, and the spatially aligned visual features are used as the key and value. The attention weight of audio on visual features is calculated to achieve cross-modal perception of "audio-guided vision" and to enhance visual features in the spatial region related to audio. In the Image→Audio attention block, spatially aligned visual features are used as query terms, and temporally aligned acoustic feature sequences are used as keys and values ​​to achieve reverse guidance of "visual feedback audio", so that acoustic features are enhanced in the time segment corresponding to the visual event. Finally, the attention outputs from both directions and the global semantic embedding are concatenated along the channel dimension and input to the feedforward network for nonlinear feature transformation and semantic compression. The output is then superimposed on the input features through residual connections to form the final multimodal device feature representation.

7. The video recording device recognition system based on multimodal fusion according to claim 1, characterized in that, The classification and discrimination module is implemented as follows: The identification of recording devices is achieved through two fully connected layers and a Softmax classifier. The fully connected layers are used for non-linear feature mapping. Each layer is followed by a ReLU activation function and a Dropout layer. Finally, the Softmax classifier outputs the predicted probability distribution of each recording device category.

8. A method for identifying video recording devices based on multimodal fusion, characterized in that, Includes the following modules: Step 1: Data preprocessing, separating and extracting the audio stream and intra-coded frame sequence from the input video; The video is demultiplexed using a multimedia parsing tool to obtain the audio stream and the intra-coded frame sequence; then, the intra-coded frame sequence is subjected to denoising, normalization, and brightness equalization; at the same time, the audio stream is formatted and subjected to spectral transformation. Step 2: Select the most representative set of frames from the preprocessed intra-coded frame sequence; By comprehensively analyzing inter-frame motion changes, texture complexity, and sensor fingerprint energy, a comprehensive score is calculated, and a greedy strategy is used to select the representative set of frames with the most information. Step 3: Image Feature Extraction Used to extract visual device features from representative frames; combining image fingerprint enhancement structure and improved Transformer network to extract visual features from representative frames, thereby obtaining visual device features that can reflect differences in imaging sensors; Step 4: Audio Feature Extraction Extracting acoustic features of the recording device from the audio stream; By integrating convolutional structures and attention mechanisms, local spectral features and global temporal dependencies are captured simultaneously, and acoustic device features reflecting microphone and analog-to-digital conversion characteristics are extracted. Step 5: Integrate visual device features with acoustic device features in the temporal and spatial dimensions; A synchronous-deformable bidirectional cross-modal attention structure is adopted. Through temporal synchronization, spatially deformable sampling and bidirectional cross-modal attention mechanism, the interaction and complementarity of audio and video information are realized, so that acoustic device features guide visual feature representation, while visual features inversely enhance acoustic feature expression, thereby generating multimodal device feature representation; Step 6: Discriminate the multimodal device features obtained after processing by the cross-modal fusion module to achieve category identification and confidence assessment of the recording device.

Images