A micro video place classification method and system based on multi-modal feature fusion

Abstract

The application discloses a micro-video site classification method and system based on multi-modal feature fusion, and the method comprises the following steps: acquiring a visual feature vector and an auditory feature vector of a micro-video; performing main-path fusion on the visual feature vector and the auditory feature vector to obtain main-path fusion features; performing slave-path fusion on the visual feature vector and the auditory feature vector to obtain slave-path basic features; inputting the main-path fusion features and the slave-path basic features into a gating network; the gating network outputs a dynamic compensation coefficient according to a difference vector between the main-path fusion features and the slave-path basic features and an element-by-element product vector of the main-path fusion features and the slave-path basic features, and the dynamic compensation coefficient is applied to the slave-path basic features to obtain slave-path compensation features; combining the main-path fusion features and the slave-path compensation features to obtain final fusion features; and outputting a micro-video site category according to the final fusion features, which significantly improves the robustness of the model in a modal inconsistency scene.

Description

Technical Field

[0001] This invention relates to the field of micro-video classification technology, specifically to a micro-video location classification method and system based on multimodal feature fusion. Background Technology

[0002] Micro-videos (such as user-generated content on short video platforms) contain both visual and auditory modal information. Automatically identifying the filming location of micro-videos (such as restaurants, offices, ski resorts, and train stations) has significant application value in content recommendation, security monitoring, and advertising. Effectively integrating visual and auditory features is key to improving location classification performance.

[0003] Existing multimodal feature fusion methods in micro-video location classification mainly employ two strategies: one is to directly concatenate visual and auditory features along the channel dimension and then feed them into the classification network; the other is to assign a weight to each of the visual and auditory features and then perform a weighted summation, with the weight remaining fixed after the model training is completed.

[0004] However, in real-world applications, the correlation between visual and auditory modalities dynamically changes depending on the sample. For example, in a restaurant scene video, background noise might include discussions about skiing unrelated to the location, leading to semantic conflicts between the two modalities. Similarly, in silent or quiet videos, effective auditory modal information is missing. When the correlation between these modalities is low or conflicting, existing fusion methods cannot perceive these dynamic changes and adjust their fusion strategies accordingly. This results in the fused features containing interfering information, leading to decreased classification accuracy and robustness.

[0005] Therefore, in micro-video location classification, existing technologies cannot achieve adaptive multimodal feature fusion to improve the robustness and accuracy of micro-video location classification when the correlation between visual and auditory modalities changes dynamically. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention provides a micro-video location classification method and system based on multimodal feature fusion to solve the problems in existing technologies.

[0007] To achieve the above objectives, the present invention provides the following technical solution: A micro-video location classification method based on multimodal feature fusion includes: Obtain the visual and auditory feature vectors of the micro-video; The visual feature vector and auditory feature vector are fused together to obtain the main path fused feature. The visual feature vector and auditory feature vector are fused from the path to obtain the basic features from the path; The main road fusion feature and the secondary road basic feature are input into a gating network. The gating network outputs a dynamic compensation coefficient based on the difference vector between the main road fusion feature and the secondary road basic feature and the element-wise product vector of the two. The dynamic compensation coefficient is then applied to the secondary road basic feature to obtain the secondary road compensation feature. The main road fusion feature is combined with the secondary road compensation feature to obtain the final fusion feature; The micro-video location category is output based on the final fusion features; During model training, orthogonal constraints are applied to reduce the correlation between the fused features of the main road and the basic features of the secondary road. The orthogonal constraints and the gating network reinforce each other during joint training, including: The orthogonal constraint increases the difference vector between the main road fusion feature and the secondary road basic feature; The gated network outputs dynamic compensation coefficients based on the increased difference vector and the element-wise product vector; The gradient of the dynamic compensation coefficient updates the parameters of the path fusion during backpropagation.

[0008] In one embodiment, the gated network is a multi-layer fully connected network whose output is limited to between 0 and 1 by a sigmoid function.

[0009] In one embodiment, the orthogonal constraint is achieved by minimizing the absolute value of the cosine similarity between the main path fusion feature and the secondary path basic feature; The orthogonal constraints are applied to multiple feature subspaces of the main road fusion feature and the secondary road basic feature, respectively, with different weights of orthogonal constraint coefficients applied to different subspaces.

[0010] In one embodiment, the main path fusion includes: The visual feature vector and the auditory feature vector are concatenated, enhanced by channel attention, and then reduced in dimensionality by a multilayer perceptron. The channel attention enhancement uses a Squeeze-and-Excitation module.

[0011] In one embodiment, the path fusion includes: The visual feature vector and the auditory feature vector are transformed into a common space through linear projection and activation function, respectively, and then weighted and summed using learnable weights; The activation function is the tanh function, and the learnable weights are restricted to between 0 and 1 by the sigmoid function.

[0012] In one embodiment, the visual feature vector and the auditory feature vector are obtained through parallel temporal aggregation networks, which are NNeXtVLAD networks, temporal attention pooling networks, or deformable convolutional temporal aggregation networks; wherein, the NNeXtVLAD network adds ReLU activation to the front end of the NeXtVLAD structure and L2 normalization to the back end.

[0013] In one embodiment, the orthogonal constraints are applied to multiple feature subspaces of the main road fusion feature and the secondary road basic feature, respectively, with different weights of orthogonal constraint coefficients applied to different subspaces.

[0014] In one embodiment, the input to the gated network also includes the cosine similarity between the master path fusion feature and the slave path basic feature.

[0015] In one embodiment, during training, the contribution of a single modality in the basic features of the secondary road is randomly discarded with a preset probability, while the normal propagation of the fused features of the main road is preserved; the preset probability is dynamically adjusted according to the real-time correlation between the visual feature vector and the auditory feature vector: when the correlation is lower than a threshold, the modality discard probability is reduced, and when the correlation is higher than the threshold, the modality discard probability is increased.

[0016] A micro-video location classification system based on multimodal feature fusion, comprising: The feature extraction module is used to obtain the visual and auditory feature vectors of the micro-video; The main road fusion module is used to generate main road fusion features; The path fusion module is used to generate basic path features; The gated network module is used to output dynamic compensation coefficients based on the difference vector between the main path fusion features and the secondary path basic features and the element-wise product vector of the two, and to apply the dynamic compensation coefficients to the secondary path basic features to obtain secondary path compensation features. A combination module is used to combine the main road fusion feature and the secondary road compensation feature into a final fusion feature; The classification module is used to output the location category based on the final fusion features; The training constraint module is used to apply orthogonal constraints during training to reduce the correlation between the fused features of the main path and the basic features of the secondary path; A mutual enhancement module is used to mutually enhance the orthogonal constraints and the gating network during joint training, including: the orthogonal constraints increasing the difference vector between the fused features of the main path and the basic features of the secondary path; the gating network outputting dynamic compensation coefficients based on the increased difference vector and the element-wise product vector; and the gradient of the dynamic compensation coefficients updating the parameters of the fused secondary path during backpropagation.

[0017] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention uses a gated network to take the difference vector, element-wise product vector, and cosine similarity between the fused features of the main path and the basic features of the secondary path as input, and outputs a dynamic compensation coefficient that adapts to the correlation between the two. When the correlation between the main and secondary path features is high (modal consistency), the compensation coefficient approaches 0, and the compensation features of the secondary path are suppressed to avoid redundant information interference. When the correlation is low (modal conflict or missing features), the compensation coefficient increases, and the compensation features of the secondary path are enhanced, thereby effectively compensating for the discriminative information lost by the main path due to modal conflict. This transforms the fusion strategy from static preset to sample adaptive, thus achieving dynamic adaptive fusion and solving the fusion mismatch problem caused by changes in modal correlation.

[0018] This invention employs a dual-path structure that combines main-path fusion and secondary-path fusion. The main path utilizes channel attention enhancement and multilayer perceptron dimensionality reduction, focusing on capturing globally correlated semantic features between visual and auditory modalities. The secondary path uses linear projection and learnable weighted summation, focusing on transforming the two modalities into a common space while preserving their independent discriminative cues. The two paths are structurally differentiated: the main path emphasizes modal interaction and compression, while the secondary path emphasizes modal preservation and weighting. Together, they form a synergistic relationship where the main path provides a stable foundation, and the secondary path stores compensating potential. When modal correlation is high, the contribution of the secondary path is gating and suppressed, and the main path dominates classification; when modal correlation is low, the complementary information carried in the secondary path is dynamically released, effectively compensating the main path. This achieves a synergistic and complementary main-secondary dual-path structure, taking into account both global semantics and potential compensating information.

[0019] This invention applies orthogonal constraints during model training. By minimizing the absolute value of the cosine similarity between the fused features of the main path and the basic features of the secondary path, the two are forced to become orthogonal in the feature space. This forces the basic features of the secondary path to no longer repeat the information already captured by the main path, but instead learn feature representations that are differentiated and orthogonal to the main path. More importantly, this orthogonal constraint and the gating network synergistically enhance each other during joint training: the orthogonal constraint amplifies the differences between the features of the main and secondary paths, making the difference vector a more sensitive discriminative signal; the gating network learns appropriate dynamic compensation coefficients based on these differences and backpropagates the gradients back to the secondary path fusion module and feature extraction module, further encouraging the secondary path to learn complementary and orthogonal features to the main path. The joint optimization of these two approaches breaks the limitations of traditional independent regularization terms and prediction modules, achieving enhancement from forced separation to dynamic compensation and then to gradient feedback, significantly improving the robustness of the model in modal inconsistency scenarios. By jointly training with orthogonal constraints and gating networks, feature separation is forced and compensation selectivity is enhanced, avoiding the problem that the features of the master and slave paths tend to become homogeneous during training, causing the slave paths to lose their independent compensation value.

[0020] This invention introduces a random modality discarding strategy during training, randomly setting the contribution of visual or auditory modalities from the basic features of the secondary path to zero with a certain probability, while maintaining normal propagation of the main path fusion. This allows the secondary path fusion module to learn to provide meaningful features even when any modality is missing, and the gating network also learns to adaptively adjust the compensation coefficients when a modality is missing. This strategy, combined with the aforementioned orthogonal constraints and dynamic gating, gives the overall model a natural resistance to incomplete modal inputs, eliminating the need for additional missing modality processing branches, resulting in a simple structure and significant effects. This random modality discarding strategy enhances robustness to single-modal missing features in real-world scenarios, effectively addressing the issue of sensitivity to missing modalities after deployment, which often occurs in real-world micro-videos due to the default assumption of complete bimodal information during training, when there is often silence, static images, or audio noise pollution.

[0021] In the feature extraction stage, this invention employs an improved NNeXtVLAD network, which enhances nonlinearity through front-end ReLU activation and stabilizes the distribution through back-end L2 normalization, thereby improving feature separability while maintaining the original aggregation capability. The SE module in the main path fusion performs channel attention enhancement with minimal computational overhead, and the multilayer perceptron compresses the 512-dimensional features to 256 dimensions for subsequent combination. The entire inference process requires only one forward propagation, without iterative optimization or external memory, meeting the real-time requirements of micro-video classification scenarios; through compact and efficient feature aggregation and dimensionality reduction design, real-time inference capability is guaranteed. Attached Figure Description

[0022] Figure 1The flowchart shows a micro-video location classification method based on multimodal feature fusion provided in one embodiment of the present invention. Figure 2 This is a flowchart illustrating the framework of a micro-video location classification method based on multimodal feature fusion, provided in one embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] like Figures 1 to 2 As shown, the present invention provides a micro-video location classification method based on multimodal feature fusion, comprising the following steps: S100. Obtain the visual and auditory feature vectors of the micro-video; specifically: S110. Collect a micro-video dataset. For example, each video should be 10–30 seconds long, with a resolution of at least 720p and an audio sampling rate of 44.1kHz. Location categories should be manually labeled, with a total of 20 categories. The training set should contain 8000 videos, the validation set 1000 videos, and the test set 2000 videos.

[0025] S120. Perform visual preprocessing. Specifically, use FFmpeg to uniformly extract 30 keyframes at 25 frames per second, scale each frame to 224×224 pixels, and standardize them (mean values ​​are 0.485, 0.456, 0.406, and standard deviation values ​​are 0.229, 0.224, 0.225).

[0026] S130. Perform auditory preprocessing. Specifically, extract the audio stream, resample it to 16kHz mono, divide it into 200 equal-length segments (each segment approximately 0.96 seconds), perform a short-time Fourier transform on each segment and convert it into a log-Mel spectrum, which is then used as input to VGGish.

[0027] S140. Extract visual features. For example, using EfficientNet-B5 pre-trained on ImageNet as the backbone network, remove its top classification layer and extract feature vectors (e.g., 2048 dimensions) for each keyframe. Stack the features of each frame into a visual feature matrix.

[0028] S150. Extract auditory features, for example, using a VGGish model pre-trained on AudioSet to extract feature vectors (e.g., 128-dimensional) for each audio segment and stack them into an auditory feature matrix.

[0029] S160. The visual feature matrix and the auditory feature matrix are aggregated using a parallel NNeXtVLAD network to obtain visual feature vector and auditory feature vector (for example, the dimensions of the two can be set to be the same, such as 256 dimensions).

[0030] It should be noted that the NNeXtVLAD network is an improved structure based on NeXtVLAD. Specifically, its forward computation process is as follows: Input: Feature matrix (T is the number of frames / segments, and D is the feature dimension of each frame).

[0031] Front-end ReLU activation: Perform ReLU on the input features element by element. This operation, performed before grouping, aims to increase nonlinearity and suppress negative value interference.

[0032] Grouping and Residual Aggregation: The data is divided into G groups along the feature dimension, and residual aggregation based on soft allocation is performed within each group. Taking the g-th group as an example: Let its input feature matrix be ,in, =D / G.

[0033] This group has K learnable cluster centers. , ,..., }, each .

[0034] Characteristics of time t First, calculate the soft assignment weight for its belonging to the k-th cluster center: in, L2 norm, cluster center The number of K is , For summation index variables.

[0035] Then calculate the residual aggregation vector of the k-th cluster: in, Thus far, this group has yielded a total of K residual vectors { , ,..., }, where T represents the time step of the input feature matrix, which is the number of keyframes for the visual branch and the number of audio segments for the auditory branch.

[0036] Intra-group feature compression: This involves compressing the K residual vectors (total dimension) of each group. This is compressed into a fixed-length vector, and a learnable concatenation layer (without bias or activation function) is introduced: Concatenate all residual vectors of this group along the feature dimension to obtain a length of vector: It is mapped to a K-dimensional vector through a fully connected layer: in, Grouping The weights of the fully connected layer, output .

[0037] Different groups can use different methods independently. Alternatively, they can share the same fully connected layer to reduce the number of parameters (in which case all groups are required to...). Similarly, this condition is met in this embodiment.

[0038] Cross-group concatenation: output of all G groups By concatenating along the dimensions, we obtain the aggregated result: That is, the dimension is .

[0039] For example, consider a visual feature matrix: enter If the number of groups is set to G=16, then the feature dimension of each group is... Each group has K=16 cluster centers.

[0040] Each group yields 16 residual vectors (each with 128 dimensions), which are concatenated to form a 16×128=2048-dimensional vector. Fully connected layer Map it to a 16-dimensional vector .

[0041] Concatenating the 16-dimensional vectors of the 16 groups yields a 16×16=256-dimensional vector. When flattened, it becomes a 256-dimensional visual feature vector.

[0042] For example, consider a visual feature matrix: enter Set the number of groups G=32, and the feature dimension of each group. =128 / 32=4, K=8 in each group; Each group yields 8 residual vectors (each 4-dimensional), which are concatenated to form an 8×4=32-dimensional vector. Fully connected layer Map it to an 8-dimensional vector .

[0043] Concatenating the 8-dimensional vectors of the 32 groups yields a 32 × 8 = 256-dimensional vector. After being flattened, it becomes a 256-dimensional auditory feature vector.

[0044] Backend L2 normalization: flattens the aggregation result into Then perform L2 normalization: The final feature vector is obtained. L2 normalization projects feature vectors onto a unit hypersphere, which can eliminate interference caused by differences in feature magnitudes between different video segments, allowing the classifier to focus more on directional information.

[0045] Among them, the above-mentioned cluster centers and fully connected layer weights All of these are learnable parameters.

[0046] In this embodiment, the ReLU activation function added to the front end of the NNeXtVLAD network can suppress negative noise in the features and enhance the nonlinear expressive power of the features; the L2 normalization added to the back end eliminates the feature magnitude difference, making the subsequent classifier pay more attention to the directional information of the features, thereby improving the separability of the features and the generalization performance of the model.

[0047] S200. Perform main path fusion on the visual feature vector and auditory feature vector to obtain the main path fused feature; specifically: S210, Perform channel stitching. Specifically, stitch the visual feature vectors... and auditory feature vectors By splicing along the channel dimension (along the column direction), a shape is obtained. The matrix (fusion vector) For example, if ,but , This represents the dimension of the feature vector; in this example, it is set to 256. for A real matrix with 2 rows and 2 columns.

[0048] S220. Perform channel attention enhancement. This is done using the Squeeze-and-Excitation (SE) module, specifically: S221, Execute the Squeeze operation. Specifically, for... Global average pooling is performed on each channel to obtain the channel description vector. Its dimension is equal to the number of channels (2 in this case). That is... ,in, , .

[0049] S222, Perform the Excitation operation. Specifically, this involves capturing inter-channel dependencies through a two-layer fully connected network and generating channel weights. : ; in, These are the weights (importance coefficients) of the two channels, respectively. This is the channel description vector output by the Squeeze operation. , , This represents the compression ratio of the SE module. This operation correlates the weights of the two channels, achieving attention enhancement.

[0050] For example, compression ratio It can be set to 1, in which case the two fully connected networks achieve cross-channel information interaction with the minimum number of parameters; or a larger compression ratio can be set according to the feature dimension, which can be selected by those skilled in the art as needed.

[0051] S223. Perform the Scale operation. Specifically, scale the weights... Multiplying by the original features of the corresponding channel yields the enhanced visual features. and auditory features Then, the two are concatenated along the channel dimension to obtain the enhanced feature. : .

[0052] S230, Perform MLP dimensionality reduction. Specifically, Flattened into a one-dimensional vector, it is fed into a two-layer fully connected network, and the output is the main path fusion feature. : in, Main road integration features This is the weight matrix of the first layer of the MLP; This is the weight matrix of the second layer of the MLP; To be The flattened vector, For the first layer bias, For the second layer bias; .

[0053] For example, if the original feature vector has a dimension of 256, then the flattened dimension is 512. MLP can reduce its dimension to 256, that is... ∈ To ensure dimensionality matching when combining features with secondary features later, this embodiment uses fused features for the primary path. With road basic features All are designed with the same dimensions (e.g., 256 dimensions), so that subsequent element-by-element addition operations can be performed smoothly.

[0054] In this embodiment, the effective channels are adaptively enhanced through the SE module, the MLP compression dimension facilitates subsequent combination, and the main path features focus on global semantics.

[0055] S300. Perform path fusion on the visual feature vector and auditory feature vector to obtain the path basic features; S310, Project onto a public space, for example, using linear projection and tanh activation: in, , Output , Same as the input dimension (e.g., 256 dimensions).

[0056] S320, Learnable weighted summation, for example, introducing a learnable scalar. and (Through sigmoid constraints), calculate the basic features of the path: in, This is based on the fundamental features of the path, i.e., the original output before weighting. Dimensions and , Same, for example, , , These are the learnable weighted coefficients for the visual and auditory modalities, respectively, with dimensions / value ranges of [missing information]. By using sigmoid constraint, It is a hyperbolic tangent activation function with an output range of (-1, 1). , These are the weight matrices for the visual and auditory projection layers, respectively. , These are the bias vectors for the corresponding projection layers. , These are visual and auditory feature vectors, respectively.

[0057] In this embodiment, by eliminating modal distribution differences through a projection layer, the initial contributions of visual and auditory perception can be learned to balance the weights, and potential compensation information for the main road can be provided from road features.

[0058] S400. Input the main road fusion features and the secondary road basic features into the gating network. The gating network outputs a dynamic compensation coefficient based on the difference vector between the main road fusion features and the secondary road basic features and the element-wise product vector of the two. The dynamic compensation coefficient is applied to the secondary road basic features to obtain the secondary road compensation features. The value of the dynamic compensation coefficient changes with the correlation between the main road fusion features and the secondary road basic features.

[0059] Specifically, the input to the gating network is constructed. The core task of the gating network is to perceive the fused features of the main path. With road basic features The differences and synergies between them are analyzed to output appropriate dynamic compensation coefficients. To this end, this embodiment constructs the following three sets of input features with clear physical meanings, instead of directly inputting the main path fusion features. With road basic features This is to avoid information redundancy and reduce the difficulty of learning online: Element-wise absolute difference vector This vector directly reflects the degree of difference between the two features in each dimension; Element-wise product vector (⊙ represents the Hadamard product). This product vector can capture the co-activation patterns of two features across various dimensions: if both features have large positive or negative values ​​with large absolute values ​​in a certain dimension, the product result is large, indicating that the information in that dimension is consistent; conversely, if the signs are opposite, the product result is negative, indicating a conflict. This helps gating networks determine the local correlation between modalities.

[0060] element-wise absolute difference vector element-wise product vector The vectors are concatenated sequentially to form a long vector, which serves as the basic input to the gated network. This allows the gating network to directly obtain the differences and collaboration information between the master and slave paths without having to extract it from the original features, thereby improving the training efficiency and the accuracy of the compensation coefficients of the gating network.

[0061] For example, assuming the feature dimension N=256, then the element-wise absolute difference vector element-wise product vector Both are 256-dimensional vectors, concatenated. The dimension is 512.

[0062] In another preferred embodiment, to further enhance the gating network's ability to perceive global correlations, dynamically calculated cosine similarity can be added to the input. (Scalar).

[0063] For example, suppose the feature dimension N=256, then , All are 256-dimensional vectors. As a 1-dimensional scalar, after splicing The dimension is 256 + 256 + 1 = 513.

[0064] Gated networks can employ a two-layer fully connected network: First layer: , Second layer: .

[0065] in, ∈(0,1) represents the dynamic compensation coefficient, and the final path compensation feature is: .

[0066] It should be noted that the dynamic compensation coefficient in this embodiment is in scalar form. The entire set of basic features of the slave path is uniformly scaled, which dynamically adjusts the injection intensity of slave path information based on the global correlation between master and slave path features. When the correlation between master and slave path features is high, When the correlation approaches 0, the basic characteristics of the road are suppressed to avoid interference from redundant information; when the correlation is low, The overall contribution of road-based features is enhanced, effectively improving classification robustness.

[0067] Depending on the needs, the output of the gating network can also be changed to a vector with the same dimension as the features. and with Element-wise multiplication is performed to achieve more refined adaptive compensation.

[0068] The weights of the first layer of the gating network, For the second layer weights of the gating network, For the first layer bias, This is the second-level bias (scalar). This is the gated input vector.

[0069] S500: Combine the main road fusion features with the secondary road compensation features to obtain the final fusion features, and output the micro-video location category based on the final fusion features; For example, the main road fusion features and the secondary road compensation features are added element by element: , Specifically, by The data is fed into a classifier (e.g., a fully connected network connected to Softmax) and outputs the probability distribution of location categories in the micro-video.

[0070] This includes orthogonal constraints and joint training (including mutual reinforcement mechanisms) during the training process; specifically, During model training, orthogonal constraints are applied to minimize the fusion features of the main path. With road basic features The absolute value of the cosine similarity between them reduces their correlation: The total loss function is the sum of the classification cross-entropy loss and the orthogonality constraint regularization term: ; in, This is the regularization coefficient, which can be set to 0.01 for example. Cross-entropy classification loss, The orthogonal constraint loss value. ;in, The first for the real label Each component (one-hot encoded) has a value range of [value range missing]. , This is a category index, where K is the total number of venue categories. The first output of Softmax Class probability, with a range of values. , , It is the natural logarithm.

[0071] It is important to emphasize that orthogonal constraints and gating networks can create a synergistic enhancement effect during joint training. Specifically: Orthogonal constraint loss By minimizing and The absolute value of the cosine similarity between the features learned from the main road forces the features learned from the main road to tend to be orthogonal in the feature space.

[0072] Meanwhile, since the output of the NNeXtVLAD network in this scheme has adopted L2 normalization, and the main path fusion characteristics are... With road basic features Both inherit this normalization property; therefore, their magnitudes are constrained to the unit hypersphere. Under this condition, the difference vectors in the orthogonal directions... The modulus length will increase as the included angle increases (reaching its maximum value when the included angle is 90°). This makes the difference vector a more sensitive discriminative signal for modal inconsistency.

[0073] The gated network learns and outputs appropriate dynamic compensation coefficients based on this difference and element-wise product vectors. Furthermore, the gradient is backpropagated to the slave path fusion module and the feature extraction module through backpropagation, further encouraging the slave path to learn features that are complementary and orthogonal to the main path. The joint optimization of the two breaks the limitation of the traditional independent regularization term and prediction module, and realizes the synergistic enhancement from forced separation to dynamic compensation.

[0074] Due to the orthogonality constraint, the difference vector Similarity to cosine Becoming more sensitive and information-rich, gating networks can more easily learn from difference vectors. Element-wise product vector (and optional cosine similarity) ) to compensation coefficient The mapping outputs more accurate dynamic compensation coefficients. More importantly, Through multiplication It also participates in the calculation of subsequent classification loss, and its gradient updates the parameters of the path fusion module through backpropagation. Classification loss The gradient will encourage Carrying something that is beneficial to classification and has not been The captured discriminative information (complementary information) is then compared with... Orthogonal.

[0075] In one embodiment, to further enhance the flexibility of orthogonal constraints, this embodiment can incorporate main path fusion features. and from the basic features of the road The feature space is divided into multiple feature subspaces, and orthogonal constraints are applied independently to each subspace. The subspaces are divided equally: if the total feature dimension is N and the number of subspaces is M, then the dimension of each subspace is N / M (N must be divisible by M). Different subspaces can use different weighting coefficients. Details are as follows: set up Given N=256, M=4, and each subspace has a dimension of 64, meaning it is divided into 4 consecutive subvectors, each with a dimension of 64: For each subspace Calculate cosine similarity And configure independent weighting coefficients The total orthogonal constraint loss is: Weighting coefficient It can be preset (e.g.) =0.01, =0.02, =0.005, =0.01), which can also be used as a learnable parameter to be adaptively adjusted during training.

[0076] It should be noted that the above subspace partitioning method is independent of the grouping operation in the aforementioned NNeXtVLAD network; the two belong to different levels of feature processing. Those skilled in the art can flexibly divide the number and dimensions of subspaces according to the feature dimensions.

[0077] By employing subspace orthogonal constraints compared to global single constraints, the redundant correlation of master-slave path features in local dimensions can be further reduced, and the fine-grained adjustment capability of dynamic compensation coefficients can be improved, especially in scenarios with modal conflicts.

[0078] In one embodiment, during training, contributions from a single modality in the road-based features are randomly discarded with a preset probability, while the normal propagation of the main road fusion features is preserved.

[0079] To further enhance the model's robustness to real-world scenarios with missing single-modal information (such as silent videos, static images, or strong noise pollution) during training, a modality random discarding strategy is introduced. This strategy only applies to the secondary path fusion branch; the primary path fusion branch always uses the complete original features, as detailed below: During the forward propagation of each training sample, a decision is made whether to discard the sample with a preset probability p (e.g., 0.2). If not, normal computation proceeds. If executed, the visual modality or the auditory modality will be discarded with an equal probability of 0.5 (at most one modality will be discarded each time, and not two at the same time).

[0080] In obtaining projection features and This is performed afterward and before the weighted sum is calculated. If it is determined that the visual modality should be discarded, then it is forced to... Set to with All-zero tensors of the same shape (i.e., 0∈ Auditory projection features If the auditory modality is determined to be discarded, then... Set to zero. It remains unchanged. The zeroing operation takes effect during forward propagation, and the corresponding backpropagation gradient is also zero.

[0081] Visual feature vectors in the main path fusion branch and auditory feature vectors The primary path does not participate in modality dropout and always maintains the original values. It provides stable, complete modality-based classification capabilities, avoiding training instability caused by random dropout. The secondary path, on the other hand, is forced to learn to generate meaningful compensating features even when a single modality is missing, while the gating network also learns to adaptively adjust the compensation coefficients when a modality is missing. Together, they ensure the model neither overly relies on bimodal completeness nor fails to leverage the advantages of bimodal completeness.

[0082] In a preferred embodiment, the preset probability pp can be dynamically adjusted based on the real-time correlation between the visual feature vector and the auditory feature vector to further enhance the model's adaptability to modal correlation. The specific adjustment method is as follows: Calculate the cosine similarity between the visual and auditory feature vectors of the current sample. .

[0083] When cosine similarity When the value is less than the first preset threshold (low modal correlation, prone to conflict or missing), the discard probability p=0.1 is reduced to retain more trail information for compensation; When cosine similarity When the value exceeds the second preset threshold (high modal correlation and information redundancy), the discard probability p=0.3 is increased to enhance the robustness of the slave path to single modes. When cosine similarity Greater than the first preset threshold, and the cosine similarity When the value is less than the second preset threshold, the preset probability p=0.2 is maintained. Through this dynamic adjustment strategy, the model can automatically adjust the regularization strength based on the modal consistency of the samples, further improving training efficiency and final generalization performance. The first preset threshold (e.g., 0.3) is less than the second preset threshold (e.g., 0.7), and can be set accordingly as needed.

[0084] By employing this modality random drop-off strategy, the model adapts to modality missing conditions during the training phase and exhibits high robustness to single-modal inputs without any special processing during inference.

[0085] Other training settings (such as optimizer, learning rate, data augmentation, etc.) can be chosen from conventional options in the field, such as: Adam optimizer, initial learning rate of 0.001, batch size of 32, random horizontal flipping of visual frames, brightness adjustment, addition of Gaussian noise to audio, early stopping strategy, etc.

[0086] The application of the present invention is illustrated by the following examples: For the micro-videos to be categorized, perform the following steps: During the training phase, visual and auditory feature matrices are extracted, and the visual feature vector is obtained through NNeXtVLAD. Auditory feature vector ; Calculate the main road fusion features and from the basic features of the road Will and Input the gating network to obtain the dynamic compensation coefficient γ Calculate y′= + Feed y′ into the softmax classifier and output the location category with the highest probability. The entire reasoning process requires only one forward propagation, no orthogonal constraints are needed, and there is no additional iteration overhead.

[0087] The application of this invention will be illustrated below using the 'restaurant micro-video' scenario as an example: Video content: The visuals show a dining table, tableware, and food; the auditory elements include the sounds of tableware clattering and people talking, with the content consistent with the restaurant.

[0088] Processing procedure: The main path fusion captures strongly correlated visual and auditory features (e.g., visual features of the dining table and auditory sounds of cutlery, which are strongly correlated), y1 has high discriminative power, and it is based on the path's fundamental features. It also contains similar information, but with The differences are small, and the difference vector in the input of the gating network is relatively small. The cosine similarity is small, close to 1, and the output compensation coefficient γ is very small (e.g., close to 0.1). The final feature y′= +0.7 ≈ The main route is still prioritized to avoid redundancy. The classifier correctly outputs "restaurant".

[0089] In this embodiment, dynamic gating automatically reduces interference from the slave path, ensuring stable classification.

[0090] The following example illustrates the application of this invention using a modal inconsistency scenario where the video is from a restaurant but the audio is a skiing discussion: Video content: The visuals are still of a restaurant, but the background audio shows people excitedly discussing skiing (possibly from the restaurant TV or conversations at neighboring tables).

[0091] Processing procedure: During the main road fusion, visual features point to "restaurant," but auditory features point to "skiing" or "outdoors," creating a conflict. The main road features... Fuzzy (modal conflict, low classification confidence), from road-based features Through projection and weighting, some auditory information is still retained, but it differs greatly from visual features, as shown in the difference vector. The cosine similarity is large, close to 0, and the gating network perceives this difference, outputting a large compensation coefficient γ (e.g., 0.7). Finally, the features are fused. = +0.7 The auditory compensation features of the road were significantly enhanced, making up for the discriminative power lost due to modal conflict on the main road. The classifier integrates visual restaurant cues and anomalous information in the auditory context (but after training, the model has learned to ignore irrelevant skiing discussions and still outputs "restaurant"), correctly outputting "restaurant".

[0092] In this embodiment, dynamic gating automatically increases compensation when modal inconsistencies occur, avoiding misjudgments caused by auditory interference in the fixed fusion method.

[0093] The application of this invention will be illustrated below using the scenario of 'silent video' with modal absence as an example: Video content: The user-uploaded video has no sound (no audio at all).

[0094] Processing procedure: During the training phase, a modality-based random dropout strategy was adopted, as the model had already encountered similar situations; auditory feature vectors Near-zero vectors, from the basic features of the road It mainly relies on visual projection, and due to the lack of hearing, the main path fusion is still effective (visual is dominant). The discrimination is good, and the differences between the basic road features and the main road are small (due to the lack of auditory perturbation); the gating network outputs a small γ, and the final features are mainly based on the main road, resulting in correct classification.

[0095] In this embodiment, the model is made robust to single-modal inputs by random modal dropout training, and dynamic gating automatically suppresses invalid slave paths.

[0096] Through joint training with dynamic gating and orthogonal constraints, this invention significantly improves the robustness and classification accuracy of the model in modal inconsistency and modality loss scenarios. Compared to the comparison model without orthogonal constraints, applying orthogonal constraints significantly reduces the correlation between the basic features of the secondary road and the fused features of the primary road, making it more sensitive to difference signals. The gating network can output more accurate dynamic compensation coefficients, thereby effectively supplementing the primary road information and avoiding redundant interference. The modality random discarding strategy further enhances the model's adaptability to single-modality loss in real-world scenarios. The overall scheme has real-time inference capabilities, requiring only one forward propagation to complete classification.

[0097] The foregoing description of specific exemplary embodiments of the present invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it is obvious that many changes and variations can be made based on the above teachings. Although embodiments of the invention have been shown and described, these specific embodiments are merely explanations of the invention and are not intended to limit it. The specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. The purpose of selecting and describing exemplary embodiments is to explain the specific principles of the invention and its practical application, so that those skilled in the art, after reading this specification, can make modifications, substitutions, variations, and various choices and changes to the embodiments as needed without departing from the principles and spirit of the invention, provided that such modifications, substitutions, variations, and choices and changes are within the scope of the claims of the invention and are protected by patent law.

Claims

1. A micro-video location classification method based on multimodal feature fusion, characterized in that, include: Obtain the visual and auditory feature vectors of the micro-video; The visual feature vector and auditory feature vector are fused together to obtain the main path fused feature. The visual feature vector and auditory feature vector are fused from the path to obtain the basic features from the path; The main road fusion feature and the secondary road basic feature are input into a gating network. The gating network outputs a dynamic compensation coefficient based on the difference vector between the main road fusion feature and the secondary road basic feature and the element-wise product vector of the two. The dynamic compensation coefficient is then applied to the secondary road basic feature to obtain the secondary road compensation feature. The main road fusion feature is combined with the secondary road compensation feature to obtain the final fusion feature; The micro-video location category is output based on the final fusion features; During model training, orthogonal constraints are applied to reduce the correlation between the fused features of the main road and the basic features of the secondary road. The orthogonal constraints and the gating network reinforce each other during joint training, including: The orthogonal constraint increases the difference vector between the main road fusion feature and the secondary road basic feature; The gated network outputs dynamic compensation coefficients based on the increased difference vector and the element-wise product vector; The gradient of the dynamic compensation coefficient updates the parameters of the path fusion during backpropagation.

2. The micro-video location classification method based on multimodal feature fusion according to claim 1, characterized in that, The gated network is a multi-layer fully connected network, and its output is limited to between 0 and 1 by the sigmoid function.

3. The micro-video location classification method based on multimodal feature fusion according to claim 2, characterized in that, The orthogonal constraint is achieved by minimizing the absolute value of the cosine similarity between the fused features of the main road and the basic features of the secondary road.

4. The micro-video location classification method based on multimodal feature fusion according to claim 1, characterized in that, The main road fusion includes: The visual feature vector and the auditory feature vector are concatenated, enhanced by channel attention, and then reduced in dimensionality by a multilayer perceptron. The channel attention enhancement uses a Squeeze-and-Excitation module.

5. The micro-video location classification method based on multimodal feature fusion according to claim 4, characterized in that, The path fusion includes: The visual feature vector and the auditory feature vector are transformed into a common space through linear projection and activation function, respectively, and then weighted and summed using learnable weights; The activation function is the tanh function, and the learnable weights are restricted to between 0 and 1 by the sigmoid function.

6. The micro-video location classification method based on multimodal feature fusion according to claim 5, characterized in that, The visual feature vector and auditory feature vector are obtained through parallel temporal aggregation networks, which are NNeXtVLAD networks, temporal attention pooling networks, or deformable convolutional temporal aggregation networks. The NNeXtVLAD network adds ReLU activation to the front end of the NeXtVLAD structure and L2 normalization to the back end.

7. The micro-video location classification method based on multimodal feature fusion according to claim 2, characterized in that, The orthogonal constraints are applied to multiple feature subspaces of the main road fusion feature and the secondary road basic feature, respectively, with different weights of orthogonal constraint coefficients applied to different subspaces.

8. The micro-video location classification method based on multimodal feature fusion according to claim 1, characterized in that, The input to the gated network also includes the cosine similarity between the main path fusion features and the secondary path basic features.

9. The micro-video location classification method based on multimodal feature fusion according to claim 1, characterized in that, During training, the contribution of a single modality in the basic features of the secondary path is randomly discarded with a preset probability, while the normal propagation of the fused features of the main path is preserved. The preset probability is dynamically adjusted according to the real-time correlation between the visual feature vector and the auditory feature vector: when the correlation is lower than the threshold, the modality discard probability is reduced, and when the correlation is higher than the threshold, the modality discard probability is increased.

10. A micro-video location classification system based on multimodal feature fusion, characterized in that, include: The feature extraction module is used to obtain the visual and auditory feature vectors of the micro-video; The main road fusion module is used to generate main road fusion features; The path fusion module is used to generate basic path features; The gated network module is used to output dynamic compensation coefficients based on the difference vector between the main path fusion features and the secondary path basic features and the element-wise product vector of the two, and to apply the dynamic compensation coefficients to the secondary path basic features to obtain secondary path compensation features. A combination module is used to combine the main road fusion feature and the secondary road compensation feature into a final fusion feature; The classification module is used to output the location category based on the final fusion features; The training constraint module is used to apply orthogonal constraints during training to reduce the correlation between the fused features of the main path and the basic features of the secondary path; A mutual enhancement module is used to mutually enhance the orthogonal constraints and the gating network during joint training, including: the orthogonal constraints increasing the difference vector between the fused features of the main path and the basic features of the secondary path; the gating network outputting dynamic compensation coefficients based on the increased difference vector and the element-wise product vector; and the gradient of the dynamic compensation coefficients updating the parameters of the fused secondary path during backpropagation.

Images