Short video homogenization identification method and system based on multiple modes, and medium

Abstract

The invention relates to the technical field of computers, and provides a short video homogenization recognition method and device based on multiple modalities and a medium, and the method comprises the steps: extracting video multi-granularity features and text multi-granularity features of a to-be-recognized sample pair; calculating the initial similarity of each granularity level based on the video and text multi-granularity features, and generating a feature guide vector based on the initial similarity; based on the feature guide vector, screening out a target feature from the features of the plurality of selected granularity hierarchies, and fusing the target feature and the features of the selected granularity hierarchies to generate a semantic enhancement feature; fuzzy semantic modeling is carried out on the semantic enhancement features, a feature weight distribution result is obtained through implicit rule reasoning, and the new similarity of the selected granularity level is determined based on the feature weight distribution result; and fusing the new similarity of the selected granularity hierarchy and the weighted similarity of the remaining granularity hierarchy to generate a homogenization recognition result of the to-be-recognized sample pair. According to the scheme, the accuracy and robustness of short video homogenization recognition are improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, system, and medium for identifying homogenization in short videos based on multimodality. Background Technology

[0002] With the widespread adoption of short video platforms and the lowering of content creation barriers, users upload millions of videos daily. This explosive growth has also led to a surge in copycat creations aimed at gaining traffic, characterized by similar visual layouts, titles, and captions, as well as highly similar copywriting – a phenomenon known as short video homogenization. This homogenization results in two main problems: firstly, the platform is flooded with low-cost copied videos, reducing the visibility of original content, squeezing the space for high-quality content dissemination, and causing an imbalance in the content ecosystem; secondly, it leads to frequent copyright infringements and high costs of legal action, harming the rights of creators.

[0003] However, existing methods for identifying homogenization in short videos primarily rely on feature distance as the criterion. This approach has technical shortcomings in at least the following aspects: First, it struggles to uncover the correlation between semantic expression and the interwoven changes in visual events, resulting in a lack of multimodal semantic depth. Second, fine-grained discrimination mechanisms are insufficient, and coarse-grained features easily obscure key details. Furthermore, the high-dimensional feature vectors extracted by the encoder suffer from semantic boundary ambiguity, and there is currently no suitable strategy for handling feature ambiguity, leading to insufficient ability to distinguish the importance of key features. Summary of the Invention

[0004] To address the problems of existing methods for identifying homogenized short videos, such as lack of multimodal semantic depth, coarse-grained features easily obscuring key details, and blurred semantic boundaries, this invention proposes a method, system, and medium for identifying homogenized short videos based on multimodality.

[0005] To achieve the above objectives, one aspect of the present invention provides a method for short video homogenization identification based on multimodality, comprising: Extract multi-granularity features from the video and text of the sample pairs to be identified; The initial similarity at each granularity level is calculated based on the video multi-granularity features and the text multi-granularity features, and a feature guidance vector is generated based on the initial similarity. Based on the feature guidance vector, target features are selected from features at several selected granularity levels, and the target features and features at the selected granularity levels are fused to generate semantically enhanced features; The semantic enhancement features are modeled using fuzzy semantics, and the feature weight allocation results are obtained through implicit rule reasoning. Based on the feature weight allocation results, a new similarity at the selected granularity level is determined. By fusing the new similarity at the selected granularity level with the weighted initial similarity at the remaining granularity levels, a homogeneous identification result for the sample pair to be identified is generated.

[0006] In some embodiments, performing fuzzy semantic modeling on the semantically enhanced features and obtaining the feature weight allocation result through implicit rule reasoning includes: The membership degree of the semantic enhancement features at the selected granularity level in each fuzzy set is calculated using a composite Gaussian membership function. Based on preset fuzzy rules, the membership degree of the semantic enhancement features in the fuzzy set involved by the corresponding fuzzy rule is fused to obtain the activation intensity of each fuzzy rule; The weights of each fuzzy rule are constructed based on the membership degree distribution and the peak values ​​of each fuzzy set. The weights of each fuzzy rule, their corresponding response values, and activation intensities are weighted and fused to generate the feature contribution of the semantic enhancement feature at the selected granularity level. The feature contribution of the semantic enhancement features at the selected granularity level is normalized to obtain the feature weight allocation result of the selected granularity level.

[0007] In some embodiments, the step of fusing the new similarity at the selected granularity level and the weighted initial similarity at the remaining granularity levels to generate a result for homogenization identification of the sample pairs to be identified includes: The initial similarity of the remaining granularity level is multiplied by the corresponding learnable similarity weight to obtain the weighted initial similarity of the remaining granularity level, wherein the learnable similarity weight is generated through parameterized mapping. The new similarity at the selected granularity level is weighted and summed with the weighted initial similarity at the remaining granularity levels to obtain the comprehensive similarity for the homogeneous identification of the sample to be identified. In response to the overall similarity exceeding a preset threshold, the sample pair to be identified is determined to be homogeneous content; If the overall similarity does not exceed a preset threshold, the sample pair to be identified is determined to be non-homogeneous content.

[0008] In some embodiments, determining the new similarity of the selected granularity level based on the feature weight allocation result includes: Based on the semantic enhancement features of the selected granularity level, the feature similarity of the selected granularity level is recalculated; The new similarity of the selected granularity level is obtained by multiplying the recalculated feature similarity of the selected granularity level with the feature weight allocation result of the selected granularity level.

[0009] In some embodiments, the step of selecting target features from features at several selected granularity levels based on the feature guidance vector, and fusing the target features and the features at the selected granularity levels to generate semantically enhanced features includes: Calculate the similarity between the feature guidance vector and all features at several selected granularity levels, and select a preset number of features with the highest similarity at the selected granularity levels as target features; The target feature is semantically consistent with the global features at its corresponding granularity level, and then aggregated features are generated through normalized weighting. The aggregated features are concatenated with all features of the selected granularity level and then jointly encoded to generate the semantically enhanced features of the selected granularity level.

[0010] In some embodiments, generating the feature guidance vector based on the initial similarity includes: The similarity at all granular levels is combined into an input vector and fed into the multilayer perceptron. Based on the input vector, the weight distribution of each granularity level is predicted through the first set of learnable parameters. The similarity of each granularity level is weighted based on the weight distribution to obtain the weighted similarity. The weighted similarity mapping is used to generate a semantic cue vector, which serves as a feature guidance vector, through the second set of learnable parameters. The learnable parameters are optimized through video classification supervision.

[0011] In some embodiments, calculating the initial similarity at each granularity level based on the video multi-granularity features and the text multi-granularity features includes: Based on the video multi-granularity features and the text multi-granularity features, the feature vectors of the sample pairs to be identified at the same granularity level are normalized respectively. Calculate the cosine similarity between the normalized feature vectors of the sample pairs to be identified at each same granularity level, and use it as the initial similarity for each granularity level.

[0012] In some embodiments, the extraction of video multi-granularity features and text multi-granularity features of the short video sample pairs to be identified includes: Based on the video data and text data of the short video to be identified in the sample pair to be identified, extract the corresponding visual multi-granularity features and text multi-granularity features; Based on the video data and text data of the reference video in the sample pair to be identified, the corresponding visual multi-granularity features and text multi-granularity features are extracted.

[0013] Another aspect of the present invention provides a system for identifying homogenization in short videos based on multimodal methods, comprising: The first module is used to extract video multi-granularity features and text multi-granularity features of the sample pairs to be identified; The second module is used to calculate the initial similarity of each granularity level based on the video multi-granularity features and the text multi-granularity features, and to generate a feature guidance vector based on the initial similarity. The third module is used to select target features from features at several selected granularity levels based on the feature guidance vector, and to fuse the target features and the features at the selected granularity levels to generate semantically enhanced features. The fourth module is used to perform fuzzy semantic modeling on the semantic enhancement features, obtain the feature weight allocation result through implicit rule reasoning, and determine the new similarity of the selected granularity level based on the feature weight allocation result; The fifth module is used to fuse the new similarity of the selected granularity level and the weighted initial similarity of the remaining granularity levels to generate the homogeneous recognition result of the sample pair to be identified.

[0014] In another aspect of the present invention, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the method for multimodal short video homogenization identification described above.

[0015] This invention offers at least the following advantages: It proposes a multimodal method for identifying homogenization in short videos. This method extracts multi-granular features from the video (for representing spatiotemporal layout) and multi-granular features from the text (for representing semantic expression), enabling joint analysis of visual events and text. This overcomes the limitations of traditional single-modal feature distance discrimination and can accurately capture homogenization patterns amidst multimodal interweaving and change. Based on feature-guided vectors, target features (such as key image blocks / text words) are selected and fused to generate semantically enhanced features. These features focus on local details that play a decisive role in homogenization discrimination, avoiding the obscuring of key semantics by coarse-grained features and reducing misjudgments caused by overlooking subtle differences. By modeling the semantically enhanced features through implicit rule reasoning and outputting feature weight allocation results, the method effectively avoids the rule explosion risk of traditional fuzzy systems, adapts to the fuzzy characteristics of high-dimensional feature semantic boundaries, and quantifies the importance differences of different features within the same granularity, thereby enhancing the ability to focus on key features. By fusing the new similarity at the selected granularity level (optimizing local similarity) with the weighted initial similarity at the remaining granularity levels (preserving global similarity), this approach balances local detail optimization with global consistency, avoiding misjudgments caused by single-granularity biases and improving the accuracy of homogenization recognition for complex short videos. Overall, this solution significantly improves the accuracy and robustness of short video homogenization recognition through multi-granularity feature collaborative optimization and fuzzy semantic modeling.

[0016] Meanwhile, the system and medium for homogenization recognition of short videos based on multimodal modes of the present invention can also achieve the above-mentioned technical effects, which will not be elaborated here. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0018] Figure 1 The flowchart shown is a method for identifying homogenization of short videos based on multimodality, provided in the first embodiment of the present invention. Figure 2 The flowchart shown is for another embodiment of multimodal homogenization identification provided by the present invention; Figure 3 The flowchart shown is for another embodiment of implicit high-dimensional fuzzy inference provided by the present invention; Figure 4 A schematic diagram of the structure of a system for identifying homogenization of short videos based on multimodal modes, provided by another embodiment of the present invention, is shown. Figure 5 The diagram shown is a schematic representation of a computer-readable storage medium provided according to another embodiment of the present invention. Detailed Implementation

[0019] The following describes embodiments of the present invention. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various alternative forms.

[0020] Furthermore, it should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or apparatus that comprises a list of elements may include not only those elements but also elements not expressly listed or inherent to such process, method, article, or apparatus.

[0021] One or more embodiments of this application will now be described with reference to the accompanying drawings.

[0022] Based on the above objectives, the first aspect of the present invention proposes an embodiment of a method for identifying homogenization of short videos based on multimodality. Figure 1 The flowchart shown is a method for short video homogenization identification based on multimodality, provided as an embodiment of the present invention. Figure 1 As shown, a method for identifying homogenization in short videos based on multimodal methods includes: S1. Extract video multi-granularity features and text multi-granularity features of the sample pairs to be identified.

[0023] Specifically, the sample pairs to be identified include the short video to be identified and its corresponding reference video. Video multi-granularity features include, but are not limited to, global video features, frame-level video features, and image patch-level features. Text multi-granularity features include, but are not limited to, sentence-level text features and word-level text features.

[0024] S2. Calculate the initial similarity of each granularity level based on video multi-granularity features and text multi-granularity features, and generate feature guidance vectors based on the initial similarity.

[0025] Specifically, the feature guidance vector is a semantic cue vector generated from the similarity of each granularity level in the video multi-granularity features and the text multi-granularity features, used to guide local feature optimization. For example, in a specific embodiment, the video multi-granularity features include video global features, video frame-level features, and image block-level features, and the text multi-granularity features include text sentence-level features and text word-level features. Then, the feature guidance vector is a semantic cue vector generated from the similarity of the above five granularity levels.

[0026] S3. Based on the feature-guided vector, select target features from several selected granularity levels, fuse the target features and the selected granularity levels to generate semantically enhanced features.

[0027] Specifically, the selected granularity level is one or more fine-grained levels from both video multi-granularity features and text multi-granularity features, such as the finest one or two levels. For example, in one specific embodiment, video multi-granularity features include video global features, video frame-level features, and image block-level features; text multi-granularity features include text sentence-level features and text word-level features. The selected granularity level specifically refers to the three types of fine-grained levels: video frame-level features, image block-level features, and text word-level features. The remaining granularity levels (those not selected) specifically refer to the two types of coarse-grained levels: video global features and text sentence-level features.

[0028] S4. Perform fuzzy semantic modeling on the semantic enhancement features, obtain the feature weight allocation result through implicit rule reasoning, and determine the new similarity at the selected granularity level based on the feature weight allocation result.

[0029] Specifically, implicit rule reasoning refers to a fuzzy reasoning process based on the fusion of composite Gaussian membership functions, Dombi T norms, and rule weights. Implicit rule reasoning does not manually define any specific rules; instead, it uses a unified, differentiable mathematical formula. The system adjusts learnable parameters in the formula, such as the membership function center c. At Rule response value B tThis allows the model to learn the optimal reasoning strategy from the data, essentially providing an infinitely large, adaptively generated rule base. Fuzzy semantic modeling maps the semantically ambiguous numerical features extracted by deep learning models to human-understandable semantic concepts. For example, a feature of an image patch with a certain dimension value of 0.7 is neither absolutely high nor absolutely low similarity. Fuzzy modeling calculates the degree to which it belongs to high, medium, and low semantic concepts; for instance, it might calculate a degree of 0.8 for high similarity and 0.2 for medium similarity.

[0030] S5. Fuse the new similarity of the selected granularity level with the weighted initial similarity of the remaining granularity levels to generate a homogeneous identification result for the sample pairs to be identified.

[0031] Specifically, the remaining granularity levels are unselected granularity levels. Their initial similarity processing is not calculated through a fuzzy inference system, but rather processed using a multilayer perceptron mapping and a normalized exponential function. The weighted similarity of the remaining granularity levels = initial similarity × learnable similarity weight. This learnable similarity weight is generated through parameter mapping; more specifically, it is a learnable weight obtained through an MLP (Multilayer Perceptron) and a normalized exponential function. Therefore, the processing method for the initial similarity of the remaining granularity levels differs from that of the selected granularity levels. The weighted similarity obtained from processing the initial similarity of the remaining granularity levels is then weighted and summed with the new similarity of the selected granularity levels to obtain a comprehensive similarity. Based on the comprehensive similarity and the judgment result of a preset threshold, the homogeneity recognition result of the sample pairs to be identified is determined. Specifically, in the sample pair to be identified (including the short video to be identified and the reference video), the preset threshold is the similarity judgment value for determining whether the short video to be identified and the reference video are homogeneous. In a specific embodiment, if the overall similarity exceeds the preset threshold, it means that the content similarity between the short video to be identified and the reference video is extremely high, and the short video to be identified and the reference video are determined to be homogeneous. Conversely, if the content similarity between the short video to be identified and the reference video is low, it means that the short video to be identified and the reference video are determined to be non-homogeneous.

[0032] The aforementioned method for identifying homogenization in short videos based on multimodal approaches significantly improves the accuracy and robustness of this identification through multi-granularity feature collaborative optimization and fuzzy semantic modeling.

[0033] Please refer to several embodiments of the present invention. Figure 2In S1, video multi-granularity features and text multi-granularity features of the short video sample pair to be identified are extracted, including: extracting corresponding visual multi-granularity features and text multi-granularity features based on the video data and text data of the short video to be identified in the sample pair to be identified; and extracting corresponding visual multi-granularity features and text multi-granularity features based on the video data and text data of the reference video in the sample pair to be identified.

[0034] Specifically, visual multi-granularity features can be extracted using an image-time encoder, and text multi-granularity features can be extracted using a text encoder. Both the image-time encoder and the text encoder are pre-trained encoders, and their training process is not the focus of this application, so it will not be described in detail here.

[0035] The above method extracts multi-granular features from videos (to represent spatiotemporal layout) and multi-granular features from text (to represent semantic expression) to achieve joint analysis of visual events and text, breaking through the limitations of traditional single-modal feature distance discrimination and accurately capturing homogeneous patterns in multimodal interweaving and change.

[0036] Please refer to several embodiments of the present invention. Figure 2 In S2, the initial similarity of each granularity level is calculated based on video multi-granularity features and text multi-granularity features, including: normalizing the feature vectors of the sample pairs to be identified at the same granularity level based on video multi-granularity features and text multi-granularity features; calculating the cosine similarity between the normalized feature vectors of the sample pairs to be identified at each same granularity level, as the initial similarity of each granularity level.

[0037] As a feasible implementation, since the similarity performance of different sample pairs varies across different modalities and granularities when analyzing video homogenization, it is necessary to calculate the similarity of sample pairs across the aforementioned five granularities. During the calculation, all features involved undergo L2 regularization normalization, such as video global features, frame features, image patch features, text sentence-level features, and text word-level features, each represented by a corresponding symbol. The similarity of sample pairs across these five granularities is calculated as follows: (Formula 1) (Formula 2) (Formula 3) (Formula 4) (Formula 5) in, This is the result of L2 regularization; the rest are similar representations. , , , , These represent the video global features, frame features, image patch features, text sentence-level features, and text word-level features after regularization and normalization. Nvi represents the number of video frames, C represents the number of image patches per frame, and Nti represents the number of text words. Sv,Sf,Sp,Ss Sw The initial similarity scores for each granular level are calculated by using the video global, frame-level, image patch-level, and text sentence-level and word-level similarity scores for the sample pairs.

[0038] The steps described above for calculating the initial similarity at each granularity level help ensure the comparability of cross-modal features.

[0039] Please refer to several embodiments of the present invention. Figure 2 In S2, a feature guidance vector is generated based on the initial similarity, including: combining the similarities of all granular levels into an input vector and inputting it into a multilayer perceptron; based on the input vector, predicting the weight distribution of each granular level through a first set of learnable parameters, and weighting the similarity of each granular level based on the weight distribution to obtain a weighted similarity; mapping the weighted similarity to generate a semantic cue vector through a second set of learnable parameters, which serves as the feature guidance vector; wherein, the learnable parameters are optimized through video classification supervision, and the prediction strategy of granular level weights can be dynamically adjusted.

[0040] Specifically, the first set of learnable parameters is used to implement the multilayer perceptron structure for granular weight prediction; the second set of learnable parameters is used to implement the multilayer perceptron structure for dimensionality reduction and cross-granularity association extraction; the video classification supervision optimization adopts the cross-entropy loss function, and the optimization objective is to improve the accuracy of weight allocation at the discriminative granularity.

[0041] As a specific implementation, the obtained five-dimensional similarity vector [Sv, Sf, Sp, Ss, Sw] is input into a similarity-aware cue generator. The first branch of this generator, a multilayer perceptron, predicts the weight distribution of the five granularities. After weighting the obtained weights to the similarity scores of the corresponding granularities, the vectors are input into a second branch, a multilayer perceptron, to extract semantic cue vectors. These semantic cue vectors are used for subsequent feature refinement. To optimize this process, a classification task is introduced: a lightweight multilayer perceptron network is trained to classify videos, and supervised training is performed using the cross-entropy loss function. Through this training, the model can continuously optimize the granularity weight allocation strategy during training, strengthening the most discriminative granularities, and guiding subsequent feature enhancement through the semantic cue vectors, ultimately achieving reverse adjustment of the multi-granularity weighting strategy from the classification objective.

[0042] The aforementioned method for generating feature guidance vectors enables adaptive reinforcement of discriminative granularity through learnable parameters. Classification supervision helps guide attention to granularities that are strongly correlated with homogeneity, and overall, it helps to achieve dynamic optimization of the balance of contributions from multiple granularities.

[0043] Please refer to several embodiments of the present invention. Figure 2 In S3, based on the feature guidance vector, target features are selected from features at several selected granularity levels. The target features and features at the selected granularity levels are then fused to generate semantically enhanced features. This includes: calculating the similarity between the feature guidance vector and all features at several selected granularity levels; selecting a preset number of features with the highest similarity at the selected granularity level as target features; performing semantic consistency verification on the target features and the global features at their corresponding granularity levels, and then generating aggregated features through normalization and weighting; and concatenating the aggregated features with all features at the selected granularity level and jointly encoding them to generate semantically enhanced features at the selected granularity level.

[0044] Specifically, target feature selection involves calculating the cosine similarity between the semantic cue vector and the feature, and selecting the top K features with the highest similarity, where K is a preset parameter. Semantic consistency verification measures the similarity between the target feature and the corresponding global feature, and generates aggregated features through Softmax normalization and weighting. Joint encoding involves concatenating the aggregated features with the original features and inputting the result into a multi-layer neural network encoder.

[0045] Among them, the selected granularity levels are video frame level, image block level, and text word level. The feature guidance vector is the semantic cue vector. The target feature refers to the feature that plays a key role in determining video homogeneity. The semantic enhancement feature is the feature that is more semantically consistent after processing.

[0046] As a feasible implementation, the similarity between the feature guidance vector and all features at the selected granularity level can be calculated using cosine similarity, as detailed below. (Formula 6) Here, `prompt` represents the semantic prompt vector, and `v` represents the feature vector at the selected granularity level. Since feature refinement is only performed at the video frame level, image block level, and text word level, `v` here represents the video frame-level, image block-level, and text word-level feature vectors, respectively. It is understood that in other embodiments, the meaning of `v` will differ depending on the selected granularity level. Multiple features with the highest similarity at the selected granularity level (a predetermined number) are selected as target features; for example, the top K most similar features are selected.

[0047] As a specific implementation, the semantic consistency between the target feature and its corresponding granularity level global features is checked. The semantic consistency between the target feature vector and the global feature vector is further measured, and the similarity between the two is processed by a normalized exponential function as the feature weight, as detailed below. (Formula 7) Among them, z ci z represents a feature vector at this granularity. cls This represents the global feature vector at this granularity.

[0048] Then, after normalization and weighting, aggregated features, i.e., coarse-level aggregated features, are generated, as follows: (Formula 8) For example, aggregated features are concatenated with all features at a selected granularity level and then jointly encoded to generate semantically enhanced features at the selected granularity level. Specifically, coarse-level aggregated features are concatenated with the original feature sequences at the corresponding granularity level and fed into a multi-layer feature encoder for joint modeling. The aggregated features serve as prior information to guide the original features in learning contextual relationships to enhance semantics, thereby generating a more structure-sensitive and semantically consistent refined representation, i.e., semantically enhanced features.

[0049] The steps described above for generating semantically enhanced signals include target feature selection to identify discriminative local segments, such as specific image blocks or keywords; global feature semantic consistency verification to eliminate the risk of misidentification; and joint encoding to fuse contextual information, which helps to enhance feature expressiveness and effectively overcome the bottleneck of fine-grained semantic mining.

[0050] Please refer to several embodiments of the present invention. Figure 3 In S4, fuzzy semantic modeling is performed on the semantic enhancement features, and the feature weight allocation results are obtained through implicit rule reasoning. This includes: calculating the membership degree of the selected granularity level semantic enhancement features in each fuzzy set using a composite Gaussian membership function; fusing the membership degrees of the semantic enhancement features in the fuzzy sets involved by the corresponding fuzzy rules based on preset fuzzy rules to obtain the activation intensity of each fuzzy rule; constructing the weight of each fuzzy rule based on the membership degree distribution and the peak value of each fuzzy set; weighting and fusing the weight of each fuzzy rule with its corresponding response value and activation intensity to generate the feature contribution of the selected granularity level semantic enhancement features; and normalizing the feature contribution of the selected granularity level semantic enhancement features to obtain the feature weight allocation results of the selected granularity level.

[0051] Specifically, fuzzy sets are the semantic foundation of fuzzy inference. The system predefines multiple fuzzy sets, each representing a specific semantic concept, such as extremely high similarity, moderate similarity, and extremely low similarity. These fuzzy sets are mathematically defined using a composite Gaussian membership function. For each feature value x in the semantically enhanced feature vector, we use the following improved composite Gaussian membership function to calculate its membership degree in the t-th fuzzy set. After obtaining the membership degrees of each fuzzy set, inference is performed based on the predetermined fuzzy rule base to calculate the activation intensity of the fuzzy rule. The activation intensity ε of the t-th fuzzy rule can be obtained by fusing the membership degrees of all fuzzy sets involved in the antecedent of the rule using the Dombi T norm. t Considering the varying importance of different rules, a weight w needs to be constructed for each fuzzy rule. t The weight is calculated based on the membership distribution μ of the current feature vector in each fuzzy set and the peak value of each fuzzy set.

[0052] Specifically, the core parameters of the composite Gaussian membership function are the center position and standard deviation. Rule activation strength is generated by fusing multidimensional membership degrees using the Dombi T norm, avoiding underflow of high-dimensional rules. Rule weight construction relies on two inherent parameters: membership degree distribution (step output) and the peak value of the fuzzy set (function definition attribute). Rule contribution = ∑(rule weight × response value × activation strength), where the response value is a system-preset learnable parameter.

[0053] Among them, the selected granularity levels are video frame level, image block level and text word level. Semantic enhancement features are features with greater semantic consistency after preprocessing. Fuzzy rules are preset rules used to characterize the semantic fuzziness of features. The feature weight allocation results are used to reflect the importance of different features to homogeneous recognition.

[0054] As a feasible implementation, the membership degree of the semantic enhancement features at the selected granularity level in each fuzzy set is calculated using a composite Gaussian membership function. The expression for the composite Gaussian membership function is as follows: (Formula 9) in, c At =i(ba), representing the Gaussian center corresponding to the t-th rule in dimension m of the n-th feature vector at this granularity. σ t =2a / 3 represents the standard deviation of the Gaussian function, where a is the core parameter of the membership function width, and i = 0, ±1, ..., ±N.

[0055] The activation intensity of the preset fuzzy rule is obtained by fusing the membership degrees from each fuzzy set, and is calculated using the Dombi T norm, as shown in the following formula. (Formula 10) in, μ t,nm (x) λ represents the membership degree of the m-th dimension in the t-th rule within the n-th feature vector. M is the feature vector dimension, and λ is a learnable parameter whose definition was determined through derivation and verification.

[0056] As a specific implementation, in order to reduce computational load, a unified λ is pursued across all dimensions. Through derivation, its definition is as follows: (Formula 11) Where ε is the theoretical lower bound of the membership function, and K is experimentally verified to be optimally 10. Under the current definition of the Gaussian membership function, ε is 0, and the resulting λ is not as good as directly setting it to 1. Therefore, we consider changing the Gaussian membership function to a composite Gaussian membership function, i.e., Equation 9 above.

[0057] As a specific implementation, the weights of each fuzzy rule are constructed based on the membership degree and peak value of each fuzzy set. The peak value within the t-th rule under the corresponding feature vector is then used. pk t,nm The weights of the t-th rule are constructed using the membership degree and then normalized using a normalized exponential function, as detailed below. (Formula 12) The weights of each fuzzy rule, along with their corresponding response values ​​and activation strengths, are weighted and fused to generate the feature contribution of the semantic enhancement feature at the selected granularity level. The rule response values ​​are defined as follows: (Formula 13) in, v k Let t represent the core learnable parameter of the t-th rule, i.e., the baseline response value, l be the step size parameter for adjusting the rule response value, and i be the offset.

[0058] First obtain rule R i The trigger strength is calculated using the following formula. (Formula 14) The rule contribution value is then obtained from the trigger strength and rule response value, as shown in the following formula. (Formula 15) Finally, the importance of the feature vector in the fuzzy semantic space, i.e., the feature contribution, is calculated through the defuzzification layer. The specific formula is as follows. (Formula 16) The feature contribution of semantic enhancement features at the selected granularity level is normalized to obtain the feature weight allocation result for that level. The importance of all feature vectors is then processed using a normalized exponential function, the formula of which is as follows. (Formula 17) Through the above process, the semantic enhancement features at the video frame level, image block level, and text word level are processed respectively to obtain the feature weight allocation results at each granularity. This enhances the semantic expressive power of the features and the saliency of key features, providing a more accurate basis for subsequent homogenization judgment.

[0059] The method described above for obtaining feature weight allocation results effectively solves the parameter failure problem caused by the fuzzy membership boundary through the composite Gaussian function. The Dombi T norm can avoid underflow of high-dimensional rules, and the rule weights focus on key semantic regions, which helps to improve the interpretability of homogeneity discrimination. Overall, it can achieve accurate quantification of the importance of high-dimensional fuzzy features.

[0060] Please refer to several embodiments of the present invention. Figure 3 In S4, a new similarity for the selected granularity level is determined based on the feature weight allocation result, including: recalculating the feature similarity of the selected granularity level based on the semantic enhancement features of the selected granularity level; multiplying the recalculated feature similarity of the selected granularity level with the feature weight allocation result of the selected granularity level to obtain the new similarity of the selected granularity level.

[0061] Specifically, the recalculation of similarity is performed based on the semantically enhanced features, using cosine similarity calculation. The new similarity = recalculated similarity × feature weight assignment value. The feature weight assignment result is the importance weight of each feature obtained through implicit fuzzy inference.

[0062] As a feasible implementation, based on the semantic enhancement features at the selected granularity level, the feature similarity at the selected granularity level is recalculated. For example, the feature similarity between the short video to be identified and the reference video is calculated at five granularities: global video, frame level, image patch level, and text sentence level and word level, respectively, resulting in five similarity scores. The calculation method can refer to the following formula. (Formula 18) (Formula 19) (Formula 20) (Formula 21) (Formula 22) As a specific implementation, the recalculated feature similarity at the selected granularity level is multiplied by the feature weight allocation result at the selected granularity level to obtain a new similarity at the selected granularity level. It should be noted that the video global feature weights... f ew and text sentence-level feature weights f sw The learnable parameters are obtained through multilayer perceptron mapping and normalized exponential function processing, rather than being calculated via a fuzzy inference system. These parameters are then multiplied by the corresponding global video feature similarity and sentence-level text feature similarity to obtain new similarity values ​​at these two granular levels. For the video frame-level weights... f fw Image block weights f pw Text word weights f ww All of these are obtained through implicit high-dimensional fuzzy reasoning. After being processed by this system, they have gained higher feature expressiveness. The recalculated feature similarity is directly multiplied by the feature weight allocation result obtained through fuzzy reasoning to obtain a new similarity at the corresponding granular level.

[0063] The above steps of calculating the new similarity at the selected granularity level, and recalculating the similarity based on semantic enhancement features, can effectively improve the sensitivity of fine-grained discrimination. The weights are applied to similarity rather than features, avoiding information loss and helping to ensure the integrity of the discrimination signal.

[0064] Please refer to several embodiments of the present invention. Figure 2 In S5, the new similarity at the selected granularity level and the weighted initial similarity at the remaining granularity levels are fused to generate the result of homogenization identification of the sample pairs to be identified. This includes: multiplying the initial similarity at the remaining granularity levels by the corresponding learnable similarity weights to obtain the weighted initial similarity at the remaining granularity levels, wherein the learnable similarity weights are generated through parameterized mapping; summing the new similarity at the selected granularity level and the weighted initial similarity at the remaining granularity levels to obtain the comprehensive similarity of homogenization identification of the sample pairs to be identified; determining that the sample pairs to be identified are homogenized content in response to the comprehensive similarity exceeding a preset threshold; and determining that the sample pairs to be identified are not homogenized content in response to the comprehensive similarity not exceeding the preset threshold.

[0065] Specifically, the weighted summation applies to the similarity across all granularity levels. For selected granularity levels, a new similarity is used; for levels not selected, a weighted similarity is used. The preset threshold θ for threshold determination is determined through validation set parameter tuning.

[0066] In one specific embodiment, video multi-granularity features include video global features, video frame-level features, and image block-level features; text multi-granularity features include text sentence-level features and text word-level features. The selected granularity levels specifically refer to three fine-grained levels: video frame-level features, image block-level features, and text word-level features. The remaining unselected granularity levels are video global features and text sentence-level features. Therefore, the three fine-grained levels of video frame-level features, image block-level features, and text word-level features use a new similarity metric. The video global features and text sentence-level features use a weighted similarity metric obtained from the initial similarity. The weighted similarity at the video global level = initial video global similarity × video global weight; the weighted similarity at the text sentence level = initial text sentence level similarity × text sentence level weight. The video global weight and text sentence level weight are learnable weight values ​​obtained through parameter mapping, for example, in one specific embodiment, learnable weight values ​​obtained through MLP and a normalized exponential function. The weighted similarity of the video global features and text sentence-level features is summed with the new similarity of the video frame-level features, image block-level features, and text word-level features to obtain the comprehensive similarity, which is then used as the comprehensive similarity of the sample to be identified for homogeneous identification.

[0067] To obtain the results of homogeneous identification of the samples to be identified and to effectively train the system, this embodiment achieves this by fusing relevant similarities and setting a loss function. In some embodiments, the new similarity at a selected granularity level and the weighted similarity at the remaining granularity levels are fused to generate the results of homogeneous identification of the samples to be identified. The selected granularity levels are video frame level, image patch level, text word level, etc., after feature weight allocation processing; the remaining granularity levels are video global level, text sentence level, etc. The new similarity is obtained by combining the feature weight allocation results; the weighted similarity is obtained by multiplying the initial similarity by the learnable similarity weights; and the comprehensive similarity is the fused result used to determine whether the content is homogeneous.

[0068] As a feasible implementation, the new similarity at the selected granularity level is summed with the weighted similarity at the remaining granularity levels to obtain the comprehensive similarity. The final feature similarity score is as follows: (Formula 23) If the overall similarity exceeds a preset threshold, the sample pair to be identified is determined to be homogeneous content; if the overall similarity does not exceed the preset threshold, the sample pair to be identified is determined to be non-homogeneous content.

[0069] As a specific implementation, to effectively train the mechanism, triplet loss is used as the main loss function, and multiple triplets are constructed. Each triplet includes a baseline, positive samples, and negative samples. The baseline and positive samples are the video to be analyzed and its homogeneous videos, while the negative samples are dissimilar videos. Simultaneously, to maintain consistency in similarity judgments across different granularities, variance regularization loss is applied to the similarity scores of the five granularities, constraining them to be semantically consistent. The total loss function expression is as follows: (Formula 24) By setting such a loss function, the accuracy of homogeneity identification can be improved.

[0070] The steps described above for generating the results of homogeneous identification of the samples to be identified provide a practically feasible homogeneous decision-making mechanism. Multi-granularity similarity weighted fusion can balance global consistency and local discriminativity, while threshold determination achieves efficient binary classification.

[0071] A second aspect of the present invention proposes a system for identifying homogenization in short videos based on multimodal methods. Figure 4 The diagram illustrates a system for identifying homogenization in short videos based on multimodal data, as provided in an embodiment of the present invention. Figure 4 As shown, the system includes: a first module 101, used to extract video multi-granularity features and text multi-granularity features of the sample pair to be identified; a second module 102, used to calculate the initial similarity of each granularity level based on the video multi-granularity features and text multi-granularity features, and generate a feature guidance vector based on the initial similarity; a third module 103, used to select target features from features of several selected granularity levels based on the feature guidance vector, and fuse the target features and features of the selected granularity levels to generate semantically enhanced features; a fourth module 104, used to perform fuzzy semantic modeling on the semantically enhanced features, obtain feature weight allocation results through implicit rule reasoning, and determine the new similarity of the selected granularity level based on the feature weight allocation results; and a fifth module 105, used to fuse the new similarity of the selected granularity level and the weighted similarity of the remaining granularity levels to generate a homogenized identification result for the sample pair to be identified.

[0072] The present invention also provides a computer-readable storage medium. Figure 5 The diagram shown is a structural schematic of a computer-readable storage medium provided in an embodiment of the present invention. Figure 5 As shown, computer-readable storage medium 031 stores a computer program 032 that, when executed by a processor, performs the steps of the method for multimodal short video homogenization identification as described above. The method performed is the same as described above.

[0073] Finally, it should be noted that those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program for setting system parameters can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The storage medium for the program can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. The above computer program embodiments can achieve the same or similar effects as any of the corresponding foregoing method embodiments.

[0074] Furthermore, the method disclosed in the embodiments of the present invention can also be implemented as a computer program executed by a processor, which may be stored in a computer-readable storage medium. When the computer program is executed by the processor, it performs the functions defined in the method disclosed in the embodiments of the present invention.

[0075] Furthermore, the above-described method steps and system units can also be implemented using a controller and a computer-readable storage medium for storing a computer program that enables the controller to perform the functions of the above-described steps or units.

[0076] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the functionality of various illustrative components, blocks, modules, circuits, and steps has been generally described. Whether this functionality is implemented as software or as hardware depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the functionality in various ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the embodiments disclosed herein.

[0077] In one or more exemplary designs, functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, functionality may be stored as one or more instructions or code on or transmitted via a computer-readable medium. Computer-readable media include computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one location to another. Storage media may be any available medium accessible to a general-purpose or special-purpose computer. By way of example, and not limitation, computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, disk storage devices or other magnetic storage devices, or any other medium that may be used to carry or store the required program code in the form of instructions or data structures and is accessible to a general-purpose or special-purpose computer or a general-purpose or special-purpose processor. Furthermore, any connection may be appropriately referred to as computer-readable media. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the aforementioned coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are all included in the definition of media. As used herein, disks and optical discs include compact discs (CDs), laser discs, optical discs, digital multifunction discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0078] The above are exemplary embodiments disclosed in this invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments of this invention as defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular number.

[0079] It should be understood that, as used herein, the singular form “a” is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, “and / or” refers to any and all possible combinations of one or more of the associated listed items.

[0080] The embodiment numbers disclosed in the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0081] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0082] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of different aspects of the invention exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A method for identifying homogenization in short videos based on multimodal methods, characterized in that, include: Extract multi-granularity features from the video and text of the sample pairs to be identified; The initial similarity at each granularity level is calculated based on the video multi-granularity features and the text multi-granularity features, and a feature guidance vector is generated based on the initial similarity. Based on the feature guidance vector, target features are selected from features at several selected granularity levels, and the target features and features at the selected granularity levels are fused to generate semantically enhanced features; The semantic enhancement features are modeled using fuzzy semantics, and the feature weight allocation results are obtained through implicit rule reasoning. Based on the feature weight allocation results, a new similarity at the selected granularity level is determined. By fusing the new similarity at the selected granularity level with the weighted initial similarity at the remaining granularity levels, a homogeneous identification result for the sample pair to be identified is generated.

2. The method for homogenization identification of short videos based on multimodal processing according to claim 1, characterized in that, The step of performing fuzzy semantic modeling on the semantically enhanced features and obtaining the feature weight allocation result through implicit rule reasoning includes: The membership degree of the semantic enhancement features at the selected granularity level in each fuzzy set is calculated using a composite Gaussian membership function. Based on the preset fuzzy rules, the membership degree of the semantic enhancement features in the fuzzy set involved by the corresponding fuzzy rules is fused to obtain the activation intensity of each fuzzy rule; The weights of each fuzzy rule are constructed based on the membership degree distribution and the peak values ​​of each fuzzy set. The weights of each fuzzy rule, their corresponding response values, and activation intensities are weighted and fused to generate the feature contribution of the semantic enhancement feature at the selected granularity level. The feature contribution of the semantic enhancement features at the selected granularity level is normalized to obtain the feature weight allocation result of the selected granularity level.

3. The method for homogenization identification of short videos based on multimodal processing according to claim 1, characterized in that, The process of fusing the new similarity at the selected granularity level and the weighted initial similarity at the remaining granularity levels to generate a result for homogenization identification of the sample pairs to be identified includes: The initial similarity of the remaining granularity level is multiplied by the corresponding learnable similarity weight to obtain the weighted initial similarity of the remaining granularity level, wherein the learnable similarity weight is generated through parameterized mapping. The new similarity at the selected granularity level is weighted and summed with the weighted initial similarity at the remaining granularity levels to obtain the comprehensive similarity for the homogeneous identification of the sample to be identified. In response to the overall similarity exceeding a preset threshold, the sample pair to be identified is determined to be homogeneous content; If the overall similarity does not exceed a preset threshold, the sample pair to be identified is determined to be non-homogeneous content.

4. The method for homogenization identification of short videos based on multimodal processing according to claim 1, characterized in that, Determining the new similarity at the selected granularity level based on the feature weight allocation result includes: Based on the semantic enhancement features of the selected granularity level, the feature similarity of the selected granularity level is recalculated; The new similarity of the selected granularity level is obtained by multiplying the recalculated feature similarity of the selected granularity level with the feature weight allocation result of the selected granularity level.

5. The method for identifying homogenization of short videos based on multimodal processing according to claim 1, characterized in that, The process of selecting target features from features at several selected granularity levels based on the feature guidance vector, fusing the target features and the features at the selected granularity levels to generate semantically enhanced features includes: Calculate the similarity between the feature guidance vector and all features at several selected granularity levels, and select a preset number of features with the highest similarity at the selected granularity levels as target features; The target feature is semantically consistent with the global features at its corresponding granularity level, and then aggregated features are generated through normalized weighting. The aggregated features are concatenated with all features of the selected granularity level and then jointly encoded to generate the semantically enhanced features of the selected granularity level.

6. The method for identifying homogenization of short videos based on multimodal processing according to claim 1, characterized in that, The generation of the feature guidance vector based on the initial similarity includes: The similarity at all granular levels is combined into an input vector and fed into the multilayer perceptron. Based on the input vector, the weight distribution of each granularity level is predicted through the first set of learnable parameters. The similarity of each granularity level is weighted based on the weight distribution to obtain the weighted similarity. The weighted similarity mapping is used to generate a semantic cue vector, which serves as a feature guidance vector, through the second set of learnable parameters. The learnable parameters are optimized through video classification supervision.

7. The method for homogenization identification of short videos based on multimodal processing according to claim 1, characterized in that, The calculation of the initial similarity at each granularity level based on the video multi-granularity features and text multi-granularity features includes: Based on the video multi-granularity features and the text multi-granularity features, the feature vectors of the sample pairs to be identified at the same granularity level are normalized respectively. Calculate the cosine similarity between the normalized feature vectors of the sample pairs to be identified at each granularity level, and use it as the initial similarity for each granularity level.

8. The method for homogenization identification of short videos based on multimodal processing according to claim 1, characterized in that, The extraction of multi-granularity video features and multi-granular text features of the short video sample pairs to be identified includes: Based on the video data and text data of the short video to be identified in the sample pair to be identified, extract the corresponding visual multi-granularity features and text multi-granularity features; Based on the video data and text data of the reference video in the sample pair to be identified, the corresponding visual multi-granularity features and text multi-granularity features are extracted.

9. A system for identifying homogenization in short videos based on multimodal methods, characterized in that, include: The first module is used to extract video multi-granularity features and text multi-granularity features of the sample pairs to be identified; The second module is used to calculate the initial similarity of each granularity level based on the video multi-granularity features and the text multi-granularity features, and to generate a feature guidance vector based on the initial similarity. The third module is used to select target features from features at several selected granularity levels based on the feature guidance vector, and to fuse the target features and the features at the selected granularity levels to generate semantically enhanced features. The fourth module is used to perform fuzzy semantic modeling on the semantic enhancement features, obtain the feature weight allocation result through implicit rule reasoning, and determine the new similarity of the selected granularity level based on the feature weight allocation result; The fifth module is used to fuse the new similarity of the selected granularity level and the weighted initial similarity of the remaining granularity levels to generate a homogeneous identification result for the sample pair to be identified.

10. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method for multimodal short video homogenization identification as described in any one of claims 1-8.

Images