A Structure-Aware Multimedia Semantic Recognition Method Based on TSK Fuzzy System

Abstract

This invention discloses a structure-aware multimedia semantic recognition method based on the TSK fuzzy system, belonging to the field of multimedia information processing and intelligent recognition technology. The invention first acquires multimedia content data and extracts features. Then, it uses TSK fuzzy rules to map the original features to a fuzzy rule space, obtaining a fuzzy feature matrix. Next, representative samples are selected in the fuzzy rule space to construct a local label co-occurrence structure matrix, while simultaneously constructing a global feature dependency structure and a global label flow matrix. Finally, multi-label regression loss, low-rank sparse consequent parameter constraints, global label flow constraints, and global-local feature flow constraints are unified into a single optimization objective to learn the consequent parameter matrix, which is used to output semantic label prediction results for the multimedia content to be recognized. This invention can synergistically utilize global label correlation, global feature dependency relationships, and local representative sample information, significantly improving the accuracy, consistency, and robustness of multimedia multi-label semantic recognition.

Description

Technical Field

[0001] This invention belongs to the field of multimedia information processing and intelligent recognition technology, specifically relating to a structure-aware multimedia semantic recognition method based on TSK. Background Technology

[0002] With the rapid development of the internet, multimedia platforms, and smart terminals, multimedia content such as images, audio, video, and media text exhibits characteristics such as large scale, diverse types, and semantic complexity. Accurate semantic recognition of multimedia content is a key technology in applications such as automatic image annotation, music emotion recognition, video content understanding, intelligent retrieval, and content recommendation. In real-world scenarios, a multimedia object often does not correspond to a single semantic category but simultaneously contains multiple semantic tags. For example, an image may simultaneously contain tags such as "sky," "beach," and "people"; a piece of music may simultaneously have emotion tags such as "soothing," "romantic," and "pleasant"; and a video may simultaneously involve content tags such as "sports," "crowds," and "outdoors." Therefore, multimedia semantic recognition typically manifests as a typical multi-tag recognition problem.

[0003] Existing multi-label recognition methods for multimedia mainly fall into two categories: problem transformation and algorithm adaptation. Problem transformation methods decompose the multi-label task into multiple independent single-label or binary classification tasks. While simple to implement, these methods sever the inherent relationships between semantic labels, easily leading to inconsistent prediction results. Algorithm adaptation methods, although capable of directly processing multi-label outputs, struggle to simultaneously model label correlations, feature structures, and local sample distribution information when dealing with complex multimedia data, thus limiting recognition accuracy. In particular, multimedia semantic labels commonly exhibit co-occurrence, mutual exclusion, or semantic dependencies; ignoring these relationships severely impacts the predictive consistency of the model.

[0004] Furthermore, multimedia data generally exhibits characteristics such as high dimensionality, nonlinearity, noise interference, and uneven label distribution. Traditional linear models struggle to fully characterize the complex nonlinear mapping between multimedia features and semantic labels, and are susceptible to interference from redundant features and noisy samples. The TSK fuzzy system maps original features to a rule-driven fuzzy feature space through fuzzy rules, possessing good nonlinear expressive power and interpretability. However, existing TSK-based multi-label methods mostly focus on rule construction or single label relevance learning, failing to collaboratively utilize three key structural information types in the fuzzy rule space: 1. global label relevance, 2. global feature dependencies, and 3. semantic co-occurrence structure reflected by locally representative samples. The lack of these three types of information makes it difficult for the model to effectively coordinate between the overall data distribution and local semantic patterns.

[0005] Meanwhile, in multimodal datasets, the contributions of different samples to model learning vary significantly. Representative samples located in the core region of the data distribution can better reflect local semantic co-occurrence relationships, while marginal or noisy samples may interfere with model training. Existing methods generally fail to effectively screen and utilize the local structural information contained in these representative samples, resulting in insufficient ability of the model to capture key semantic patterns.

[0006] Therefore, there is an urgent need for a semantic recognition method for multimedia content that can simultaneously integrate global label relevance, global feature dependency, and local representative sample co-occurrence structure in the TSK fuzzy rule space to achieve collaborative modeling of the three, thereby improving the accuracy, robustness, and generalization ability of multimedia multi-label semantic recognition. Summary of the Invention

[0007] To address the aforementioned technical problems, this application provides a structure-aware multimedia semantic recognition method based on the TSK fuzzy system, the detailed technical solution of which is as follows: A structure-aware multimedia semantic recognition method based on the TSK fuzzy system includes the following steps: S1: Acquire multimedia content data and construct a multi-label training set; extract features from the multimedia content data to obtain a feature matrix. and its corresponding label matrix ; S2: Based on the TSK fuzzy rules, the feature matrix is... Mapping to the fuzzy rule space yields the fuzzy feature matrix. ; S3: In the fuzzy feature matrix Based on this, representative samples are selected, and a local label co-occurrence structure matrix is ​​constructed according to the label submatrix corresponding to the representative samples. ; S4: Based on the fuzzy feature matrix Construct a global feature dependency structure and then combine it with the local label co-occurrence structure matrix. By combining these methods, global-local feature flow constraints can be constructed. S5: Based on the label matrix Construct a tag relevance matrix, and further construct a global tag flow matrix. ; S6: Unify the multi-label regression loss, low-rank sparse consequent parameter constraints, the global label flow constraints, and the global-local feature flow constraints into a single optimization objective, and learn the consequent parameter matrix. ; S7: Input the multimedia content to be identified into the consequent parameter matrix. A defined multi-label recognition model yields the corresponding semantic label prediction results.

[0008] Compared with the prior art, the TSK-based structure-aware multimedia semantic recognition method proposed in claim 1 of this invention has the following significant technical effects: 1. Improved the accuracy of multimedia semantic recognition.

[0009] By mapping the original multimedia features to the fuzzy rule space through TSK fuzzy rules, the complex nonlinear relationship between features and multiple semantic labels is effectively characterized, overcoming the shortcomings of insufficient expressive power of linear models.

[0010] 2. Enhanced semantic consistency in multi-label prediction.

[0011] By constructing a global label flow constraint, the co-occurrence and correlation relationships between semantic labels are explicitly utilized, so that the prediction results of highly correlated labels remain consistent, avoiding the problem of contradictory label predictions in traditional methods.

[0012] 3. Improved the robustness and generalization ability of the model.

[0013] By selecting representative samples and constructing global-local feature flow constraints, the interference from noisy and marginal samples is effectively suppressed, enabling the model to focus on the core semantic patterns in the data. Simultaneously, low-rank sparse constraints make the model parameters more compact, reducing the risk of overfitting.

[0014] 4. A unified multi-label learning framework has been formed.

[0015] By incorporating multi-label regression loss, low-rank sparse constraints, global label flow constraints, and global-local feature flow constraints into the same optimization objective, collaborative learning of multiple structural information is achieved, resulting in superior recognition performance compared to existing methods on multiple public multimedia datasets. Attached Figure Description

[0016] Figure 1 This is a flowchart of the method execution of the present invention.

[0017] Figure 2 This invention is in the parameters Sensitivity analysis of parameters under change.

[0018] Figure 3 This invention is in the parameters Sensitivity analysis of parameters under change.

[0019] Figure 4 This invention is in the parameters Sensitivity analysis of parameters under change.

[0020] Figure 5This invention is in the parameters Sensitivity analysis of parameters under change.

[0021] Figure 6 This invention is in the parameters Sensitivity analysis of parameters under change.

[0022] Figure 7 This is an ablation experiment analysis diagram of the AP index of this invention.

[0023] Figure 8 This is an ablation experiment analysis diagram of the present invention in terms of the HL index.

[0024] Figure 9 This is an ablation experiment analysis diagram of the present invention on the RL index.

[0025] Figure 10 This is an ablation experiment analysis diagram of the OE index of this invention.

[0026] Figure 11 This is an ablation experiment analysis diagram of the Coverage index based on the present invention.

[0027] Figure 12 This is a graph showing the Bonferroni-Dunn test results of the present invention and the comparison method on the AP index.

[0028] Figure 13 This is a graph showing the Bonferroni-Dunn test results of the present invention and the comparative method on the HL index.

[0029] Figure 14 This is a graph showing the Bonferroni-Dunn test results of the present invention and the comparison method on the OE index.

[0030] Figure 15 This is a graph showing the Bonferroni-Dunn test results of the present invention and the comparison method on the RL index.

[0031] Figure 16 This is a graph showing the Bonferroni-Dunn test results of the present invention and the comparison method on the Coverage index.

[0032] Figure 17 This is a convergence analysis graph of the present invention on the Birds dataset.

[0033] Figure 18 This is a convergence analysis graph of the present invention on the CAL500 dataset.

[0034] Figure 19 This is a convergence analysis diagram of the present invention on the Rcv1s1 dataset. Detailed Implementation

[0035] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be described in detail below with reference to the accompanying drawings. The description in this part is only exemplary and explanatory, and should not be used to limit the scope of protection of the present invention in any way.

[0036] Example 1: This embodiment provides a structure-aware multimedia semantic recognition method based on TSK, corresponding to the overall technical solution described in claim 1. The method includes the following steps: S1: Acquire multimedia content data and construct a multi-label training set.

[0037] Given a containing A dataset of multimedia samples ,in, Indicates the first The feature vector of a multimedia sample, wherein the multimedia sample includes one or more of images, audio, video, or media text; Indicates the first The semantic label vector corresponding to each multimedia sample This represents the total number of semantic labels. The feature matrix is ​​composed of the feature vectors of all samples. The label vectors of all samples are combined to form a label matrix. , among which, if the first The multimedia sample has the first A semantic tag is represented as ,otherwise Therefore, image scene tags, music emotion tags, video event tags, and media text topic tags can all be uniformly represented as a multi-tag semantic recognition problem.

[0038] S2: Based on the TSK fuzzy rules, the feature matrix is... Mapping to the fuzzy rule space yields the fuzzy feature matrix. .

[0039] Suppose that the TSK fuzzy system includes A fuzzy rule. For the sample , No. ( The number of rules (=1,2…R) can be represented as: , in, Indicates the first Rule No. Fuzzy sets on dimensional features This is the consequent parameter corresponding to the rule.

[0040] The antecedent of the rule is described using the Gaussian membership function, namely: , in, and They represent the first Rule No. The center and width parameters of the dimensional feature. The center and width parameters can be obtained through fuzzy C-means clustering.

[0041] sample For the first The activation strength of a rule is defined as the product of the membership degrees of each feature dimension: ; To ensure the comparability of activation strengths across different rules, they are normalized: ; Furthermore, the original samples are expanded into vectors with bias terms: ; Then the sample The local fuzzy feature under the r-th rule is represented as: ; By concatenating the local fuzzy features corresponding to all rules, a sample is obtained. TSK fuzzy rule features: ; in, Arrange the TSK fuzzy rule features of all samples row-wise to obtain the fuzzy feature matrix: ; Therefore, the multimedia semantic tag prediction function can be expressed as: ,in This is the TSK consequent parameter matrix. To predict the label score matrix. Thus, the complex nonlinear relationship from the original multimedia features to the semantic labels is transformed into a consequent parameter learning problem in the TSK fuzzy rule space.

[0042] S3: In the fuzzy feature matrix Based on this, representative samples are selected, and a local label co-occurrence structure matrix is ​​constructed according to the label submatrix corresponding to the representative samples. .

[0043] In multimedia, multi-labeled data, different samples contribute differently to model learning. To highlight key samples that reflect local semantic structure, representative samples are selected from the TSK fuzzy rule space.

[0044] For fuzzy features First, calculate its local density: , in, This is the neighborhood scale parameter.

[0045] Subsequently, the first is defined The minimum distance between a sample and all samples with a density higher than its own density: , For the sample with the highest density, their relative distance is defined as: .

[0046] Further, calculate the sample representativeness score: .

[0047] When a sample simultaneously exhibits high local density and large relative distance, it indicates that the sample not only resides in a locally high-density region but also possesses a degree of distributional independence, thus making it more suitable as a locally representative sample. All samples are then categorized according to... Sort in descending order and select the first few. Each sample constitutes a representative sample set. Let the label submatrix corresponding to the representative sample be... Then the local label co-occurrence structure matrix is: .

[0048] To explain more clearly The meaning of , expanding its elements as follows: , in, In the representative sample set, the first... The semantic tag and the first The number of times each semantic label co-occurs. This matrix can reflect the co-occurrence relationship between labels in local key samples.

[0049] S4: Based on the fuzzy feature matrix Construct a global feature dependency structure and then combine it with the local label co-occurrence structure matrix. By combining these approaches, a global-local feature flow constraint can be constructed.

[0050] In obtaining the fuzzy feature matrix Next, a global feature dependency structure is constructed in the TSK rule space. First, a sample similarity matrix is ​​constructed: , Constructing a centralized matrix: , in, It is the identity matrix. A vector consisting entirely of 1s.

[0051] Further construct the global structure matrix in the TSK fuzzy rule space: , Among them, matrix This is used to describe the global dependencies between different feature dimensions in the TSK fuzzy feature space. The global structure matrix... Perform eigenvalue decomposition to obtain the feature projection basis. And construct a global feature flow matrix: , Furthermore, the global feature flow matrix Co-occurrence structure matrix of local labels induced by representative samples Construct global-local feature flow constraints: , in, Represents the trace operation of a matrix. This represents the TSK consequent parameter matrix.

[0052] To illustrate the meaning of this set trace operation, let... ,in, Indicates the first The consequent parameter vector corresponding to each semantic label, due to Given the label co-occurrence matrix, the above global-local feature flow constraint formula can be further expanded as follows: , in, Indicates the first The fuzzy feature dimension and the first Global dependencies between fuzzy feature dimensions Indicates the first The semantic tag in the first The consequent parameters are located on a fuzzy feature dimension. As can be seen from the expanded formula, when two labels have a strong co-occurrence relationship in representative samples, their corresponding consequent parameters will remain coordinated under the constraints of the global feature structure, thus achieving joint modeling of global feature dependencies and local label co-occurrence relationships.

[0053] S5: Based on the label matrix Construct a tag relevance matrix, and further construct a global tag flow matrix. .

[0054] To explicitly utilize the global relevance between multimedia semantic tags, based on the tag matrix Construct a tag relevance matrix. Let the first tag be... The label vectors of the first label and the b-th label in the training set are respectively and A label relationship matrix is ​​constructed using cosine similarity. Construct the label relationship matrix: , Furthermore, based on the aforementioned tag correlation matrix Construct the degree matrix of the label graph Its a-th diagonal element is: , The Laplace matrix of the label graph is then... .

[0055] Tag graph Laplace matrix This describes the global structural relationships in the label space. To maintain consistency of related labels in the TSK prediction output space, a global label flow constraint is constructed: .

[0056] S6: Unify the multi-label regression loss, low-rank sparse consequent parameter constraints, the global label flow constraints, and the global-local feature flow constraints into a single optimization objective, and learn the consequent parameter matrix. .

[0057] Combining the above components, TSK fuzzy rule mapping, multi-label regression loss, low-rank sparse consequent parameter constraints, global label flow constraints, and global-local feature flow constraints are unified into a single optimization framework. The objective function is defined as: , in, Describing the Frobenius norm, Represents the nuclear norm. for Norm, , and For regularization parameters, To balance the nuclear norm and The weighting coefficients of the norm.

[0058] S7: Input the multimedia content to be identified into the consequent parameter matrix. A defined multi-label recognition model yields the corresponding semantic label prediction results.

[0059] For the multimedia content to be identified, features are first extracted in the same way as in the training phase, and then fuzzy rule features are obtained through TSK fuzzy rule mapping. Then, based on the learned consequent parameter matrix... Calculate its multi-label prediction score: , Finally, the output tag set is determined based on the preset threshold or the tag sorting result: , in, Threshold for label determination This is a set of semantic tags corresponding to the multimedia content to be identified.

[0060] The technical solution provided in this embodiment first utilizes TSK fuzzy rules to map the original multimedia features to a fuzzy rule space, effectively characterizing the complex nonlinear feature-label relationships in multimedia content such as images, audio, video, and media text. Based on this, global label flow constraints are used to explicitly utilize the co-occurrence and semantic correlation relationships between semantic labels, ensuring consistent prediction results for related labels. Simultaneously, a representative sample selection mechanism extracts key samples that reflect local semantic structures, and their label co-occurrence relationships are combined with the global feature dependency structure to construct global-local feature flow constraints, effectively suppressing interference from noisy and marginal samples. Furthermore, low-rank sparse constraints are introduced in the consequent parameter matrix learning, enabling the model to learn the shared structure between different semantic labels while suppressing redundant parameters. Finally, these multiple mechanisms are unified into a single optimization framework for collaborative learning, achieving a comprehensive improvement in the accuracy, stability, and generalization ability of multi-label semantic recognition for multimedia content.

[0061] Example 2: Based on Example 1, this example provides a detailed description of the specific construction method of the TSK fuzzy rule mapping in step S2.

[0062] First, a TSK fuzzy system containing R rules is constructed. The membership degree of sample features to the fuzzy set of each rule's antecedent is calculated using a Gaussian membership function: , in, and They represent the first Rule No. The center and width parameters on the dimensional features can be obtained through fuzzy C-means clustering.

[0063] Then, calculate the activation strength of each rule for the sample: , Then, normalization is performed to obtain the standardized activation intensity: , Expand the original samples into vectors with bias terms: , Then the sample The local fuzzy feature under the r-th rule is represented as: , By concatenating the local fuzzy features corresponding to all rules, a sample is obtained. TSK fuzzy rule features: , in, The TSK fuzzy rule features of all samples constitute the fuzzy feature matrix.

[0064] The optional process for obtaining the center and width parameters through fuzzy C-means clustering is as follows: First, fuzzy C-means clustering is performed on the feature matrix X to obtain R cluster centers. ,in Let represent the antecedent center vector of the r-th rule.

[0065] Then, based on the clustering results, determine the width parameter of each rule on different feature dimensions. This allows the membership degrees of multimedia samples under different rules to reflect their distribution differences in the original feature space. A feasible way to determine the width parameter is as follows: for the j-th dimension feature of the r-th rule, the width parameter... Let it be some measure of the distance between this cluster center and other cluster centers in the j-th dimension, for example: , Here, β is an adjustable hyperparameter, typically taking the value of 1 or 2.

[0066] Finally, the center parameter and width parameter Substitute the Gaussian membership function to calculate the membership degree of each multimedia sample to each rule, and then obtain the TSK fuzzy rule features corresponding to each multimedia sample.

[0067] The TSK fuzzy rule mapping method provided in this embodiment maps the original multimedia features to a high-dimensional fuzzy rule space. This process effectively characterizes the complex nonlinear mapping relationship between multimedia content features and multiple semantic tags in a way that "approximates global nonlinearity with local linearity." Fuzzy C-means clustering is used to determine the center and width parameters, enabling the fuzzy rule division to adaptively reflect the inherent distribution structure of the data. This avoids the subjectivity of manually setting parameters and lays a solid feature foundation for subsequent semantic recognition.

[0068] Example 3: Based on Example 1, this example further refines the process of selecting representative samples and constructing a local label co-occurrence structure matrix based on the fuzzy feature matrix Φ. The specific steps.

[0069] First, for fuzzy features First, calculate its local density: , in, This is the neighborhood scale parameter.

[0070] A preferred method for determining this is to calculate the Euclidean distance between all sample pairs and take the 2nd percentile as the mean. .

[0071] Then, calculate the minimum distance between the i-th sample and all samples with a density higher than its density: , For the sample with the highest density, their relative distance is defined as: , Next, a representative score for each sample is calculated based on the product of local density and relative distance: .

[0072] According to the representative score Sort the samples in descending order and select the top q samples to form a representative sample set. .

[0073] Finally, extract the corresponding label submatrix based on the representative sample set S. And construct a local label co-occurrence structure matrix: , elements In the representative sample set, the first... The semantic tag and the first The number of times each semantic tag appears together.

[0074] This embodiment further refines the specific steps for constructing global-local feature flow constraints.

[0075] First, based on the fuzzy feature matrix Construct the sample similarity matrix: , Constructing a centralized matrix: , in, It is the identity matrix. A vector consisting entirely of 1s.

[0076] Then, construct the global structure matrix in the TSK fuzzy rule space: , Among them, matrix Used to describe the global dependencies between different feature dimensions in the TSK fuzzy feature space.

[0077] For the global structure matrix Perform eigenvalue decomposition to obtain the feature projection basis. And construct a global feature flow matrix: , Finally, the global feature flow matrix Co-occurrence structure matrix of local labels induced by representative samples Construct global-local feature flow constraints: , in, Represents the trace operation of a matrix. This represents the TSK consequent parameter matrix.

[0078] To illustrate the meaning of this set trace operation, let... ,in, Indicates the first The consequent parameter vector corresponding to each semantic label, due to Given the label co-occurrence matrix, the above global-local feature flow constraint formula can be further expanded as follows: , in, Indicates the first The fuzzy feature dimension and the first Global dependencies between fuzzy feature dimensions Indicates the first The semantic tag in the first The consequent parameters are located on a fuzzy feature dimension. As can be seen from the expanded formula, when two labels have a strong co-occurrence relationship in representative samples, their corresponding consequent parameters will remain coordinated under the constraints of the global feature structure, thus achieving joint modeling of global feature dependencies and local label co-occurrence relationships.

[0079] This embodiment further refines the construction of the global tag flow matrix. The specific steps for global tag flow constraints.

[0080] First, let the first... The label vectors of the first label and the b-th label in the training set are respectively and A label relationship matrix is ​​constructed using cosine similarity. Construct the label relationship matrix: , Then, based on the label correlation matrix Construct the degree matrix of the label graph Its a-th diagonal element is: , The Laplace matrix of the label graph is then... .

[0081] Tag graph Laplace matrix It is used to describe the global structural relationships in the tag space, namely the global tag flow matrix.

[0082] To maintain consistency of relevant labels in the TSK prediction output space, a global label flow constraint is constructed: .

[0083] This constraint can be expanded as follows: , in, Indicates the first The sample at the th The model calculates the predicted scores for each label. When two semantic labels are highly correlated, the model constrains their predicted outputs to remain similar across all samples, thereby improving the consistency of multimedia multi-label semantic recognition results.

[0084] Through the structure-aware module provided in this embodiment, this method can simultaneously utilize three types of key structural information in multimedia data. Specifically, the representative sample screening mechanism effectively suppresses the interference of noisy and marginal samples, allowing the model to focus on the core semantic patterns in the data; the global-local feature flow constraint organically unifies the local semantic patterns revealed by the representative samples with the global feature structure, realizing collaborative modeling of global and local structures; and the global label flow constraint explicitly utilizes the co-occurrence and dependency relationships between semantic labels, ensuring consistent prediction results for related labels. The synergistic effect of these three mechanisms significantly improves the accuracy, consistency, and robustness of multimedia multi-label semantic recognition.

[0085] Example 4: This embodiment adopts the following unified optimization objective: , Among them, the first item The multi-label regression loss is used to minimize the difference between the predicted output and the true label; the second term... For low-rank sparse consequent parameter constraints, where the kernel norm is... Promote the low-rank structure of parameter matrices to mine shared information among labels. norm Promote parameter sparsity to suppress redundant features; the third term This is a global label flow constraint used to maintain the consistency of related labels in the prediction space; the fourth term It serves as a global-local feature flow constraint, used to collaboratively model the relationship between global feature dependencies and local label co-occurrence. , For regularization parameters, Used to balance nuclear norm and The influence of norms.

[0086] Since the objective function simultaneously includes nuclear norm terms and The norm is difficult to solve directly for P. This embodiment introduces an auxiliary variable Q, letting P = Q, and rewrites the unified optimization objective as: , Then construct the augmented Lagrangian function: , Where Λ is the Lagrange multiplier matrix, >0 is the penalty parameter.

[0087] Update sequentially using an alternating iterative strategy. , Λ and : (1) Update :fixed And Λ, update The subproblems are: , set up Then, using the singular value soft threshold operator, we obtain: , in, This indicates that the threshold is... The singular value contraction operator.

[0088] (2) Update :fixed And Λ, with the related subproblems containing smooth and non-smooth terms, let the gradient of the smooth part be: , make The accelerated proximal gradient method is used to update... : , in, It is an element-level soft thresholding operator. This is the upper bound of the Lipschitz constant.

[0089] (3) Update the Lagrange multiplier and penalty parameters: , , in The penalty parameter scaling factor, The upper bound of the penalty parameter.

[0090] (4) Convergence judgment: Alternatively, it can stop iterating when the maximum number of iterations is reached, and output the consequent parameter matrix. .

[0091] Through the unified optimization objective and solution algorithm provided in this embodiment, multiple beneficial mechanisms (multi-label regression loss, low-rank sparse constraints, global label flow constraints, and global-local feature flow constraints) are incorporated into the same optimization framework to synergistically affect the learning process of consequent parameters, avoiding suboptimal solutions that may result from staged optimization. Low-rank sparse constraints enable the model to learn shared structures between different semantic labels while suppressing redundant parameters, improving the model's compactness and generalization ability. By employing an alternating iterative strategy of introducing auxiliary variables and augmented Lagrangian functions, combined with an accelerated proximal gradient method, efficient solutions to complex optimization objectives are achieved.

[0092] Example 5: Based on Example 1, this example provides a detailed description of the prediction output steps.

[0093] For the multimedia content to be identified First, feature vectors are extracted using the same method as in the training phase, and then mapped to fuzzy rule features using the TSK fuzzy rule mapping method described in Example 2. Specifically, for the sample to be identified... First, calculate its membership degree to each rule to obtain the standardized activation strength. Then expand it into a vector with a bias term. Calculate local fuzzy features Finally, the fuzzy rule features are obtained by splicing them together. .

[0094] Then, based on the learned consequent parameter matrix Calculate the multi-label prediction score: , Finally, the output tag set is determined based on the preset threshold or the tag sorting result: , in, Threshold for label determination This is a set of semantic tags corresponding to the multimedia content to be identified.

[0095] threshold The threshold can be determined by one of the following methods: (1) Fixed threshold method: such as setting it to 0.5, which is suitable for scenarios where the label distribution is relatively balanced; (2) Validation set search method: search for the threshold that makes the evaluation index optimal on the validation set; (3) Label ranking method: sort the predicted scores from high to low and select the top k labels as output, where k is the average number of labels for each sample determined according to the validation set.

[0096] Through the prediction output steps provided in this embodiment, the multimedia content to be identified can quickly obtain multi-label prediction results through a single forward propagation. This prediction process is simple and efficient, requiring no iterative calculations, and meets the needs of real-time application scenarios. Multiple threshold setting methods enable this method to adapt to different application scenarios and evaluation metrics, exhibiting good flexibility and adjustability.

[0097] 1. Experimental setup: Six representative multimedia multi-label datasets were selected for experiments, covering various multimedia semantic recognition tasks such as audio, music emotion, image content, and image scene. Specifically, the selected datasets include Birds (bird sound recognition), CAL500 (music semantic recognition), Emotions (music emotion recognition), Flags (image content recognition), Image (image recognition), and Scene (image scene annotation). Detailed information for each dataset is shown in the table below: Table 1: Detailed information on six multimedia multi-label datasets:

[0098] During training, a five-fold cross-validation strategy was employed for both training and testing. Each time, four subsets were selected as the training set, and the remaining subset was used as the test set. This process was repeated five times, ensuring that each subset served as a test set. For each set of candidate parameters, model training and performance evaluation were performed within the five-fold cross-validation framework, and the average of the five-fold results was used as the final performance.

[0099] The evaluation metrics used are five standard multi-label metrics: AP (Average Precision), HL (Hamming Loss), OE (One Error), RL (Ranking Loss), and Coverage. A higher AP indicates better performance; lower HL, OE, RL, and Coverage also indicate better performance.

[0100] The parameter settings are shown in Table 2.

[0101] Table 2 Parameter Settings:

[0102] 2. Compare the experimental results.

[0103] To verify the effectiveness of this invention, the method was compared with eight mainstream multi-label learning algorithms on six public datasets. The comparison methods included: CC (Canonical Relevance Analysis based on Question Transformation), EMC (Multi-label Classification based on Ensemble Learning), GLOCAL (Multi-label Learning based on Local Label Co-occurrence), HNOML (Multi-label Learning based on Higher-Order Label Correlation), HTTN (Network Based on Hierarchical Label Transformation), MLSF (Multi-label Learning based on Label-Specific Features), MLBE-ICF (Multi-label Learning based on Iterative Relevance Feedback), and ML-LSS (Multi-label Learning based on Sparse Label Structure).

[0104] The experimental results are shown in Table 3. The bolded parts in the table represent the best performance under the corresponding indicators.

[0105]

[0106]

[0107] As shown in Table 3, the present invention (SGL-TSK-ML) exhibits the best performance on most evaluation metrics. For example, on the Emotions dataset, the AP of the present invention reaches 0.8042, significantly outperforming methods such as CC (0.7531) and EMC (0.6921); the HL is 0.2081, lower than all the comparison methods, indicating that the present invention has the lowest prediction error rate. On the CAL500 dataset (containing high-dimensional multi-label data with 174 labels), the AP of the present invention is 0.5152, outperforming all the comparison methods, verifying the robustness of the present invention in handling complex multi-label tasks.

[0108] 3. Ablation test results.

[0109] To verify the effectiveness of each structural component in this invention, an ablation experiment was conducted in this embodiment. The complete model was used as the experimental group, and the global label flow constraint and the global-local feature flow constraint were removed respectively as the control group.

[0110] In the experiment, the complete model simultaneously included multi-label regression loss, low-rank sparse consequent parameter constraints, global label flow constraints, and global-local feature flow constraints. Control group 1 used the parameters corresponding to the global label flow constraints. Setting it to 0 removes global label correlation modeling; control group two sets the parameters corresponding to the global-local feature flow constraints. Setting it to 0 removes the joint constraint between the local label co-occurrence structure induced by representative samples and the global feature dependency. Other parameters remain consistent.

[0111] Ablation test results as follows Figures 7 to 11 As shown in the figure. The figure involves: (1). Core information.

[0112] Chart type: Radar chart (polar coordinate chart), used to compare performance differences across multiple dimensions (datasets) simultaneously.

[0113] Evaluation indicators: AP: Average Precision, is a core metric for multi-label classification. The higher the value, the better the model performance.

[0114] HL; Hamming loss, which measures the proportion of mismatches between predicted and true labels. The lower the value, the better the model performance.

[0115] RL: Ranking Loss, which measures the proportion of errors the model makes in ranking "relevant labels before irrelevant labels". The lower the value, the stronger the model's label ranking ability and the better the performance.

[0116] OE: Single Error Rate. It measures the proportion of samples where the model predicts the label with the highest confidence but is not the true label. The lower the value, the better the model performance.

[0117] Coverage: This measures the average number of label locations the model needs to predict to cover all truly relevant labels. A lower value indicates a more efficient and better-performing model.

[0118] Comparison objects (illustration): Green line: Full model.

[0119] Red line: Setting it to 0 removes global relevance modeling for labels. Blue line: Setting it to 0 removes the joint constraint between the local label co-occurrence structure induced by representative samples and the global feature dependency relationship.

[0120] (2). Datasets on the axis (public datasets commonly used for multi-label classification).

[0121] Each axis of the radar chart represents an independent test dataset, ranging from arts to image, totaling 10 datasets and covering different types of multi-label tasks, including text, image, and multimodal tasks. Text classes: bibtex, rcv1sl, yeast.

[0122] Image categories: arts, birds, CAL500, emotions, flags, scene, image (CAL500 is a music tag dataset, and emotions is an emotion tag dataset).

[0123] The ablation experiments show that the complete model outperforms the model without the single structural constraint on most datasets and most evaluation metrics. For example, the complete model outperforms or matches the two control groups on all six datasets using the AP metric; similarly, the complete model demonstrates the best performance on HL, RL, OE, and Coverage metrics. These results indicate that the global label flow effectively utilizes the overall correlation between semantic labels, while the global-local feature flow further combines the global feature structure in the fuzzy rule space with the local label co-occurrence information induced by representative samples. When both work together, the model can more fully utilize the structural information in multimedia data, thereby improving multimedia semantic recognition performance.

[0124] 4. Results of parameter sensitivity experiment.

[0125] To analyze the sensitivity of this invention to different parameters, this embodiment conducts parameter sensitivity experiments on the CAL500 dataset. In each experiment, only the value of one parameter is changed, while the other parameters remain unchanged, and the changing trends of five evaluation indicators—AP, HL, OE, RL, and Coverage—are observed.

[0126] The analysis ranges for each parameter are as follows: , , , , .in, Controlling the strength of low-rank sparse constraints Controlling the strength of global tag flow constraints Controlling the strength of global-local feature flow constraints, Control the number of TSK fuzzy rules. Control the width of fuzzy rules.

[0127] The sensitivity evaluation indicators used are as follows: Ranking Loss: This measures the proportion of incorrectly sorted label pairs; the lower the better.

[0128] Coverage: The average number of tags retrieved during prediction; the lower the better.

[0129] Hamming Loss: The proportion of incorrect matches between predicted labels and true labels; the lower the better.

[0130] One Error: The percentage of labels that are predicted with the highest confidence level but are not the true labels; the lower the better.

[0131] Average Precision: Average accuracy, the average accuracy of multi-label prediction, which comprehensively measures ranking and classification performance; the higher the better.

[0132] Experimental results are as follows Figures 2 to 6 As shown: (1) Model parameters and Not sensitive. Within a wide range of values, these five evaluation indicators only fluctuate slightly, indicating that the proposed method has good stability and robustness in terms of overall regularization strength and the selection of parameters related to TSK fuzzy rule construction. (2) Parameters The impact on model performance is more pronounced. When or As the value of gradually increases, AP shows a significant decreasing trend, while HL, RL, OE, and Coverage increase significantly, indicating that the model performance has deteriorated significantly. (3) This phenomenon shows that excessively large This can lead to overly strong constraints, weakening the model's ability to learn discriminative information and negatively impacting the final multi-label prediction performance. Therefore, in practical applications, λ, and Set within a relatively wide range, while or More careful selection is needed, and smaller values ​​generally yield better experimental results. Overall, these experimental results demonstrate that the proposed method is robust to most parameters, and also validate the importance of setting key parameters appropriately for improving model performance.

[0133] 5. Convergence test results.

[0134] To verify the convergence of this invention, this embodiment selects representative multimedia datasets for convergence analysis. Specifically, Birds, CAL500, and Rcv1s1 datasets are selected as examples, representing audio multimedia data, music multimedia data, and image scene multimedia data, respectively. During each iteration, the absolute value of the difference between two adjacent objective function values ​​is recorded: , in, Indicates the first The objective function value at the next iteration. If If the value gradually decreases and approaches 0 as the number of iterations increases, it indicates that the model optimization process has good convergence.

[0135] The experimental results are shown in Figures 17 to 19 Experimental results show that the present invention can achieve stable convergence within a finite number of iterations on various types of multimedia datasets. The objective function converges relatively quickly on the Birds and CAL500 datasets, while the objective function descent on the Rcv1s1 dataset, due to its higher feature dimensionality and more complex label distribution, is relatively slow, but still exhibits a stable convergence trend overall. These results demonstrate that the alternating optimization strategy employed in this invention can effectively solve the TSK consequent parameter matrix learning problem.

[0136] 6. Results of statistical significance test.

[0137] To further verify the statistical significance difference between the present invention and the comparative methods, this embodiment uses the Friedman test and the Bonferroni-Dunn post-hoc test to statistically analyze the experimental results. If the Friedman test statistic is greater than the critical value at the corresponding significance level, the null hypothesis that there is no significant difference in the performance of all methods is rejected. Furthermore, using the present invention as a control method, the Bonferroni-Dunn test is used to compare the average ranking difference between the present invention and other comparative methods.

[0138] The critical difference CD is defined as: .

[0139] in, significance level The critical value below, Indicates the number of comparison methods. This indicates the number of data sets. A statistically significant difference is considered to exist between the two methods when the average ranking difference between the present invention and a certain comparison method is greater than CD. Table 4 presents the test results for 10 methods on five indicators. The results show that the five indicators... All values ​​were above the critical value. This indicates that SGL-TSK-ML differs statistically from the nine comparative methods in terms of AP, HL, RL, OE, and CV.

[0140] Table 4. Friedman's statistical results:

[0141] Figures 12-16 The average ranking of the SGL-TSK-ML (the standard in the figure is Ours) of the present invention with nine comparative methods is shown. The smaller the average ranking value, the better the method performs.

[0142] The nine comparative methods include: CC (Canonical Relevance Analysis based on question transformation), EMC (Multi-label Classification based on Ensemble Learning), GLOCAL (Multi-label Learning based on Local Label Co-occurrence), HNOML (Multi-label Learning based on Higher-Order Label Correlation), HTTN (Hierarchical Label Transformation Network), MLSF (Multi-label Learning based on Label-Specific Features), MLBE-ICF (Multi-label Learning based on Iterative Relevance Feedback), ML-LSS (Multi-label Learning based on Label Sparse Structure), and RLFSCL (Robust Label and Feature Space Co-learning). Statistical test results show that the average ranking of this invention is superior to most comparative methods across multiple evaluation metrics, and statistically significant differences exist between it and several comparative methods on some metrics. This indicates that this invention not only exhibits superior performance on a single dataset, but also demonstrates good stability and generalization ability across multiple multimedia multi-label datasets.

[0143] Comparative experiments on six public datasets verified the superiority of this invention compared to existing mainstream multi-label learning methods. This invention achieved leading performance in five evaluation metrics: AP, HL, OE, RL, and Coverage, demonstrating its comprehensive advantages in accuracy, consistency, and ranking performance for multimedia semantic recognition. In particular, for complex high-dimensional, multi-label multimedia data (such as the CAL500 dataset containing 174 labels), this invention still achieved optimal AP and OE metrics, verifying its robustness and effectiveness in handling complex multi-label tasks. Ablation experiments verified the necessity of each module, while parameter sensitivity and convergence experiments further demonstrated the stability and reproducibility of this invention. Statistical tests, from a statistical perspective, proved that the advantages of this invention are not accidental.

[0144] The above examples are merely illustrative to aid in understanding the method and core ideas of this invention. The above descriptions are only preferred embodiments of this invention. It should be noted that due to the limitations of written expression, there are objectively infinite specific structures. For those skilled in the art, various improvements, modifications, or variations can be made without departing from the principles of this invention, and the above technical features can be combined in appropriate ways. These improvements, modifications, variations, or combinations, or the direct application of the inventive concept and technical solution to other situations without modification, should all be considered within the scope of protection of this invention.

Claims

1. A structure-aware multimedia semantic recognition method based on a TSK fuzzy system, characterized in that, It includes the following steps: S1: Acquire multimedia content data and construct a multi-label training set; extract features from the multimedia content data to obtain a feature matrix. and its corresponding label matrix ; S2: Based on the TSK fuzzy rules, the feature matrix is... Mapping to the fuzzy rule space yields the fuzzy feature matrix. ; S3: In the fuzzy feature matrix Based on this, representative samples are selected, and a local label co-occurrence structure matrix is ​​constructed according to the label submatrix corresponding to the representative samples. ; S4: Based on the fuzzy feature matrix Construct a global feature dependency structure and then combine it with the local label co-occurrence structure matrix. By combining these methods, global-local feature flow constraints can be constructed. S5: Based on the label matrix Construct a tag relevance matrix, and further construct a global tag flow matrix. ; S6: Unify the multi-label regression loss, low-rank sparse consequent parameter constraints, the global label flow constraints, and the global-local feature flow constraints into a single optimization objective, and learn the consequent parameter matrix. ; S7: Input the multimedia content to be identified into the consequent parameter matrix. A defined multi-label recognition model yields the corresponding semantic label prediction results.

2. The structure-aware multimedia semantic recognition method based on the TSK fuzzy system as described in claim 1, characterized in that, Step S2 specifically includes: S21: Construct a TSK fuzzy system containing R rules, and use the Gaussian membership function to calculate the membership degree of the sample features to the fuzzy set of each rule's antecedent; S22: Calculate the activation intensity of each rule for the sample and normalize it to obtain the standardized activation intensity; S23: Expand the original sample into a vector with a bias term and multiply it by the standardized activation intensity to obtain the local fuzzy features of the sample under each rule; S24: Concatenate the local fuzzy features to obtain the TSK fuzzy rule features for each sample. The TSK fuzzy rule features of all samples constitute the fuzzy feature matrix. .

3. The structure-aware multimedia semantic recognition method based on the TSK fuzzy system as described in claim 2, characterized in that, The center parameter and width parameter of the Gaussian membership function are used to perform fuzzy C-means clustering on the feature matrix. It was determined after clustering.

4. The structure-aware multimedia semantic recognition method based on the TSK fuzzy system as described in claim 1, characterized in that, Step S3 specifically includes: S31: For fuzzy features Calculate its local density and the minimum distance between higher density samples ; S32: Calculate the representative score for each sample based on the product of the local density and the minimum distance. ; S33: Select the top representative scores Each sample constitutes a representative sample set. And extract the corresponding tag submatrix. ; S34: Construct the local label co-occurrence structure matrix .

5. The structure-aware multimedia semantic recognition method based on the TSK fuzzy system as described in claim 1, characterized in that, Step S4 specifically includes: S41: Based on the fuzzy feature matrix Construct a sample similarity matrix ; S42: Constructing a centralized matrix ,in It is the identity matrix. A vector consisting entirely of 1s; and using the aforementioned centering matrix Based on the sample similarity matrix, construct the global structure matrix in the TSK fuzzy rule space. ; S43: For the global structure matrix Perform eigenvalue decomposition to obtain the feature projection basis. And construct a global feature flow matrix. ; S44: Transfer the global feature flow matrix Co-occurrence structure matrix with the local labels By combining these methods, a global-local feature flow constraint can be constructed. .

6. The structure-aware multimedia semantic recognition method based on the TSK fuzzy system as described in claim 1, characterized in that, Step S5 specifically includes: S51: Calculate the cosine similarity between the label vectors of any two labels and construct the label relevance matrix. ; S52: Based on the aforementioned tag correlation matrix Construct the degree matrix of the label graph And further construct a global tag stream matrix. ; S53: Based on the global tag stream matrix Constructing global tag flow constraints .

7. The structure-aware multimedia semantic recognition method based on the TSK fuzzy system as described in claim 1, characterized in that, The unified optimization objective in step S6 is: ， in, Describing the Frobenius norm, Represents the nuclear norm. for Norm, , and For regularization parameters, To balance the nuclear norm and The weighting coefficients of the norm.

8. The structure-aware multimedia semantic recognition method based on the TSK fuzzy system as described in claim 7, characterized in that, The unified optimization objective is solved by introducing an auxiliary variable Q and constructing an augmented Lagrange function. During the solution process, an alternating iterative strategy is used to update Q, P, the Lagrange multiplier Λ, and the penalty parameter μ sequentially until the convergence condition is met.

9. The structure-aware multimedia semantic recognition method based on the TSK fuzzy system as described in claim 8, characterized in that, The alternating iteration strategy includes: With P and Λ fixed, Q is updated using the singular value soft thresholding operator; With Q and Λ fixed, calculate the gradient of the smooth portion and update P using the accelerated proximal gradient method; Update the Lagrange multiplier Λ and the penalty parameter μ; when Alternatively, it can stop iterating when the maximum number of iterations is reached.

10. The structure-aware multimedia semantic recognition method based on the TSK fuzzy system as described in claim 1, characterized in that, Step S7 specifically includes: S71: Feature vectors of the same dimension as those extracted during the training phase for multimedia content to be recognized. ; S72: Transfer the feature vector Mapping to fuzzy rule features ; S73: Calculate the multi-label prediction score based on the learned consequent parameter matrix P. It outputs the final set of semantic tags based on a preset threshold.

Images