Method and system for multi-modal decoupling and hierarchical alignment recognition of fiber materials

Abstract

The application discloses a fiber material multi-modal decoupling and layered alignment identification method and system, relates to the technical field of fiber material intelligent identification and multi-modal machine learning, and is used for solving the problems of insufficient recognition accuracy, stability and robustness caused by the superposition of shared semantics and modal private information, insufficient cross-modal alignment and fixed fusion weight in the existing fiber material multi-modal identification. The method comprises the following steps: acquiring multi-modal data of the fiber material; performing feature extraction on each modal data to obtain original features; decoupling shared / private features; establishing a private structure representation based on the private features and performing private heterogeneous alignment; establishing a shared semantic structure based on the shared features and performing shared homogeneous alignment; performing layered cross-modal fusion on the shared features after homogeneous alignment; generating shared branch weights and private branch weights; and performing fusion according to the shared branch weights and the private branch weights to obtain final classification features and output an identification result.

Description

Technical Field

[0001] This application relates to the field of intelligent identification and multimodal machine learning technology for fiber materials, specifically to a method and system for multimodal decoupling and hierarchical alignment identification of fiber materials. Background Technology

[0002] Fiber material identification is a crucial technical aspect in textile industry processes such as quality inspection, raw material identification, fiber classification, blended material analysis, and intelligent sorting. With the development of machine vision, spectral detection, and machine learning technologies, fiber material identification is gradually evolving from manual judgment and single-instrument detection towards intelligent, multi-source information fusion. Existing identification methods typically utilize visible light images to obtain surface information such as fiber texture and morphological contours, near-infrared spectroscopy to obtain chemical absorption characteristics, or Raman spectroscopy to obtain molecular vibrational characteristics. Different modal data reflect the physical morphology, chemical composition, and molecular structural characteristics of fiber materials from different perspectives.

[0003] However, existing fiber material identification technologies still have certain shortcomings. On the one hand, single-modal identification methods typically rely on only one source of information: images, near-infrared spectroscopy, or Raman spectroscopy. When different fiber materials exhibit similar appearances, similar compositions, subtle differences in spectral peak shapes, strong noise interference, or incomplete sample information, the accuracy and stability of identification can easily decline. On the other hand, existing multimodal identification methods often integrate features from different modalities using feature splicing, fixed-weighted fusion, or general attention fusion. While these methods can introduce multi-source information, they often fail to adequately distinguish between shared information reflecting category semantics across different modalities and the unique structural information inherent in each modality. This can easily lead to the mixing of cross-modal common information, modality-specific information, and noise information. Furthermore, different modal data differ significantly in their expression, feature distribution, semantic granularity, and noise characteristics. Simply merging or directly fusing features is insufficient to stably maintain consistency in category relationships and structural distribution across different modalities. Simultaneously, fixed fusion weights are also difficult to adaptively adjust the contributions of each modality or branch based on the structural clarity and identification uncertainty of different samples. Summary of the Invention

[0004] To address this, this application provides a method and system for multimodal decoupling and hierarchical alignment recognition of fiber materials, in order to solve the problems of insufficient recognition accuracy, stability and robustness caused by the mixing of shared semantics and modal private information, insufficient cross-modal alignment and fixed fusion weights in existing multimodal recognition of fiber materials.

[0005] To achieve the above objectives, this application provides the following technical solution: A method for multimodal decoupling and hierarchical alignment identification of fiber materials includes: Acquire multimodal data of fiber materials, wherein the multimodal data includes at least two of visible light image modes, near-infrared spectral modes, and Raman spectral modes; Feature extraction is performed on each modality of data to obtain the original features corresponding to each modality; The shared features and private features of each modality are decoupled to obtain the shared features and private features corresponding to each modality. A private structure representation is established based on the private features of each modality, and the private structure distributions between different modalities are heterogeneously aligned according to the private structure representation, and structural uncertainty information is obtained from the private structure representation; A shared semantic structure is established based on the shared features of each modality, and the shared semantics between different modalities are homogeneously aligned according to the shared semantic structure. Hierarchical cross-modal fusion is performed on the homogeneously aligned shared features to obtain fused shared features; Private branch features are obtained based on the private features of each modality, and shared branch weights and private branch weights are generated based on the fused shared features, the private branch features and the structural uncertainty information. Based on the shared branch weight and the private branch weight, the fused shared features and the private branch features are fused to obtain the final classification features, and the fiber material identification result is output based on the final classification features.

[0006] Optionally, the shared features and private features of the original features of each modality are decoupled to obtain the shared features and private features corresponding to each modality, including: The original features of each modality are input into the shared mapping head and the private mapping head respectively. The shared mapping head outputs the shared features of the corresponding modality, and the private mapping head outputs the private features of the corresponding modality. Decoupling constraints are applied to shared and private features under the same modality to reduce information aliasing between the shared and private features; The decoupling constraints include orthogonal constraints to reduce sample-level directional consistency and cross-covariance constraints to reduce batch-level statistical correlation.

[0007] Optionally, a private structure representation is established based on the private features of each modality, and structural uncertainty information is obtained from the private structure representation, including: For each modality's private features, construct multiple fuzzy prototype centers; The membership degree of each sample relative to each fuzzy prototype center is calculated based on the distance between the private features of each sample and the multiple fuzzy prototype centers. The cluster quality distribution of the corresponding mode is obtained based on the membership degree, and the fuzzy structure summary is obtained based on the membership degree and the fuzzy prototype center. Structural uncertainty information is obtained based on the degree of dispersion of the membership degree distribution.

[0008] Optionally, heterogeneous alignment of the private structure distributions among different modalities based on the private structure representation includes: For any two modalities involved in the identification, a prototype cost matrix is ​​constructed to characterize the matching cost between different fuzzy prototype centers based on the fuzzy prototype centers corresponding to each of the two modalities. Based on the cluster quality distribution corresponding to each of the two modes and the prototype cost matrix, the transmission matrix between the two modes is solved based on optimal transmission. Based on the transmission matrix and the prototype cost matrix, the private heterogeneous alignment loss between the two modes is determined, and the private structure distribution between different modes is constrained based on the private heterogeneous alignment loss.

[0009] Optionally, a shared semantic structure is established based on the shared features of each modality, and the shared semantics between different modalities are homogeneously aligned according to the shared semantic structure, including: Based on the shared features of each modality, the category center of each category in each modality is calculated separately; Based on the shared features of the samples and the relationship between each category center, a sample-category center relationship vector is constructed, and relationship consistency constraints are applied based on the sample-category center relationship vectors corresponding to the same sample under different modalities. A category topology matrix is ​​constructed based on the distance between the category centers, and topology consistency constraints are applied based on the category topology matrix under different modalities. Based on the relational consistency constraint and the topological consistency constraint, homogeneous alignment is performed on the shared semantics between different modalities.

[0010] Optionally, hierarchical cross-modal fusion is performed on the homogeneously aligned shared features to obtain fused shared features, including: The shared features of each modality after homogeneous alignment are used to construct corresponding modality tags; Construct interactive tags to represent intermodal interaction information based on the combination relationships between different modalities; A classification label is introduced to aggregate cross-modal fusion information, and the modal label, the interaction label, and the classification label are subjected to interaction fusion processing based on an attention mechanism; The fused shared features are obtained based on the output corresponding to the classification label.

[0011] Optionally, based on the fusion shared features, the private branch features, and the structural uncertainty information, shared branch weights and private branch weights are generated, and fusion is performed according to the shared branch weights and the private branch weights, including: Private branch features are obtained based on the private features of each modality; The fusion shared features, the private branch features, and the structural feature information obtained from the private structure representation are combined to obtain the gated input features. The structural feature information includes fuzzy structure summary, membership degree distribution, and structural uncertainty information. Dynamic gating processing is performed based on the gating input features to generate shared branch weights and private branch weights. The contribution of the fused shared features is adjusted according to the shared branch weights, and the contribution of the private branch features is adjusted according to the private branch weights to obtain the final classification features; The shared branch weights and the private branch weights are determined based on the structural uncertainty information, and the structural uncertainty information is used to adjust the contribution ratio of the fused shared features and the private branch features in the final classification features.

[0012] Optionally, it also includes a model training step: Obtain multimodal training samples of fiber materials with category labels; Auxiliary classification is performed based on the private features of each modality to obtain the private auxiliary classification result corresponding to each modality, and the private auxiliary classification loss is determined based on the private auxiliary classification result and the category label. The classification loss is determined based on the fiber material identification results and the category label; The decoupling loss is determined based on the decoupling process of shared features and private features, the private heterogeneous alignment loss is determined based on the heterogeneous alignment process of private structure distribution, the shared homogeneous alignment loss is determined based on the homogeneous alignment process of shared semantics, and the gating constraint loss is determined based on the adjustment relationship between shared branch weights and private branch weights based on structural uncertainty information. The fiber material identification model is trained based on the classification loss, the private auxiliary classification loss, the decoupling loss, the private heterogeneous alignment loss, the shared homogeneous alignment loss, and the gating constraint loss.

[0013] Optionally, acquiring multimodal data of the fiber material and extracting features from each modal data separately includes: Different modal data corresponding to the same fiber material sample are paired according to sample identifier, collection batch or collection time to form multimodal data corresponding to the same fiber material sample; When the multimodal data includes a visible light image mode, the visible light image is subjected to at least one of the following processing methods: cropping, scaling, grayscale conversion, normalization, and image enhancement. When the multimodal data includes near-infrared spectral modes, the near-infrared spectrum is processed by at least one of band alignment, denoising, scattering correction, standardization, and normalization. When the multimodal data includes Raman spectral modes, the Raman spectra are processed by at least one of the following: baseline correction, smoothing filtering, peak alignment, intensity normalization, and noise reduction. Feature extraction is performed on the preprocessed modal data using encoders corresponding to each modality to obtain the original features corresponding to each modality, and the original features corresponding to each modality are mapped to a feature space of a unified dimension.

[0014] This application also discloses a multimodal decoupling and layer alignment identification system for fiber materials, including: A multimodal data processing module is used to acquire multimodal data of fiber materials, wherein the multimodal data includes at least two of visible light image modes, near-infrared spectral modes and Raman spectral modes, and to pair and preprocess different modal data corresponding to the same fiber material sample; The feature encoding module is used to extract features from each modality data to obtain the original features corresponding to each modality. The shared feature and private feature decoupling module is used to decouple the shared features and private features of the original features of each modality to obtain the shared features and private features corresponding to each modality; The private structure representation module is used to construct fuzzy prototype centers based on the private features of each modality, and obtain membership distribution, cluster quality distribution, fuzzy structure summary and structural uncertainty information; A private heterogeneous alignment module is used to construct a prototype cost matrix based on fuzzy prototype centers of different modalities, and to perform heterogeneous alignment of private structure distributions between different modalities through optimal transmission according to the cluster quality distribution of different modalities and the prototype cost matrix. The shared homogeneous alignment module is used to calculate the category center based on the shared features of each modality, construct the sample-category center relationship vector and the category topology matrix, and perform homogeneous alignment of the shared semantics between different modalities according to the sample-category center relationship vector and the category topology matrix; The hierarchical cross-modal fusion module is used to construct modality tags, interaction tags, and classification tags from the homogeneously aligned shared features, and to perform attention-based interaction fusion processing on the modality tags, interaction tags, and classification tags to obtain fused shared features; The dynamic gating fusion module is used to generate shared branch weights and private branch weights based on the fusion shared features, private branch features obtained from the private features of each modality, the fuzzy structure summary, the membership degree distribution and the structural uncertainty information, and to fuse the fusion shared features and the private branch features according to the shared branch weights and the private branch weights to obtain the final classification features; The classification output module is used to output the fiber material identification result based on the final classification features.

[0015] Compared with the prior art, this application has at least the following beneficial effects: To address the information differences exhibited by fiber materials under different detection methods, this study first acquires multi-source characterization information of the same fiber material using at least two of the visible light imaging modal, near-infrared spectroscopy modal, and Raman spectroscopy modal, enabling the shared utilization of the fiber material's morphology, chemical absorption characteristics, and molecular structure features. Then, feature extraction is performed on the data from each modality, and the original features of each modality are decoupled into shared features and private features, allowing for the differentiation between common information reflecting fiber categories across different modalities and the unique information retained by each modality. Based on this, on the one hand, a private structure representation is established based on the private features, and the distribution of private structures across different modalities is heterogeneously aligned, enabling the establishment of matching relationships between private structure information that differs in form but has corresponding relationships across different modalities. On the other hand, a shared semantic structure is established based on the shared features, and the shared semantics across different modalities are homogeneously aligned, ensuring consistency in the semantic expression of fiber categories across different modalities. Furthermore, hierarchical cross-modal fusion is performed on the homogeneously aligned shared features to obtain fused shared features that integrate common information from different modalities. Simultaneously, private branch features are obtained based on private features, and shared branch weights and private branch weights are generated by combining structural uncertainty information. This adaptively adjusts the contribution of shared and private information to the final classification features according to the structural state of the sample. Therefore, compared to direct splicing, fixed weighting, or single attention fusion methods, this application can reduce the aliasing between different modal information, improve the stability of cross-modal feature alignment, and enhance the recognition accuracy and robustness for fiber material samples with similar appearances, subtle spectral differences, or blurred structural boundaries. Attached Figure Description

[0016] To illustrate the prior art and this application more intuitively, several exemplary figures are provided below.

[0017] Figure 1 A schematic flowchart illustrating the multimodal decoupling and layer alignment identification method for fiber materials provided in this application embodiment; Figure 2 This is a technical roadmap for the fiber material multimodal decoupling and layer alignment identification method provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] Example 1: refer to Figure 1-2 This application discloses a method for multimodal decoupling and layer alignment identification of fiber materials, including: Acquire multimodal data of the fiber material, including at least two of the visible light image mode, near-infrared spectral mode, and Raman spectral mode; Feature extraction is performed on each modality of data to obtain the original features corresponding to each modality; The shared features and private features of each modality are decoupled to obtain the shared features and private features corresponding to each modality. A private structure representation is established based on the private features of each modality, and the private structure distribution among different modalities is heterogeneously aligned according to the private structure representation, and structural uncertainty information is obtained from the private structure representation; A shared semantic structure is established based on the shared features of each modality, and the shared semantics between different modalities are homogeneously aligned according to the shared semantic structure; Hierarchical cross-modal fusion is performed on the homogeneously aligned shared features to obtain fused shared features; Private branch features are obtained based on the private features of each modality, and shared branch weights and private branch weights are generated based on the fusion of shared features, private branch features and structural uncertainty information. Based on the shared branch weight and the private branch weight, the shared features and the private branch features are fused to obtain the final classification features, and the fiber material identification results are output based on the final classification features.

[0020] To address the information differences exhibited by fiber materials under different detection methods, this study first acquires multi-source characterization information of the same fiber material using at least two of the visible light imaging modal, near-infrared spectroscopy modal, and Raman spectroscopy modal, enabling the shared utilization of the fiber material's morphology, chemical absorption characteristics, and molecular structure features. Then, feature extraction is performed on the data from each modality, and the original features of each modality are decoupled into shared features and private features, allowing for the differentiation between common information reflecting fiber categories across different modalities and the unique information retained by each modality. Based on this, on the one hand, a private structure representation is established based on the private features, and the distribution of private structures across different modalities is heterogeneously aligned, enabling the establishment of matching relationships between private structure information that differs in form but has corresponding relationships across different modalities. On the other hand, a shared semantic structure is established based on the shared features, and the shared semantics across different modalities are homogeneously aligned, ensuring consistency in the semantic expression of fiber categories across different modalities. Furthermore, hierarchical cross-modal fusion is performed on the homogeneously aligned shared features to obtain fused shared features that integrate common information from different modalities. Simultaneously, private branch features are obtained based on private features, and shared branch weights and private branch weights are generated by combining structural uncertainty information. This adaptively adjusts the contribution of shared and private information to the final classification features according to the structural state of the sample. Therefore, compared to direct splicing, fixed weighting, or single attention fusion methods, this application can reduce the aliasing between different modal information, improve the stability of cross-modal feature alignment, and enhance the recognition accuracy and robustness for fiber material samples with similar appearances, subtle spectral differences, or blurred structural boundaries.

[0021] It should be noted that, in this embodiment, multimodal data refers to data obtained from different detection methods for the same fiber material sample; shared features refer to features used to characterize common category information among different modalities, while private features refer to features used to retain modality-specific information in the corresponding modality; private structure representation refers to a representation formed based on private features to describe the distribution of modality-specific structures; structural uncertainty information refers to information used to reflect the clarity or ambiguity of the private structure representation; heterogeneous alignment refers to aligning the distribution of private structures with different expression forms in different modalities, while homogeneous alignment refers to aligning shared semantics with the same category semantic orientation in different modalities.

[0022] The shared features and private features of the original features of each modality are decoupled to obtain the shared features and private features corresponding to each modality, including: The original features of each modality are input into the shared mapping head and the private mapping head respectively. The shared mapping head outputs the shared features of the corresponding modality, and the private mapping head outputs the private features of the corresponding modality. Decoupling constraints are applied to shared and private features within the same modality to reduce information aliasing between shared and private features; Decoupling constraints include orthogonal constraints to reduce sample-level directional consistency and cross-covariance constraints to reduce batch-level statistical correlation.

[0023] For cases where the original features of the same modality simultaneously contain cross-modal common information and modality-specific information, the original features are mapped using shared and private mapping heads respectively, thus separating the original features into shared features and private features. The shared features carry information that can be used for category discrimination across different modalities, while the private features retain the structural differences inherent in the corresponding modality. Furthermore, by applying decoupling constraints to the shared and private features within the same modality, their sample-level directional relationships and batch-level statistical relationships are separated as much as possible, thereby reducing cross-interference between the shared and private features. Therefore, this implementation avoids directly mixing common semantic information, modality-specific information, and noise information, which is beneficial for improving the stability and accuracy of subsequent feature alignment, feature fusion, and fiber material category identification.

[0024] It should be noted that the shared mapping head and the private mapping head are learnable mapping structures used to project the original features in different directions; the orthogonal constraint is used to suppress the consistency of the shared features and private features of the same sample in the feature direction, so that the two express different information as much as possible; the cross covariance constraint is used to suppress the statistical correlation between the shared features and private features within a training batch, so that the shared features and private features are further decoupled in the overall distribution.

[0025] A private structure representation is established based on the private features of each modality, and structural uncertainty information is obtained from the private structure representation, including: For each modality's private features, construct multiple fuzzy prototype centers; Calculate the membership degree of each sample relative to each fuzzy prototype center based on the distance between the private features of each sample and multiple fuzzy prototype centers; The cluster quality distribution of the corresponding mode is obtained based on the membership degree, and the fuzzy structure summary is obtained based on the membership degree and the fuzzy prototype center. Information on structural uncertainty is obtained based on the degree of dispersion of membership distribution.

[0026] To address the modality-specific structural differences in the private features of each modality, multiple fuzzy prototype centers are first constructed in the private feature space of each modality to represent representative structural types within the modality's private features. Then, based on the distance between the sample's private features and each fuzzy prototype center, the membership degree of the sample relative to each fuzzy prototype center is calculated. This ensures that the sample is not fixed to a single structural center, but rather corresponds to multiple structural centers with varying degrees of membership. Furthermore, the cluster quality distribution of different structural clusters under the corresponding modality can be obtained based on the membership degree. Combining the membership degree and fuzzy prototype centers, a fuzzy structural summary is obtained, thus forming a private structure representation that characterizes the distribution state of the modality's private structure. Simultaneously, structural uncertainty information is obtained based on the dispersion of the membership degree distribution, reflecting whether the sample's private structure affiliation is clear. Therefore, this implementation method can structurally represent the private features of different modalities, avoiding direct treatment of private features as ordinary feature vectors. This facilitates subsequent alignment of the private structure distribution of different modalities and provides a reliable basis for dynamic weight allocation, thereby improving the stability and adaptability of the complex fiber sample recognition process.

[0027] It should be noted that fuzzy prototype centers refer to the central features used to represent typical private structures in the private feature space of the corresponding modality; membership degree refers to the degree to which a sample's private features belong to each fuzzy prototype center; cluster quality distribution refers to the distribution state of samples among multiple fuzzy prototype centers; fuzzy structure summary refers to the general information about private structures obtained based on membership degree and fuzzy prototype centers; and structural uncertainty information refers to the information obtained based on the dispersion of membership degree distribution, used to characterize the clarity or fuzziness of the private structure affiliation of samples.

[0028] Heterogeneous alignment of private structure distributions across different modalities based on private structure representation includes: For any two modalities involved in the identification, a prototype cost matrix is ​​constructed to characterize the matching cost between different fuzzy prototype centers based on the fuzzy prototype centers corresponding to each of the two modalities. Based on the cluster quality distribution and prototype cost matrix corresponding to the two modes, the transmission matrix between the two modes is solved based on optimal transmission. Based on the transfer matrix and the prototype cost matrix, the private heterogeneous alignment loss between the two modes is determined, and the private structure distribution between different modes is constrained based on the private heterogeneous alignment loss.

[0029] After obtaining the private structure representations for each modality, instead of directly narrowing down the private features of different modalities point by point, the fuzzy prototype centers of each modality are first used as representative points of the corresponding modal private structure distributions, and the matching cost between the fuzzy prototype centers of different modalities is characterized by the prototype cost matrix. Furthermore, combining the cluster quality distributions of each modality, the private structure distributions of different modalities can be regarded as two structure distributions to be matched, and the transfer matrix between them is solved through optimal transfer, thereby determining the correspondence between the fuzzy prototype centers in one modality and the fuzzy prototype centers in another modality. Based on the transfer matrix and the prototype cost matrix, the private heterogeneous alignment loss is determined, which can constrain the private structure distributions of different modalities to form a stable correspondence during training. Therefore, this implementation avoids forcibly performing numerical uniformity processing on different types of private information such as image texture structures, near-infrared absorption structures, and Raman vibration structures, but instead achieves heterogeneous alignment through structure distribution matching, thereby better preserving the differences and effectiveness of the private information of each modality and improving the adaptability of the subsequent fusion recognition process to modal differences, boundary fuzzy samples, and complex fiber samples.

[0030] It should be noted that the prototype cost matrix is ​​a matrix used to characterize the matching cost between fuzzy prototype centers of two modalities. The matching cost can be determined based on the distance or similarity between different fuzzy prototype centers. The cluster quality distribution refers to the distribution weight of samples in the corresponding modality on each fuzzy prototype center. The transfer matrix is ​​a matrix obtained through optimal transfer and used to characterize the matching relationship between fuzzy prototype centers of two modalities. The private heterogeneous alignment loss is a loss term determined based on the transfer matrix and the prototype cost matrix and used to constrain the alignment of private structure distributions of different modalities.

[0031] A shared semantic structure is established based on the shared features of each modality, and the shared semantics between different modalities are homogeneously aligned according to the shared semantic structure, including: Based on the shared features of each modality, the category center of each category in each modality is calculated separately; Based on the shared features of the samples and the relationship between each category center, a sample-category center relationship vector is constructed, and relationship consistency constraints are applied based on the sample-category center relationship vectors corresponding to the same sample under different modalities. A category topology matrix is ​​constructed based on the distance between the category centers, and topology consistency constraints are applied based on the category topology matrix under different modalities. Based on relational consistency constraints and topological consistency constraints, homogeneous alignment is performed on shared semantics between different modalities.

[0032] To address the issue that while shared features across different modalities all reflect the semantics of fiber material categories, inconsistencies in category relationships may exist within their respective feature spaces, this approach first calculates the category center for each category within its corresponding modality based on the shared features of each modality. This ensures that each category has a corresponding semantic representative position in each modality. Then, a sample-category center relationship vector is constructed based on the relationship between the sample's shared features and the category centers. This vector represents the proximity or similarity of the sample relative to different categories within the corresponding modality, and relationship consistency constraints ensure that the category relationship of the same sample remains consistent across different modalities. Furthermore, a category topology matrix is ​​constructed using the distances between category centers to represent the relative geometric relationships between different categories, and topological consistency constraints ensure that the structure between categories remains consistent across different modalities. Therefore, this implementation does not simply numerically narrow down the shared features across different modalities; instead, it homogeneously aligns the shared semantics from two levels: the relationship between samples and categories, and the topological structure between categories. This reduces the interference caused by inconsistencies in the shared semantic expressions across different modalities on fusion recognition, improving the semantic consistency of cross-modal shared features and the stability of fiber material recognition.

[0033] It should be noted that the category center refers to the representative central feature of the shared features of samples of the same category in the corresponding modal feature space; the sample-category center relationship vector is a vector used to characterize the distance or similarity relationship between the shared features of samples and each category center; the category topology matrix is ​​a matrix formed based on the distance between different category centers and used to characterize the relative geometric relationship between categories; the relationship consistency constraint is used to ensure that the relationship of the same sample with respect to each category center remains consistent in different modalities, and the topology consistency constraint is used to ensure that the geometric structure between each category center remains consistent in different modalities.

[0034] Hierarchical cross-modal fusion is performed on the homogeneously aligned shared features to obtain fused shared features, including: The shared features of each modality after homogeneous alignment are used to construct corresponding modality tags; Construct interactive tags to represent intermodal interaction information based on the combination relationships between different modalities; We introduce classification tags for aggregating cross-modal fusion information and perform attention-based interaction fusion processing on modal tags, interaction tags, and classification tags. The fused shared features are obtained from the output corresponding to the classification labels.

[0035] Based on the homogeneous alignment of shared features across different modalities, a further hierarchical fusion of these shared features and their interrelationships is performed. Specifically, the homogeneously aligned shared features of each modality are constructed into corresponding modal tags, allowing the shared semantic information of each modality to participate in subsequent fusion as an independent information unit. Interaction tags are constructed based on the combination relationships between different modalities, explicitly representing the association information between different modalities and avoiding reliance on only a single modal feature while ignoring complementary relationships between modalities. Introducing classification tags and performing attention-based interaction fusion processing on modal tags, interaction tags, and classification tags allows the classification tags to aggregate shared features and intermodal interaction information during the interaction process, ultimately forming the fused shared features from the output corresponding to the classification tags. Therefore, this implementation method can further enhance the information interaction and complementary utilization capabilities between different modalities based on shared semantic alignment. Compared to direct splicing or simple weighted fusion methods, it can more fully extract cross-modal shared semantics, improve the expressive power of fused features for fiber material categories, and contribute to improving the accuracy and stability of subsequent identification results.

[0036] It should be noted that modality tags refer to feature units constructed from the shared features of the corresponding modality to represent the shared semantic information of that modality; interaction tags refer to feature units constructed based on the combination relationship between different modalities to represent the interaction relationship between modalities; classification tags refer to feature units used to aggregate cross-modal fusion information and output fused shared features; and attention-based interaction fusion processing refers to the process of weighted interaction and fusion of the information carried by different tags based on the correlation between the tags.

[0037] Based on the fusion of shared features, private branch features, and structural uncertainty information, shared branch weights and private branch weights are generated, and then fused according to the shared branch weights and private branch weights, including: Private branch features are obtained based on the private features of each modality; By combining the shared features, private branch features, and structural feature information obtained from the private structure representation, gated input features are obtained. The structural feature information includes fuzzy structure summary, membership degree distribution, and structural uncertainty information. Dynamic gating processing is performed based on gating input features to generate shared branch weights and private branch weights; The contribution of the fused shared features is adjusted according to the weight of the shared branches, and the contribution of the private branch features is adjusted according to the weight of the private branches to obtain the final classification features. The weights of the shared branch and the private branch are determined based on structural uncertainty information, which is used to adjust the contribution ratio of the fused shared features and private branch features in the final classification features.

[0038] Based on the obtained shared features and modal-specific private features, the shared and private branches are further adaptively weighted according to the private structural state of the sample itself. Specifically, private branch features are first obtained based on the private features of each modality, so that the unique structural information retained in each modality can serve as a supplementary basis for the final identification. Then, the shared features, private branch features, and fuzzy structural summaries, membership distributions, and structural uncertainty information obtained from the private structural representation are combined to form gated input features, enabling dynamic gating processing to perceive not only shared semantic information and private structural information, but also the clarity and distribution of the private structure. Furthermore, shared branch weights and private branch weights are generated through the gated input features, and the contribution of the shared features in the final classification features is adjusted using the shared branch weights, while the contribution of the private branch features in the final classification features is adjusted using the private branch weights. Thus, this implementation can adaptively adjust the fusion ratio of shared and private information according to the structural uncertainty of different fiber samples, avoiding misjudgments caused by fixed-weight fusion when sample differences are large, modal information reliability is inconsistent, or private structural boundaries are blurred, thereby improving the adaptability, stability, and accuracy of fiber material identification.

[0039] It should be noted that the fusion shared features refer to the features obtained after homogeneous alignment and hierarchical cross-modal fusion, used to characterize the common category semantics of different modalities; the private branch features refer to the features obtained based on the private features of each modality, used to characterize the modality-specific structural information; the structural feature information refers to the information obtained from the private structure representation to describe the state of the private structure, which includes fuzzy structure summary, membership degree distribution, and structural uncertainty information; the gated input features refer to the input features used to generate the shared branch weights and private branch weights; the shared branch weights and private branch weights are used to adjust the contribution ratio of the fusion shared features and private branch features in the final classification features, respectively.

[0040] The multimodal decoupling and hierarchical alignment identification method for fiber materials also includes a model training step: Obtain multimodal training samples of fiber materials with category labels; Auxiliary classification is performed based on the private features of each modality to obtain the private auxiliary classification results corresponding to each modality, and the private auxiliary classification loss is determined based on the private auxiliary classification results and the category labels. The classification loss is determined based on the fiber material identification results and category labels; The decoupling loss is determined based on the decoupling process of shared features and private features, the private heterogeneous alignment loss is determined based on the heterogeneous alignment process of private structure distribution, the shared homogeneous alignment loss is determined based on the homogeneous alignment process of shared semantics, and the gating constraint loss is determined based on the adjustment relationship between shared branch weights and private branch weights based on structural uncertainty information. A fiber material identification model is trained based on classification loss, private auxiliary classification loss, decoupling loss, private heterogeneous alignment loss, shared homogeneous alignment loss, and gating constraint loss.

[0041] During model training, supervised optimization of the fiber material recognition model is performed using multimodal training samples of fiber materials with category labels. This enables collaborative learning of feature decoupling, private structure alignment, shared semantic alignment, dynamic gating fusion, and classification output within the model. Specifically, auxiliary classification is performed based on the private features of each modality. This ensures that the private features of each modality not only retain modality-specific structural information but also retain discriminative information related to the fiber material category, preventing private features from degenerating into invalid residuals or noisy features during training. The classification loss, determined based on fiber material identification results and category labels, directly constrains the accuracy of the final identification result. The decoupling loss, determined based on the decoupling process of shared and private features, constrains the degree of separation between shared and private information. The private heterogeneous alignment loss, determined based on the heterogeneous alignment process of private structure distribution, constrains the formation of stable correspondences among private structure distributions of different modalities. The shared homogeneous alignment loss, determined based on the homogeneous alignment process of shared semantics, constrains the consistency of shared category semantics across different modalities. The gating constraint loss, determined based on the adjustment relationship between shared branch weights and private branch weights using structural uncertainty information, constrains the dynamic gating process to match the sample's private structure state. Therefore, this implementation does not rely solely on a single classification loss to train the model; instead, it uses multiple losses to jointly constrain different functional branches of the model, enabling collaborative optimization of each module under a unified training objective. This improves the feature representation capability, cross-modal alignment stability, fusion decision adaptability, and final identification accuracy of the fiber material identification model.

[0042] It should be noted that: category label refers to the true category information corresponding to the multimodal training samples of fiber materials; private auxiliary classification loss refers to the loss term determined based on the difference between the auxiliary classification result corresponding to the private features of each modality and the category label; classification loss refers to the loss term determined based on the difference between the final fiber material identification result and the category label; decoupling loss refers to the loss term used to constrain the degree of separation between shared features and private features; private heterogeneous alignment loss refers to the loss term used to constrain the alignment of the private structure distribution of different modalities; shared homogeneous alignment loss refers to the loss term used to constrain the consistency of the shared semantic structure of different modalities; and gating constraint loss refers to the loss term used to constrain the adjustment relationship between structural uncertainty information and the weights of shared branches and private branches.

[0043] Acquiring multimodal data of fiber materials and extracting features from each modality, including: Different modal data corresponding to the same fiber material sample are paired according to sample identifier, collection batch or collection time to form multimodal data corresponding to the same fiber material sample; When the multimodal data includes visible light image modalities, the visible light image is subjected to at least one of the following processing methods: cropping, scaling, grayscale conversion, normalization, and image enhancement. When multimodal data includes near-infrared spectral modes, at least one of the following processes is performed on the near-infrared spectrum: band alignment, denoising, scattering correction, standardization, and normalization. When multimodal data includes Raman spectral modes, at least one of the following processing methods is applied to the Raman spectra: baseline correction, smoothing filtering, peak alignment, intensity normalization, and noise reduction. Feature extraction is performed on the preprocessed modal data using encoders corresponding to each modality to obtain the original features corresponding to each modality, and the original features corresponding to each modality are mapped to a feature space of a unified dimension.

[0044] Before identifying multimodal fiber materials, data collected from the same fiber material sample under different modes are paired to ensure that visible light images, near-infrared spectra, and Raman spectra correspond to the same identification object, avoiding data mismatch between different samples that could affect subsequent cross-modal alignment and fusion. After data pairing, preprocessing is performed on the representation format and noise sources of different modal data. Visible light images are processed through cropping, scaling, grayscale conversion, normalization, or image enhancement to improve the consistency of image size, brightness, and texture representation. Near-infrared spectra are processed through band alignment, denoising, scattering correction, standardization, or normalization to reduce the impact of acquisition conditions, scattering interference, and intensity scale differences on spectral characteristics. Raman spectra are processed through baseline correction, smoothing filtering, peak alignment, intensity normalization, or denoising to reduce identification errors caused by baseline drift, random noise, and peak shift. Furthermore, by extracting the original features of each modality from the encoders corresponding to each modality, and mapping these original features to a feature space of a unified dimension, modal data from different sources and in different formats can be converted into feature representations that facilitate subsequent decoupling, alignment, and fusion processing. Therefore, this implementation improves the correspondence accuracy and feature representation consistency between multimodal data, reduces the impact of original data noise, scale differences, and acquisition bias on the recognition results, and provides a stable data foundation for subsequent cross-modal feature decoupling, alignment, and fusion.

[0045] It should be noted that sample identification, collection batch, or collection time are used to determine whether different modal data correspond to the same fiber material sample; visible light image mode is used to characterize the appearance, morphology, texture, or surface structure information of fiber materials; near-infrared spectral mode is used to characterize the chemical absorption characteristics of fiber materials; Raman spectral mode is used to characterize the molecular vibrational characteristics of fiber materials; encoder refers to the feature extraction structure used to convert the corresponding modal data into a computable feature representation; unified dimensional feature space refers to mapping the original features corresponding to different modes to a feature representation space with consistent dimensionality or capable of subsequent cross-modal processing.

[0046] Example 2: This application provides a method for multimodal decoupling and hierarchical alignment identification of fiber materials. This method is used for intelligent identification of fiber materials, where the multimodal data of the fiber materials includes at least two or three of the following: visible light image modes, near-infrared spectral modes, and Raman spectral modes.

[0047] First, a multimodal sample dataset of the fiber material to be identified is obtained. Each sample is then labeled with its category, and the data for different modalities is preprocessed. Let the sample dataset be: ; in, , , They represent the first The visible light image mode input, near-infrared spectral mode input, and Raman spectral mode input for each sample. Indicates category label, Indicates the number of samples. This indicates the number of categories. If only two modalities are used in the actual recognition scenario, the missing modalities can be excluded from subsequent processing.

[0048] In this embodiment, preprocessing includes at least one of the following: cropping, scaling, enhancement, and normalization of visible light image modes; band alignment, denoising, standard normal variable transformation, scattering correction, or normalization of near-infrared spectral modes; and baseline drift removal, smoothing filtering, peak alignment, intensity normalization, or denoising of Raman spectral modes. Simultaneously, samples of each mode are paired according to sample number or timestamp to form a unified multimodal training sample. Through the above processing, the data of the same fiber material sample in different modes can form a corresponding relationship, and the impact of scale differences, noise interference, and acquisition bias in the original data on subsequent recognition results can be reduced.

[0049] Then, feature encoders corresponding to each modality are constructed to extract the original features of each modality. For the th modality... The first sample Modal input Its original characteristics can be represented as: ; in, Indicates the first The feature encoder corresponding to each modality This represents the encoded original features. Specifically, the image encoder corresponding to the visible light image mode is used to extract fiber surface texture, color, and morphology information; the near-infrared spectral encoder corresponding to the near-infrared spectral mode is used to extract near-infrared absorption features; and the Raman spectral encoder corresponding to the Raman spectral mode is used to extract molecular vibrational features. The modal features output by each encoder are mapped to a unified dimension representation space to facilitate subsequent decoupling of shared and private features, cross-modal alignment, and feature fusion.

[0050] Furthermore, the original features of each modality are input into the shared feature and private feature decoupling module to obtain the shared features and private features corresponding to each modality. For the first... The sample at the th Original features under each modality Its shared and private characteristics can be represented as: ; ; in, Indicates the first The shared feature mapping function corresponding to each modality Indicates the first Private feature mapping function corresponding to each modality Indicates shared features, Private features are used to represent private features. Shared features are used to carry information that can be used together for class discrimination across different modalities, while private features are used to retain the structural differences that exist within the corresponding modality itself.

[0051] To reduce redundant coupling between shared and private features, this embodiment applies decoupling constraints to the shared and private features. These decoupling constraints include cosine orthogonality constraints and cross-covariance constraints. The cosine orthogonality constraint can be expressed as: ; in, Indicates batch size, This represents the L2 norm. This cosine orthogonal constraint is used to suppress the consistency between shared features and private features in the feature direction within the same sample.

[0052] The cross-covariance constraint can be expressed as: ; in, and These represent the shared and private characteristics after centralization, respectively. Describing the Frobenius norm, and These represent the shared feature matrix and the private feature matrix, respectively.

[0053] Decoupling loss function: ; By using the decoupling loss described above, shared features and private features can be separated in terms of both sample-level directional relationships and batch-level statistical relationships, thereby reducing the overlap of common information, modal private information, and noise information.

[0054] In this embodiment, to enhance the category discrimination capability of private branches, private auxiliary classification constraints can also be introduced. For each modality Private characteristics Through private auxiliary classification head Obtain auxiliary classification output: ; in, Indicates the first Each sample in modality The private auxiliary classification loss is logits. The corresponding unimodal private auxiliary classification loss can be expressed as: ; in, This represents the cross-entropy loss. Averaging over all activated modes yields the private auxiliary classification loss: ; in, This represents the set of activation modes involved in the identification. This represents the number of activated modes. By using private auxiliary classification constraints, private features can retain not only the modality-specific structural information but also the discriminative information related to fiber categories, thus preventing private features from degenerating into invalid residuals.

[0055] After obtaining the private features of each modality, a fuzzy prototype representation is constructed based on the private features of each modality. Specifically, for the ... The private features of each modality are used to divide prototype clusters using fuzzy clustering methods, resulting in... Center of a fuzzy prototype cluster: ; For the sample The membership degree to each cluster center can be calculated using fuzzy clustering: ; in, Indicates sample In modality The following belongs to the The membership degree of each prototype cluster, where q>1 is the fuzzy coefficient; To prevent extremely small positive numbers with a denominator of zero, membership degree calculation is used. Instead of being fixedly assigned to a single prototype cluster, samples can correspond to multiple fuzzy prototype cluster centers with different membership degrees.

[0056] In-batch prototype centers can be constructed based on membership degrees: ; To enhance training stability, a momentum update operation center is adopted: ; in To update momentum. The final effective center is defined as: ; By combining intra-batch prototype centers, running centers, and effective centers, the impact of fluctuations in the distribution of individual training batch samples on prototype centers can be reduced, enabling prototype centers to more stably represent typical structures in the corresponding modality's private feature space.

[0057] Based on the fuzzy prototype, cluster quality distribution, fuzzy structure summary, and structural uncertainty information can be further obtained. The cluster quality distribution can be represented as: ; The fuzzy structural summary can be represented as: ; Structural uncertainty information can be represented as: ; Among them, cluster quality distribution is used to characterize the distribution state of sample private structure among multiple prototype clusters in the corresponding mode; fuzzy structure summary is used to characterize the comprehensive association between sample private structure and multiple prototype cluster centers; structural uncertainty information is used to characterize the clarity or fuzziness of sample private structure affiliation.

[0058] To achieve private heterogeneous alignment between private structure distributions of different modalities, a prototype cost matrix is ​​defined between modality a and modality b: ; in The cosine cost weight is set to 0.5 in the code. Represents the private feature dimension. and Representing modes and modality The effective prototype center in.

[0059] Furthermore, based on cluster quality distribution and cost matrix The private heterogeneous alignment loss is obtained by using optimal transmission: ; in, Represents a set of modal pairs. This represents the optimal transmission with entropy regularization. This represents the transmission regularization coefficient. Specifically: ; in: ; Represents the transmission matrix. Indicates from modality The Cluster-oriented modes The The transport quality allocated to each cluster, express A one-dimensional vector. The optimal transport with entropy regularization can be solved iteratively using the Sinkhorn algorithm. This private heterogeneous alignment loss can constrain the distribution of private structures of different modalities to form a stable correspondence, without forcibly unifying the point-by-point values ​​of private features of different modalities.

[0060] After obtaining the shared features across modalities, category centers are constructed based on these shared features, and homogeneous alignment is performed. For the shared features... For each category Calculate its modal The following category center: ; in, Indicates category The sample index set.

[0061] To achieve homogeneous alignment of shared semantic structures, a relation alignment loss is constructed using the relative distance between samples and the centers of each category, thus constraining the consistency of relation vectors for the same sample across different modalities. For samples... Define its modality The following relation vector is: ; in, This represents the normalization operation. The sample-class center consistency loss can then be defined as: ; in, These are the sample weights. The sample weights are guided by the uncertainty of the fuzzy structure: ; ; in, This indicates the sample-level average structural uncertainty. This represents the fuzzy guidance coefficient. This sample weight allows structural uncertainty information to participate in the process of aligning shared semantic relationships.

[0062] Furthermore, to characterize the geometric topological relationships between categories, a category center-category center topological matrix is ​​constructed using the pairwise distances between category centers: ; in, Let be the normalized class centers. Then the topological consistency loss can be expressed as: ; Ultimately, the shared homogeneous alignment loss consists of the two parts mentioned above: ; By sharing homogeneous alignment loss, shared features across different modalities can maintain consistency at both the sample-class relationship level and the class-class topology level, thereby improving the consistency of shared semantic representations across modalities.

[0063] After completing the shared homogeneous alignment, hierarchical cross-modal fusion is performed on the homogeneously aligned shared features. First, the shared features of each modality are constructed into modality labels: ; in, This represents modal embedding. When modal interaction enhancement is enabled, for any modal pair... Construct interactive tags: ; in, This represents element-wise multiplication. Indicates interactive mapping, This represents modality pair embeddings. Then, a classification token is introduced. ), forming the input sequence: ; The sequence is input into a hierarchical cross-modal fusion module, which may include... Layer Informerblock, obtained by stacking attention mechanism and feedforward network: ; Finally, the classification label output is used as the fused shared feature: ; in, This represents the fused shared features obtained through hierarchical cross-modal fusion. The Informer fusion module employs a sparse attention mechanism to jointly model the aligned shared semantic information and explicit modal interaction information, outputting global fused shared features.

[0064] After obtaining the fusion-shared features, dynamic gating fusion is performed based on the fusion-shared features, private features, fuzzy structural summaries, membership degree distributions, and structural uncertainty information. First, modal averaging is performed on the private features to obtain the private summaries: ; Next, the fuzzy structure summary is averaged to obtain: ; If fuzzy private alignment is enabled, the private decision representation is defined as follows: ; If fuzzy private alignment is not enabled, then: ; The average membership distribution can be denoted as: ; The average structural uncertainty can be denoted as: ; By concatenating shared features, private decision representations, fuzzy structure summaries, average membership distributions, and average structural uncertainty information, the gated input is obtained: ; The shared branch weight and private branch weight are output through dynamic gating fusion processing: ; Then, sample-level adaptive fusion is performed: ; in, Indicates shared branch weights. This represents the weights of the private branches. Dynamic gating fusion processing can be implemented through the Cluster-aware Gate module, which inputs shared features, private features, fuzzy structural summaries, membership distributions, and structural uncertainty information into a lightweight multilayer perceptron to output shared branch weights and private branch weights. The structural uncertainty information is used to adjust the contribution ratio of shared branches and private branches in the final classification features.

[0065] During model training, gating consistency constraints can also be defined: ; in, This is used to constrain dynamic gating fusion processing by adaptively allocating shared branch weights and private branch weights based on structural uncertainty information.

[0066] Finally, the gated shared branch representation is concatenated with the private branch representation to obtain the final classification feature: ; The final classification features are input into the classifier, which outputs the sample class prediction results: ; The primary classification loss can be expressed as: ; When using cross-entropy loss It can be represented as the first The one-hot class label vector of each sample. This represents the predicted probability of the class output by the classifier.

[0067] The total loss function in this embodiment can be expressed as: ; in, Indicate the weight of each loss; This represents the classification loss, used to constrain the fiber material identification results; This represents the decoupling loss, used to constrain the separation of shared features and private features; This represents the private heterogeneous alignment loss, used to constrain the consistency of the distribution of private structures across different modalities; This represents the shared homogeneous alignment loss, used to constrain the consistency of shared semantic structures across different modalities; This represents the private auxiliary classification loss, used to enhance the discriminative power of private branches; This represents the gating consistency loss, used to constrain dynamic gating fusion processing to adaptively allocate shared branch weights and private branch weights based on structural uncertainty information.

[0068] Classification loss weights The weight is fixed at 1, and the remaining weights are determined through a discrete grid search on the validation set. Among these, the decoupling loss weights... This value controls the degree of separation between shared and private features. If it's too small, the decoupling constraints are insufficient; if it's too large, it may force the shared and private representations to become excessively orthogonal, resulting in a loss of effective discriminative information. Therefore, a value of 0.1 is preferred. Private Alignment Loss Weight This is used for heterogeneous structure alignment of private features. Since the private branches themselves need to preserve mode-specific information between the image, near-infrared, and Raman spectra, this value should not be set too high; therefore, 0.05 is preferred. Shared alignment loss weight. Used to constrain the consistency of the shared semantic space, shared features undertake the task of cross-modal semantic alignment, therefore their weight can be slightly higher than the private heterogeneous alignment loss, preferably set to 0.1. Private auxiliary classification loss weights. This term, used to enhance the class discrimination ability of private features, is an auxiliary supervision measure. If its weight is too high, each private branch will classify independently, thus weakening the effect of multimodal fusion. Therefore, a weight of 0.1 is preferred. Gating consistency loss weight. This parameter is used to constrain the consistency between the predicted gating weights and the target gating weights. Since this parameter mainly plays a regularization role rather than directly optimizing the classification results, it is preferably set to 0.05. See Table 1 for details.

[0069] In this embodiment, training using the aforementioned joint loss function enables the fiber material recognition model to be collaboratively optimized in multiple aspects, including the final classification task, decoupling of shared and private features, alignment of private structure distribution, alignment of shared semantic structure, enhancement of private branch discrimination capability, and dynamic gating fusion. After training, inputting multimodal data of the fiber material to be identified allows for the generation of fiber material recognition results through the aforementioned feature encoding, decoupling of shared and private features, alignment of private heterogeneous features, alignment of shared homogeneous features, hierarchical cross-modal fusion, dynamic gating fusion, and classification output processes.

[0070] Example 3: The identification method of this application can be trained and validated based on a real-world multimodal fiber material dataset. The multimodal fiber material dataset can include six types of fiber materials: cotton, spandex, nylon, polyester, polypropylene, and wool. For each type of fiber material, 100 sets of valid sample data are constructed, resulting in a total of 600 sets of multimodal data. Each set of multimodal data includes a set of near-infrared spectral data, a set of Raman spectral data, and a set of microscopic morphology image data corresponding to the same fiber material sample. When constructing the sample dataset, the near-infrared and Raman spectral data are denoised, standardized, and scaled; the microscopic morphology image data are normalized in size and enhanced in image quality; and different modal data are matched one-to-one according to the sample category labels to form a unified multimodal fiber material identification dataset. Furthermore, ablation experiments can be conducted to verify the contribution of each functional module in this application to the recognition performance. Experimental results show that, with the retention of shared and private feature decoupling, private heterogeneous alignment, shared homogeneous alignment, hierarchical cross-modal fusion, dynamic gating fusion, and joint loss training, the model's accuracy is 0.9333, macro F1 is 0.9314, and macro AUC is 0.9917; when homogeneous alignment is removed, the model's accuracy is 0.9222, macro F1 is 0.9214, and macro AUC is 0.9895; when heterogeneous alignment is removed, the model's accuracy is 0.8944, macro F1 is 0.8924, and macro AUC is 0.9898; when dynamic gating is removed, the model's accuracy is 0.8778, macro F1 is 0.8716, and macro AUC is 0.9917. The AUC is 0.9863; when the private auxiliary discrimination is removed, the model's accuracy is 0.9056, macro F1 is 0.9052, and macro AUC is 0.9839; when the decoupling module is removed, the model's accuracy is 0.9000, macro F1 is 0.8949, and macro AUC is 0.9915; when all modules are removed, the model's accuracy is 0.8556, macro F1 is 0.8538, and macro AUC is 0.9852. These experimental results show that this application, through decoupling shared features from private features, heterogeneous alignment of private structure distributions, homogeneous alignment of shared semantic structures, and dynamic gating fusion based on structural uncertainty information, can improve the accuracy, stability, and robustness of fiber material identification results.

[0071] This application also discloses a multimodal decoupling and layer alignment identification system for fiber materials, including: The multimodal data processing module is used to acquire multimodal data of fiber materials, including at least two of visible light image modes, near-infrared spectral modes and Raman spectral modes, and to pair and preprocess different modal data corresponding to the same fiber material sample. The feature encoding module is used to extract features from each modality data to obtain the original features corresponding to each modality. The shared feature and private feature decoupling module is used to decouple the shared features and private features of the original features of each modality to obtain the shared features and private features corresponding to each modality; The private structure representation module is used to construct fuzzy prototype centers based on the private features of each modality, and obtain membership distribution, cluster quality distribution, fuzzy structure summary and structural uncertainty information; The private heterogeneous alignment module is used to construct the prototype cost matrix based on the fuzzy prototype centers of different modalities, and to perform heterogeneous alignment of the private structure distribution between different modalities through optimal transmission according to the cluster quality distribution and prototype cost matrix of different modalities. The shared homogeneous alignment module is used to calculate the category center based on the shared features of each modality, construct the sample-category center relationship vector and the category topology matrix, and perform homogeneous alignment of the shared semantics between different modalities based on the sample-category center relationship vector and the category topology matrix; The hierarchical cross-modal fusion module is used to construct modality labels, interaction labels, and classification labels from the homogeneously aligned shared features, and to perform attention-based interaction fusion processing on the modality labels, interaction labels, and classification labels to obtain fused shared features; The dynamic gating fusion module is used to generate shared branch weights and private branch weights based on fusion shared features, private branch features obtained from private features of each modality, fuzzy structure summary, membership degree distribution and structural uncertainty information, and to fuse fusion shared features and private branch features according to shared branch weights and private branch weights to obtain the final classification features; The classification output module is used to output the fiber material identification results based on the final classification features.

[0072] Through data flow and collaborative processing among multiple functional modules, the decoupling, alignment, fusion, and identification of multimodal data of fiber materials are achieved. Specifically, the multimodal data processing module first acquires at least two types of data from the visible light image, near-infrared spectrum, and Raman spectrum corresponding to the same fiber material sample, and completes pairing and preprocessing to enable different modal data to correspond to the same identification object; the feature encoding module further converts the different forms of modal data into raw features, providing a feature basis for subsequent unified processing. The shared and private feature decoupling module decomposes the original features of each modality into shared and private features, enabling cross-modal common information and modality-specific information to be processed separately. The private structure representation module and the private heterogeneous alignment module construct fuzzy prototypes, cluster quality distributions, and structural uncertainty information based on private features, and establish the correspondence between private structure distributions of different modalities through optimal transmission. The shared homogeneous alignment module constructs category centers, sample-category center relationship vectors, and category topology matrices based on shared features, ensuring that the semantics of shared categories remain consistent across different modalities. The hierarchical cross-modal fusion module further fuses intra-modal information and inter-modal interaction information of the homogeneously aligned shared features to obtain fused shared features. The dynamic gating fusion module generates shared branch weights and private branch weights based on the fused shared features, private branch features, and structural uncertainty information, and obtains the final classification features accordingly. The classification output module finally outputs the fiber material identification results based on the final classification features. Therefore, this implementation method can reduce information aliasing caused by direct fusion of different modal information through systematic module collaboration, enhance the stability of shared semantic alignment and private structure alignment, and adaptively adjust the contribution ratio of shared information and private information according to the sample structure state, thereby improving the accuracy, robustness and adaptability to complex samples of fiber material identification.

[0073] It should be noted that each module in this embodiment can be implemented by one or more processors, memory, and program instructions stored in memory and executed by the processor. Alternatively, it can be implemented by software modules, hardware modules, or a combination of software and hardware modules in a server, industrial control computer, embedded computing device, or edge computing device. The modules can be connected sequentially according to the data processing order described above, or they can interact with each other on the same computing platform through data interfaces, cache units, or feature tensor transfer methods. Specifically, the shared homogeneous alignment module and the private heterogeneous alignment module can process the corresponding shared and private features in parallel, while the hierarchical cross-modal fusion module and the dynamic gating fusion module are used to complete the final fusion decision based on the alignment results.

[0074] The technical features of the above embodiments can be combined in any way (as long as there is no contradiction in the combination of these technical features). For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described; these embodiments not explicitly written should also be considered to be within the scope of this specification.

[0075] The present application has been described in a relatively specific and detailed manner above through general descriptions and specific embodiments. It should be understood that, based on the technical concept of the present application, several conventional adjustments or further innovations can be made to these specific embodiments; however, as long as they do not depart from the technical concept of the present application, the technical solutions obtained by these conventional adjustments or further innovations also fall within the protection scope of the claims of the present application.

Claims

1. A method for multimodal decoupling and layer alignment identification of fiber materials, characterized in that, include: Acquire multimodal data of fiber materials, wherein the multimodal data includes at least two of visible light image modes, near-infrared spectral modes, and Raman spectral modes; Feature extraction is performed on each modality of data to obtain the original features corresponding to each modality; The shared features and private features of each modality are decoupled to obtain the shared features and private features corresponding to each modality. A private structure representation is established based on the private features of each modality, and the private structure distributions between different modalities are heterogeneously aligned according to the private structure representation, and structural uncertainty information is obtained from the private structure representation; A shared semantic structure is established based on the shared features of each modality, and the shared semantics between different modalities are homogeneously aligned according to the shared semantic structure. Hierarchical cross-modal fusion is performed on the homogeneously aligned shared features to obtain fused shared features; Private branch features are obtained based on the private features of each modality, and shared branch weights and private branch weights are generated based on the fused shared features, the private branch features and the structural uncertainty information. Based on the shared branch weight and the private branch weight, the fused shared features and the private branch features are fused to obtain the final classification features, and the fiber material identification result is output based on the final classification features.

2. The method for multimodal decoupling and layer alignment identification of fiber materials according to claim 1, characterized in that, The shared features and private features of the original features of each modality are decoupled to obtain the shared features and private features corresponding to each modality, including: The original features of each modality are input into the shared mapping head and the private mapping head respectively. The shared mapping head outputs the shared features of the corresponding modality, and the private mapping head outputs the private features of the corresponding modality. Decoupling constraints are applied to shared and private features under the same modality to reduce information aliasing between the shared and private features; The decoupling constraints include orthogonal constraints to reduce sample-level directional consistency and cross-covariance constraints to reduce batch-level statistical correlation.

3. The method for multimodal decoupling and layer alignment identification of fiber materials according to claim 1, characterized in that, A private structure representation is established based on the private features of each modality, and structural uncertainty information is obtained from the private structure representation, including: For each modality's private features, construct multiple fuzzy prototype centers; The membership degree of each sample relative to each fuzzy prototype center is calculated based on the distance between the private features of each sample and the multiple fuzzy prototype centers. The cluster quality distribution of the corresponding mode is obtained based on the membership degree, and the fuzzy structure summary is obtained based on the membership degree and the fuzzy prototype center. Structural uncertainty information is obtained based on the degree of dispersion of the membership degree distribution.

4. The method for multimodal decoupling and layer alignment identification of fiber materials according to claim 3, characterized in that, Heterogeneous alignment of the private structure distributions across different modalities based on the aforementioned private structure representation includes: For any two modalities involved in the identification, a prototype cost matrix is ​​constructed to characterize the matching cost between different fuzzy prototype centers based on the fuzzy prototype centers corresponding to each of the two modalities. Based on the cluster quality distribution corresponding to each of the two modes and the prototype cost matrix, the transmission matrix between the two modes is solved based on optimal transmission. Based on the transmission matrix and the prototype cost matrix, the private heterogeneous alignment loss between the two modes is determined, and the private structure distribution between different modes is constrained based on the private heterogeneous alignment loss.

5. The method for multimodal decoupling and layer alignment identification of fiber materials according to claim 1, characterized in that, A shared semantic structure is established based on the shared features of each modality, and the shared semantics between different modalities are homogeneously aligned according to the shared semantic structure, including: Based on the shared features of each modality, the category center of each category in each modality is calculated separately; Based on the shared features of the samples and the relationship between each category center, a sample-category center relationship vector is constructed, and relationship consistency constraints are applied based on the sample-category center relationship vectors corresponding to the same sample under different modalities. A category topology matrix is ​​constructed based on the distance between the category centers, and topology consistency constraints are applied based on the category topology matrix under different modalities. Based on the relational consistency constraint and the topological consistency constraint, homogeneous alignment is performed on the shared semantics between different modalities.

6. The method for multimodal decoupling and layer alignment identification of fiber materials according to claim 5, characterized in that, Hierarchical cross-modal fusion is performed on the homogeneously aligned shared features to obtain fused shared features, including: The shared features of each modality after homogeneous alignment are used to construct corresponding modality tags; Construct interactive tags to represent intermodal interaction information based on the combination relationships between different modalities; A classification label is introduced to aggregate cross-modal fusion information, and the modal label, the interaction label, and the classification label are subjected to interaction fusion processing based on an attention mechanism; The fused shared features are obtained based on the output corresponding to the classification label.

7. The method for multimodal decoupling and layer alignment identification of fiber materials according to claim 6, characterized in that, Based on the fusion shared features, the private branch features, and the structural uncertainty information, shared branch weights and private branch weights are generated, and fusion is performed according to the shared branch weights and private branch weights, including: Private branch features are obtained based on the private features of each modality; The fusion shared features, the private branch features, and the structural feature information obtained from the private structure representation are combined to obtain the gated input features. The structural feature information includes fuzzy structure summary, membership degree distribution, and structural uncertainty information. Dynamic gating processing is performed based on the gating input features to generate shared branch weights and private branch weights. The contribution of the fused shared features is adjusted according to the shared branch weights, and the contribution of the private branch features is adjusted according to the private branch weights to obtain the final classification features; The shared branch weights and the private branch weights are determined based on the structural uncertainty information, and the structural uncertainty information is used to adjust the contribution ratio of the fused shared features and the private branch features in the final classification features.

8. The method for multimodal decoupling and layer alignment identification of fiber materials according to claim 1, characterized in that, It also includes the model training step: Obtain multimodal training samples of fiber materials with category labels; Auxiliary classification is performed based on the private features of each modality to obtain the private auxiliary classification result corresponding to each modality, and the private auxiliary classification loss is determined based on the private auxiliary classification result and the category label. The classification loss is determined based on the fiber material identification results and the category label; The decoupling loss is determined based on the decoupling process of shared features and private features, the private heterogeneous alignment loss is determined based on the heterogeneous alignment process of private structure distribution, the shared homogeneous alignment loss is determined based on the homogeneous alignment process of shared semantics, and the gating constraint loss is determined based on the adjustment relationship between shared branch weights and private branch weights based on structural uncertainty information. The fiber material identification model is trained based on the classification loss, the private auxiliary classification loss, the decoupling loss, the private heterogeneous alignment loss, the shared homogeneous alignment loss, and the gating constraint loss.

9. The method for multimodal decoupling and layer alignment identification of fiber materials according to claim 1, characterized in that, Acquiring multimodal data of fiber materials and extracting features from each modality, including: Different modal data corresponding to the same fiber material sample are paired according to sample identifier, collection batch or collection time to form multimodal data corresponding to the same fiber material sample; When the multimodal data includes a visible light image mode, the visible light image is subjected to at least one of the following processing methods: cropping, scaling, grayscale conversion, normalization, and image enhancement. When the multimodal data includes near-infrared spectral modes, the near-infrared spectrum is processed by at least one of band alignment, denoising, scattering correction, standardization, and normalization. When the multimodal data includes Raman spectral modes, the Raman spectra are processed by at least one of the following: baseline correction, smoothing filtering, peak alignment, intensity normalization, and noise reduction. Feature extraction is performed on the preprocessed modal data using encoders corresponding to each modality to obtain the original features corresponding to each modality, and the original features corresponding to each modality are mapped to a feature space of a unified dimension.

10. A multimodal decoupling and layer alignment identification system for fiber materials, characterized in that, include: A multimodal data processing module is used to acquire multimodal data of fiber materials, wherein the multimodal data includes at least two of visible light image modes, near-infrared spectral modes and Raman spectral modes, and to pair and preprocess different modal data corresponding to the same fiber material sample; The feature encoding module is used to extract features from each modality data to obtain the original features corresponding to each modality. The shared feature and private feature decoupling module is used to decouple the shared features and private features of the original features of each modality to obtain the shared features and private features corresponding to each modality; The private structure representation module is used to construct fuzzy prototype centers based on the private features of each modality, and obtain membership distribution, cluster quality distribution, fuzzy structure summary and structural uncertainty information; A private heterogeneous alignment module is used to construct a prototype cost matrix based on fuzzy prototype centers of different modalities, and to perform heterogeneous alignment of private structure distributions between different modalities through optimal transmission according to the cluster quality distribution of different modalities and the prototype cost matrix. The shared homogeneous alignment module is used to calculate the category center based on the shared features of each modality, construct the sample-category center relationship vector and the category topology matrix, and perform homogeneous alignment of the shared semantics between different modalities according to the sample-category center relationship vector and the category topology matrix; The hierarchical cross-modal fusion module is used to construct modality tags, interaction tags, and classification tags from the homogeneously aligned shared features, and to perform attention-based interaction fusion processing on the modality tags, interaction tags, and classification tags to obtain fused shared features; The dynamic gating fusion module is used to generate shared branch weights and private branch weights based on the fusion shared features, private branch features obtained from the private features of each modality, the fuzzy structure summary, the membership degree distribution and the structural uncertainty information, and to fuse the fusion shared features and the private branch features according to the shared branch weights and the private branch weights to obtain the final classification features; The classification output module is used to output the fiber material identification result based on the final classification features.

Images