Knowledge data classification multi-modal reasoning evolution system based on depth model

Abstract

The invention provides a knowledge data classification multi-modal reasoning evolution system based on a deep model, and the system comprises a dynamic semantic memory network module which is used for storing multi-source heterogeneous data through a memory network, and according to the relevance between the newly input multi-source heterogeneous data and the existing knowledge content in the memory network, obtaining the knowledge content of the multi-source heterogeneous data; dynamically updating the knowledge content in the memory network; the cross-modal collaborative classification enhancement module is used for extracting feature vectors of different modals from the multi-source heterogeneous data, performing feature alignment, and performing feature fusion on the feature vectors after feature alignment to generate comprehensive feature vectors fused with multi-modal information; and the multi-level layered attention classification module is used for processing the comprehensive feature vector by adopting a multi-level attention mechanism to understand the overall content and context information of the multi-source heterogeneous data so as to make a classification decision. According to the method, intelligent processing and classification decision making of complex information are realized, and a solid technical support is provided for knowledge evolution of a Chinese graph classification system.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a multimodal reasoning evolution system for knowledge data classification based on a deep model. Background Art

[0002] The core of the China Library Classification System, as a classification system widely used in the fields of books, documents, information retrieval, etc., is to provide a unified and standardized classification framework for various knowledge resources. This system usually includes multiple levels and categories, each of which corresponds to a specific knowledge field or theme, so as to achieve effective organization and management of knowledge resources.

[0003] In the prior art, when processing multi-source heterogeneous data such as text and images, the China Library Classification System usually adopts an independent processing method. This leads to the lack of an effective cross-modal fusion mechanism, and the system cannot fully utilize the complementary relationship between different modal data, and cannot accurately capture and express the multi-dimensional characteristics of knowledge. As a result, it is unable to cope with the processing of multi-source heterogeneous data such as text and images. Therefore, in the classification and reasoning process, the accuracy and efficiency of the system are limited to a certain extent. Summary of the invention

[0004] In response to the shortcomings of the prior art, the present application provides a multimodal reasoning evolution system for knowledge data classification based on a deep model, so as to at least solve the problem of low accuracy of Chinese image classification caused by insufficient processing capabilities of cross-modal multi-source heterogeneous data in the prior art.

[0005] In order to achieve the above objectives and other advantages, the present application adopts the following technical solutions:

[0006] In a first aspect, the present application provides a knowledge data classification multimodal reasoning evolution system based on a deep model, comprising:

[0007] A dynamic semantic memory network module, used to receive multi-source heterogeneous data including text data, image data and structure data, store the multi-source heterogeneous data using a memory network, and dynamically update the knowledge content in the memory network according to the relevance of the newly input multi-source heterogeneous data and the existing knowledge content in the memory network;

[0008] A cross-modal collaborative classification enhancement module is used to extract text feature vectors, image feature vectors, structural feature vectors and reference relationship feature vectors from the multi-source heterogeneous data and perform feature alignment, and perform feature fusion on the feature vectors after feature alignment to generate a comprehensive feature vector that fuses multi-modal information;

[0009] The multi-level hierarchical attention classification module is used to process the comprehensive feature vector using a multi-level attention mechanism to understand the overall content and contextual information of the multi-source heterogeneous data, and then make classification decisions to classify the newly input multi-source heterogeneous data into the corresponding level of the China Library Classification System.

[0010] According to a knowledge data classification multimodal reasoning evolution system based on a deep model provided by the present application, the memory network includes: a short-term memory module and a long-term memory module, and the dynamic semantic memory network module includes: a memory update unit, a semantic entropy calculation unit, an entropy weight update unit and a knowledge mapping unit;

[0011] The memory update unit is used to integrate the outputs of the short-term memory module, the long-term memory module and the knowledge distillation module to achieve dynamic updating and optimization of knowledge;

[0012] The semantic entropy calculation unit is used to calculate the information entropy of the semantic unit and evaluate the occurrence probability of each semantic unit;

[0013] The entropy weight updating unit dynamically adjusts the weight based on the semantic change amount to ensure the dynamic update of knowledge;

[0014] The knowledge mapping unit is used to map the knowledge in the memory network into the space of the knowledge graph in the form of key-value pairs to achieve structured knowledge representation.

[0015] According to a multimodal reasoning evolution system for knowledge data classification based on a deep model provided by the present application, the multi-level hierarchical attention classification module includes: an attention calculation unit and a classification decision unit;

[0016] The attention calculation unit is used to respectively calculate the local semantic features and the global multimodal features of the comprehensive feature vector using the local attention weight matrix and the global attention weight matrix, and fuse the local semantic features with the global multimodal features;

[0017] The classification decision unit is used to process the weighted fused comprehensive feature vector through a classification mapping function to obtain a classification decision.

[0018] According to a multimodal reasoning evolution system for knowledge data classification based on a deep model provided by the present application, the cross-modal collaborative classification enhancement module includes: a feature extraction unit, a feature alignment unit and a feature fusion unit;

[0019] The feature extraction unit is used to extract text feature vectors, image feature vectors, structural feature vectors and reference relationship feature vectors from the multi-source heterogeneous data;

[0020] The feature alignment unit is used to map feature vectors of different modalities into a unified feature space;

[0021] The feature fusion unit is used to perform feature fusion on the feature vectors after feature alignment to generate a comprehensive feature vector that fuses multi-modal information.

[0022] According to a deep model-based knowledge data classification multimodal reasoning evolution system provided by the present application, it also includes: a bidirectional knowledge distillation module;

[0023] The bidirectional knowledge distillation module includes a large model for storing long-term memory knowledge in a dynamic semantic memory network and a small model for processing short-term memory knowledge in a dynamic semantic memory network;

[0024] The bidirectional knowledge distillation module provides a bidirectional knowledge migration mechanism between the large model and the small model, promotes the co-evolution between the large model and the small model through the knowledge migration results, and provides a knowledge consistency constraint mechanism in the knowledge migration process, and minimizes the differences in the knowledge migration process through the consistency constraint results;

[0025] The bidirectional knowledge distillation module also provides a knowledge dynamic update adaptation mechanism to dynamically balance the knowledge migration results and the consistency constraint results.

[0026] According to a knowledge data classification multimodal reasoning evolution system based on a deep model provided by the present application, it also includes: a knowledge graph enhancement module, the knowledge graph enhancement module includes: a graph structure definition unit, a graph reasoning unit and a knowledge enhancement unit;

[0027] The graph structure definition unit is used to define a knowledge graph, which includes a category node set, a relationship edge set, and a relationship type set, wherein the category node set represents a multi-level category structure of the China Library Classification System, the relationship edge set represents a semantic association of a dynamic semantic memory network, and the relationship type set represents a multimodal relationship of multi-source heterogeneous data;

[0028] The graph reasoning unit is used to perform dynamic reasoning based on the knowledge graph to generate reasoning results;

[0029] The knowledge enhancement unit is used to perform multi-dimensional enhancement on the nodes, relationships and types of the knowledge graph to improve the representation capability of the graph.

[0030] According to a knowledge data classification multimodal reasoning evolution system based on a deep model provided by the present application, it also includes: a confidence assessment module, the confidence assessment module includes: a comprehensive assessment unit, an uncertainty estimation unit and a decision guidance unit;

[0031] The comprehensive evaluation unit is used to perform a multi-dimensional evaluation on the classification result based on the memory knowledge of the dynamic semantic memory network, the multimodal features of the multi-source heterogeneous data, and the knowledge graph to obtain a comprehensive evaluation score;

[0032] The uncertainty estimation unit is used to quantify the uncertainty of the classification result based on a preset uncertainty balance factor to obtain an uncertainty estimation value;

[0033] The decision guidance unit is used to generate a decision guidance value for collaborative decision-making based on the comprehensive evaluation score and the uncertainty estimate.

[0034] According to a knowledge data classification multimodal reasoning evolution system based on a deep model provided by the present application, it also includes: a collaborative decision-making module, the collaborative decision-making module includes: a decision fusion unit, a decision optimization unit and a decision adjustment unit;

[0035] The decision fusion unit is used to integrate decision information from different sources to generate a decision optimization result, wherein the decision information includes a large model output, a small model output, and a knowledge graph output;

[0036] The decision optimization unit is used to construct a loss function of multi-task learning based on knowledge distillation loss, graph reasoning loss and feature alignment loss, so as to define an optimization target of multi-task learning by minimizing the loss function;

[0037] The decision adjustment unit is used to adjust the weight according to the feedback information in the decision process. When there is a deviation between the decision optimization result and the actual situation, the weight is fine-tuned according to the gradient information of the loss function.

[0038] According to a knowledge data classification multimodal reasoning evolution system based on a deep model provided by the present application, it also includes: a self-evolution classification module, the self-evolution classification module includes: a parameter updating unit, a mutation probability calculation unit and a multi-module optimization unit;

[0039] The parameter updating unit is used to perform feedforward correction update on the current parameters in the knowledge distillation result based on gradient optimization and mutation mechanism, and update the memory network based on the parameter update result;

[0040] The mutation probability calculation unit is used to dynamically calculate the mutation probability, and continuously adjust the mutation intensity during the adaptive evolution process to find the most suitable optimization strategy to obtain the mutation optimization result;

[0041] The multi-module optimization unit is used to balance the parameter update results and the mutation optimization results to achieve collaborative optimization among multiple modules.

[0042] According to a knowledge data classification multimodal reasoning evolution system based on a deep model provided by the present application, it also includes: a classification optimization module, the classification optimization module includes: a category optimization unit, a feedback optimization loss unit and an optimization integration unit;

[0043] The category optimization unit is used to optimize the Chinese Library Classification categories by adjusting the balance coefficient based on the comprehensive evaluation score, the decision optimization result and the feedforward correction item provided by the self-evolution classification module, and generate a category optimization result;

[0044] The feedback optimization loss unit constructs a feedback optimization loss function that imposes multiple constraints on the classification results based on the conditional probability of the decision optimization result, the rule satisfaction of the consistency constraint result, and the historical consistency score reflecting the past classification results, and obtains the feedback optimization loss;

[0045] The optimization integration unit is used to perform weighted summation on the category optimization result and the feedback optimization loss to obtain a final optimization result.

[0046] The present application provides a multimodal reasoning evolution system for knowledge data classification based on a deep model, which includes: a dynamic semantic memory network module, which is used to receive multi-source heterogeneous data including text data, image data and structural data, store the multi-source heterogeneous data using a memory network, and dynamically update the knowledge content in the memory network according to the correlation between the newly input multi-source heterogeneous data and the existing knowledge content in the memory network; a cross-modal collaborative classification enhancement module, which is used to extract text feature vectors, image feature vectors, structural feature vectors and reference relationship feature vectors from the multi-source heterogeneous data and perform feature alignment, and perform feature fusion on the feature vectors after feature alignment to generate a comprehensive feature vector that fuses multimodal information; a multi-level hierarchical attention classification module, which is used to use a multi-level attention mechanism to process the comprehensive feature vector to understand the overall content and contextual information of the multi-source heterogeneous data, and then make classification decisions to classify the newly input multi-source heterogeneous data into the corresponding level of the China Library Classification System. By integrating technologies such as dynamic semantic memory, cross-modal collaborative classification enhancement, and multi-level hierarchical attention classification, this application enables the system to more accurately grasp the core features of multi-source heterogeneous data, enhance the system's understanding and adaptability to each level of the China Library Classification System, and realize intelligent processing and classification decisions for complex information. This can effectively improve classification accuracy, reduce error propagation rate, significantly improve processing efficiency, and provide solid technical support for the knowledge evolution of the China Library Classification System. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other implementation methods can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 It is a structural diagram of a multimodal reasoning evolution system for knowledge data classification based on a deep model provided in an embodiment of the present application;

[0049] Figure 2 is a logic processing flow chart of the knowledge distillation module provided in an embodiment of the present application;

[0050] Figure 3 is a logic processing flow chart of the self-evolution classification module provided in an embodiment of the present application;

[0051] Figure 4 It is a logical processing flow chart of the collaborative decision-making module provided in the embodiment of the present application;

[0052] Figure 5 This is a real-time classification prediction effect diagram of a knowledge data classification multimodal reasoning evolution system based on a deep model provided by this application. DETAILED DESCRIPTION

[0053] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following specifically cites the preferred embodiments and describes them in detail with the accompanying drawings.

[0054] It should be noted that it is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict. Unless otherwise defined, the technical terms or scientific terms involved in this application should be the usual meanings understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a kind of", "the" and the like involved in this application do not represent quantity restrictions and may represent the singular or plural. The terms "including", "comprising", "having" and any of their variations involved in this application are intended to cover non-exclusive inclusions; the terms "first", "second", "third", etc. involved in this application are merely to distinguish similar objects and do not represent a specific ordering of objects.

[0055] Reference Figure 1 As shown, the embodiment of the present application provides a knowledge data classification multimodal reasoning evolution system based on a deep model, including:

[0056] The dynamic semantic memory network module is used to receive multi-source heterogeneous data including text data, image data and structure data, use the memory network to store the multi-source heterogeneous data, and dynamically update the knowledge content in the memory network according to the correlation between the newly input multi-source heterogeneous data and the existing knowledge content in the memory network.

[0057] Among them, multi-source heterogeneous data include: text data: such as title text, abstract text and directory text, etc.; image data: such as cover pictures, content illustrations and structural charts, etc.; structural data: such as chapter structure, citation relationships and thesaurus, etc.

[0058] In this embodiment, the memory network includes: a short-term memory module and a long-term memory module, and the dynamic semantic memory network module includes: a memory update unit, a semantic entropy calculation unit, an entropy weight update unit and a knowledge mapping unit.

[0059] Among them, the dynamic semantic memory network module in the China Library Classification System mainly stores and updates the following types of knowledge: such as text knowledge, image knowledge and structural knowledge.

[0060] The memory network can be a BiLSTM model, including a short-term memory module and a long-term memory module. Among them, the short-term memory module adopts the following specific technical parameters: the input layer dimension is set to 768, the hidden layer dimension is set to 256, the memory capacity is 10,000, the update cycle is 24 hours, and the entropy threshold is set to 0.3. The long-term memory module adopts the following technical parameters: the node dimension is set to 256, the edge dimension is set to 64, a dynamically updated heterogeneous graph network structure is adopted, and the time threshold is set to 30 days. The two memory modules maintain the timeliness and importance of information through a dynamic update mechanism.

[0061] The function defining the short-term memory module can be: Ms(t) = {(k i ,v i ,e i )|ei>eth}

[0062] The function defining the long-term memory module can be: M1(t) = {(k j ,v j ,e j )|Tj>Tth}

[0063] Where: k i , k j is the semantic key vector with dimension d1; vi, vj are memory value vectors with dimension d2; e i 、e j is the semantic entropy weight, ranging from [0, 1]; eth is the entropy threshold; Tth is the time threshold; t is the current time step; T jThe duration of the memory item.

[0064] The memory update unit is used to integrate the outputs of the short-term memory module, the long-term memory module, and the knowledge distillation module to achieve dynamic updating and optimization of knowledge.

[0065] Specifically, the memory update unit adopts a memory update mechanism that cooperates with the knowledge distillation module, and the memory update function set can be:

[0066] Where: M′(t) is the output of the memory network; Ms(t) is the output of the short-term memory module; Ml(t) is the output of the long-term memory module; KD(t) is the output of the knowledge distillation module; α is the memory balance coefficient, ranging from [0, 1]; β is the weight coefficient of knowledge distillation, ranging from [0, 1]; is the memory fusion operator.

[0067] The memory update function is used to achieve collaborative work with the knowledge distillation module. The memory network integrates the short-term memory module output Ms(t), the long-term memory module output Ml(t) and the knowledge distillation module output KD(t) through the fusion operator, where β controls the influence of knowledge distillation, thereby achieving dynamic updating and optimization of knowledge.

[0068] The semantic entropy calculation unit is used to calculate the information entropy of the semantic unit and evaluate the occurrence probability of each semantic unit.

[0069] Specifically, the semantic entropy calculation unit provides an entropy calculation mechanism for calculating semantic importance, and the semantic entropy calculation function set may be: S(x)=-∑p(x i )×log(p(x i ))

[0070] Where: S(x) is the semantic entropy; x i is the i-th semantic unit; p(x i ) is the occurrence probability of the semantic unit.

[0071] The semantic entropy calculation function provides an importance measure for memory updating and knowledge graph construction. By calculating the information entropy of semantic units, it provides knowledge importance evaluation for the knowledge distillation module and provides a quantitative basis for the importance of knowledge graph nodes.

[0072] The entropy weight updating unit dynamically adjusts the weight based on the semantic change to ensure the dynamic updating of knowledge.

[0073] Specifically, the entropy weight updating unit provides a weight updating mechanism for maintaining knowledge timeliness, and the entropy weight updating function set may be: e′(t)=e(t-1)×exp(-ζ·ΔS(t))

[0074] Where: e′(t) is the entropy weight; ζ is the attenuation coefficient, which ranges from (0, 1); ΔS(t) is the semantic change at time t; e(t-1) is the entropy weight at the previous moment.

[0075] The entropy weight update function dynamically adjusts the information entropy weight of the semantic unit based on the semantic change ΔS(t) to ensure the dynamic update of knowledge. It provides a timeliness reference for the knowledge distillation module and guides the dynamic evolution of the knowledge graph.

[0076] The knowledge mapping unit is used to map the knowledge in the memory network into the space of the knowledge graph in the form of key-value pairs to achieve structured knowledge representation.

[0077] Specifically, the knowledge mapping unit provides a data support mechanism for knowledge graph enhancement, and the knowledge mapping function set can be: D(M) = {(k i , v i )|k i ∈K,v i )∈V}

[0078] Where: D(M) is the result of knowledge mapping; K is the key space of the knowledge graph; V is the value space of the knowledge graph; k i is the semantic key vector; v i is the memory value vector.

[0079] Through this knowledge mapping function, the knowledge in the memory network is represented by key-value pairs (k i , v i ) is mapped into the space of the knowledge graph in the form of , providing a structured knowledge representation for the knowledge graph to support the construction and update of the knowledge graph.

[0080] The dynamic semantic memory network module also includes: a feature deep fusion unit, which provides a fusion mechanism for deep fusion with the multi-level hierarchical attention classification module. The feature deep fusion function set can be:

[0081]

[0082] Where: F(M, AL) is the result of deep feature fusion; M′(t) is the output of the memory network; AL(x) is the output of the multi-level hierarchical attention classification module; γ1 and γ2 are fusion weight coefficients, ranging from [0, 1]; is the feature fusion operator.

[0083] Through this feature deep fusion function, the semantic feature M′(t) of the memory network and the output AL(x) of the attention classification mechanism are combined through the feature fusion operator Deep fusion is performed, and the fusion weights γ1 and γ2 dynamically adjust the importance of the two, providing fused semantic feature support for classification decisions.

[0084] Through the collaborative work of the above multiple mechanisms, a complete knowledge management closed loop is formed. The fusion mechanism and the multi-level hierarchical attention classification module realize classification feature enhancement, the memory update mechanism and the knowledge distillation module realize knowledge optimization, and the data support mechanism and the knowledge graph enhancement module realize knowledge structuring, which together provide a reliable basic guarantee for the classification decision and knowledge evolution of the China Library Classification System.

[0085] The cross-modal collaborative classification enhancement module is used to extract text feature vectors, image feature vectors, structural feature vectors and reference relationship feature vectors from multi-source heterogeneous data and perform feature alignment, and perform feature fusion on the feature vectors after feature alignment to generate a comprehensive feature vector that integrates multi-modal information.

[0086] In this embodiment, the cross-modal collaborative classification enhancement module includes: a feature extraction unit, a feature alignment unit and a feature fusion unit.

[0087] The feature extraction unit is used to extract text feature vectors, image feature vectors, structural feature vectors and reference relationship feature vectors from multi-source heterogeneous data.

[0088] Specifically, multi-source heterogeneous data of different modalities can be a variety of data types including text data, image data, and structured data. These data types may come from different data sources or systems and need to be integrated, analyzed, and processed to achieve multi-layer accurate classification of Chinese Library Classification (AZ).

[0089] The feature extraction unit provides a feature extraction mechanism that works in conjunction with the dynamic semantic memory network, processing four types of features respectively: text features are extracted using a pre-trained language model with a dimension of 768; image features are extracted using a pre-trained visual model with a dimension of 256; structural features are extracted using a graph neural network with a dimension of 64; reference relationship features are determined by analyzing the reference relationship network between documents, extracting the citation and citation relationship between documents, and constructing a citation relationship graph to analyze the reference list. The set multimodal feature extraction function can be: F(d) = [Tt(d), Vi(d), Ss(d), Rr(d)]

[0090] Among them: F(d) is a multimodal feature; Tt(d) is a text feature vector with dimension dt, which extracts text information such as title and abstract, and provides text features for the dynamic semantic memory network; Vi(d) is an image feature vector with dimension di, which extracts image information such as charts and formulas, and provides image features for the multi-level hierarchical attention classification module; Ss(d) is a structural feature vector with dimension ds, which extracts organizational information such as directory structure, and provides structural features for the knowledge distillation module; Rr(d) is a reference relationship feature vector with dimension dr, which extracts related information such as citation relationship, and provides reference relationship features for the collaborative decision-making module.

[0091] The feature alignment unit is used to map feature vectors of different modalities into a unified feature space.

[0092] Specifically, the feature alignment unit provides a cross-modal alignment mechanism that cooperates with the knowledge distillation module. The set cross-modal alignment function can be:

[0093] Among them: A(x, y) is the aligned multimodal feature; x and y are different modal features; is the mapping function of the modality x, which is used to realize the feature space conversion with the dynamic semantic memory network; is the mapping function of the mode y, which is used to realize the feature space conversion with the knowledge distillation module; Φ xy (·) is the cross-modal interaction function, which is used to establish feature associations between features of different modalities.

[0094] The feature fusion unit is used to perform feature fusion on the feature vectors after feature alignment to generate a comprehensive feature vector that fuses multimodal information.

[0095] Specifically, the feature fusion unit provides a feature fusion mechanism that cooperates with the classification optimization module and the collaborative decision-making module. The set feature fusion function can be:

[0096] M(F)=W1×Tt(d)⊙W2×Vi(d)⊙W3×Ss(d)⊙W4×Rr(d)

[0097] Among them: M(F) is the feature fusion result; W1, W2, W3, W4 are feature weight matrices, which are regulated by the classification optimization module; ⊙ is the weighted fusion operator, which is used to realize feature fusion with the collaborative decision-making module.

[0098] The cross-modal collaborative classification enhancement module also includes: a feature update unit, which provides a feedback mechanism for optimizing the overall feature processing. The feature update function set can be:

[0099] F′(d)=F(d)+λ×G(M(F))

[0100] Where: F′(d) is the updated multimodal feature representation; F(d) is the current multimodal feature representation; G(·) is the optimization gradient provided by the classification optimization module; λ is the learning rate parameter.

[0101] In this way, a complete multimodal feature processing system is constructed to provide feature support for accurate recognition of Chinese image classification.

[0102] The multi-level hierarchical attention classification module is used to process the comprehensive feature vector using a multi-level attention mechanism to understand the overall content and contextual information of multi-source heterogeneous data, and then make classification decisions to classify the newly input multi-source heterogeneous data into the corresponding level of the China Library Classification System.

[0103] In this embodiment, the multi-level hierarchical attention classification module includes: an attention calculation unit and a classification decision unit;

[0104] The attention calculation unit is used to respectively calculate the local semantic features and the global multimodal features of the comprehensive feature vector using the local attention weight matrix and the global attention weight matrix, and to fuse the local semantic features with the global multimodal features.

[0105] Specifically, the attention calculation unit provides a multi-level accurate classification attention calculation mechanism, including: local attention mechanism, global attention mechanism and hierarchical attention mechanism. The local attention mechanism is used to extract the local semantic features of the semantic memory network, and the set function can be: ψl(x) = softmax(Wl×tanh(Ul×x+b1)), where Wl and U1 are local attention weight matrices for processing local semantic features; bl is a bias vector.

[0106] The local attention mechanism is used to process the global multimodal features provided by the cross-modal collaborative classification enhancement module. The set function can be: ψg(x) = sigmoid(Wg×x+bg), where Wg is the global attention weight matrix; bg is the bias vector.

[0107] The hierarchical attention mechanism is used to fuse the local semantic feature ψ1(x) and the global multimodal feature ψg(x). The function set can be: Among them, ψh(x) is the comprehensive eigenvector, is the Hadamard product operation.

[0108] At the same time, the hierarchical attention mechanism also integrates the knowledge graph structure of the knowledge graph enhancement module, thereby providing weight distribution for the confidence assessment module and providing a reliable basis for the collaborative decision-making module.

[0109] The classification decision unit is used to process the weighted fused comprehensive feature vector through a classification mapping function to obtain a classification decision.

[0110] Specifically, the classification decision unit provides a mapping mechanism to realize the fusion classification of multi-source features, comprehensively processes multi-source features through the classification mapping function, provides hierarchical classification results for the confidence assessment module, and provides decision-making basis for the collaborative decision module. The set classification mapping function can be:

[0111] C(x)=∑ψm(x)×Am(x)

[0112] Where: C(x) is the multi-layer classification result; x is a comprehensive feature vector, which integrates the semantic features provided by the dynamic semantic memory network module, the multimodal features provided by the cross-modal collaborative classification enhancement module, and the knowledge graph features provided by the knowledge graph enhancement module; ψm(x) is the attention weight of the mth layer, which is used to adjust the importance of features at different levels; Am(x) is the classification result of the mth layer, corresponding to the output of each level of the China Library Classification System; m∈{1, 2, 3, 4, 5} represents the hierarchical index of the five-level classification of the China Library Classification System.

[0113] The classification decision unit adopts a multi-layer classification structure, in which the L1 layer (AZ major categories) is implemented using the BiLSTM structure, the L2 layer (sub-categories) is implemented using a hierarchical softmax structure, and the L3-L5 layers are implemented using recursive fine-grained classifiers.

[0114] Therefore, through the collaborative work of the above-mentioned classification mapping mechanism and attention calculation mechanism, this module realizes the multi-level precise classification of the Chinese Library Classification System from L1 to L5, ensuring the accuracy of the classification results and the rationality of the hierarchical understanding.

[0115] In this embodiment, it also includes: a two-way knowledge distillation module;

[0116] The bidirectional knowledge distillation module includes a large model for storing long-term memory knowledge in the dynamic semantic memory network and a small model for processing short-term memory knowledge in the dynamic semantic memory network;

[0117] The bidirectional knowledge distillation module provides a bidirectional knowledge transfer mechanism between the large model and the small model. It promotes the co-evolution between the large model and the small model through the knowledge transfer results. It also provides a knowledge consistency constraint mechanism during the knowledge transfer process. It minimizes the differences in the knowledge transfer process through the consistency constraint results.

[0118] The bidirectional knowledge distillation module also provides a knowledge dynamic update adaptation mechanism to dynamically balance the knowledge transfer results and consistency constraint results.

[0119] Specifically, refer to Figure 2 As shown, the module provides a bidirectional knowledge transfer mechanism, and the knowledge transfer function set can be:

[0120] KT(t)=ρa×T(ML→MS)+ρb×R(MS→ML)

[0121] Where: KT(t) is the result of knowledge transfer; ML is a large model used to store the long-term memory knowledge of the memory network; MS is a small model used to process the short-term memory knowledge of the memory network; ρa and ρb are dynamic weight coefficients, which are regulated by the self-evolution classification module; T(·) is the forward knowledge transfer function, which realizes the knowledge transfer from ML to MS; R(·) is the reverse knowledge supplement function, which realizes the knowledge supplement from MS to ML.

[0122] The knowledge transfer function provides a knowledge basis for the dynamic semantic memory network module through a two-way knowledge transfer mechanism, while receiving feedback guidance from the classification optimization module.

[0123] This module also provides a knowledge consistency constraint mechanism that cooperates with the self-evolution classification module and the classification optimization module. The knowledge consistency constraints set by this constraint mechanism can be:

[0124] CC(K)=||KLS-KSL|| 2 +λc×DKL(PL||PS)

[0125] Among them: CC(K) is the consistency constraint result; KLS is the amount of knowledge transfer from large to small, which is optimized by the parameters of the self-evolution classification module; KSL is the amount of knowledge transfer from small to large, which is used for knowledge update in the dynamic semantic memory network module; PL and PS are the probability distributions of large and small models, which are used for knowledge consistency evaluation; λc is the consistency balance coefficient, which is dynamically adjusted by the classification optimization module; DKL is the KL divergence, which is used to measure the difference in knowledge distribution.

[0126] This knowledge consistency constraint mechanism ensures the consistency of knowledge transfer in the collaborative process with the dynamic semantic memory network module, the self-evolving classification module and the classification optimization module by minimizing knowledge differences.

[0127] The module also provides an adaptation mechanism for realizing dynamic knowledge update. The knowledge adaptation function set by the adaptation mechanism can be: KA(t) = n×KT(t) + (1-n)×CC(K)

[0128] Among them: KA(t) is the knowledge fitness; η is the adaptation weight coefficient; KT(t) is the knowledge transfer result; CC(K) is the consistency constraint result.

[0129] The adaptation mechanism achieves collaborative optimization with related modules by dynamically balancing knowledge migration and consistency constraints.

[0130] Therefore, the bidirectional knowledge distillation module provides close collaboration between the above-mentioned mechanisms and related modules, breaks through the limitations of traditional one-way distillation, builds a complete knowledge transfer and optimization system, realizes knowledge complementarity and co-evolution between large and small models, and provides reliable guarantee for the knowledge optimization of China Library Classification.

[0131] In this embodiment, it also includes: a self-evolution classification module, which includes: a parameter updating unit, a mutation probability calculation unit and a multi-module optimization unit.

[0132] The parameter update unit is used to perform feedforward correction updates on the current parameters in the knowledge distillation results based on gradient optimization and mutation mechanisms, and to update the memory network based on the parameter update results.

[0133] Specifically, refer to Figure 3 As shown, the parameter updating unit provides a parameter updating mechanism coordinated with the knowledge distillation result. The parameter updating function set by the parameter updating mechanism can be:

[0134]

[0135] Among them: θc is the current parameter, inherited from the knowledge distillation result in the knowledge distillation module; θn is the updated parameter, used to update the memory network in the dynamic semantic memory network module; nl is the learning rate, which is dynamically adjusted by the confidence evaluation module; is the gradient of the loss function, reflecting the optimization objective in the classification optimization module; σm is the mutation intensity, which is controlled by the mutation probability calculation mechanism; I(·) is the parameter mutation function, which realizes the adaptive adjustment of parameters.

[0136] The feedforward correction function set by the parameter update mechanism can be:

[0137] F(θ)=ω×tanh(W×θc+b)

[0138] Where: F(θ) is the feedforward correction term; ω is the correction weight matrix; W is the transformation matrix; b is the bias vector.

[0139] This parameter update function achieves collaborative optimization with the dynamic semantic memory network module, knowledge distillation module, confidence assessment module, and classification optimization module through gradient optimization and mutation mechanism.

[0140] The mutation probability calculation unit is used to dynamically calculate the mutation probability and continuously adjust the mutation intensity during the adaptive evolution process to find the most suitable optimization strategy to obtain the mutation optimization result.

[0141] Specifically, the mutation probability calculation unit provides a mutation probability calculation mechanism for implementing adaptive evolution, and the set mutation probability calculation function can be:

[0142] pl(t)=sgmoid(ξa×ΔJ+ωb)

[0143] Where: pl(t) is the mutation probability; ξ is the loss sensitivity coefficient, reflecting the evaluation result of the confidence assessment module; ΔJ is the loss change, which comes from the optimization feedback of the classification optimization module; ωb is the basic mutation probability, which is determined by the knowledge distribution of the knowledge distillation module; t is the number of iterations, which is used to control the adaptive evolution process.

[0144] The mutation probability calculation function ensures the stability of the adaptive evolution process by adaptively adjusting the mutation intensity.

[0145] The multi-module optimization unit is used to balance the parameter update results and the mutation optimization results to achieve collaborative optimization among multiple modules.

[0146] Specifically, the multi-module optimization unit provides an integration mechanism for coordinating multi-module optimization, and the set optimization integration function can be:

[0147] O(θ)=β1×U(θ)+β2×P(θ)

[0148] Among them: O(θ) is the integrated optimization result; U(θ) is the parameter update result; P(θ) is the mutation optimization result; β1 and β2 are the weight coefficients of integrated optimization.

[0149] This integrated mechanism achieves collaborative optimization with related modules by balancing parameter update results and mutation optimization results.

[0150] Therefore, the self-evolving classification module provides the above-mentioned multiple mechanisms to achieve close collaboration with related modules, builds a complete adaptive evolution system, realizes the continuous evolution of the model in a complex environment, and provides a reliable guarantee for the performance improvement of Chinese image classification.

[0151] In this embodiment, it also includes: a knowledge graph enhancement module, and the knowledge graph enhancement module includes: a graph structure definition unit, a graph reasoning unit and a knowledge enhancement unit.

[0152] The graph structure definition unit is used to define the knowledge graph, which includes a category node set, a relationship edge set and a relationship type set. The category node set represents the multi-level category structure of the China Library Classification System, the relationship edge set represents the semantic association of the dynamic semantic memory network, and the relationship type set represents the multimodal relationship of multi-source heterogeneous data.

[0153] Specifically, the graph structure definition unit provides a knowledge graph definition mechanism that cooperates with the dynamic semantic memory network. The knowledge graph structure can be defined as: G = (VN, EG, RG)

[0154] Among them: G is the knowledge graph; VN is the category node set, corresponding to the five-level category structure of the China Library Classification System; EG is the relationship edge set, representing the semantic association of the memory network; RG is the relationship type set, including the multimodal relationship in the cross-modal collaborative classification enhancement module.

[0155] This knowledge graph provides a structured knowledge basis for the multi-level classification of the multi-level hierarchical attention classification module through formal expression.

[0156] The graph reasoning unit is used to perform dynamic reasoning based on the knowledge graph and generate reasoning results.

[0157] Specifically, the graph reasoning unit provides a reasoning process mechanism that cooperates with the cross-modal collaborative classification enhancement module and the multi-level hierarchical attention classification module. The graph reasoning function set can be:

[0158] s n =GRUθ(s n -1, [rn; e n ])

[0159] Where: s n is the n-th step state vector, integrating the multimodal features in the cross-modal collaborative classification enhancement module; r n is the relation vector, inheriting the semantic association in the dynamic semantic memory network module; n is the entity vector, corresponding to the category information of the Chinese library classification system; GRUθ is the gated recurrent unit, which realizes the dynamic reasoning process.

[0160] The graph reasoning function provides dynamic reasoning support for the collaborative decision-making module and classification optimization module through recursive updates.

[0161] The knowledge enhancement unit is used to perform multi-dimensional enhancement on the nodes, relationships, and types of the knowledge graph to improve the representation capabilities of the graph.

[0162] Specifically, the knowledge enhancement unit provides a graph enhancement mechanism for optimizing knowledge representation, and the set knowledge enhancement function can be:

[0163] K(G)=μ1×S(VN)+μ2×R(EG)+μ3×T(RG)

[0164] Among them: K(G) is the optimized representation of the knowledge graph G; S(·) is the node enhancement function; R(·) is the relationship enhancement function; T(·) is the type enhancement function; μ1, μ2, μ3 are the enhancement weight coefficients.

[0165] This enhancement mechanism improves the representation capability of the knowledge graph through multi-dimensional optimization.

[0166] Therefore, the knowledge graph enhancement module provides the above-mentioned multiple mechanisms to achieve close collaboration with related modules, and builds a complete knowledge graph reasoning system through graph definition, reasoning process and graph enhancement, providing reliable guarantee for knowledge support and reasoning optimization of Chinese library classification.

[0167] In this embodiment, it also includes: a confidence assessment module, which includes: a comprehensive assessment unit, an uncertainty estimation unit and a decision guidance unit.

[0168] The comprehensive evaluation unit is used to perform multi-dimensional evaluation on the classification results based on the memory knowledge of the dynamic semantic memory network, the multimodal characteristics of multi-source heterogeneous data, and the knowledge graph to obtain a comprehensive evaluation score.

[0169] Specifically, the comprehensive evaluation unit provides a comprehensive evaluation mechanism for evaluating the classification results in the multi-level hierarchical attention classification module, and the set comprehensive evaluation function can be:

[0170] Q(y)=βe×E(y)+βd×D(y)+βn×N(y)

[0171] Among them: Q(y) is the comprehensive evaluation score; E(y) is the expert rule score, which is evaluated in combination with the memory knowledge in the dynamic semantic memory network module, and its value range is [0, 1]; D(y) is the data statistical score, which is calculated based on the multimodal features in the fusion cross-modal collaborative classification enhancement module, and its value range is [0, 1]; N(y) is the proximity similarity score, which is calculated using the knowledge graph in the knowledge graph enhancement module, and its value range is [0, 1]; βe, βd, and βn are weight coefficients, which are dynamically adjusted by the optimization mechanism of the classification optimization module.

[0172] This comprehensive evaluation function provides a reliable basis for the collaborative decision-making module through multi-dimensional evaluation.

[0173] The uncertainty estimation unit is used to quantify the uncertainty of the classification result based on a preset uncertainty balance factor to obtain an uncertainty estimation value.

[0174] Specifically, the uncertainty estimation unit provides an uncertainty evaluation mechanism that cooperates with the self-evolution classification module and the classification optimization module. The uncertainty estimation function set can be:

[0175] UC(y)=-∑p(yi)×log(p(yi))+λu×var(y)

[0176] Among them: UC(y) is the uncertainty estimate; p(yi) is the category probability distribution, which comes from the classification result of the multi-level hierarchical attention classification module; var(y) is the variance, which reflects the stability of the classification result; λu is the uncertainty balance factor, which is regulated by the self-evolving classification module.

[0177] The uncertainty estimation function provides feedback guidance for the optimization process of the classification optimization module by quantifying uncertainty.

[0178] The decision guidance unit is used to generate decision guidance values ​​for collaborative decision-making based on the comprehensive evaluation score and uncertainty estimation value.

[0179] Specifically, the decision guidance unit provides a decision support mechanism for integrating the evaluation results. The set decision guidance function can be: D(y) = Q(y) × (1-UC(y))

[0180] Where: Q(y) is the comprehensive evaluation score; UC(y) is the uncertainty estimate.

[0181] The decision guidance function provides reliable support for the collaborative decision-making module by balancing the comprehensive evaluation score and uncertainty estimation value.

[0182] Therefore, the confidence assessment module provides the above-mentioned multiple mechanisms to achieve close collaboration with related modules, builds a complete quality assessment system, and provides effective guarantee for the accuracy and reliability of Chinese library classification.

[0183] In this embodiment, it also includes: a collaborative decision-making module, which includes: a decision fusion unit, a decision optimization unit and a decision adjustment unit.

[0184] The decision fusion unit is used to integrate decision information from different sources to generate decision optimization results. The decision information includes large model output, small model output and knowledge graph output.

[0185] Specifically, refer to Figure 4 As shown, the decision fusion unit provides a decision mechanism for fusing multi-source decision information, and the set decision fusion function can be: Y(x) = argmax(πL×OL(x)+πS×OS(x)+πC×OC(x))

[0186] Among them: Y(x) is the decision fusion accuracy; OL(x) is the output of the large model, which comes from the large model of the knowledge distillation module; OS(x) is the output of the small model, which comes from the small model of the knowledge distillation module; OC(x) is the collaborative decision output, which integrates the multimodal features in the cross-modal collaborative classification enhancement module and the reasoning results of the knowledge graph enhancement module; πL, πS, πC are adaptive weights, which are dynamically adjusted by the self-evolution classification module.

[0187] The decision fusion function integrates multi-source information from the processing results of the cross-modal collaborative classification enhancement module, the knowledge distillation module, the knowledge graph enhancement module, and the confidence assessment module, providing a decision-making basis for the optimization mechanism in the classification optimization module.

[0188] The decision optimization unit is used to construct the loss function of multi-task learning based on knowledge distillation loss, graph reasoning loss and feature alignment loss, and to define the optimization objective of multi-task learning by minimizing the loss function.

[0189] Specifically, the decision optimization unit provides a decision optimization mechanism that cooperates with the knowledge distillation module, the knowledge graph enhancement module, and the classification optimization module. The loss function set can be:

[0190] min Loss=LKD+κa×LCRO+κb×LFA

[0191] Among them: LKD is the knowledge distillation loss, which comes from the knowledge distillation process of the knowledge distillation module; LCRO is the collaborative reasoning loss, which is generated by the knowledge graph reasoning of the knowledge graph enhancement module; LFA is the feature alignment loss, which is based on the feature processing result of the cross-modal collaborative classification enhancement module; Ka and Kb are loss balance coefficients, which are regulated by the optimization mechanism of the classification optimization module; the min function represents the minimization of the loss function Loss.

[0192] The optimization goal of multi-task learning is obtained by minimizing the value of the loss function Loss to achieve continuous optimization of the collaborative decision-making process.

[0193] The decision adjustment unit is used to adjust the weights according to the feedback information in the decision-making process. When there is a deviation between the decision optimization result and the actual situation, the weights are fine-tuned according to the gradient information of the loss function.

[0194] Specifically, the decision adjustment unit provides an adjustment mechanism for implementing decision feedback, and the weight update function set may be:

[0195] Where: π′ is the weight vector after decision feedback adjustment; π is the current weight vector [πL, πS, πC]; γ is the learning step size; is the gradient of the loss function Loss.

[0196] The adjustment mechanism performs feedback optimization of weights through the weight update function. When the classification decision result deviates from the actual situation, the unit will fine-tune the weights according to the gradient information of the loss function, so that the decision fusion function is more in line with the actual decision needs and ensure the adaptive adjustment of the decision process.

[0197] Therefore, the collaborative decision-making module provides the above-mentioned multiple mechanisms to achieve close collaboration with related modules, builds a complete collaborative decision-making system, and achieves a dual improvement in the accuracy and reliability of Chinese library classification decisions.

[0198] In this embodiment, it also includes: a classification optimization module, which includes: a category optimization unit, a feedback optimization loss unit and an optimization integration unit.

[0199] The category optimization unit is used to optimize the Chinese Library Classification categories by adjusting the balance coefficient based on the comprehensive evaluation score, decision optimization results and the feedforward correction items provided by the self-evolution classification module to generate category optimization results.

[0200] Specifically, the category optimization unit provides a category optimization mechanism for realizing collaboration with the confidence assessment module and the collaborative decision-making module. The set category optimization function may be: y*=argmax(Q(y)+λb×B(y)+λf×F(y))

[0201] Wherein: y* is the category optimization result; Q(y) is the comprehensive evaluation score, which is the confidence evaluation result from the confidence evaluation module; B(y) is the category balance factor, which is calculated based on the decision distribution in claim 9; F(y) is the feedforward correction term, which is provided by the feedforward correction term function of the self-evolution classification module; λb and λf are balance coefficients used to adjust the optimization intensity.

[0202] The category optimization function takes into account the comprehensive evaluation score, the category balance factor of the decision optimization result and the feedforward correction item of the self-evolution classification module in multiple dimensions, optimizes the classification categories by adjusting the balance coefficient, and achieves a dynamic balance of the Chinese Library Classification categories.

[0203] The feedback optimization loss unit constructs a feedback optimization loss function that imposes multiple constraints on the classification results based on the conditional probability of the decision optimization results, the rule satisfaction of the consistency constraint results, and the historical consistency score reflecting the past classification results, and obtains the feedback optimization loss.

[0204] Specifically, the feedback optimization loss unit provides a loss mechanism for integrating the feedback of the knowledge distillation module and the collaborative decision-making module. The feedback optimization loss function set can be:

[0205] LFB=-∑logP(y*|x)×R(y)×H(y)

[0206] Wherein: LFB is the feedback optimization loss; P(y*|x) is the conditional probability, which comes from the decision result of the collaborative decision-making module; R(y) is the rule satisfaction, which is based on the knowledge consistency constraint of the knowledge distillation module; H(y) is the historical consistency score, which reflects the temporal stability of the classification result.

[0207] Based on historical data, the system evaluates the historical stability of each classification result. The calculated historical consistency score is used as a component in the feedback optimization loss function. The feedback optimization loss function integrates knowledge constraints and historical consistency scores to impose multi-party constraints on the classification results. It uses decision results, rule satisfaction, and temporal stability information to calculate the feedback optimization loss to ensure the rationality of the optimization process and the accuracy of the classification results.

[0208] The optimization integration unit is used to perform weighted summation of the category optimization results and feedback optimization losses to obtain the final optimization result.

[0209] Specifically, the optimization integration unit provides an integration mechanism for coordinating multi-module optimization, and the set optimization integration function is: O(y) = α × y* + (1-α) × exp(-LFB)

[0210] Where: y * is the category optimization result; LFB is the feedback optimization loss; α is the integrated weight coefficient.

[0211] By optimizing the integrated function to balance the category optimization results and feedback optimization losses, the system performance can be continuously improved. It weights the category optimization results and feedback optimization losses according to the integrated weight coefficient to obtain the final optimization result, thereby coordinating the optimization goals of multiple modules and improving the overall performance of the classification system.

[0212] In this way, the classification optimization module provides the above-mentioned multiple mechanisms to achieve close collaboration with related modules, builds a complete classification optimization system, and provides reliable guarantee for the continuous improvement of the accuracy of China Library Classification results.

[0213] like Figure 5 As shown, this embodiment is tested on 100,000 Chinese library classification data, and the following experimental results are obtained:

[0214] Among them, the classification accuracy is: L1 layer: 93%; L2 layer: 89%; L3 layer: 85%; L4 layer: 81%; L5 layer: 77%.

[0215] The system performance is as follows: average response time: 150ms; memory usage: 4GB; concurrent processing capacity: 100 requests / second; system availability: 99.99%; error rate: less than 0.1%.

[0216] Improvement effect: Overall classification accuracy increased by 8.5%; error propagation rate reduced by 35%; processing speed increased by 2.3 times; labor costs reduced by 60%.

[0217] The experimental results show that through this collaborative mechanism of multi-level precise classification, the system performance of classification accuracy exceeding 82%, processing speed increased by more than 2 times, and error propagation rate reduced by 35% was finally achieved, providing solid technical support for the intelligent reasoning and knowledge evolution of the China Library Classification System. Therefore, this system can effectively improve the classification accuracy, reduce the error propagation rate, and significantly improve the processing efficiency, which has important practical value.

[0218] In summary, an embodiment of the present application provides a multimodal reasoning evolution system for knowledge data classification based on a deep model, which includes: a dynamic semantic memory network module, which is used to receive multi-source heterogeneous data including text data, image data and structural data, and use a memory network to store the multi-source heterogeneous data, and dynamically update the knowledge content in the memory network according to the correlation between the newly input multi-source heterogeneous data and the existing knowledge content in the memory network; a cross-modal collaborative classification enhancement module, which is used to extract text feature vectors, image feature vectors, structural feature vectors and reference relationship feature vectors from the multi-source heterogeneous data and perform feature alignment, and perform feature fusion on the feature vectors after feature alignment to generate a comprehensive feature vector that fuses multimodal information; a multi-level hierarchical attention classification module, which is used to use a multi-level attention mechanism to process the comprehensive feature vector to understand the overall content and contextual information of the multi-source heterogeneous data, and then make classification decisions to classify the newly input multi-source heterogeneous data into the corresponding level of the China Library Classification System. By integrating technologies such as dynamic semantic memory, cross-modal collaborative classification enhancement, and multi-level hierarchical attention classification, this application enables the system to more accurately grasp the core features of multi-source heterogeneous data, enhance the system's understanding and adaptability to each level of the China Library Classification System, and realize intelligent processing and classification decisions for complex information. This can effectively improve classification accuracy, reduce error propagation rate, significantly improve processing efficiency, and provide solid technical support for the knowledge evolution of the China Library Classification System.

[0219] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.

[0220] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0221] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily mention changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims, and the above embodiments should be regarded as exemplary and non-restrictive.

Claims

1. A multimodal reasoning evolution system for knowledge data classification based on deep models, characterized in that: include: A dynamic semantic memory network module, used to receive multi-source heterogeneous data including text data, image data and structure data, store the multi-source heterogeneous data using a memory network, and dynamically update the knowledge content in the memory network according to the relevance of the newly input multi-source heterogeneous data and the existing knowledge content in the memory network; A cross-modal collaborative classification enhancement module is used to extract text feature vectors, image feature vectors, structural feature vectors and reference relationship feature vectors from the multi-source heterogeneous data and perform feature alignment, and perform feature fusion on the feature vectors after feature alignment to generate a comprehensive feature vector that fuses multi-modal information; The multi-level hierarchical attention classification module is used to process the comprehensive feature vector using a multi-level attention mechanism to understand the overall content and contextual information of the multi-source heterogeneous data, and then make classification decisions to classify the newly input multi-source heterogeneous data into the corresponding level of the China Library Classification System.

2. The multimodal reasoning evolution system for knowledge data classification based on deep model according to claim 1 is characterized in that: The memory network includes: a short-term memory module and a long-term memory module, and the dynamic semantic memory network module includes: a memory update unit, a semantic entropy calculation unit, an entropy weight update unit and a knowledge mapping unit; The memory update unit is used to integrate the outputs of the short-term memory module, the long-term memory module and the knowledge distillation module to achieve dynamic updating and optimization of knowledge; The semantic entropy calculation unit is used to calculate the information entropy of the semantic unit and evaluate the occurrence probability of each semantic unit; The entropy weight updating unit dynamically adjusts the weight based on the semantic change amount to ensure the dynamic update of knowledge; The knowledge mapping unit is used to map the knowledge in the memory network into the space of the knowledge graph in the form of key-value pairs to achieve structured knowledge representation.

3. The knowledge data classification multimodal reasoning evolution system based on deep model according to claim 1 is characterized in that: The multi-level hierarchical attention classification module includes: an attention calculation unit and a classification decision unit; The attention calculation unit is used to respectively calculate the local semantic features and the global multimodal features of the comprehensive feature vector using the local attention weight matrix and the global attention weight matrix, and fuse the local semantic features with the global multimodal features; The classification decision unit is used to process the weighted fused comprehensive feature vector through a classification mapping function to obtain a classification decision.

4. The knowledge data classification multimodal reasoning evolution system based on deep model according to claim 1 is characterized in that: The cross-modal collaborative classification enhancement module includes: a feature extraction unit, a feature alignment unit and a feature fusion unit; The feature extraction unit is used to extract text feature vectors, image feature vectors, structural feature vectors and reference relationship feature vectors from the multi-source heterogeneous data; The feature alignment unit is used to map feature vectors of different modalities into a unified feature space; The feature fusion unit is used to perform feature fusion on the feature vectors after feature alignment to generate a comprehensive feature vector that fuses multi-modal information.

5. The knowledge data classification multimodal reasoning evolution system based on deep model according to claim 1 is characterized in that: Also includes: Bidirectional knowledge distillation module; The bidirectional knowledge distillation module includes a large model for storing long-term memory knowledge in a dynamic semantic memory network and a small model for processing short-term memory knowledge in a dynamic semantic memory network; The bidirectional knowledge distillation module provides a bidirectional knowledge migration mechanism between the large model and the small model, promotes the co-evolution between the large model and the small model through the knowledge migration results, and provides a knowledge consistency constraint mechanism in the knowledge migration process, and minimizes the differences in the knowledge migration process through the consistency constraint results; The bidirectional knowledge distillation module also provides a knowledge dynamic update adaptation mechanism to dynamically balance the knowledge migration results and the consistency constraint results.

6. The multimodal reasoning evolution system for knowledge data classification based on deep model according to claim 5 is characterized in that: Also includes: A knowledge graph enhancement module, the knowledge graph enhancement module comprising: a graph structure definition unit, a graph reasoning unit and a knowledge enhancement unit; The graph structure definition unit is used to define a knowledge graph, which includes a category node set, a relationship edge set, and a relationship type set, wherein the category node set represents a multi-level category structure of the China Library Classification System, the relationship edge set represents a semantic association of a dynamic semantic memory network, and the relationship type set represents a multimodal relationship of multi-source heterogeneous data; The graph reasoning unit is used to perform dynamic reasoning based on the knowledge graph to generate reasoning results; The knowledge enhancement unit is used to perform multi-dimensional enhancement on the nodes, relationships and types of the knowledge graph to improve the representation capability of the graph.

7. The multimodal reasoning evolution system for knowledge data classification based on deep model according to claim 6 is characterized in that: It also includes: a confidence assessment module, which includes: a comprehensive assessment unit, an uncertainty estimation unit and a decision guidance unit; The comprehensive evaluation unit is used to perform a multi-dimensional evaluation on the classification result based on the memory knowledge of the dynamic semantic memory network, the multimodal features of the multi-source heterogeneous data, and the knowledge graph to obtain a comprehensive evaluation score; The uncertainty estimation unit is used to quantify the uncertainty of the classification result based on a preset uncertainty balance factor to obtain an uncertainty estimation value; The decision guidance unit is used to generate a decision guidance value for collaborative decision-making based on the comprehensive evaluation score and the uncertainty estimate.

8. The multimodal reasoning evolution system for knowledge data classification based on deep model according to claim 7 is characterized in that: Also includes: A collaborative decision-making module, the collaborative decision-making module includes: a decision fusion unit, a decision optimization unit and a decision adjustment unit; The decision fusion unit is used to integrate decision information from different sources to generate a decision optimization result, wherein the decision information includes a large model output, a small model output, and a knowledge graph output; The decision optimization unit is used to construct a loss function of multi-task learning based on knowledge distillation loss, graph reasoning loss and feature alignment loss, so as to define an optimization target of multi-task learning by minimizing the loss function; The decision adjustment unit is used to adjust the weight according to the feedback information in the decision process. When there is a deviation between the decision optimization result and the actual situation, the weight is fine-tuned according to the gradient information of the loss function.

9. The knowledge data classification multimodal reasoning evolution system based on deep model according to claim 8 is characterized in that: Also includes: A self-evolution classification module, the self-evolution classification module comprising: a parameter updating unit, a mutation probability calculation unit and a multi-module optimization unit; The parameter updating unit is used to perform feedforward correction update on the current parameters in the knowledge distillation result based on gradient optimization and mutation mechanism, and update the memory network based on the parameter update result; The mutation probability calculation unit is used to dynamically calculate the mutation probability, and continuously adjust the mutation intensity during the adaptive evolution process to find the most suitable optimization strategy to obtain the mutation optimization result; The multi-module optimization unit is used to balance the parameter update results and the mutation optimization results to achieve collaborative optimization among multiple modules.

10. The multimodal reasoning evolution system for knowledge data classification based on deep model according to claim 9, characterized in that: It also includes: a classification optimization module, which includes: a category optimization unit, a feedback optimization loss unit and an optimization integration unit; The category optimization unit is used to optimize the Chinese Library Classification categories by adjusting the balance coefficient based on the comprehensive evaluation score, the decision optimization result and the feedforward correction item provided by the self-evolution classification module, and generate a category optimization result; The feedback optimization loss unit constructs a feedback optimization loss function that imposes multiple constraints on the classification results based on the conditional probability of the decision optimization result, the rule satisfaction of the consistency constraint result, and the historical consistency score reflecting the past classification results, and obtains the feedback optimization loss; The optimization integration unit is used to perform weighted summation on the category optimization result and the feedback optimization loss to obtain a final optimization result.

Images