Archive information extraction management method and system based on multi-modal learning

Abstract

The invention relates to the technical field of archive information management, and provides an archive information extraction management method and system based on multi-modal learning, and the method comprises the steps: collecting multi-modal archive data, carrying out the preprocessing of the multi-modal archive data, and obtaining the processed multi-modal archive data; performing feature extraction on the multi-modal archive data, and obtaining an archive comprehensive feature vector through weight distribution and feature alignment; based on a preset archive knowledge graph, performing multi-modal entity recognition on the archive comprehensive feature vector, constructing an archive relation graph, and performing iterative calculation through a graph neural network to obtain an archive association rule; updating a preset archive knowledge graph according to the archive association rule to obtain an archive enhanced knowledge graph, and performing iterative optimization on the archive association rule to form and store an archive management strategy network; and constructing an archive information index system based on the archive enhanced knowledge graph and the archive management strategy network. According to the invention, efficient management and accurate retrieval of archive information are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of archive information management, and in particular to an archive information extraction management method and system based on multimodal learning. Background Art

[0002] With the rapid development of information technology, archive management is facing increasingly complex challenges. Modern archives include not only traditional text documents, but also data in various forms such as images, audio, and video. The amount of these multimodal archive data has shown explosive growth. The explosive growth of data volume has made current archive management extremely prone to omissions and errors.

[0003] The existing technology manages archives by establishing a simple database system and adopts basic methods such as keyword matching to retrieve and classify archives. However, this method cannot effectively process and integrate multimodal archive data, and each type of data is often processed separately, resulting in low information relevance. Due to the lack of intelligent feature extraction capabilities, it is difficult to deeply explore the correlation between archive contents. The archive management strategy is fixed and cannot be dynamically optimized and adjusted according to the actual application effect, resulting in low efficiency of archive management and difficulty in meeting the growing demand for intelligent management. Summary of the invention

[0004] In view of this, the present invention proposes to realize intelligent management and efficient retrieval of archival information through multimodal feature extraction, knowledge graph construction and association rule learning, which solves the problem that the existing technology cannot effectively process and integrate multimodal archival data, resulting in low efficiency of archival management.

[0005] The technical solution of the present invention is implemented as follows: In the first aspect, the present invention provides a method for extracting and managing archive information based on multimodal learning, comprising the following steps:

[0006] Collecting multimodal archival data, and preprocessing the multimodal archival data to obtain processed text data, image data, audio data, and video data;

[0007] Feature extraction is performed on text data, image data, audio data and video data respectively, and after weight allocation and feature alignment, a comprehensive feature vector of the archive is obtained;

[0008] Based on the preset archive knowledge graph, multimodal entity recognition is performed on the archive comprehensive feature vector, and an archive relationship graph is constructed. The archive association rules are obtained by iterative calculation through the graph neural network;

[0009] According to the archive association rules, the preset archive knowledge graph is updated to obtain the archive enhanced knowledge graph, and the archive association rules are iteratively optimized to form and store the archive management strategy network;

[0010] An archive information index system is constructed based on the archive enhanced knowledge graph and the archive management strategy network.

[0011] On the basis of the above technical solution, preferably, the collecting of multimodal archival data, preprocessing of the multimodal archival data to obtain processed text data, image data, audio data and video data specifically includes:

[0012] A multimodal data acquisition interface is established, and multimodal archival data is collected through the multimodal data acquisition interface. The multimodal archival data is organized in a time-series manner, processed in a unified format, and subjected to quality assessment and filtering to obtain processed text data, image data, audio data, and video data. The multimodal educational archival data includes text data, image data, audio data, and video data; the text data includes structured documents and unstructured text, the image data includes scanned images and digital photos, the audio data includes voice records and sound signals, and the video data includes dynamic image records.

[0013] On the basis of the above technical solution, preferably, the feature extraction of text data, image data, audio data and video data is performed respectively, and after weight distribution and feature alignment, a comprehensive feature vector of the archive is obtained, which specifically includes:

[0014] Feature extraction is performed on text data, image data, audio data and video data respectively to obtain archive feature vectors representing archive content, and an attention mechanism is used to dynamically allocate weights between modalities for the archive feature vectors to obtain weight-optimized modal feature vectors. Each modal feature of the weight-optimized modal feature vector is mapped to a unified feature space through cross-modal feature alignment to obtain an archive comprehensive feature vector.

[0015] The weights are assigned according to the importance of each modal feature vector. The calculation formula for dynamic weight assignment is:

[0016]

[0017] Among them, W i is the weight coefficient of the i-th mode, is the eigenvector of the ith mode, is the feature vector of the jth mode, α1 is the adjustable attention coefficient, exp(·) is the exponential function, is the query vector, N1 is the number of modalities;

[0018] The calculation formula for the cross-modal feature alignment is:

[0019]

[0020] in, is the i-th modal eigenvector after mapping, M i is the transformation matrix of the i-th mode, tanh(·) is the hyperbolic tangent activation function, is the bias vector;

[0021] The calculation formula of the comprehensive feature vector of the archive is:

[0022]

[0023] in, is the comprehensive feature vector of the archive, W i is the weight coefficient of the i-th mode, is the i-th modal eigenvector after mapping.

[0024] On the basis of the above technical solution, preferably, based on the preset archive knowledge graph, multimodal entity recognition is performed on the archive comprehensive feature vector, and an archive relationship graph is constructed, and iterative calculation is performed through a graph neural network to obtain archive association rules, which specifically include:

[0025] The multimodal entity recognition sub-step performs multimodal entity recognition on the archive comprehensive feature vector based on a preset archive knowledge graph to obtain an archive entity feature set;

[0026] The archive relationship graph construction and feature extraction sub-step constructs an archive relationship graph based on the archive entity feature set, uses the entity features of the archive relationship graph as graph nodes, and the associations between entities as graph edges, iterates the archive relationship graph through a graph neural network to obtain archive regular features, matches and analyzes the archive regular features with a preset archive knowledge graph, and generates archive association rules.

[0027] On the basis of the above technical solution, preferably, the multimodal entity recognition sub-step specifically includes:

[0028] Use natural language processing algorithms to perform named entity recognition on text data, use image recognition algorithms to detect and classify objects in image data, use audio analysis algorithms to extract keywords from audio data, and combine video analysis algorithms to identify activities and events in video data, generate multimodal entity feature vectors, match the multimodal entity feature vectors based on a preset archive knowledge graph, use similarity calculation methods to determine the consistency of entities, and integrate the matching entities into a unified archive entity feature set;

[0029] The calculation formula of the similarity calculation method is:

[0030]

[0031] Among them, S(e a,e b ) is entity e a With entity e b The similarity of and Entity e a With entity e b The eigenvector of

[0032] The archive relationship graph construction and feature extraction sub-steps specifically include:

[0033] According to the entity attributes and associations in the archival entity feature set, a graph construction algorithm is used to establish an archival relationship graph, the attributes of the graph nodes and graph edges are determined, and the archival relationship graph is embedded in nodes and graphs using a graph neural network. Content features, association features, and time series features between nodes are extracted through multi-layer iterative calculations to form archival regular features, and the archival association rules are generated by comparing them with the preset archival knowledge graph through matching analysis;

[0034] The calculation formula for the node embedding is:

[0035]

[0036] in, is the embedding vector of node v at layer k, is the embedding vector of node v at the k-1th layer, N(v) is the set of neighbor nodes of node v, W is the trainable weight matrix, is the embedding vector of node u in the k-1 layer, σ(·) is the activation function, and node u is the neighbor node of node v;

[0037] The calculation formula of the archive regularity characteristics is:

[0038]

[0039] Among them, F 规律 For the regular characteristics of the archives, is the embedding vector of node v at the Kth level, V is the set of all nodes in the archive relationship graph, and concat(·) is the vector concatenation function;

[0040] The calculation formula of the file association rule is:

[0041] R={(F 规律 ,U)|similarity(F 规律 ,K)>θ};

[0042] Among them, R is the generated archive association rule set, U is the knowledge block in the preset archive knowledge graph, similarity(·) is the similarity calculation function, and θ is the similarity threshold.

[0043] On the basis of the above technical solution, preferably, the updating of the preset archive knowledge graph according to the archive association rules to obtain the archive enhanced knowledge graph, iteratively optimizing the archive association rules to form and store the archive management strategy network specifically includes:

[0044] Update the preset archive knowledge graph according to the archive association rules to obtain the archive enhanced knowledge graph;

[0045] Merge the new entities and relations in the archive association rules with the preset archive knowledge graph to obtain the archive enhanced knowledge graph;

[0046] Based on historical application effect feedback, the archive association rules are iteratively optimized to form an archive management strategy network, and the archive enhanced knowledge graph and the archive management strategy network are stored;

[0047] Adjust the weights and parameters of archive association rules based on historical feedback through machine learning algorithms;

[0048] Construct archive management strategy network through archive association rules.

[0049] On the basis of the above technical solution, preferably, the archival information index system is constructed based on the archival enhanced knowledge graph and the archival management strategy network, specifically including:

[0050] A multi-dimensional archive retrieval catalog is constructed based on the entity relationships in the archive enhanced knowledge graph, archive access rights and archive use policies are set based on the archive management policy network, an archive information rapid retrieval mechanism is established, and the extracted archive information is classified, organized and dynamically updated.

[0051] In a second aspect, the present invention further provides an archive information extraction and management system based on multimodal learning, the system comprising:

[0052] A data processing module, used for collecting multimodal archival data, preprocessing the multimodal archival data, and obtaining processed text data, image data, audio data, and video data;

[0053] The feature extraction module is used to extract features from text data, image data, audio data and video data respectively, and obtain a comprehensive feature vector of the archive through weight distribution and feature alignment;

[0054] The association analysis module is used to perform multimodal entity recognition on the archive comprehensive feature vector based on the preset archive knowledge graph, and to construct an archive relationship graph, and to perform iterative calculations through a graph neural network to obtain archive association rules;

[0055] A knowledge updating module is used to update a preset archive knowledge graph according to the archive association rules to obtain an archive enhanced knowledge graph, iteratively optimize the archive association rules, form an archive management strategy network and store it;

[0056] The archive management module is used to build an archive information index system based on the archive enhanced knowledge graph and the archive management strategy network.

[0057] In a third aspect, the present invention further provides an electronic device, comprising: at least one processor, at least one memory, a communication interface and a bus;

[0058] Among them, the processor, memory, and communication interface communicate with each other through the bus, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement steps such as a method for archival information extraction and management based on multimodal learning.

[0059] In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions enable a computer to implement steps of a method for extracting and managing archive information based on multimodal learning.

[0060] The archival information extraction and management method and system based on multimodal learning of the present invention have the following beneficial effects compared with the prior art:

[0061] (1) Through unified preprocessing, feature extraction and alignment of multimodal archival data, combined with knowledge graph and graph neural network, intelligent recognition and association analysis of archival entities are carried out. By dynamically updating the knowledge graph and optimizing the management strategy, a complete archival information index system is built, realizing efficient management and accurate retrieval of archival information.

[0062] (2) By extracting features from multimodal data such as text data, image data, audio data, and video data, the attention mechanism is used to dynamically assign weights to the feature vectors of each modality, and the optimized modal features are mapped to a unified feature space through cross-modal feature alignment to generate a comprehensive archival feature vector, thus achieving deep fusion and unified expression of multimodal data and improving the accuracy of archival information feature extraction and the reliability of association analysis.

[0063] (3) By constructing an archival relationship graph based on the archival entity feature set and using a graph neural network to perform multi-layer iterative calculations on the relationship graph, the content features, association features, and time series features between nodes are extracted to generate archival regularity features. These features are then matched and analyzed with the preset archival knowledge graph to generate archival association rules. This enables in-depth mining and rule extraction of complex relationships between archival entities, and improves the association analysis capabilities of archival information and the efficiency of knowledge discovery. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 A flow chart of an archive information extraction and management method based on multimodal learning of the present invention;

[0066] Figure 2 This is a structural diagram of an archive information extraction and management system based on multimodal learning according to the present invention. DETAILED DESCRIPTION

[0067] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] See also Figure 1 The present invention provides a method for extracting and managing archive information based on multimodal learning, comprising the following steps:

[0069] Collecting multimodal archival data, and preprocessing the multimodal archival data to obtain processed text data, image data, audio data, and video data;

[0070] Feature extraction is performed on text data, image data, audio data and video data respectively, and after weight allocation and feature alignment, a comprehensive feature vector of the archive is obtained;

[0071] Based on the preset archive knowledge graph, multimodal entity recognition is performed on the archive comprehensive feature vector, and an archive relationship graph is constructed. The archive association rules are obtained by iterative calculation through the graph neural network;

[0072] According to the archive association rules, the preset archive knowledge graph is updated to obtain the archive enhanced knowledge graph, and the archive association rules are iteratively optimized to form and store the archive management strategy network;

[0073] An archive information index system is constructed based on the archive enhanced knowledge graph and the archive management strategy network.

[0074] Specifically, this embodiment performs unified preprocessing, feature extraction and alignment of multimodal archival data, combines knowledge graphs and graph neural networks to perform intelligent identification and association analysis of archival entities, and builds a complete archival information index system by dynamically updating the knowledge graph and optimizing management strategies, thereby achieving efficient management and accurate retrieval of archival information.

[0075] The collecting of multimodal archival data and preprocessing of the multimodal archival data to obtain processed text data, image data, audio data and video data specifically includes:

[0076] Establish a multimodal data acquisition interface, collect multimodal archival data through the multimodal data acquisition interface, organize the multimodal archival data in a time-series manner, perform format unification processing and quality assessment filtering on the multimodal archival data, and obtain processed text data, image data, audio data and video data, wherein the multimodal educational archival data includes text data, image data, audio data and video data; the text data includes structured documents and unstructured texts, the image data includes scanned images and digital photos, the audio data includes voice records and sound signals, and the video data includes dynamic image records;

[0077] The temporal organization and format unification processing includes word segmentation and semantic annotation of text data, size standardization and feature enhancement of image data, noise reduction and signal enhancement of audio data, frame extraction and scene segmentation of video data, and generation of standardized multimodal archive data;

[0078] The standardized multimodal archival data is subjected to quality assessment and data filtering, including calculating the quality score of each modality data, screening the data based on a preset quality threshold, eliminating data with unqualified quality, and obtaining processed multimodal archival data.

[0079] Specifically, this embodiment establishes a multimodal data acquisition interface, which can simultaneously collect various types of archival data such as text, images, audio and video, and organize them in a time-series and unify their formats to ensure the structural consistency and time correlation of data in different modalities.

[0080] By performing operations such as word segmentation and semantic annotation on text data, size standardization and feature enhancement on image data, noise reduction and signal enhancement on audio data, frame extraction and scene segmentation on video data, the quality of data in each modality can be significantly improved and the interference of noise and redundant information can be reduced.

[0081] Through quality assessment and data filtering mechanisms, the quality score of each modality data is calculated, and unqualified data is eliminated based on the preset quality threshold to ensure that the data input into the system has high reliability and accuracy.

[0082] Through standardized processing and quality screening of data of different modalities, multimodal data can be mapped to a unified standardized data format, solving the problem of differences in structure and quality of multimodal data.

[0083] The feature extraction of text data, image data, audio data and video data is performed respectively, and after weight distribution and feature alignment, a comprehensive feature vector of the archive is obtained, which specifically includes:

[0084] Feature extraction is performed on text data, image data, audio data and video data respectively to obtain archive feature vectors representing archive content, and an attention mechanism is used to dynamically allocate weights between modalities for the archive feature vectors to highlight key information features, thereby obtaining weight-optimized modal feature vectors. Each modal feature of the weight-optimized modal feature vector is mapped to a unified feature space through cross-modal feature alignment to obtain an archive comprehensive feature vector.

[0085] A feature extraction sub-step, performing feature extraction on the text data, image data, audio data and video data respectively to obtain a preliminary feature vector of each modality;

[0086] Text feature extraction: Use natural language processing methods to perform word segmentation, part-of-speech tagging, and semantic analysis on structured documents and unstructured texts to generate text feature vectors;

[0087] Image feature extraction, using convolutional neural networks to perform size standardization, edge detection and color analysis on scanned images and digital photos to generate image feature vectors;

[0088] Audio feature extraction: Use acoustic feature extraction algorithms to perform spectrum analysis, noise reduction, and pitch detection on voice recordings and sound signals to generate audio feature vectors;

[0089] Video feature extraction: through video frame extraction and scene segmentation algorithms, motion detection, scene classification and object recognition are performed on dynamic image records to generate video feature vectors;

[0090] In the feature optimization substep, the attention mechanism is used to dynamically allocate weights between modalities for the preliminary feature vector, highlight key information features, obtain the weight-optimized modal feature vector, and map each modal feature to a unified feature space through cross-modal feature alignment to generate a comprehensive archive feature vector;

[0091] The weights are assigned according to the importance of each modal feature vector. The calculation formula for dynamic weight assignment is:

[0092]

[0093] Among them, W i is the weight coefficient of the i-th mode, is the eigenvector of the ith mode, is the feature vector of the jth mode, α1 is the adjustable attention coefficient, exp(·) is the exponential function, is the query vector, N1 is the number of modalities;

[0094] The calculation formula for the cross-modal feature alignment is:

[0095]

[0096] in, is the i-th modal eigenvector after mapping, M i is the transformation matrix of the i-th mode, tanh(·) is the hyperbolic tangent activation function, is the bias vector;

[0097] The calculation formula of the comprehensive feature vector of the archive is:

[0098]

[0099] in, is the comprehensive feature vector of the archive, W i is the weight coefficient of the i-th mode, is the i-th modal eigenvector after mapping.

[0100] Specifically, this embodiment realizes deep feature extraction of different modal data through natural language processing, convolutional neural network, acoustic feature extraction algorithm, video analysis algorithm, etc., ensuring the integrity and accuracy of feature expression of various archival data. The attention mechanism is used for dynamic weight allocation between modalities, which can adaptively adjust the weight according to the importance of different modal features, highlight key information features, suppress the interference of irrelevant information, and improve the effectiveness of feature expression.

[0101] Through the cross-modal feature alignment algorithm, the feature vectors of different modes are mapped to a unified feature space, which solves the heterogeneity problem of multimodal features, realizes the effective fusion of features, and obtains a more comprehensive and accurate expression of archive comprehensive features. A complete processing flow from feature extraction to feature optimization is established, which improves the overall processing efficiency.

[0102] Based on the preset archive knowledge graph, multimodal entity recognition is performed on the archive comprehensive feature vector, and an archive relationship graph is constructed. The archive association rules are obtained by iterative calculation through the graph neural network, which specifically includes:

[0103] The multimodal entity recognition sub-step performs multimodal entity recognition on the archive comprehensive feature vector based on a preset archive knowledge graph to obtain an archive entity feature set;

[0104] The archive relationship graph construction and feature extraction sub-step constructs an archive relationship graph based on the archive entity feature set, uses the entity features of the archive relationship graph as graph nodes, and the associations between entities as graph edges, iterates the archive relationship graph through a graph neural network to obtain archive regular features, matches and analyzes the archive regular features with a preset archive knowledge graph, and generates archive association rules.

[0105] Specifically, this embodiment can accurately identify and extract entity information in the archive comprehensive feature vector through multimodal entity recognition based on the preset archive knowledge graph, forming a complete set of archive entity features. By taking the archive entity features as graph nodes and the associations between entities as graph edges, a complete archive relationship graph is constructed, which realizes the systematic expression of the relationships between archive entities and fully demonstrates the relevance of archive information.

[0106] Using graph neural networks to iteratively calculate the archival relationship graph can deeply explore the potential associations between archival entities and extract deeper archival regularity features. By matching and analyzing the archival regularity features with the preset archival knowledge graph, the generated archival association rules have strong interpretability and reliability.

[0107] The multimodal entity recognition sub-step specifically includes:

[0108] Use natural language processing algorithms to perform named entity recognition on text data, use image recognition algorithms to detect and classify objects in image data, use audio analysis algorithms to extract keywords from audio data, and combine video analysis algorithms to identify activities and events in video data, generate multimodal entity feature vectors, match the multimodal entity feature vectors based on a preset archive knowledge graph, use similarity calculation methods to determine the consistency of entities, and integrate the matching entities into a unified archive entity feature set;

[0109] The calculation formula of the similarity calculation method is:

[0110]

[0111] Among them, S(e a ,e b ) is entity e a With entity e b The similarity of and Entity e a With entity e b The eigenvector of

[0112] The archive relationship graph construction and feature extraction sub-steps specifically include:

[0113] According to the entity attributes and associations in the archival entity feature set, a graph construction algorithm is used to establish an archival relationship graph, the attributes of the graph nodes and graph edges are determined, and the archival relationship graph is embedded in nodes and graphs using a graph neural network. Content features, association features, and time series features between nodes are extracted through multi-layer iterative calculations to form archival regular features, and the archival association rules are generated by comparing them with the preset archival knowledge graph through matching analysis;

[0114] The calculation formula for the node embedding is:

[0115]

[0116] in, is the embedding vector of node v at layer k, is the embedding vector of node v at the k-1th layer, N(v) is the set of neighbor nodes of node v, W is the trainable weight matrix, is the embedding vector of node u in the k-1 layer, σ(·) is the activation function, and node u is the neighbor node of node v;

[0117] The calculation formula of the archive regularity characteristics is:

[0118]

[0119] Among them, F 规律 For the regular characteristics of the archives, is the embedding vector of node v at the Kth level, V is the set of all nodes in the archive relationship graph, and concat(·) is the vector concatenation function;

[0120] The calculation formula of the file association rule is:

[0121] R={(F 规律 ,U)similarity(F 规律 ,K)>θ};

[0122] Among them, R is the generated archive association rule set, U is the knowledge block in the preset archive knowledge graph, similarity(·) is the similarity calculation function, and θ is the similarity threshold.

[0123] Specifically, this embodiment uses a natural language processing algorithm to perform named entity recognition on text data, an image recognition algorithm to detect and classify objects in image data, an audio analysis algorithm to extract keywords from audio data, and a video analysis algorithm to identify activities and events in video data, which can fully cover the entity information in multimodal archival data. Combined with the preset archival knowledge graph, multimodal entity feature vectors are matched through a similarity calculation method to ensure the consistency of entity recognition and generate a unified archival entity feature set.

[0124] The method of updating the preset archive knowledge graph according to the archive association rules to obtain the archive enhanced knowledge graph, iteratively optimizing the archive association rules, forming and storing the archive management strategy network, specifically includes:

[0125] Update the preset archive knowledge graph according to the archive association rules to obtain the archive enhanced knowledge graph;

[0126] The new entities and relationships in the archive association rules are merged with the preset archive knowledge graph to obtain the archive enhanced knowledge graph:

[0127]

[0128]

[0129] in, Enhance the knowledge graph for archives, is the preset archive knowledge graph, ΔR is the relationship set based on archive association rules, e c and e d is a new entity identified by archive association rules, r is e c and e d the relationship between;

[0130] Use consistency maintenance algorithms to verify and correct the updated archive knowledge graph to ensure the integrity and accuracy of the knowledge graph;

[0131] The calculation formula for consistency verification and correction is:

[0132]

[0133] Among them, C is the consistency result. If it is inconsistent, it is corrected by the following formula:

[0134]

[0135] in, Enhance the knowledge graph for the corrected archives, Enhancing knowledge graphs for archives, R 冲突 is the set of conflicting relations detected, confidence(r1) is the credibility score of relation r1, θ 阈值 is the credibility threshold;

[0136] Based on historical application effect feedback, the archive association rules are iteratively optimized to form an archive management strategy network, and the archive enhanced knowledge graph and the archive management strategy network are stored for archive quality assessment and resource optimization configuration;

[0137] The weights and parameters of archive association rules are adjusted based on historical feedback through machine learning algorithms:

[0138]

[0139]

[0140] Among them, G 新 is the optimized archive association rule, G 旧 is the original archive association rule, η is the learning rate, ▽L is the gradient of the loss function with respect to the archive association rule;

[0141] The archive management strategy network is constructed through the optimized archive association rules, and the calculation formula is:

[0142] SN=f(G 新 );

[0143] Among them, SN is the archive management strategy network, f(G 新 ) is a rule-based policy generation function.

[0144] Specifically, this embodiment generates an archival enhanced knowledge graph by updating the preset archival knowledge graph based on archival association rules, which can dynamically integrate newly identified entities and relationships into the knowledge graph, ensuring that the knowledge graph can reflect the latest archival information and association relationships in real time, thereby improving the scalability and practicality of the knowledge graph.

[0145] The consistency maintenance algorithm is used to verify and correct the updated archive knowledge graph, which can effectively detect and resolve conflicts or redundant information that may exist in the knowledge graph. The logical consistency and data accuracy of the knowledge graph are ensured by scoring and correcting the credibility of the conflicting relationship set.

[0146] Through feedback analysis of historical application effects, the weights and parameters of archive association rules are dynamically adjusted using machine learning algorithms, which can continuously optimize the applicability and accuracy of the rules. Through the feedback-based iterative optimization mechanism, management strategies can be adaptively adjusted, which improves the intelligent level of archive management. The archive management strategy network is constructed through the optimized archive association rules, which closely combines the knowledge graph with the management strategy, providing efficient decision support for archive quality assessment, resource optimization allocation and intelligent management.

[0147] The archival information index system is constructed based on the archival enhanced knowledge graph and the archival management strategy network, specifically including:

[0148] A multi-dimensional archive retrieval catalog is constructed based on the entity relationships in the archive enhanced knowledge graph, archive access rights and archive use policies are set based on the archive management policy network, and an archive information rapid retrieval mechanism is established to support multi-dimensional information extraction based on content type, associated features, and temporal relationships; the extracted archive information is classified, organized, and dynamically updated to realize intelligent management of archive information.

[0149] Specifically, this embodiment constructs a retrieval catalog covering multiple dimensions such as content type, associated features, and temporal relationships based on the entity relationships in the archive enhanced knowledge graph, realizes multi-level organization and systematic management of archive information, and improves the comprehensiveness of archive retrieval.

[0150] This embodiment sets archive access rights and usage policies based on the archive management strategy network, realizes refined control and security management of archive access, and ensures the standardization of archive information use.

[0151] This embodiment establishes a fast retrieval mechanism, supports multi-dimensional information extraction, and performs precise retrieval based on multiple dimensions such as content type, associated features, and temporal relationships, thereby improving the efficiency of archive retrieval. It supports information extraction based on multiple dimensions such as content type, associated features, and temporal relationships, realizes multi-angle analysis and utilization of archive information, and meets the archive information needs in different scenarios.

[0152] This embodiment ensures the timeliness of the archival information and achieves continuous optimization of archival information management by classifying and dynamically updating the extracted archival information.

[0153] See also Figure 2 The present invention also provides an archive information extraction and management system based on multimodal learning, the system comprising:

[0154] A data processing module, used for collecting multimodal archival data, preprocessing the multimodal archival data, and obtaining processed text data, image data, audio data, and video data;

[0155] The feature extraction module is used to extract features from text data, image data, audio data and video data respectively, and obtain a comprehensive feature vector of the archive through weight distribution and feature alignment;

[0156] The association analysis module is used to perform multimodal entity recognition on the archive comprehensive feature vector based on the preset archive knowledge graph, and to construct an archive relationship graph, and to perform iterative calculations through a graph neural network to obtain archive association rules;

[0157] A knowledge updating module is used to update a preset archive knowledge graph according to the archive association rules to obtain an archive enhanced knowledge graph, iteratively optimize the archive association rules, form an archive management strategy network and store it;

[0158] The archive management module is used to build an archive information index system based on the archive enhanced knowledge graph and the archive management strategy network.

[0159] Specifically, the archival information extraction and management system based on multimodal learning in this embodiment decomposes the archival information processing task into five functional modules: data processing, feature extraction, association analysis, knowledge update, and archival management. Each module has a clear division of labor and cooperates with each other. The data processing module realizes the standardized processing of multimodal data, the feature extraction module extracts high-quality features, the association analysis module performs deep feature learning based on graph neural networks, the knowledge update module supports the dynamic optimization of knowledge graphs, and the archival management module provides intelligent retrieval management functions, which improves the automation and intelligence level of archival information processing and ensures the reliability of system operation.

[0160] The present invention also discloses an electronic device, comprising: at least one processor, at least one memory communication interface and a bus: wherein the processor, memory and communication interface communicate with each other through the bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement a distributed photovoltaic power station drone inspection method.

[0161] The present invention also discloses a computer-readable storage medium, which stores computer instructions, and the computer instructions enable the computer to implement all or part of the steps of a distributed photovoltaic power station drone inspection method described in an embodiment of the present invention. The storage medium includes: a U disk, a mobile hard disk, a read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk, and other media that can store program codes.

[0162] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for extracting and managing archive information based on multimodal learning, characterized in that: The following steps are involved: Collecting multimodal archival data, and preprocessing the multimodal archival data to obtain processed text data, image data, audio data, and video data; Feature extraction is performed on text data, image data, audio data and video data respectively, and after weight allocation and feature alignment, a comprehensive feature vector of the archive is obtained; Based on the preset archive knowledge graph, multimodal entity recognition is performed on the archive comprehensive feature vector, and an archive relationship graph is constructed. The archive association rules are obtained by iterative calculation through the graph neural network; According to the archive association rules, the preset archive knowledge graph is updated to obtain the archive enhanced knowledge graph, and the archive association rules are iteratively optimized to form and store the archive management strategy network; An archive information index system is constructed based on the archive enhanced knowledge graph and the archive management strategy network.

2. The method for extracting and managing archive information based on multimodal learning as claimed in claim 1, characterized in that: The collecting of multimodal archival data and preprocessing of the multimodal archival data to obtain processed text data, image data, audio data and video data specifically includes: A multimodal data acquisition interface is established, and multimodal archival data is collected through the multimodal data acquisition interface. The multimodal archival data is organized in a time-series manner, processed in a unified format, and subjected to quality assessment and filtering to obtain processed text data, image data, audio data, and video data. The multimodal educational archival data includes text data, image data, audio data, and video data; the text data includes structured documents and unstructured text, the image data includes scanned images and digital photos, the audio data includes voice records and sound signals, and the video data includes dynamic image records.

3. The method for extracting and managing archive information based on multimodal learning as claimed in claim 1, characterized in that: The feature extraction of text data, image data, audio data and video data is performed respectively, and after weight distribution and feature alignment, a comprehensive feature vector of the archive is obtained, which specifically includes: Feature extraction is performed on text data, image data, audio data and video data respectively to obtain archive feature vectors representing archive content, and an attention mechanism is used to dynamically allocate weights between modalities for the archive feature vectors to obtain weight-optimized modal feature vectors. Each modal feature of the weight-optimized modal feature vector is mapped to a unified feature space through cross-modal feature alignment to obtain an archive comprehensive feature vector. The weights are assigned according to the importance of each modal feature vector. The calculation formula for dynamic weight assignment is: Among them, W i is the weight coefficient of the i-th mode, is the eigenvector of the ith mode, is the feature vector of the jth mode, α1 is the adjustable attention coefficient, exp(·) is the exponential function, is the query vector, N1 is the number of modes; The calculation formula for the cross-modal feature alignment is: in, is the i-th modal eigenvector after mapping, M i is the transformation matrix of the i-th mode, tanh(·) is the hyperbolic tangent activation function, is the bias vector; The calculation formula of the comprehensive feature vector of the archive is: in, is the comprehensive feature vector of the archive, W i is the weight coefficient of the i-th mode, is the i-th modal eigenvector after mapping.

4. The method for extracting and managing archive information based on multimodal learning as claimed in claim 1, characterized in that: Based on the preset archive knowledge graph, multimodal entity recognition is performed on the archive comprehensive feature vector, and an archive relationship graph is constructed. The archive association rules are obtained by iterative calculation through the graph neural network, which specifically includes: The multimodal entity recognition sub-step performs multimodal entity recognition on the archive comprehensive feature vector based on a preset archive knowledge graph to obtain an archive entity feature set; The archive relationship graph construction and feature extraction sub-step constructs an archive relationship graph based on the archive entity feature set, uses the entity features of the archive relationship graph as graph nodes, and the associations between entities as graph edges, iterates the archive relationship graph through a graph neural network to obtain archive regular features, matches and analyzes the archive regular features with a preset archive knowledge graph, and generates archive association rules.

5. The method for extracting and managing archive information based on multimodal learning as claimed in claim 4, characterized in that: The multimodal entity recognition sub-step specifically includes: Use natural language processing algorithms to perform named entity recognition on text data, use image recognition algorithms to detect and classify objects in image data, use audio analysis algorithms to extract keywords from audio data, and combine video analysis algorithms to identify activities and events in video data, generate multimodal entity feature vectors, match the multimodal entity feature vectors based on a preset archive knowledge graph, use similarity calculation methods to determine the consistency of entities, and integrate the matching entities into a unified archive entity feature set; The calculation formula of the similarity calculation method is: Among them, S(e a ,e b ) is entity e a With entity e b The similarity of and Entity e a With entity e b The eigenvector of The archive relationship graph construction and feature extraction sub-steps specifically include: According to the entity attributes and associations in the archival entity feature set, a graph construction algorithm is used to establish an archival relationship graph, the attributes of the graph nodes and graph edges are determined, and the archival relationship graph is embedded in nodes and graphs using a graph neural network. Content features, association features, and time series features between nodes are extracted through multi-layer iterative calculations to form archival regular features, and the archival association rules are generated by comparing them with the preset archival knowledge graph through matching analysis; The calculation formula for the node embedding is: in, is the embedding vector of node v at layer k, is the embedding vector of node v at the k-1th layer, N(v) is the set of neighbor nodes of node v, W is the trainable weight matrix, is the embedding vector of node u in the k-1 layer, σ(·) is the activation function, and node u is the neighbor node of node v; The calculation formula of the archive regularity characteristics is: Among them, F 规律 For the regular characteristics of the archives, is the embedding vector of node v at the Kth level, V is the set of all nodes in the archive relationship graph, and concat(·) is the vector concatenation function; The calculation formula of the file association rule is: R={(F 规律 ,U)similarity(F 规律 ,K)>θ}; Among them, R is the generated archive association rule set, U is the knowledge block in the preset archive knowledge graph, similarity(·) is the similarity calculation function, and θ is the similarity threshold.

6. The method for extracting and managing archive information based on multimodal learning as claimed in claim 1, characterized in that: The method of updating the preset archive knowledge graph according to the archive association rules to obtain the archive enhanced knowledge graph, iteratively optimizing the archive association rules, forming and storing the archive management strategy network, specifically includes: Update the preset archive knowledge graph according to the archive association rules to obtain the archive enhanced knowledge graph; Merge the new entities and relations in the archive association rules with the preset archive knowledge graph to obtain the archive enhanced knowledge graph; Based on historical application effect feedback, the archive association rules are iteratively optimized to form an archive management strategy network, and the archive enhanced knowledge graph and the archive management strategy network are stored; Adjust the weights and parameters of archive association rules based on historical feedback through machine learning algorithms; Construct archive management strategy network through archive association rules.

7. The method for extracting and managing archive information based on multimodal learning as claimed in claim 1, characterized in that: The archival information index system is constructed based on the archival enhanced knowledge graph and the archival management strategy network, specifically including: A multi-dimensional archive retrieval catalog is constructed based on the entity relationships in the archive enhanced knowledge graph, archive access rights and archive use policies are set based on the archive management policy network, an archive information rapid retrieval mechanism is established, and the extracted archive information is classified, organized and dynamically updated.

8. An archive information extraction and management system based on multimodal learning, characterized in that: The system comprises: A data processing module, used for collecting multimodal archival data, preprocessing the multimodal archival data, and obtaining processed text data, image data, audio data, and video data; The feature extraction module is used to extract features from text data, image data, audio data and video data respectively, and obtain a comprehensive feature vector of the archive through weight distribution and feature alignment; The association analysis module is used to perform multimodal entity recognition on the archive comprehensive feature vector based on the preset archive knowledge graph, and to construct an archive relationship graph, and to perform iterative calculations through a graph neural network to obtain archive association rules; A knowledge updating module is used to update a preset archive knowledge graph according to the archive association rules to obtain an archive enhanced knowledge graph, iteratively optimize the archive association rules, form an archive management strategy network and store it; The archive management module is used to build an archive information index system based on the archive enhanced knowledge graph and the archive management strategy network.

9. An electronic device, characterized in that: include: at least one processor, at least one memory, a communication interface, and a bus; The processor, memory, and communication interface communicate with each other via the bus, the memory stores program instructions executable by the processor, and the processor calls the program instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions enable a computer to implement the method according to any one of claims 1 to 7.

Images