A multi-source data resource key information mining method and system

Abstract

The application discloses a kind of multi-source data resource key information mining method and system, comprising: obtaining each form type multi-source data, and pre-processing;According to data type, it is divided;Word frequency statistics and word association analysis are carried out to literal data, and the numerical value meaning of digital data is extracted by context semantic association;Based on each data type key information dataset, cross-modal key information dataset is constructed;Design multi-branch long short-term memory neural network model, input model is carried out training to cross-modal key information dataset, and key information mining model for different fields is generated by dynamically adjusting network parameters;Optimal model is selected to carry out key information real-time extraction and update, generate knowledge graph, and carry out visual operation.The application has the advantages that: by multi-source data fusion and processing, long short-term memory neural network model is combined and knowledge graph is generated, and field-specific key information mining and visual operation are efficiently carried out.

Description

Technical Field

[0001] The present invention relates to information mining technology, and in particular to a method and system for mining key information of multi-source data resources. Background Art

[0002] With the rapid development of information technology, various types of data resources are constantly increasing, such as text, images, videos and other forms of data. How to quickly and accurately mine key information from massive multi-source data has become an important research topic.

[0003] Traditional information mining methods are often limited to a single data source and are unable to meet the needs of the current multi-source data environment. Faced with a wide variety of data sources with varying formats, which are difficult to unify and integrate; large data volumes and redundant information make it difficult to quickly extract key information; the correlations between different data sources require in-depth analysis and mining; and the extraction of key information requires consideration of specific application scenarios and needs. Therefore, there is an urgent need to develop a key information mining method and system that can effectively integrate multi-source data resources to overcome the challenges of multi-source data resource key information mining. Summary of the Invention

[0004] To improve existing methods for mining key information from multi-source data resources, this paper presents a method and system for mining key information from multi-source data resources. This method, through multi-source data fusion and processing, combined with a long-short-term memory neural network model, extracts and updates key information from text and numeric data in real time. By dynamically adjusting model parameters and generating a knowledge graph, it enables efficient domain-specific key information mining and visualization.

[0005] In order to achieve the above objects, the technical solution adopted by the present invention is:

[0006] A method for mining key information of multi-source data resources, comprising:

[0007] Acquire multi-source data of various forms and types, and perform pre-processing and data fusion;

[0008] Based on the processed multi-source data, the data is divided into text data and numerical data according to the data type; the word frequency statistics and word association analysis are performed on the text data, and the numerical meaning of the numerical data is extracted through contextual semantic association;

[0009] Construct a cross-modal key information dataset based on key information datasets of each data type;

[0010] Design a multi-branch long short-term memory neural network model, input a cross-modal key information dataset into the model for training, and generate key information mining models for different fields by dynamically adjusting network parameters;

[0011] Based on the data characteristics of the target field, the corresponding optimal model is selected to extract and update key information in real time, generate a knowledge graph, and perform visualization operations.

[0012] Preferably, the acquisition of multi-source data of various forms and types, and pre-processing and data fusion specifically include:

[0013] The various types of multi-source data specifically include structured data, semi-structured data and unstructured data;

[0014] Perform missing value filling and outlier correction on structured data;

[0015] Use word segmentation, stop word removal, and named entity recognition operations on unstructured text data;

[0016] Perform metadata analysis on each type of processed data to identify the original encoding format of each data source, and automatically match the optimal conversion rules through the dynamic encoding identifier to convert to a unified encoding format;

[0017] Obtain the timestamp field of each data source, perform data fusion by aligning multi-source data based on timestamps, and obtain fused data.

[0018] Preferably, the processed multi-source data is divided into text data and numerical data according to data type; word frequency statistics and word relevance analysis are performed on the text data, and the numerical meaning of the numerical data is extracted through contextual semantic association, specifically including:

[0019] Based on the processed multi-source data, regular expressions are used to identify numeric text, mixed text, and text paragraphs. The text containing numerical values ​​is then structured and parsed to separate the text description and numerical value parts.

[0020] Based on plain text paragraphs, the TF-IDF enhanced algorithm introduces position weights when calculating term frequencies, obtains inverse document frequencies through smoothing, and outputs a term frequency-inverse document frequency matrix for each document.

[0021] Scan the text through a sliding window, count the number of co-occurrences of word pairs, and perform co-occurrence analysis;

[0022] By extracting the main structure of the sentence and recording the subject-verb-object triples, dependency syntax analysis is performed;

[0023] Based on the obtained word frequency-inverse document frequency matrix, co-occurrence analysis, and dependency syntax analysis, a plain text paragraph key information dataset is constructed;

[0024] Based on the numeric text, locate the position of the value in the text and intercept the n characters before and after it as the context fragment;

[0025] Use the BiLSTM-CRF model for named entity recognition to extract the numerical subject, unit dimension, and change trend in the context;

[0026] Obtain numerical meaning analysis based on matching the extracted data with standard terms in the industry knowledge base;

[0027] Construct a digital text key information dataset based on the acquired digital text data.

[0028] Preferably, the constructing of a cross-modal key information dataset based on key information datasets of various data types specifically includes:

[0029] Based on the obtained plain text paragraph key information dataset, text modality feature encoding is performed;

[0030] Numerical modal feature encoding is performed based on the acquired digital text key information dataset;

[0031] By mapping text features and numerical features to the same time window by timestamp, alignment in the time dimension is performed;

[0032] Use a globally unique identifier to associate multimodal data of the same entity and align the entity dimensions;

[0033] Based on the aligned multimodal data, weighted fusion is performed through a dual-channel attention mechanism to construct a cross-modal key information dataset.

[0034] Preferably, the multi-branch long short-term memory neural network model is designed, a cross-modal key information dataset is input into the model for training, and key information mining models for different fields are generated by dynamically adjusting network parameters. Specifically, the following steps are involved:

[0035] Based on the input cross-modal key information dataset, a multi-branch long short-term memory neural network model structure is designed according to text branches and numerical branches;

[0036] Add a lightweight domain classifier before each branch, generate a routing weight matrix based on the domain label of the input data, and perform dynamic parameter adjustment;

[0037] Based on a multi-stage training strategy, the first stage is general pre-training to minimize the multi-task loss function, and the second stage is domain fine-tuning to maximize domain adaptation capabilities through adversarial training strategies;

[0038] Automatically select the optimal convolution kernel combination during pre-training based on neural architecture search, perform learning rate adaptation through cosine annealing scheduler, and dynamically optimize hyperparameters;

[0039] Based on the parameter training optimization results of each branch of the long short-term memory neural network model, multiple groups of key information mining models in different fields are constructed.

[0040] Preferably, the selecting of the corresponding optimal model based on the data characteristics of the target domain to extract and update key information in real time, generate a knowledge graph, and perform visualization operations specifically includes:

[0041] Metadata feature extraction is performed based on the data structure and statistical characteristics of each field, including: data modality ratio, field type distribution, mean / variance / skewness of numerical fields, average length of text fields, and information entropy;

[0042] By calculating KL divergence to monitor the changes in numerical distribution, data drift detection can be performed on the data stream of the world;

[0043] Based on data characteristics and multiple sets of key information mining models in different fields, model matching is performed to obtain the model with the highest degree of matching of data characteristics;

[0044] Extract key information in real time based on key information mining model and encapsulate the generated results;

[0045] Based on the encapsulation results and the BiLSTM-CRF model, graph entity relationships are acquired, knowledge graphs are constructed, and real-time conflict detection is performed;

[0046] Based on the acquired knowledge graph, we design the front-end framework and interactive functions, build a visualization engine, and render visualization templates through graph layout optimization and time series data linkage.

[0047] Furthermore, a multi-source data resource key information mining system is proposed, including:

[0048] Multi-source data acquisition and preprocessing module: The multi-source data acquisition and preprocessing module is responsible for preprocessing operations, including acquiring structured, semi-structured, and unstructured data from heterogeneous data sources, completing missing value filling, text segmentation, code unification, and timestamp alignment;

[0049] Data type classification module: The data type classification module is mainly used to automatically classify data into text paragraphs and text containing numerical values ​​through regular expressions and structured parsing;

[0050] Text key information extraction module: Based on the TF-IDF enhanced algorithm, word pair co-occurrence statistics and dependency syntax analysis, the text key information extraction module generates a text word frequency matrix and semantic triples to construct a text key information dataset;

[0051] Numerical key information parsing module: The module is mainly used to extract entities, dimensions and trends from numerical contexts using the BiLSTM-CRF model, match numerical semantics with the industry knowledge base, and generate a digital text key information dataset;

[0052] Cross-modal alignment and fusion module: The cross-modal alignment and fusion module is mainly used to associate multimodal data through time window alignment and global entity identifiers, and adopts a dual-channel attention mechanism to achieve weighted fusion of text and numerical features;

[0053] Multi-branch LSTM model training module: The multi-branch LSTM model training module is mainly used to design a multi-branch neural network with dynamic routing weights, combine adversarial training and neural architecture search to optimize model parameters, and generate a domain-adapted key information mining model;

[0054] Real-time knowledge graph construction module: The real-time knowledge graph construction module is mainly used to dynamically update the graph through entity relationship extraction and conflict detection, encapsulate structured data and associate it with historical data to generate an extensible knowledge network;

[0055] Visualization engine and interaction module: The visualization engine and interaction module are mainly used to design visualization templates that support dynamic layout optimization and time-series linkage, render the knowledge graph into an interactive graphical interface, and support multi-dimensional data drilling and filtering;

[0056] Processor: The processor is mainly used for the calculation process of each formula and the construction calculation process of each model.

[0057] Compared with the prior art, the advantages of the present invention are:

[0058] By preprocessing and fusing multi-source data, text and numerical data from different sources can be effectively integrated, making the information more comprehensive and representative. Word frequency statistics and correlation analysis of textual data help to explore potential relationships and key concepts in the text, while extracting the numerical meaning of numerical data through contextual semantic associations helps to accurately understand the actual meaning behind the numbers. Based on this processed key information, a cross-modal dataset is constructed and trained using a multi-branch long short-term memory neural network (LSTM) model. The network parameters can be dynamically adjusted to generate specialized key information mining models for different fields. In particular, the method selects the optimal model in real time to respond to the data characteristics of different target fields, making the extraction of key information not only accurate but also efficient. In addition, the generated knowledge graph can intuitively display the relationship between information, helping users to quickly understand and make decisions through visual operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A schematic diagram of the method proposed in the present invention;

[0060] Figure 2 This is a schematic diagram of data preprocessing and fusion proposed by the present invention;

[0061] Figure 3 Schematic diagram of data division and analysis proposed by the present invention;

[0062] Figure 4 This is a schematic diagram of obtaining a cross-modal key information dataset proposed in the present invention;

[0063] Figure 5 A schematic diagram of the key information mining model proposed in the present invention;

[0064] Figure 6 Schematic diagram of model matching and visualization proposed by the present invention;

[0065] Figure 7 This is a diagram of the architecture of the electronic equipment in this solution;

[0066] Figure 8 This is a schematic diagram of the computer-readable storage medium structure in this solution. DETAILED DESCRIPTION

[0067] The following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are merely examples, and those skilled in the art may conceive of other obvious variations.

[0068] A multi-source data resource key information mining system, comprising:

[0069] Multi-source data acquisition and preprocessing module: The multi-source data acquisition and preprocessing module is responsible for preprocessing operations, including acquiring structured, semi-structured, and unstructured data from heterogeneous data sources, completing missing value filling, text segmentation, code unification, and timestamp alignment;

[0070] Data type classification module: The data type classification module is mainly used to automatically classify data into text paragraphs and text containing numerical values ​​through regular expressions and structured parsing;

[0071] Text key information extraction module: Based on the TF-IDF enhanced algorithm, word pair co-occurrence statistics and dependency syntax analysis, the text key information extraction module generates a text word frequency matrix and semantic triples to construct a text key information dataset;

[0072] Numerical key information parsing module: The module is mainly used to extract entities, dimensions and trends from numerical contexts using the BiLSTM-CRF model, match numerical semantics with the industry knowledge base, and generate a digital text key information dataset;

[0073] Cross-modal alignment and fusion module: The cross-modal alignment and fusion module is mainly used to associate multimodal data through time window alignment and global entity identifiers, and adopts a dual-channel attention mechanism to achieve weighted fusion of text and numerical features;

[0074] Multi-branch LSTM model training module: The multi-branch LSTM model training module is mainly used to design a multi-branch neural network with dynamic routing weights, combine adversarial training and neural architecture search to optimize model parameters, and generate a domain-adapted key information mining model;

[0075] Real-time knowledge graph construction module: The real-time knowledge graph construction module is mainly used to dynamically update the graph through entity relationship extraction and conflict detection, encapsulate structured data and associate it with historical data to generate an extensible knowledge network;

[0076] Visualization engine and interaction module: The visualization engine and interaction module are mainly used to design visualization templates that support dynamic layout optimization and time-series linkage, render the knowledge graph into an interactive graphical interface, and support multi-dimensional data drilling and filtering;

[0077] Processor: The processor is mainly used for the calculation process of each formula and the construction calculation process of each model.

[0078] See Figure 1 As shown, a method for mining key information of multi-source data resources includes:

[0079] Step 1: Obtain multi-source data of various types and perform pre-processing and data fusion;

[0080] Step 2: Based on the processed multi-source data, the data is divided into text data and numerical data according to the data type; word frequency statistics and word association analysis are performed on the text data, and the numerical meaning of the numerical data is extracted through contextual semantic association;

[0081] Step 3: Construct a cross-modal key information dataset based on the key information datasets of each data type;

[0082] Step 4: Design a multi-branch long short-term memory neural network model, input the cross-modal key information dataset into the model for training, and generate key information mining models for different fields by dynamically adjusting network parameters;

[0083] Step 5: Based on the data characteristics of the target field, select the corresponding optimal model to extract and update key information in real time, generate a knowledge graph, and perform visualization operations.

[0084] See Figure 2 As shown, obtaining multi-source data of various forms and types, and performing preprocessing and data fusion specifically include:

[0085] The various types of multi-source data specifically include structured data, semi-structured data and unstructured data;

[0086] Perform missing value filling and outlier correction on structured data;

[0087] Use word segmentation, stop word removal, and named entity recognition operations on unstructured text data;

[0088] Perform metadata analysis on each type of processed data to identify the original encoding format of each data source, and automatically match the optimal conversion rules through the dynamic encoding identifier to convert to a unified encoding format;

[0089] Obtain the timestamp field of each data source, perform data fusion by aligning multi-source data based on timestamps, and obtain fused data.

[0090] Specifically, for filling in missing data in structured data, linear interpolation is performed according to the time sorting order. For processing unstructured text data, the text is divided into word sequences through word segmentation tools (such as Jieba and NLTK). For processing semi-structured text data, the nested structure is converted into a flat table and structured data cleaning methods are applied to parse it into key-value pairs or tables.

[0091] When aligning multi-source data based on timestamps, extract the timestamp fields of each data source , unified to UTC format, use linear interpolation for non-aligned timestamps, the formula is:

[0092]

[0093] Sliding window aggregation is used for discrete event data, and the formula is:

[0094]

[0095] in, To align the interpolation result to the time point t, Timestamp and The corresponding observation value, is the aggregate value within the window, is the number of data in the window, The value of each data.

[0096] See Figure 3 As shown in the figure, based on the processed multi-source data, it is divided into text data and digital data according to the data type; the word frequency statistics and word association analysis are performed on the text data, and the numerical meaning of the digital data is extracted through contextual semantic association. Specifically, the following are included:

[0097] Based on the processed multi-source data, regular expressions are used to identify numeric text, mixed text, and text paragraphs. The text containing numerical values ​​is then structured and parsed to separate the text description and numerical value parts.

[0098] Based on plain text paragraphs, the TF-IDF enhanced algorithm introduces position weights when calculating term frequencies, obtains inverse document frequencies through smoothing, and outputs a term frequency-inverse document frequency matrix for each document.

[0099] Scan the text through a sliding window, count the number of co-occurrences of word pairs, and perform co-occurrence analysis;

[0100] By extracting the main structure of the sentence and recording the subject-verb-object triples, dependency syntax analysis is performed;

[0101] Based on the obtained word frequency-inverse document frequency matrix, co-occurrence analysis, and dependency syntax analysis, a plain text paragraph key information dataset is constructed;

[0102] Based on the numeric text, locate the position of the value in the text and intercept the n characters before and after it as the context fragment;

[0103] Use the BiLSTM-CRF model for named entity recognition to extract the numerical subject, unit dimension, and change trend in the context;

[0104] Obtain numerical meaning analysis based on matching the extracted data with standard terms in the industry knowledge base;

[0105] Construct a digital text key information dataset based on the acquired digital text data.

[0106] Specifically, in the regular expression pattern, pure numeric text matches integers or floating-point numbers, and text paragraphs are separated by periods, question marks, and exclamation marks.

[0107] The formula for calculating word frequency in the TF-IDF enhanced algorithm is:

[0108]

[0109] in, is the target term, d is the current document or text paragraph, is the index of the position of the term in the paragraph, is the total number of words in the paragraph, For terms In position The frequency of is the decay factor, which controls the rate at which the position weight decays with distance. is the center position index of the paragraph, Indicates the position of a term With central location absolute distance.

[0110] The inverse document frequency formula is:

[0111]

[0112] in, is the total number of documents, To include terms The number of documents, is the smoothing factor.

[0113] In the process of numerical context extraction and entity recognition, the context information is encoded through bidirectional LSTM, and the optimal label sequence is decoded based on the CRF layer. The loss function is:

[0114]

[0115] in, is the global score of the label sequence y for the input sequence x, including the state score and the transition score, is the exponential sum of the scores of all possible label sequences y', which is the partition function.

[0116] See Figure 4 As shown in FIG, constructing a cross-modal key information dataset based on the key information dataset of each data type specifically includes:

[0117] Based on the obtained plain text paragraph key information dataset, text modality feature encoding is performed;

[0118] Numerical modal feature encoding is performed based on the acquired digital text key information dataset;

[0119] By mapping text features and numerical features to the same time window by timestamp, alignment in the time dimension is performed;

[0120] Use a globally unique identifier to associate multimodal data of the same entity and align the entity dimensions;

[0121] Based on the aligned multimodal data, weighted fusion is performed through a dual-channel attention mechanism to construct a cross-modal key information dataset.

[0122] Specifically, for text modalities, a pre-trained language model (such as BERT) is used to semantically encode plain text paragraphs, sentence vectors are extracted through the embedding layer, and the self-attention mechanism is used to capture context-related features to form a text feature matrix with unified dimensions; numerical data sets are standardized and numerical features are extracted, with a focus on retaining key statistical properties such as trends and periodicity.

[0123] During the time dimension alignment phase, a sliding window mechanism is used to divide text feature sequences and numerical feature sequences into segments of the same time granularity, using millisecond-level timestamps as a benchmark. Linear interpolation or nearest neighbor sampling is used to address incomplete timestamp matching. During the entity dimension alignment phase, identifiers are used to establish cross-modal data association indexes, and hash mapping is used to enable rapid matching and concatenation of data from different modalities for the same entity.

[0124] The fusion stage employs a dual-channel attention mechanism: the modal interaction channel calculates cross-modal attention weights between text and numerical features, filtering complementary information through a gating mechanism; the temporal correlation channel captures the dynamic dependencies of multimodal features along the time axis. Finally, a cross-modal fusion vector is formed through weighted concatenation, constructing a multimodal dataset of key information that incorporates spatiotemporal correlations and semantic enhancement, providing joint representation support for downstream tasks.

[0125] See Figure 5 As shown in the figure, a multi-branch long short-term memory neural network model is designed, and a cross-modal key information dataset is input into the model for training. By dynamically adjusting the network parameters, key information mining models for different fields are generated. Specifically, the following are included:

[0126] Based on the input cross-modal key information dataset, a multi-branch long short-term memory neural network model structure is designed according to text branches and numerical branches;

[0127] Add a lightweight domain classifier before each branch, generate a routing weight matrix based on the domain label of the input data, and perform dynamic parameter adjustment;

[0128] Based on a multi-stage training strategy, the first stage is general pre-training to minimize the multi-task loss function, and the second stage is domain fine-tuning to maximize domain adaptation capabilities through adversarial training strategies;

[0129] Automatically select the optimal convolution kernel combination during pre-training based on neural architecture search, perform learning rate adaptation through cosine annealing scheduler, and dynamically optimize hyperparameters;

[0130] Based on the parameter training optimization results of each branch of the long short-term memory neural network model, multiple groups of key information mining models in different fields are constructed.

[0131] Understandably, the performance of lightweight domain classifiers may not be sufficient to accurately generate routing weight matrices, especially in complex multi-domain scenarios, where classifier accuracy may be affected by data noise. Multi-level classifiers can be used for ensemble learning, or transfer learning can be employed to improve the generalization and accuracy of domain classifiers. Regular evaluation and adjustment of classifiers is essential to ensure their adaptability to data from various domains.

[0132] The Neural Architecture Search (NAS) process consumes large amounts of computational resources and the search space is too broad, making the optimization process time-consuming and difficult to find the optimal architecture. More efficient NAS algorithms (such as gradient-based NAS or NAS that combines reinforcement learning with evolutionary algorithms) can be used, and an upper limit on the search space can be set based on actual needs to reduce the computational burden. Furthermore, a phased search strategy and empirical heuristics can be used to narrow the search space and improve efficiency.

[0133] The computational and storage consumption of training multi-branch long short-term memory neural network models is very high, which may lead to excessive resource consumption in the training process. Model compression techniques (such as knowledge distillation and weight pruning) can be used to reduce the size and computational complexity of the model. At the same time, a distributed training framework can be used during training to speed up training and optimize resource utilization.

[0134] See Figure 6 As shown in the figure, based on the data characteristics of the target domain, the corresponding optimal model is selected to extract and update key information in real time, generate a knowledge graph, and perform visualization operations, including:

[0135] Metadata feature extraction is performed based on the data structure and statistical characteristics of each field, including: data modality ratio, field type distribution, mean / variance / skewness of numerical fields, average length of text fields, and information entropy;

[0136] By calculating KL divergence to monitor the changes in numerical distribution, data drift detection can be performed on the data stream of the world;

[0137] Based on data characteristics and multiple sets of key information mining models in different fields, model matching is performed to obtain the model with the highest degree of matching of data characteristics;

[0138] Extract key information in real time based on key information mining model and encapsulate the generated results;

[0139] Based on the encapsulation results and the BiLSTM-CRF model, graph entity relationships are acquired, knowledge graphs are constructed, and real-time conflict detection is performed;

[0140] Based on the acquired knowledge graph, we design the front-end framework and interactive functions, build a visualization engine, and render visualization templates through graph layout optimization and time series data linkage.

[0141] Specifically, the formula for calculating the average length of text field features is:

[0142]

[0143] The information entropy calculation formula is:

[0144]

[0145] in, is a set of characters / vocabulary, is the frequency of occurrence of the character.

[0146] When detecting data drift, the numerical field is discretized into K intervals, and the KL divergence is calculated by calculating the reference distribution PP and the current distribution Q.

[0147] At the framework level, we adopt a component-based architecture with React / Vue at its core, achieving multi-level graphics acceleration through hybrid rendering technology of WebGL and Canvas. Interaction design includes core functional modules such as multi-dimensional filtering and relationship path tracing.

[0148] Graph layout optimization utilizes a strategy that combines an improved force-directed algorithm with community discovery technology, innovatively introducing a dynamic weight adjustment mechanism to optimize spatial distribution in real time based on parameters such as node degree and relationship strength. A time slicing engine and incremental update algorithm were developed to address the characteristics of time series data. A timeline controller was used to implement temporal analysis capabilities such as historical version comparison and evolution tracking.

[0149] Furthermore, the method according to the embodiment of the present application can also be used with the aid of Figure 7 The electronic device architecture shown in FIG. Figure 7 As shown, the electronic device 500 may include a bus 501, one or more CPUs 502, a read-only memory (ROM) 503, a random access memory (RAM) 504, a communication port 505 connected to a network, an input / output component 506, a hard disk 507, etc. The storage device in the electronic device 500, such as the ROM 503 or the hard disk 507, may store a multi-source data resource key information mining method and system provided by the present application. The electronic device 500 may also include a terminal interface 508. Of course, Figure 7 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 7 One or more components of an electronic device are shown.

[0150] Figure 8 This is a schematic diagram of the computer-readable storage medium structure provided by an embodiment of the present application. Figure 8As shown, a computer-readable storage medium 600 according to one embodiment of the present application is shown. Computer-readable instructions are stored on the computer-readable storage medium 600. When the computer-readable instructions are executed by the processor, a multi-source data resource key information mining method and system according to an embodiment of the present application described with reference to the above figures can be executed. The storage medium 600 includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0151] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. Furthermore, the foregoing descriptions of specific embodiments of this specification are provided. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential sequence shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0152] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0153] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for mining key information of multi-source data resources, characterized in that: include: Acquire multi-source data of various forms and types, and perform pre-processing and data fusion; Based on the processed multi-source data, the data is divided into text data and numerical data according to the data type; the word frequency statistics and word association analysis are performed on the text data, and the numerical meaning of the numerical data is extracted through contextual semantic association; Construct a cross-modal key information dataset based on key information datasets of each data type; Based on the input cross-modal key information dataset, a multi-branch long short-term memory neural network model structure is designed according to text branches and numerical branches; Add a lightweight domain classifier before each branch, generate a routing weight matrix based on the domain label of the input data, and perform dynamic parameter adjustment; Based on a multi-stage training strategy, the first stage is general pre-training to minimize the multi-task loss function, and the second stage is domain fine-tuning to maximize domain adaptation capabilities through adversarial training strategies; Automatically select the optimal convolution kernel combination during pre-training based on neural architecture search, perform learning rate adaptation through cosine annealing scheduler, and dynamically optimize hyperparameters; Based on the parameter training and optimization results of each branch of the long short-term memory neural network model, multiple groups of key information mining models in different fields are constructed; Based on the data characteristics of the target field, the corresponding optimal model is selected to extract and update key information in real time, generate a knowledge graph, and perform visualization operations.

2. A method for mining key information of multi-source data resources according to claim 1, characterized in that: The acquisition of multi-source data of various forms and types, and the pre-processing and data fusion specifically include: The various types of multi-source data specifically include structured data, semi-structured data and unstructured data; Fill missing values ​​and correct outliers for structured data; Use word segmentation, stop word removal, and named entity recognition operations on unstructured text data; Perform metadata analysis on each type of processed data to identify the original encoding format of each data source, and automatically match the optimal conversion rules through the dynamic encoding identifier to convert to a unified encoding format; Obtain the timestamp field of each data source, perform data fusion by aligning multi-source data based on timestamps, and obtain fused data.

3. The method for mining key information of multi-source data resources according to claim 1, characterized in that: The processed multi-source data is divided into text data and digital data according to data type; Perform word frequency statistics and word relevance analysis on text data, and extract numerical meanings from numerical data through contextual semantic associations. Specifically, the following are performed: Based on the processed multi-source data, regular expressions are used to identify numeric text, mixed text, and text paragraphs. The text containing numerical values ​​is then structured and parsed to separate the text description and numerical value parts. Based on plain text paragraphs, the TF-IDF enhanced algorithm introduces position weights when calculating term frequencies, obtains inverse document frequencies through smoothing, and outputs a term frequency-inverse document frequency matrix for each document. Scan the text through a sliding window, count the number of co-occurrences of word pairs, and perform co-occurrence analysis; By extracting the main structure of the sentence and recording the subject-verb-object triples, dependency syntax analysis is performed; Based on the obtained word frequency-inverse document frequency matrix, co-occurrence analysis, and dependency syntax analysis, a plain text paragraph key information dataset is constructed; Based on the numeric text, locate the position of the value in the text and intercept the n characters before and after it as the context fragment; Use the BiLSTM-CRF model for named entity recognition to extract the numerical subject, unit dimension, and change trend in the context; Obtain numerical meaning analysis based on matching the extracted data with standard terms in the industry knowledge base; Construct a digital text key information dataset based on the acquired digital text data.

4. The method for mining key information of multi-source data resources according to claim 1, characterized in that: The step of constructing a cross-modal key information dataset based on key information datasets of various data types specifically includes: Based on the obtained plain text paragraph key information dataset, text modality feature encoding is performed; Numerical modal feature encoding is performed based on the acquired digital text key information dataset; By mapping text features and numerical features to the same time window by timestamp, alignment in the time dimension is performed; Use a globally unique identifier to associate multimodal data of the same entity and align the entity dimensions; Based on the aligned multimodal data, weighted fusion is performed through a dual-channel attention mechanism to construct a cross-modal key information dataset.

5. The method for mining key information of multi-source data resources according to claim 1, characterized in that: The process of selecting the optimal model based on the data characteristics of the target domain to extract and update key information in real time, generate a knowledge graph, and perform visualization operations specifically includes: Metadata feature extraction is performed based on the data structure and statistical characteristics of each field, including: data modality ratio, field type distribution, mean / variance / skewness of numerical fields, average length of text fields, and information entropy; Monitor changes in numerical distribution by calculating KL divergence and perform data drift detection on real-time data streams; Based on data characteristics and multiple sets of key information mining models in different fields, model matching is performed to obtain the model with the highest degree of matching of data characteristics; Extract key information in real time based on key information mining model and encapsulate the generated results; Based on the encapsulation results and the BiLSTM-CRF model, graph entity relationships are acquired, knowledge graphs are constructed, and real-time conflict detection is performed; Based on the acquired knowledge graph, we design the front-end framework and interactive functions, build a visualization engine, and render visualization templates through graph layout optimization and time series data linkage.

6. A multi-source data resource key information mining system, used to implement a multi-source data resource key information mining method according to any one of claims 1 to 5, characterized in that: include: Multi-source data acquisition and preprocessing module: The multi-source data acquisition and preprocessing module is responsible for preprocessing operations, including acquiring structured, semi-structured, and unstructured data from heterogeneous data sources, completing missing value filling, text segmentation, code unification, and timestamp alignment; Data type classification module: The data type classification module is mainly used to automatically classify data into text paragraphs and text containing numerical values ​​through regular expressions and structured parsing; Text key information extraction module: Based on the TF-IDF enhanced algorithm, word pair co-occurrence statistics and dependency syntax analysis, the text key information extraction module generates a text word frequency matrix and semantic triples to construct a text key information dataset; Numerical key information parsing module: The module is mainly used to extract entities, dimensions and trends from numerical contexts using the BiLSTM-CRF model, match numerical semantics with the industry knowledge base, and generate a digital text key information dataset; Cross-modal alignment and fusion module: The cross-modal alignment and fusion module is mainly used to associate multimodal data through time window alignment and global entity identifiers, and adopts a dual-channel attention mechanism to achieve weighted fusion of text and numerical features; Multi-branch LSTM model training module: The multi-branch LSTM model training module is mainly used to design a multi-branch neural network with dynamic routing weights, combine adversarial training and neural architecture search to optimize model parameters, and generate a domain-adapted key information mining model; Real-time knowledge graph construction module: The real-time knowledge graph construction module is mainly used to dynamically update the graph through entity relationship extraction and conflict detection, encapsulate structured data and associate it with historical data to generate an extensible knowledge network; Visualization engine and interaction module: The visualization engine and interaction module are mainly used to design visualization templates that support dynamic layout optimization and time-series linkage, render the knowledge graph into an interactive graphical interface, and support multi-dimensional data drilling and filtering; Processor: The processor is mainly used for the calculation process of each formula and the construction calculation process of each model.

7. An electronic device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute a multi-source data resource key information mining method as described in any one of claims 1-5.

8. A computer-readable storage medium storing computer-readable instructions, characterized in that: When the computer-readable instructions are executed by a processor, a method for mining key information of multi-source data resources according to any one of claims 1 to 5 is implemented.

Images