Intelligent analysis system based on multi-source data fusion
By using an intelligent analysis system based on multi-source data fusion and leveraging the BERT-LARGE model and deep text analysis technology, the limitations of cross-platform data collaborative analysis have been addressed, enabling efficient identification and analysis of cross-platform data and improving response speed and service efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING E HUALU INFORMATION TECH
- Filing Date
- 2025-12-25
- Publication Date
- 2026-04-21
AI Technical Summary
Existing technologies have limitations in handling collaborative analysis of multi-source data. There is a lack of effective tools and technologies to monitor and identify common concerns and potential trends across platforms in real time, which affects response speed and service efficiency.
An intelligent analysis system based on multi-source data fusion was designed, including a data preprocessing and model building module, an intelligent data fusion and event matching module, and an event monitoring and analysis module. The system utilizes the BERT-LARGE model for intelligent classification, and combines multi-threading technology and deep text analysis. Through high-precision entity recognition, semantic similarity calculation, and event association criteria, it achieves accurate fusion of cross-platform data and event matching.
It enables efficient identification and analysis of cross-platform data, accurately captures dynamic changes, improves response speed and service efficiency, and ensures the accuracy and consistency of early warnings.
Smart Images
Figure CN121901425A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a data processing system, and more particularly to an intelligent analysis system based on multi-source data fusion. Background Technology
[0002] In today's era of rapid information and digital development, organizations of all types face unprecedented challenges and opportunities. While traditional communication channels such as social media and email have become crucial bridges connecting users and service providers, existing methods have limitations in handling collaborative analysis of multi-source data in the new environment of massive information exchange. There is a lack of effective tools and technologies to monitor and identify common concerns and potential trends across platforms in real time. This not only affects response speed but also limits the ability to take effective initial measures, thus impacting service efficiency and user experience.
[0003] The aforementioned multi-source data, for traditional communication channels, includes common methods such as social media, email, telephone communication, and face-to-face communication. Each communication method has its own independent communication channel. After the communication is completed, how to effectively integrate the latest communication progress into information is a problem that needs to be addressed. Summary of the Invention
[0004] This invention provides an intelligent analysis system based on multi-source data fusion, which solves the problem of real-time monitoring and identification of common concerns and potential trends across platforms. The technical solution is as follows:
[0005] An intelligent analysis system based on multi-source data fusion includes a data preprocessing and model building module, an intelligent data fusion and event matching module, and an event monitoring and analysis module. The data preprocessing and model building module acquires enterprise feedback data and performs intelligent classification to identify and predict key events in cross-platform data. The intelligent data fusion and event matching module achieves accurate fusion and event matching of multi-source data through deep text analysis to reveal the inherent relationships between cross-platform data. The event monitoring and analysis module is used to identify important matters from high-matching events screened by the system.
[0006] The data preprocessing and model building module includes a data intelligent classification unit, a high-concurrency data crawling and sentiment analysis unit, a comprehensive data normalization and quality optimization unit, and an intelligent mapping unit. The data intelligent classification unit uses the BERT-LARGE model to achieve intelligent classification of enterprise feedback data. The high-concurrency data crawling and sentiment analysis unit utilizes multi-threading technology and a flexible page loading mechanism to improve the data acquisition rate. The comprehensive data normalization and quality optimization unit is used to denoise, standardize the format, and perform quality checks on the data. The intelligent mapping unit is used to identify and predict key events in cross-platform data.
[0007] The process by which the data intelligent classification unit intelligently classifies enterprise feedback data includes the following steps:
[0008] S1: Text preprocessing; clean the original text, removing irrelevant characters, advertising information, and noise from invalid text; unify the text encoding to UTF-8, and define the maximum text length as 512 characters; perform text truncation and deduplication operations;
[0009] S2: Dataset partitioning; Based on timestamps or random sampling methods, the data is divided into training set, validation set, and test set;
[0010] S3: Tag Optimization; Optimize the tag system using business expert knowledge to ensure it is comprehensive and logically clear; Use the 3σ rule to eliminate extremely unbalanced tags;
[0011] S4: Model selection; Comparing the performance of various algorithms such as LSTM, BERT-LARGE, and NBC under the F1-Score evaluation metric, BERT-LARGE was determined as the core classifier; Cross-validation and early stopping strategies were introduced during training.
[0012] S5: Domain-adaptive model fine-tuning; by introducing langchain, pydantic, and leading domestic open-source large models, an integrated and efficient process is built.
[0013] The high-concurrency data capture and sentiment analysis unit is designed with an efficient network data capture solution based on DrissionPage technology, suitable for high-concurrency real-time monitoring. The comprehensive data normalization and quality optimization unit covers data denoising, format standardization, and quality checks; the intelligent mapping unit, by introducing the deep feature extraction capability of a fine-tuned BERT-LARGE model, outputs the major and sub-categories of business types from multiple data sources.
[0014] The intelligent data fusion and event matching module includes a high-precision entity recognition unit, a semantic similarity calculation unit, and a cross-source event association and confirmation unit. The high-precision entity recognition unit is used to increase recognition accuracy. The semantic similarity calculation unit is used to calculate semantic similarity for cross-source data with the same entity and the same business type. The cross-source event association and confirmation unit is used to identify text similarity and ensure the accuracy of early warning through strict event association criteria.
[0015] The high-precision entity recognition unit is used to customize the training model and enhance the recognition accuracy of specific entities such as locations and organization names. By integrating a large number of specific records as a training set, an enhanced model called "Context-Aware EntityRecognition" is developed. By combining the large model strategy of langchain and pydantic, it supports the dynamic addition of entity name dictionaries.
[0016] The semantic similarity calculation unit traverses multi-source data to calculate semantic similarity for cross-source data with the same entity and business type. It introduces the Sentence BERT model and optimizes the semantic similarity evaluation between sentences based on the Transformer architecture.
[0017] The cross-source event association and confirmation unit uses event association criteria to consider two data records as associated events only when they involve the same entity, belong to the same business type, and the text similarity calculated by the SBERT model exceeds a preset threshold.
[0018] The event monitoring and analysis module has a weighted evaluation system based on three aspects: cross-platform collaborative weighting, urgency level, and emotional polarity factors. The evaluation includes:
[0019] (1) Priority assessment:
[0020] Cross-platform collaboration weight W1: Given the importance of the synergistic effect of cross-platform data in understanding the global impact of events, this factor is assigned a weight of 70%, determined by measuring the frequency of occurrence of the same event in different source systems;
[0021] (3) Time sensitivity weight W2: In order to reflect the time urgency of handling the event, the time sensitivity is set to a weight of 20%, which is evaluated based on the proximity of the time of the event.
[0022] (3) Sentimental tendency weight W3: The negative intensity of sentiment tendency is assigned a weight of 10%, which is obtained through sentiment analysis tools and used to assess the emotional response of an event or subject.
[0023] The final rating for the event is:
[0024] event score =w1*score 跨平台协同 +w2*score 时间敏感性 +score 情感倾向 ;
[0025] score 跨平台协同 The score represents the cross-platform collaboration weight W1. 时间敏感性 The score represents the time-sensitivity weighted W2 score. 情感倾向The score represents the W3 weighting for sentiment tendency.
[0026] The intelligent analysis system based on multi-source data fusion deeply integrates data from multiple information channels and utilizes natural language processing technology to efficiently identify common concerns and potential trends across channels. It can accurately capture themes appearing in different information channels and, through sophisticated text analysis, instantly perceive dynamic changes. Attached Figure Description
[0027] Figure 1 This is a schematic diagram of the framework of the intelligent analysis system based on multi-source data fusion. Detailed Implementation
[0028] like Figure 1 As shown, the intelligent analysis system based on multi-source data fusion includes a data preprocessing and model building module, an intelligent data fusion and event matching module, and an event monitoring and analysis module. The data preprocessing and model building module acquires enterprise feedback data and performs intelligent classification to identify and predict key events in cross-platform data. The intelligent data fusion and event matching module achieves accurate fusion and event matching of multi-source data through deep text analysis to reveal the inherent correlation between cross-platform data. The event monitoring and analysis module is used to identify important matters from the high-matching events screened by the system.
[0029] The data preprocessing and model building module includes a data intelligent classification unit, a high-concurrency data capture and sentiment analysis unit, a comprehensive data normalization and quality optimization unit, and an intelligent mapping unit;
[0030] The data intelligent classification unit uses the BERT-LARGE model to realize the intelligent classification of enterprise feedback data. The enterprise feedback data refers to the cross-platform data formed by the text content generated by enterprise staff through traditional communication methods, or the text content converted from speech, or the text content generated by emails and chats.
[0031] The high-concurrency data capture and sentiment analysis unit utilizes multi-threading technology and a flexible page loading mechanism to improve the data acquisition rate of the intelligent data classification unit. In this way, the high-concurrency data capture and sentiment analysis unit can meet the input of enterprise feedback data from multiple enterprise staff.
[0032] The integrated data normalization and quality optimization unit is used to denoise, standardize the format, and check the quality of the data from the data intelligent classification unit, thereby performing preliminary format processing to achieve a unified input text format and form the original text.
[0033] The intelligent mapping unit is used to identify and predict key events in cross-platform data. Based on the data source, communication target, and communication frequency of the enterprise feedback data, it determines keywords and customer information, thereby obtaining key events in cross-platform data.
[0034] Furthermore,
[0035] The process by which the data intelligent classification unit intelligently classifies enterprise feedback data includes the following steps:
[0036] S1: Text preprocessing;
[0037] The original text is cleaned to remove irrelevant characters, advertising information, invalid text and other noise; the text encoding is unified to UTF-8 and the maximum text length is defined as 512 characters; text truncation, deduplication and other operations are performed to ensure the quality of the input data.
[0038] S2: Dataset partitioning;
[0039] Data is divided into training, validation, and test sets based on timestamps or random sampling methods to ensure the fairness and reliability of model evaluation.
[0040] S3: Tag optimization;
[0041] Optimize the tagging system using business expert knowledge to ensure comprehensiveness and logical clarity; use the 3σ rule to eliminate extremely unbalanced tags to avoid category sparsity or overload.
[0042] S4: Model selection;
[0043] By comparing the performance of various algorithms such as LSTM, BERT-LARGE, and NBC under the F1-Score evaluation metric, BERT-LARGE was determined as the core classifier. The F1-Score is the weighted average of precision and recall. Cross-validation and early stopping strategies were introduced during training to prevent overfitting.
[0044] The "LSTM" algorithm stands for "Long Short-Term Memory Network". LSTM is a type of recurrent neural network with a "gating" mechanism, used to simultaneously preserve short-term patterns and long-term dependencies in sequence modeling. It is commonly used in natural language processing, speech processing, and time series tasks. Its characteristics include: (1) Short-term: capturing nearby context by passing hidden states between adjacent time steps. (2) Long-term: selectively "remembering / forgetting" early information by passing cell states over a longer span, alleviating the gradient vanishing problem of ordinary RNNs. (3) Gating mechanism: three gates—the forget gate, the input gate, and the output gate—control the flow of information, allowing the model to weight and select dependencies at different time scales. Structure and workflow.
[0045] The BERT-Large model employs a typical Transformer encoder architecture, consisting of multiple stacked functional modules. From an overall structural perspective, the model primarily comprises three core components: the embedding layer, responsible for converting input tokens into dense vector representations; the encoder layer, composed of 24 identical stacked Transformer layers, each containing a self-attention mechanism and a feedforward neural network; and the final pooling layer, which aggregates the sequence representations to generate sentence-level representations.
[0046] The NBC algorithm refers to the Naive Bayes model, a classification method based on Bayes' theorem and the assumption of conditional independence of features. The two most widely used classification models are the Decision Tree Model and the Naive Bayesian Model (NBM). Compared to the Decision Tree Model, the Naive Bayes Classifier (NBC) originates from classical mathematical theory, has a solid mathematical foundation, and offers stable classification efficiency. Furthermore, the NBC model requires fewer parameters to estimate, is less sensitive to missing data, and has a relatively simple algorithm. Theoretically, the NBC model has the lowest error rate compared to other classification methods. However, this is not always the case in practice because the NBC model assumes that attributes are independent, an assumption that often does not hold true in real-world applications, which affects the accuracy of the NBC model's classification.
[0047] S5: Domain Adaptive Model Fine-tuning;
[0048] By introducing langchain, pydantic, and leading domestic open-source large models, an integrated and efficient workflow is constructed. langchain builds a flexible data processing pipeline that supports chained operations; pydantic performs data verification and parsing, enabling the system to extract multi-class entity labels and output structured data even without labeled samples. This allows the system to quickly adapt to new application scenarios or situations lacking labeled data, enhancing its flexibility and generalization capabilities.
[0049] LangChain is an open-source Python framework designed for building applications based on Large Language Models (LLMs). It simplifies interaction with language models, integration of external data, and management of complex tasks by providing modular components and standardized interfaces.
[0050] The Pydantic model is a data validation and management tool based on Python type annotations. It defines data structures by inheriting the BaseModel class and automatically performs type checking and data validation at runtime.
[0051] The high-concurrency data crawling and sentiment analysis unit is designed with DrissionPage technology to create an efficient web data crawling solution suitable for high-concurrency real-time monitoring. It utilizes multi-threading technology and a flexible page loading mechanism to improve data collection speed, covers multiple authoritative portals and industry forums, and supports the addition of enterprise-level knowledge bases, ensuring broad and timely information sources.
[0052] The comprehensive data normalization and quality optimization unit covers data denoising, format standardization, and quality checks to ensure the purity and relevance of the analyzed data. For example, the rule engine automatically identifies and removes irrelevant characters, advertising information, invalid text, etc.; it unifies the text encoding to UTF-8 and defines the maximum text length as 512 characters; it implements data integrity and consistency verification, fills in missing items, and deletes unformatted time or text data.
[0053] The intelligent mapping unit uses the deep feature extraction capability of the BERT-LARGE model to output the major and sub-categories of business types from multiple sources of data and splice them into complete business types. This enables seamless integration of classification systems for data from different systems, allowing the system to more effectively identify and predict key events in cross-platform data and improve the accuracy and response speed of data analysis.
[0054] The intelligent data fusion and event matching module aims to achieve accurate fusion and event matching of multi-source data through deep text analysis, revealing the inherent correlations between cross-platform data and deeply exploring the intrinsic connections between different data sources. It includes a high-precision entity recognition unit, a semantic similarity calculation unit, and a cross-source event association and confirmation unit. The high-precision entity recognition unit increases recognition accuracy; the semantic similarity calculation unit calculates semantic similarity for cross-source data with the same entities and business types; and the cross-source event association and confirmation unit identifies text similarity and ensures accurate early warnings through strict event association criteria.
[0055] The high-precision entity recognition unit is used to customize the training model and enhance the recognition accuracy of specific entities (such as locations, organization names, etc.). By integrating a large number of specific records as a training set, an enhanced model called "Context-Aware EntityRecognition" is developed. By combining the large model strategy of langchain and pydantic, it supports dynamically adding entity name dictionaries to adapt to the emergence of new entities and ensure the comprehensiveness and timeliness of recognition.
[0056] Context-Aware Entity Recognition (NER) is a natural language processing technique that improves the accuracy of named entity recognition (NER) by integrating global contextual information from sentences or documents. It is particularly adept at handling ambiguous entities, and its core lies in using contextual clues to infer the correct type of an entity.
[0057] The semantic similarity calculation unit traverses multi-source data to calculate semantic similarity for cross-source data containing the same entities and business types. It introduces the Sentence BERT (SBERT) model, which optimizes semantic similarity evaluation between sentences based on the Transformer architecture. Deep semantic embeddings are obtained through pre-training, mapping textual semantic relationships in a high-dimensional space to achieve accurate similarity scoring for textual data with significant differences in expression and rich meaning. In specific implementation, the context of matching entities is encoded, and the SBERT model captures subtle semantic connections across data sources. The encoded cosine similarity is then calculated to map different text similarities, establishing a refined correspondence.
[0058] The cross-source event association and confirmation unit ensures accurate early warnings through strict event association criteria. Only when two data records involve the same entity, belong to the same business type, and their text similarity calculated by the SBERT model exceeds a preset threshold are they considered associated events. The system has a built-in dynamic threshold adjustment mechanism that automatically samples and records changes in different thresholds based on real-time feedback and historical data performance, allowing for subsequent threshold adjustments and reducing the probability of false alarms and missed alarms.
[0059] The event monitoring and analysis module focuses on further identifying critical issues requiring special attention from the high-matching events screened in the system. By introducing a rule-based priority assessment model, it automatically ranks events by urgency, considering factors such as event time sensitivity, sentiment, and the number of related events. For high-priority events triggering alerts, relevant information (including cross-platform collaborative data text, data volume, business type, entity category and name, and high-frequency time period) is pushed to relevant personnel. It employs a weighted assessment system that comprehensively considers factors such as cross-platform collaboration weight, urgency, and sentiment polarity to automatically prioritize events.
[0060] (1) Priority assessment:
[0061] Cross-Platform Collaboration Weight (W1): Given the importance of the synergistic effect of cross-platform data in understanding the global impact of events, this factor is assigned a weight of 70%. It is determined by measuring the frequency of occurrence of the same event in different source systems.
[0062] (2) Time sensitivity weight (W2): In order to reflect the time urgency of handling events, time sensitivity is set to a weight of 20%, which is evaluated based on the proximity of the events.
[0063] (3) Sentimental tendency weight (W3): The negative intensity of sentiment tendency is assigned a weight of 10%, which is obtained through sentiment analysis tools and used to assess the emotional response of an event or subject.
[0064] The final rating for the event is:
[0065] event score =w1*score 跨平台协同 +w2*score 时间敏感性 +score 情感倾向 ;
[0066] score 跨平台协同 The score represents the cross-platform collaboration weight W1. 时间敏感性 The score represents the time-sensitivity weighted W2 score. 情感倾向 The score represents the W3 weighting for sentiment tendency.
[0067] This invention has the following characteristics:
[0068] (1) Cross-platform data intelligence fusion: A highly integrated system has been innovatively designed to deeply explore the inherent connections between different data sources and achieve seamless integration and collaborative analysis. This not only fills the technical gap in cross-platform data fusion analysis, but also greatly enhances the enterprise's early warning and response capabilities in complex data environments.
[0069] (2) Deep Learning-Driven Intelligent Classification and Mapping: By fine-tuning the BERT-LARGE model, intelligent classification of enterprise feedback data and accurate mapping supporting multiple labels were achieved. This model can maintain high flexibility and accuracy in different business scenarios, ensuring consistency in cross-platform event recognition.
[0070] (3) High-precision entity recognition and semantic intelligent matching: The introduction of a context-aware entity recognition model significantly enhances the recognition accuracy of specific entities (such as locations, organization names, etc.). Combined with the Sentence BERT model, even when faced with diverse and complex text information, the system can accurately calculate the semantic similarity between cross-source data, effectively link different types of data, and improve the cross-validation and accurate matching capabilities of events.
[0071] (4) Event Priority Evaluation Model: A rule-based priority evaluation model was developed, which takes into account factors such as cross-platform collaboration weight, urgency and emotional polarity, and automatically sorts the events by priority.
Claims
1. An intelligent analysis system based on multi-source data fusion, characterized in that: The system includes a data preprocessing and model building module, an intelligent data fusion and event matching module, and an event monitoring and analysis module. The data preprocessing and model building module acquires enterprise feedback data and performs intelligent classification to identify and predict key events in cross-platform data. The intelligent data fusion and event matching module uses deep text analysis to achieve accurate fusion and event matching of multi-source data to reveal the inherent relationships between cross-platform data. The event monitoring and analysis module is used to identify important matters from high-matching events screened by the system.
2. The intelligent analysis system based on multi-source data fusion according to claim 1, characterized in that: The data preprocessing and model building module includes a data intelligent classification unit, a high-concurrency data crawling and sentiment analysis unit, a comprehensive data normalization and quality optimization unit, and an intelligent mapping unit. The data intelligent classification unit uses the BERT-LARGE model to achieve intelligent classification of enterprise feedback data. The high-concurrency data crawling and sentiment analysis unit utilizes multi-threading technology and a flexible page loading mechanism to improve the data acquisition rate. The comprehensive data normalization and quality optimization unit is used to denoise, standardize the format, and perform quality checks on the data. The intelligent mapping unit is used to identify and predict key events in cross-platform data.
3. The intelligent analysis system based on multi-source data fusion according to claim 2, characterized in that: The process by which the data intelligent classification unit intelligently classifies enterprise feedback data includes the following steps: S1: Text preprocessing; clean the original text, removing irrelevant characters, advertising information, and noise from invalid text; unify the text encoding to UTF-8, and define the maximum text length as 512 characters; perform text truncation and deduplication operations; S2: Dataset partitioning; Based on timestamps or random sampling methods, the data is divided into training set, validation set, and test set; S3: Tag Optimization; Optimize the tag system using business expert knowledge to ensure it is comprehensive and logically clear; Use the 3σ rule to eliminate extremely unbalanced tags; S4: Model selection; Comparing the performance of various algorithms such as LSTM, BERT-LARGE, and NBC under the F1-Score evaluation metric, BERT-LARGE was determined as the core classifier; Cross-validation and early stopping strategies were introduced during training. S5: Domain-adaptive model fine-tuning; by introducing langchain, pydantic, and leading domestic open-source large models, an integrated and efficient process is built.
4. The intelligent analysis system based on multi-source data fusion according to claim 1, characterized in that: The high-concurrency data capture and sentiment analysis unit is designed with an efficient network data capture solution based on DrissionPage technology, suitable for high-concurrency real-time monitoring. The comprehensive data normalization and quality optimization unit covers data denoising, format standardization, and quality checks; the intelligent mapping unit, by introducing the deep feature extraction capability of a fine-tuned BERT-LARGE model, outputs the major and sub-categories of business types from multiple data sources.
5. The intelligent analysis system based on multi-source data fusion according to claim 1, characterized in that: The intelligent data fusion and event matching module includes a high-precision entity recognition unit, a semantic similarity calculation unit, and a cross-source event association and confirmation unit. The high-precision entity recognition unit is used to increase recognition accuracy. The semantic similarity calculation unit is used to calculate semantic similarity for cross-source data with the same entity and the same business type. The cross-source event association and confirmation unit is used to identify text similarity and ensure the accuracy of early warning through strict event association criteria.
6. The intelligent analysis system based on multi-source data fusion according to claim 5, characterized in that: The high-precision entity recognition unit is used to customize the training model and enhance the recognition accuracy of specific entities such as locations and organization names. By integrating a large number of specific records as a training set, an enhanced model called "Context-Aware Entity Recognition" is developed. By combining the large model strategy of langchain and pydantic, it supports the dynamic addition of entity name dictionaries.
7. The intelligent analysis system based on multi-source data fusion according to claim 5, characterized in that: The semantic similarity calculation unit traverses multi-source data to calculate semantic similarity for cross-source data with the same entity and business type. It introduces the Sentence BERT model and optimizes the semantic similarity evaluation between sentences based on the Transformer architecture.
8. The intelligent analysis system based on multi-source data fusion according to claim 5, characterized in that: The cross-source event association and confirmation unit uses event association criteria to consider two data records as associated events only when they involve the same entity, belong to the same business type, and the text similarity calculated by the SBERT model exceeds a preset threshold.
9. The intelligent analysis system based on multi-source data fusion according to claim 1, characterized in that: The event monitoring and analysis module has a weighted evaluation system based on three aspects: cross-platform collaborative weighting, urgency level, and emotional polarity factors. The evaluation includes: (1) Priority assessment: Cross-platform collaboration weight W1: Given the importance of the synergistic effect of cross-platform data in understanding the global impact of events, this factor is assigned a weight of 70%, determined by measuring the frequency of occurrence of the same event in different source systems; (2) Time sensitivity weight W2: In order to reflect the time urgency of handling events, time sensitivity is set to a weight of 20%, which is evaluated based on the proximity of the events. (3) Sentimental tendency weight W3: The negative intensity of sentiment tendency is assigned a weight of 10%, which is obtained through sentiment analysis tools and used to assess the emotional response of an event or subject. The final rating for the event is: event score =w1*score 跨平台协同 +w2*score 时间敏感性 +score 情感倾向 ; score 跨平台协同 The score represents the cross-platform collaboration weight W1. 时间敏感性 The score represents the time-sensitivity weighted W2 score. 情感倾向 The score represents the W3 weighting for sentiment tendency.